Printing profile on sig kill. (#44)

* added skeleton for terminal logger

* done skeleton terminal writer logic... todo: actual calls to the write classes

* moved lambda to top function

* added logic to support timing

* prettier printing

* prettier printing

* lint

* separated out timing and other stats variables. printing in dict format

* keeping logging history and wiping rest

* updated logic to calculate total times instead of average times

* made stats tracker configurable

* moved stats tracker and profiler to own class file; implemented basic functionality for profiler

* lint removing duplicate function

* update readme; moved around config variables

* profiler prints on sig kill
This commit is contained in:
evonneng authored and GitHub committed 2022-04-29 11:10:34 -07:00
1 parent 47691eb291
commit 86b1011ae1
8 files changed
+46 -24

No files matched your search

+4 -4
View File
@@ -6,17 +6,18 @@ defaults:
- override hydra/hydra_logging: colorlog
logging_configs:
steps_per_log: 10
# stats tracker logs configurable training stats per iteration to terminal (e.g. rays/s, PSNR)
enable_stats: True
stats_tracker:
max_history: 10
stats_to_track: [ITER_LOAD_TIME, ITER_TRAIN_TIME] # see mattport/utils/stats_tracker.py for options
# profiler logs run times of functions and prints at end of training
enable_profiler: True
# writer logs losses and images per iteration
writer:
type: TensorboardWriter
save_dir: '/tmp/'
save_dir: './'
# profiler logs run times of functions and prints at end of training
enable_profiler: True
hydra:
run:
@@ -34,7 +35,6 @@ resume_train:
max_num_iterations: 10
steps_per_save: null
steps_per_log: 10
steps_per_test: 100
model_dir: /tmp/mattport_models/