From af58b32cc86268fe6b6eb628ada2762a771d2376 Mon Sep 17 00:00:00 2001 From: evonneng Date: Tue, 4 Oct 2022 09:43:11 -0700 Subject: [PATCH] restructuring debugging tools doc (#647) * not done; restrucutre debugging part; * 1) fixed all developer guides 2) fixed first_nerf error 3) moved run_actions * removed md --- .../{ => debugging_tools}/benchmarking.md | 57 +++++---- .../debugging_tools/index.rst | 12 ++ .../debugging_tools/local_logger.md | 48 +++++++ .../debugging_tools/profiling.md | 61 +++++++++ docs/developer_guides/logging_profiling.md | 117 ------------------ docs/index.md | 5 +- docs/quickstart/first_nerf.md | 2 +- pyproject.toml | 2 +- scripts/benchmarking/launch_eval_blender.sh | 2 +- scripts/debugging/profile.sh | 53 -------- scripts/{debugging => github}/__init__.py | 0 scripts/{debugging => github}/run_actions.py | 0 12 files changed, 161 insertions(+), 198 deletions(-) rename docs/developer_guides/{ => debugging_tools}/benchmarking.md (53%) create mode 100644 docs/developer_guides/debugging_tools/index.rst create mode 100644 docs/developer_guides/debugging_tools/local_logger.md create mode 100644 docs/developer_guides/debugging_tools/profiling.md delete mode 100644 docs/developer_guides/logging_profiling.md delete mode 100755 scripts/debugging/profile.sh rename scripts/{debugging => github}/__init__.py (100%) rename scripts/{debugging => github}/run_actions.py (100%) diff --git a/docs/developer_guides/benchmarking.md b/docs/developer_guides/debugging_tools/benchmarking.md similarity index 53% rename from docs/developer_guides/benchmarking.md rename to docs/developer_guides/debugging_tools/benchmarking.md index 31330aaa3..fbd7e62f1 100644 --- a/docs/developer_guides/benchmarking.md +++ b/docs/developer_guides/debugging_tools/benchmarking.md @@ -8,43 +8,52 @@ To start, you will need to train your NeRF on each of the blender objects. To launch training jobs automatically on each of these items, you can call: ```bash -./scripts/benchmarking/launch_train_blender.sh -c -g + +./scripts/benchmarking/launch_train_blender.sh -m {METHOD_NAME} -g {GPU_LIST} ``` Simply replace the arguments in brackets with the correct arguments. -- `-c`: name of the config you created for your new graph. For instance, if you have created the config `configs/graph_instant_ngp.yaml`, `` will be `graph_instant_ngp`. +- `-m`: name of the method you want to benchmark (e.g. `nerfacto`, `mipnerf`). - `-g`: specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs. + +:::{admonition} Tip +:class: info + +To view all the arguments and annotations, you can run `./scripts/benchmarking/launch_train_blender.sh --help` + ::: + A full example would be: ```bash + # with -g specifying gpus -./scripts/benchmarking/launch_train_blender.sh -c graph_instant_ngp -g 0 1 2 3 +./scripts/benchmarking/launch_train_blender.sh -m nerfacto -g 0 1 2 3 # without -g automatically finds available gpus -./scripts/benchmarking/launch_train_blender.sh -c graph_instant_ngp +./scripts/benchmarking/launch_train_blender.sh -m nerfacto ``` -The script will automatically launch training on all of the items and save the checkpoints in an output directory with the current date and timestamp. +The script will automatically launch training on all of the items and save the checkpoints in an output directory with the experiment name and current timestamp. ## Evaluating trained Blender models Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/launch_eval_blender.sh`. -Say we ran a benchmark on 08-10-2022 for instant_ngp. By default, the train script will save the benchmarks in the following format: +Say we ran a benchmark on 08-10-2022 for `instant-ngp`. By default, the train script will save the benchmarks in the following format: ``` outputs -└───blender_chair_08-10-2022 -| └───instant_ngp +└───blender_chair_2022-08-10 +| └───instant-ngp | └───2022-08-10_172517 -| └───train.log +| └───config.yml | ... -└───blender_drums_08-10-2022 -| └───instant_ngp +└───blender_drums_2022-08-10 +| └───instant-ngp | └───2022-08-10_172517 -| └───train.log +| └───config.yml | ... ... ``` @@ -52,25 +61,29 @@ outputs If we wanted to run the benchmark on all the blender data for the above example, we would run: ```bash -./scripts/benchmarking/launch_eval_blender.sh -c graph_instant_ngp -o ./outputs/ -m 08 -d 10 -y 2022 -s 172517 -g 4 5 6 7 + +./scripts/benchmarking/launch_eval_blender.sh -m instant-ngp -o outputs/ -t 2022-08-10_172517 ``` The flags used in the benchmarking script are defined as follows: -- `-c`: config name (e.g. `graph_instant_ngp`). This should be the same as what was passed in for -c in the train script. -- `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the hydra base dir. -- `-m`: month of benchmark (e.g. `08`) of format 02d. -- `-d`: date of benchmark (e.g. `10`) of format 02d. -- `-y`: year of benchmark (e.g. `2022`). -- `-s`: seconds timestamp of benchmark (e.g. `172517`) of format 06d. +- `-m`: config name (e.g. `instant-ngp`). This should be the same as what was passed in for -c in the train script. +- `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the `--output-dir` in the base `Config` for training. +- `-t`: timestamp of benchmark; also the identifier (e.g. `2022-08-10_172517`). - `-g`: specifies the gpus to use and if not specified (no -g flag), will automaticaly search for available gpus. The script will simultaneously run the benchmarking across all the objects in the blender dataset and calculates the PSNR/FPS/other stats. The results are saved as .json files in the `-o` directory with the following format: ``` outputs -└───instant_ngp -| └───blender_chair_08-10-2022_172517.json -| | blender_ficus_08-10-2022_172517.json +└───instant-ngp +| └───blender_chair_2022-08-10_172517.json +| | blender_ficus_2022-08-10_172517.json | | ... ``` + +:::{admonition} Warning +:class: warning + +Since we are running multiple backgrounded processes concurrently with this script, please note the terminal logs may be messy. + ::: \ No newline at end of file diff --git a/docs/developer_guides/debugging_tools/index.rst b/docs/developer_guides/debugging_tools/index.rst new file mode 100644 index 000000000..1a6aab8e4 --- /dev/null +++ b/docs/developer_guides/debugging_tools/index.rst @@ -0,0 +1,12 @@ +Debugging tools +==================== + +We document a few of the supported tooling systems and pipelines we support for debugging our models (e.g. profiling to debug speed). +As we grow, we hope to provide more updated and extensive tooling support. + +.. toctree:: + :maxdepth: 1 + + local_logger + profiling + benchmarking \ No newline at end of file diff --git a/docs/developer_guides/debugging_tools/local_logger.md b/docs/developer_guides/debugging_tools/local_logger.md new file mode 100644 index 000000000..3eef57b41 --- /dev/null +++ b/docs/developer_guides/debugging_tools/local_logger.md @@ -0,0 +1,48 @@ +# Local writer + +The `LocalWriter` simply outputs numerical stats to the terminal. +You can specify additional parameters to customize your logging experience. + +```python +@dataclass +class LocalWriterConfig(InstantiateConfig): + """Local Writer config""" + + _target: Type = writer.LocalWriter + """target class to instantiate""" + enable: bool = False + """if True enables local logging, else disables""" + stats_to_track: Tuple[writer.EventName, ...] = ( + writer.EventName.ITER_TRAIN_TIME, + writer.EventName.TRAIN_RAYS_PER_SEC, + writer.EventName.CURR_TEST_PSNR, + writer.EventName.VIS_RAYS_PER_SEC, + writer.EventName.TEST_RAYS_PER_SEC, + ) + """specifies which stats will be logged/printed to terminal""" + max_log_size: int = 10 + """maximum number of rows to print before wrapping. if 0, will print everything.""" +``` + +- `enable`: enable/disable the logger. +- `stats_to_track`: all the stats that you want to print to the terminal (see list under `EventName` in `utils/writer.py`). You can add or remove any of the defined enums. +- `max_log_size`: how much content to print onto the screen (By default, only print 10 lines onto the screen at a time). If 0, will print everything without deleting any previous lines. + +:::{admonition} Tip +:class: info + +If you want to create a new stat to track, simply add the stat name to the `EventName` enum. +- Remember to call some put event (e.g. `put_scalar` from `utils/writer.py` to place the value in the `EVENT_STORAGE`. +- Remember to add the new enum to the `stats_to_track` list + ::: + +The local writer is easily configurable via CLI. +A few common commands to use: + +```bash +# disable local writer +ns-train {METHOD_NAME} --logging.local-writer.no-enable + +# disable line wrapping +ns-train {METHOD_NAME} --logging.local-writer.max-log-size=0 +``` \ No newline at end of file diff --git a/docs/developer_guides/debugging_tools/profiling.md b/docs/developer_guides/debugging_tools/profiling.md new file mode 100644 index 000000000..dc67afea3 --- /dev/null +++ b/docs/developer_guides/debugging_tools/profiling.md @@ -0,0 +1,61 @@ +# Code profiling support + +We provide built-in performance profiling capabilities to make it easier for you to debug and assess the performance of your code. + +#### In-house profiler + +You can use our built-in profiler. By default, it is enabled and will print at the termination of the program. You can disable it via CLI using the flag `--logging.no-enable-profiler`. + + +The profiler computes the average total time of execution for any function with the `@profiler.time_function` decorator. +For instance, if you wanted to profile the total time it takes to generate rays given pixel and camera indices via the `RayGenerator` class, you might want to time its `forward()` function. In that case, you would need to add the decorator to the function. + +```python +"""nerfstudio/model_components/ray_generators.py"""" + +class RayGenerator(nn.Module): + + ... + + @profiler.time_function # <-- add the profiler decorator before the function + def forward(self, ray_indices: TensorType["num_rays", 3]) -> RayBundle: + # implementation here + ... +``` + +At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag. + +:::{admonition} Tip +:class: info + +Use this profiler if there are *specific/individual functions* you want to measure the times for. + ::: + + +#### Profiling with PySpy + +If you want to profile the entire codebase, consider using [PySpy](https://github.com/benfred/py-spy). + +Install PySpy + +```bash +pip install py-spy +``` + +To perform the profiling, you can either specify that you want to generate a flame graph or generate a live-view of the profiler. + +```bash +## flame graph: with wandb logging and our inhouse logging disabled +program="ns-train nerfacto -- --vis=wandb --logging.no-enable-profiler blender-data" +py-spy record -o {PATH_TO_OUTPUT_SVG} $program +``` + +```bash +## top-down stats: running same program configuration as above +py-spy top $program +``` +:::{admonition} Attention +:class: attention + +In defining `program`, you will need to add an extra `--` before you specify your program's arguments. + ::: \ No newline at end of file diff --git a/docs/developer_guides/logging_profiling.md b/docs/developer_guides/logging_profiling.md deleted file mode 100644 index 74efcf2fe..000000000 --- a/docs/developer_guides/logging_profiling.md +++ /dev/null @@ -1,117 +0,0 @@ -# Loggers and Profilers - -## Logging support - -We provide integration with multiple logging interfaces to log images and statistics during training. -All of these loggers are configurable via the config: - -```yaml -# e.g. configs/graphs_default.yaml - -logging: - steps_per_log: 10 # how often to log losses/images - max_buffer_size: 20 # defines how many steps to average over (if reporting averages) - writer: # definition of all writers you want to use - TensorboardWriter: - log_dir: './' # must specify the output path for all writers - LocalWriter: - log_dir: './' - stats_to_track: [ITER_TRAIN_TIME, TRAIN_RAYS_PER_SEC, CURR_TEST_PSNR] - max_log_size: 10 -``` - -Currently, we provide support for the following loggers, which can be added to the `logging.writer` section of the config. - -#### Tensorboard - -To log training stats and images with [tensorboard](https://www.tensorflow.org/tensorboard): - -``` -TensorboardWriter: - log_dir: "./" # the output path for which the event files are written -``` - -#### Weights and Biases - -To log training stats and images with [wandb](https://wandb.ai/site): - -``` -WandbWriter: - log_dir: "./" # the output path for which the run files are written -``` - -#### Local writer - -``` -LocalWriter: - log_dir: "./" # if images are saved, where the images would be saved - stats_to_track: - [ITER_TRAIN_TIME, TRAIN_RAYS_PER_SEC, CURR_TEST_PSNR] - max_log_size: 10 -``` - -The `LocalWriter` simply outputs to the terminal. However, you can specify additional parameters to customize the logging. - -- `stats_to_track`: all the stats that you want to print to the terminal (see list under `EventName` in `utils/writer.py`). You can add or remove any of the defined enums. -- `max_log_size`: how much content to print onto the screen (e.g. only print 10 lines on the screen at a time). If 0, will print everything without deleting any previous lines. - -## Code profiling support - -We provide built-in performance profiling capabilities to make it easier for you to debug and assess the performance of your code. - -#### In-house profiler - -You can use our built-in profiler by enabling the profiler in the config: - -```yaml -# e.g. configs/graphs_default.yaml - -logging: - enable_profiler: True -``` - -The profiler computes the average total time of execution for any function with the `@profiler.time_function` decorator. -For instance, if you wanted to profile the total time it takes to generate rays given pixel and camera indices via the `RayGenerator` class, you would want to evaluate its `forward()` function. In that class, you would need to add the decorator to the function. - -```python -class RayGenerator(nn.Module): - - ... - - @profiler.time_function # add the profiler decorator before the function - def forward(self, ray_indices: TensorType["num_rays", 3]) -> RayBundle: - # implementation here - ... -``` - -At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag. - -Use this profiler if there are specific functions that you want to measure the times for. - -#### Profiling with PySpy - -If you want to profile the entire codebase, we provide functionality to generate flame graphs or get real-time reports using [PySpy](https://github.com/benfred/py-spy). - -Install PySpy - -```bash -pip install py-spy -``` - -To perform the profiling, you can either specify that you want to generate a flame graph or generate a live-view of the profiler. - -```bash -## for flame graph - -./scripts/debugging/profile.sh -t flame -o flame.svg -p - -## for live view of functions - -./scripts/debugging/profile.sh -t top -p -``` - -In both cases, `` can be replaced with any command you would normally run in the terminal. For instance, you can replace `` with `scripts/train.py data/dataset=blender_lego`. Then the full command would be e.g.: - -```bash -./scripts/debugging/profile.sh -t flame -o flame.svg -p scripts/train.py data/dataset=blender_lego -``` diff --git a/docs/index.md b/docs/index.md index 5dc2cfac3..7d5a36f38 100644 --- a/docs/index.md +++ b/docs/index.md @@ -41,7 +41,7 @@ quickstart/installation quickstart/first_nerf quickstart/custom_dataset quickstart/viewer_quickstart -Contributing +Contributing ``` ```{toctree} @@ -60,8 +60,7 @@ nerfology/dataloader_components/index developer_guides/pipelines/index developer_guides/viewer/index developer_guides/config -developer_guides/logging_profiling -developer_guides/benchmarking +developer_guides/debugging_tools/index ``` ```{toctree} diff --git a/docs/quickstart/first_nerf.md b/docs/quickstart/first_nerf.md index 3c3cc6167..a739de26d 100644 --- a/docs/quickstart/first_nerf.md +++ b/docs/quickstart/first_nerf.md @@ -100,4 +100,4 @@ Calculate the psnr of your trained model and save to a json file. ns-eval --load-config={PATH_TO_CONFIG} --output-path=output.json ``` -We also provide a train and evaluation script that allows you to do benchmarking on the classical Blender dataset (see our [benchmarking workflow](../developer_guides/benchmarking.md)). \ No newline at end of file +We also provide a train and evaluation script that allows you to do benchmarking on the classical Blender dataset (see our [benchmarking workflow](../developer_guides/debugging_tools/benchmarking.md)). \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index cc138dcc9..3582410cb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -81,7 +81,7 @@ ns-download-data = "scripts.downloads.download_data:entrypoint" ns-train = "scripts.train:entrypoint" ns-eval = "scripts.eval:entrypoint" ns-render = "scripts.render:entrypoint" -ns-dev-test = "scripts.debugging.run_actions:entrypoint" +ns-dev-test = "scripts.github.run_actions:entrypoint" [options] # equivalent to using --extra-index-url with pip, which is needed for specifying the CUDA version torch and torchvision diff --git a/scripts/benchmarking/launch_eval_blender.sh b/scripts/benchmarking/launch_eval_blender.sh index 2c79856a5..85d077d8e 100755 --- a/scripts/benchmarking/launch_eval_blender.sh +++ b/scripts/benchmarking/launch_eval_blender.sh @@ -63,7 +63,7 @@ for dataset in ${DATASETS[@]}; do export CUDA_VISIBLE_DEVICES=${GPU_IDX[$idx]} config_path="${output_dir}/blender_${dataset}_${timestamp::-7}/${method_name}/${timestamp}/config.yml" ns-eval --load-config=${config_path} \ - --output-path=${output_dir}/${method_name}/blender_${timestamp}.json & + --output-path=${output_dir}/${method_name}/blender_${dataset}_${timestamp}.json & echo "Launched ${config_path} on gpu ${GPU_IDX[$idx]}" # update gpu diff --git a/scripts/debugging/profile.sh b/scripts/debugging/profile.sh deleted file mode 100755 index f3437aa85..000000000 --- a/scripts/debugging/profile.sh +++ /dev/null @@ -1,53 +0,0 @@ -#!/bin/bash - -helpFunction_flame() -{ - echo "Usage: $0 -o output_file -p program" - echo -e "\t-o output .svg filename, where to save flame graph" - echo -e "\t-p .py program we want to profile with associated arguments" - exit 1 # Exit program after printing help -} - -helpFunction_top() -{ - echo "Usage: $0 -p program" - echo -e "\t-p .py program we want to profile with associated arguments" - exit 1 # Exit program after printing help -} - - -while getopts "t:o:p" opt; do - case "$opt" in - t ) type="$OPTARG" ;; - o ) output_file="$OPTARG" ;; - p ) program="$OPTARG" ;; - ? ) helpFunction ;; - esac -done - -if [ -z "$type" ]; then - echo "Missing profiling type (flame / top)" - exit 1 -fi -shift $((OPTIND-1)) -program="$@" -program=${program/"--"/"-- --"} - -# Print helpFunction in case parameters are empty -if [[ "$type" = "flame" ]]; then - if [ -z "$output_file" ] || [ -z "$program" ]; then - echo "Some or all of the parameters are empty"; - helpFunction_flame - fi - command=`echo py-spy record -o "$output_file" python "$program"` - echo "Running..." $command - eval "$command" -elif [[ "$type" = "top" ]]; then - if [ -z "$program" ]; then - echo "Some or all of the parameters are empty"; - helpFunction_top - fi - command=`echo py-spy top python "$program"` - echo "Running..." $command - eval "$command" -fi \ No newline at end of file diff --git a/scripts/debugging/__init__.py b/scripts/github/__init__.py similarity index 100% rename from scripts/debugging/__init__.py rename to scripts/github/__init__.py diff --git a/scripts/debugging/run_actions.py b/scripts/github/run_actions.py similarity index 100% rename from scripts/debugging/run_actions.py rename to scripts/github/run_actions.py