restructuring debugging tools doc (#647)

* not done; restrucutre debugging part;

* 1) fixed all developer guides 2) fixed first_nerf error 3) moved run_actions

* removed md

docs/developer_guides/debugging_tools/benchmarking.md

@@ -8,43 +8,52 @@ To start, you will need to train your NeRF on each of the blender objects.
 To launch training jobs automatically on each of these items, you can call:
 
 ```bash
-./scripts/benchmarking/launch_train_blender.sh -c <config-name> -g <list-of-gpus>
+
+./scripts/benchmarking/launch_train_blender.sh -m {METHOD_NAME} -g {GPU_LIST}
 ```
 
 Simply replace the arguments in brackets with the correct arguments.
 
-- `-c`: name of the config you created for your new graph. For instance, if you have created the config `configs/graph_instant_ngp.yaml`, `<config-name>` will be `graph_instant_ngp`.
+- `-m`: name of the method you want to benchmark (e.g. `nerfacto`, `mipnerf`).
 - `-g`: specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs.
+  
+:::{admonition} Tip
+:class: info
+
+To view all the arguments and annotations, you can run `./scripts/benchmarking/launch_train_blender.sh --help`
+  :::
+
 
 A full example would be:
 
 ```bash
+
 # with -g specifying gpus
-./scripts/benchmarking/launch_train_blender.sh -c graph_instant_ngp -g 0 1 2 3
+./scripts/benchmarking/launch_train_blender.sh -m nerfacto -g 0 1 2 3
 
 # without -g automatically finds available gpus
-./scripts/benchmarking/launch_train_blender.sh -c graph_instant_ngp
+./scripts/benchmarking/launch_train_blender.sh -m nerfacto
 ```
 
-The script will automatically launch training on all of the items and save the checkpoints in an output directory with the current date and timestamp.
+The script will automatically launch training on all of the items and save the checkpoints in an output directory with the experiment name and current timestamp.
 
 ## Evaluating trained Blender models
 
 Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/launch_eval_blender.sh`.
 
-Say we ran a benchmark on 08-10-2022 for instant_ngp. By default, the train script will save the benchmarks in the following format:
+Say we ran a benchmark on 08-10-2022 for `instant-ngp`. By default, the train script will save the benchmarks in the following format:
 
 ```
 outputs
-└───blender_chair_08-10-2022
-|   └───instant_ngp
+└───blender_chair_2022-08-10
+|   └───instant-ngp
 |       └───2022-08-10_172517
-|           └───train.log
+|           └───config.yml
 |               ...
-└───blender_drums_08-10-2022
-|   └───instant_ngp
+└───blender_drums_2022-08-10
+|   └───instant-ngp
 |       └───2022-08-10_172517
-|           └───train.log
+|           └───config.yml
 |               ...
 ...
 ```
@@ -52,25 +61,29 @@ outputs
 If we wanted to run the benchmark on all the blender data for the above example, we would run:
 
 ```bash
-./scripts/benchmarking/launch_eval_blender.sh -c graph_instant_ngp -o ./outputs/ -m 08 -d 10 -y 2022 -s 172517 -g 4 5 6 7
+
+./scripts/benchmarking/launch_eval_blender.sh -m instant-ngp -o outputs/ -t 2022-08-10_172517   
 ```
 
 The flags used in the benchmarking script are defined as follows:
 
-- `-c`: config name (e.g. `graph_instant_ngp`). This should be the same as what was passed in for -c in the train script.
-- `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the hydra base dir.
-- `-m`: month of benchmark (e.g. `08`) of format 02d.
-- `-d`: date of benchmark (e.g. `10`) of format 02d.
-- `-y`: year of benchmark (e.g. `2022`).
-- `-s`: seconds timestamp of benchmark (e.g. `172517`) of format 06d.
+- `-m`: config name (e.g. `instant-ngp`). This should be the same as what was passed in for -c in the train script.
+- `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the `--output-dir` in the base `Config` for training.
+- `-t`: timestamp of benchmark; also the identifier (e.g. `2022-08-10_172517`).
 - `-g`: specifies the gpus to use and if not specified (no -g flag), will automaticaly search for available gpus.
 
 The script will simultaneously run the benchmarking across all the objects in the blender dataset and calculates the PSNR/FPS/other stats. The results are saved as .json files in the `-o` directory with the following format:
 
 ```
 outputs
-└───instant_ngp
-|   └───blender_chair_08-10-2022_172517.json
-|   |   blender_ficus_08-10-2022_172517.json
+└───instant-ngp
+|   └───blender_chair_2022-08-10_172517.json
+|   |   blender_ficus_2022-08-10_172517.json
 |   |   ...
 ```
+
+:::{admonition} Warning
+:class: warning
+
+Since we are running multiple backgrounded processes concurrently with this script, please note the terminal logs may be messy.
+  :::

docs/developer_guides/debugging_tools/index.rst

@@ -0,0 +1,12 @@
+Debugging tools
+====================
+
+We document a few of the supported tooling systems and pipelines we support for debugging our models (e.g. profiling to debug speed).
+As we grow, we hope to provide more updated and extensive tooling support.
+
+.. toctree::
+    :maxdepth: 1
+
+    local_logger
+    profiling
+    benchmarking

docs/developer_guides/debugging_tools/local_logger.md

@@ -0,0 +1,48 @@
+# Local writer
+
+The `LocalWriter` simply outputs numerical stats to the terminal.
+You can specify additional parameters to customize your logging experience.
+
+```python
+@dataclass
+class LocalWriterConfig(InstantiateConfig):
+    """Local Writer config"""
+
+    _target: Type = writer.LocalWriter
+    """target class to instantiate"""
+    enable: bool = False
+    """if True enables local logging, else disables"""
+    stats_to_track: Tuple[writer.EventName, ...] = (
+        writer.EventName.ITER_TRAIN_TIME,
+        writer.EventName.TRAIN_RAYS_PER_SEC,
+        writer.EventName.CURR_TEST_PSNR,
+        writer.EventName.VIS_RAYS_PER_SEC,
+        writer.EventName.TEST_RAYS_PER_SEC,
+    )
+    """specifies which stats will be logged/printed to terminal"""
+    max_log_size: int = 10
+    """maximum number of rows to print before wrapping. if 0, will print everything."""
+```
+
+- `enable`: enable/disable the logger.
+- `stats_to_track`: all the stats that you want to print to the terminal (see list under `EventName` in `utils/writer.py`). You can add or remove any of the defined enums.
+- `max_log_size`: how much content to print onto the screen (By default, only print 10 lines onto the screen at a time). If 0, will print everything without deleting any previous lines.
+
+:::{admonition} Tip
+:class: info
+
+If you want to create a new stat to track, simply add the stat name to the `EventName` enum.
+- Remember to call some put event (e.g. `put_scalar` from `utils/writer.py` to place the value in the `EVENT_STORAGE`. 
+- Remember to add the new enum to the `stats_to_track` list
+  :::
+
+The local writer is easily configurable via CLI.
+A few common commands to use:
+
+```bash
+# disable local writer
+ns-train {METHOD_NAME} --logging.local-writer.no-enable
+
+# disable line wrapping
+ns-train {METHOD_NAME} --logging.local-writer.max-log-size=0
+```

docs/developer_guides/debugging_tools/profiling.md

@@ -0,0 +1,61 @@
+# Code profiling support
+
+We provide built-in performance profiling capabilities to make it easier for you to debug and assess the performance of your code.
+
+#### In-house profiler
+
+You can use our built-in profiler. By default, it is enabled and will print at the termination of the program. You can disable it via CLI using the flag `--logging.no-enable-profiler`.
+
+
+The profiler computes the average total time of execution for any function with the `@profiler.time_function` decorator.
+For instance, if you wanted to profile the total time it takes to generate rays given pixel and camera indices via the `RayGenerator` class, you might want to time its `forward()` function. In that case, you would need to add the decorator to the function.
+
+```python
+"""nerfstudio/model_components/ray_generators.py""""
+
+class RayGenerator(nn.Module):
+
+    ...
+
+    @profiler.time_function     # <-- add the profiler decorator before the function
+    def forward(self, ray_indices: TensorType["num_rays", 3]) -> RayBundle:
+        # implementation here
+        ...
+```
+
+At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag.
+
+:::{admonition} Tip
+:class: info
+
+Use this profiler if there are *specific/individual functions* you want to measure the times for.
+  :::
+
+
+#### Profiling with PySpy
+
+If you want to profile the entire codebase, consider using [PySpy](https://github.com/benfred/py-spy).
+
+Install PySpy
+
+```bash
+pip install py-spy
+```
+
+To perform the profiling, you can either specify that you want to generate a flame graph or generate a live-view of the profiler.
+
+```bash
+## flame graph: with wandb logging and our inhouse logging disabled
+program="ns-train nerfacto -- --vis=wandb --logging.no-enable-profiler blender-data"
+py-spy record -o {PATH_TO_OUTPUT_SVG} $program
+```
+
+```bash
+## top-down stats: running same program configuration as above
+py-spy top $program
+```
+:::{admonition} Attention
+:class: attention
+
+In defining `program`, you will need to add an extra `--` before you specify your program's arguments.
+  :::

docs/developer_guides/logging_profiling.md

@@ -1,117 +0,0 @@
-# Loggers and Profilers
-
-## Logging support
-
-We provide integration with multiple logging interfaces to log images and statistics during training.
-All of these loggers are configurable via the config:
-
-```yaml
-# e.g. configs/graphs_default.yaml
-
-logging:
-  steps_per_log: 10 # how often to log losses/images
-  max_buffer_size: 20 # defines how many steps to average over (if reporting averages)
-  writer: # definition of all writers you want to use
-    TensorboardWriter:
-      log_dir: './' # must specify the output path for all writers
-    LocalWriter:
-      log_dir: './'
-      stats_to_track: [ITER_TRAIN_TIME, TRAIN_RAYS_PER_SEC, CURR_TEST_PSNR]
-      max_log_size: 10
-```
-
-Currently, we provide support for the following loggers, which can be added to the `logging.writer` section of the config.
-
-#### Tensorboard
-
-To log training stats and images with [tensorboard](https://www.tensorflow.org/tensorboard):
-
-```
-TensorboardWriter:
-    log_dir: "./" # the output path for which the event files are written
-```
-
-#### Weights and Biases
-
-To log training stats and images with [wandb](https://wandb.ai/site):
-
-```
-WandbWriter:
-    log_dir: "./" # the output path for which the run files are written
-```
-
-#### Local writer
-
-```
-LocalWriter:
-    log_dir: "./" # if images are saved, where the images would be saved
-    stats_to_track:
-        [ITER_TRAIN_TIME, TRAIN_RAYS_PER_SEC, CURR_TEST_PSNR]
-    max_log_size: 10
-```
-
-The `LocalWriter` simply outputs to the terminal. However, you can specify additional parameters to customize the logging.
-
-- `stats_to_track`: all the stats that you want to print to the terminal (see list under `EventName` in `utils/writer.py`). You can add or remove any of the defined enums.
-- `max_log_size`: how much content to print onto the screen (e.g. only print 10 lines on the screen at a time). If 0, will print everything without deleting any previous lines.
-
-## Code profiling support
-
-We provide built-in performance profiling capabilities to make it easier for you to debug and assess the performance of your code.
-
-#### In-house profiler
-
-You can use our built-in profiler by enabling the profiler in the config:
-
-```yaml
-# e.g. configs/graphs_default.yaml
-
-logging:
-  enable_profiler: True
-```
-
-The profiler computes the average total time of execution for any function with the `@profiler.time_function` decorator.
-For instance, if you wanted to profile the total time it takes to generate rays given pixel and camera indices via the `RayGenerator` class, you would want to evaluate its `forward()` function. In that class, you would need to add the decorator to the function.
-
-```python
-class RayGenerator(nn.Module):
-
-    ...
-
-    @profiler.time_function     # add the profiler decorator before the function
-    def forward(self, ray_indices: TensorType["num_rays", 3]) -> RayBundle:
-        # implementation here
-        ...
-```
-
-At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag.
-
-Use this profiler if there are specific functions that you want to measure the times for.
-
-#### Profiling with PySpy
-
-If you want to profile the entire codebase, we provide functionality to generate flame graphs or get real-time reports using [PySpy](https://github.com/benfred/py-spy).
-
-Install PySpy
-
-```bash
-pip install py-spy
-```
-
-To perform the profiling, you can either specify that you want to generate a flame graph or generate a live-view of the profiler.
-
-```bash
-## for flame graph
-
-./scripts/debugging/profile.sh -t flame -o flame.svg -p <python_command>
-
-## for live view of functions
-
-./scripts/debugging/profile.sh -t top -p <python_command>
-```
-
-In both cases, `<python_command>` can be replaced with any command you would normally run in the terminal. For instance, you can replace `<python_function>` with `scripts/train.py data/dataset=blender_lego`. Then the full command would be e.g.:
-
-```bash
-./scripts/debugging/profile.sh -t flame -o flame.svg -p scripts/train.py data/dataset=blender_lego
-```

docs/index.md

@@ -41,7 +41,7 @@ quickstart/installation
 quickstart/first_nerf
 quickstart/custom_dataset
 quickstart/viewer_quickstart
-Contributing<reference/contributing.md>
+Contributing<reference/contributing>
 ```
 
 ```{toctree}
@@ -60,8 +60,7 @@ nerfology/dataloader_components/index
 developer_guides/pipelines/index
 developer_guides/viewer/index
 developer_guides/config
-developer_guides/logging_profiling
-developer_guides/benchmarking
+developer_guides/debugging_tools/index
 ```
 
 ```{toctree}

docs/quickstart/first_nerf.md

@@ -100,4 +100,4 @@ Calculate the psnr of your trained model and save to a json file.
 ns-eval --load-config={PATH_TO_CONFIG} --output-path=output.json
 ```
 
-We also provide a train and evaluation script that allows you to do benchmarking on the classical Blender dataset (see our [benchmarking workflow](../developer_guides/benchmarking.md)).
+We also provide a train and evaluation script that allows you to do benchmarking on the classical Blender dataset (see our [benchmarking workflow](../developer_guides/debugging_tools/benchmarking.md)).

pyproject.toml

@@ -81,7 +81,7 @@ ns-download-data = "scripts.downloads.download_data:entrypoint"
 ns-train = "scripts.train:entrypoint"
 ns-eval = "scripts.eval:entrypoint"
 ns-render = "scripts.render:entrypoint"
-ns-dev-test = "scripts.debugging.run_actions:entrypoint"
+ns-dev-test = "scripts.github.run_actions:entrypoint"
 
 [options]
 # equivalent to using --extra-index-url with pip, which is needed for specifying the CUDA version torch and torchvision

scripts/benchmarking/launch_eval_blender.sh

@@ -63,7 +63,7 @@ for dataset in ${DATASETS[@]}; do
     export CUDA_VISIBLE_DEVICES=${GPU_IDX[$idx]}
     config_path="${output_dir}/blender_${dataset}_${timestamp::-7}/${method_name}/${timestamp}/config.yml"
     ns-eval --load-config=${config_path} \
-            --output-path=${output_dir}/${method_name}/blender_${timestamp}.json &
+            --output-path=${output_dir}/${method_name}/blender_${dataset}_${timestamp}.json &
     echo "Launched ${config_path} on gpu ${GPU_IDX[$idx]}"
 
     # update gpu

scripts/debugging/profile.sh

@@ -1,53 +0,0 @@
-#!/bin/bash
-
-helpFunction_flame()
-{
-   echo "Usage: $0 -o output_file -p program"
-   echo -e "\t-o output .svg filename, where to save flame graph"
-   echo -e "\t-p .py program we want to profile with associated arguments"
-   exit 1 # Exit program after printing help
-}
-
-helpFunction_top()
-{
-   echo "Usage: $0 -p program"
-   echo -e "\t-p .py program we want to profile with associated arguments"
-   exit 1 # Exit program after printing help
-}
-
-
-while getopts "t:o:p" opt; do
-    case "$opt" in
-        t ) type="$OPTARG" ;;
-        o ) output_file="$OPTARG" ;;
-        p ) program="$OPTARG" ;;
-        ? ) helpFunction ;; 
-    esac
-done
-
-if [ -z "$type" ]; then
-    echo "Missing profiling type (flame / top)"
-    exit 1
-fi
-shift $((OPTIND-1))
-program="$@"
-program=${program/"--"/"-- --"}
-
-# Print helpFunction in case parameters are empty
-if [[ "$type" = "flame" ]]; then
-    if [ -z "$output_file" ] || [ -z "$program" ]; then
-        echo "Some or all of the parameters are empty";
-        helpFunction_flame
-    fi
-    command=`echo py-spy record -o "$output_file" python "$program"`
-    echo "Running..." $command
-    eval "$command"
-elif [[ "$type" = "top" ]]; then
-    if [ -z "$program" ]; then
-        echo "Some or all of the parameters are empty";
-        helpFunction_top
-    fi
-    command=`echo py-spy top python "$program"`
-    echo "Running..." $command
-    eval "$command"
-fi