restructuring debugging tools doc (#647)

* not done; restrucutre debugging part;

* 1) fixed all developer guides 2) fixed first_nerf error 3) moved run_actions

* removed md

docs/developer_guides/debugging_tools/benchmarking.md

@@ -0,0 +1,89 @@
+# Benchmarking workflow
+
+We make it easy to benchmark your new NeRF against the standard Blender dataset.
+
+## Launching training on Blender dataset
+
+To start, you will need to train your NeRF on each of the blender objects.
+To launch training jobs automatically on each of these items, you can call:
+
+```bash
+
+./scripts/benchmarking/launch_train_blender.sh -m {METHOD_NAME} -g {GPU_LIST}
+```
+
+Simply replace the arguments in brackets with the correct arguments.
+
+- `-m`: name of the method you want to benchmark (e.g. `nerfacto`, `mipnerf`).
+- `-g`: specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs.
+  
+:::{admonition} Tip
+:class: info
+
+To view all the arguments and annotations, you can run `./scripts/benchmarking/launch_train_blender.sh --help`
+  :::
+
+
+A full example would be:
+
+```bash
+
+# with -g specifying gpus
+./scripts/benchmarking/launch_train_blender.sh -m nerfacto -g 0 1 2 3
+
+# without -g automatically finds available gpus
+./scripts/benchmarking/launch_train_blender.sh -m nerfacto
+```
+
+The script will automatically launch training on all of the items and save the checkpoints in an output directory with the experiment name and current timestamp.
+
+## Evaluating trained Blender models
+
+Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/launch_eval_blender.sh`.
+
+Say we ran a benchmark on 08-10-2022 for `instant-ngp`. By default, the train script will save the benchmarks in the following format:
+
+```
+outputs
+└───blender_chair_2022-08-10
+|   └───instant-ngp
+|       └───2022-08-10_172517
+|           └───config.yml
+|               ...
+└───blender_drums_2022-08-10
+|   └───instant-ngp
+|       └───2022-08-10_172517
+|           └───config.yml
+|               ...
+...
+```
+
+If we wanted to run the benchmark on all the blender data for the above example, we would run:
+
+```bash
+
+./scripts/benchmarking/launch_eval_blender.sh -m instant-ngp -o outputs/ -t 2022-08-10_172517   
+```
+
+The flags used in the benchmarking script are defined as follows:
+
+- `-m`: config name (e.g. `instant-ngp`). This should be the same as what was passed in for -c in the train script.
+- `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the `--output-dir` in the base `Config` for training.
+- `-t`: timestamp of benchmark; also the identifier (e.g. `2022-08-10_172517`).
+- `-g`: specifies the gpus to use and if not specified (no -g flag), will automaticaly search for available gpus.
+
+The script will simultaneously run the benchmarking across all the objects in the blender dataset and calculates the PSNR/FPS/other stats. The results are saved as .json files in the `-o` directory with the following format:
+
+```
+outputs
+└───instant-ngp
+|   └───blender_chair_2022-08-10_172517.json
+|   |   blender_ficus_2022-08-10_172517.json
+|   |   ...
+```
+
+:::{admonition} Warning
+:class: warning
+
+Since we are running multiple backgrounded processes concurrently with this script, please note the terminal logs may be messy.
+  :::

docs/developer_guides/debugging_tools/index.rst

@@ -0,0 +1,12 @@
+Debugging tools
+====================
+
+We document a few of the supported tooling systems and pipelines we support for debugging our models (e.g. profiling to debug speed).
+As we grow, we hope to provide more updated and extensive tooling support.
+
+.. toctree::
+    :maxdepth: 1
+
+    local_logger
+    profiling
+    benchmarking

docs/developer_guides/debugging_tools/local_logger.md

@@ -0,0 +1,48 @@
+# Local writer
+
+The `LocalWriter` simply outputs numerical stats to the terminal.
+You can specify additional parameters to customize your logging experience.
+
+```python
+@dataclass
+class LocalWriterConfig(InstantiateConfig):
+    """Local Writer config"""
+
+    _target: Type = writer.LocalWriter
+    """target class to instantiate"""
+    enable: bool = False
+    """if True enables local logging, else disables"""
+    stats_to_track: Tuple[writer.EventName, ...] = (
+        writer.EventName.ITER_TRAIN_TIME,
+        writer.EventName.TRAIN_RAYS_PER_SEC,
+        writer.EventName.CURR_TEST_PSNR,
+        writer.EventName.VIS_RAYS_PER_SEC,
+        writer.EventName.TEST_RAYS_PER_SEC,
+    )
+    """specifies which stats will be logged/printed to terminal"""
+    max_log_size: int = 10
+    """maximum number of rows to print before wrapping. if 0, will print everything."""
+```
+
+- `enable`: enable/disable the logger.
+- `stats_to_track`: all the stats that you want to print to the terminal (see list under `EventName` in `utils/writer.py`). You can add or remove any of the defined enums.
+- `max_log_size`: how much content to print onto the screen (By default, only print 10 lines onto the screen at a time). If 0, will print everything without deleting any previous lines.
+
+:::{admonition} Tip
+:class: info
+
+If you want to create a new stat to track, simply add the stat name to the `EventName` enum.
+- Remember to call some put event (e.g. `put_scalar` from `utils/writer.py` to place the value in the `EVENT_STORAGE`. 
+- Remember to add the new enum to the `stats_to_track` list
+  :::
+
+The local writer is easily configurable via CLI.
+A few common commands to use:
+
+```bash
+# disable local writer
+ns-train {METHOD_NAME} --logging.local-writer.no-enable
+
+# disable line wrapping
+ns-train {METHOD_NAME} --logging.local-writer.max-log-size=0
+```

docs/developer_guides/debugging_tools/profiling.md

@@ -0,0 +1,61 @@
+# Code profiling support
+
+We provide built-in performance profiling capabilities to make it easier for you to debug and assess the performance of your code.
+
+#### In-house profiler
+
+You can use our built-in profiler. By default, it is enabled and will print at the termination of the program. You can disable it via CLI using the flag `--logging.no-enable-profiler`.
+
+
+The profiler computes the average total time of execution for any function with the `@profiler.time_function` decorator.
+For instance, if you wanted to profile the total time it takes to generate rays given pixel and camera indices via the `RayGenerator` class, you might want to time its `forward()` function. In that case, you would need to add the decorator to the function.
+
+```python
+"""nerfstudio/model_components/ray_generators.py""""
+
+class RayGenerator(nn.Module):
+
+    ...
+
+    @profiler.time_function     # <-- add the profiler decorator before the function
+    def forward(self, ray_indices: TensorType["num_rays", 3]) -> RayBundle:
+        # implementation here
+        ...
+```
+
+At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag.
+
+:::{admonition} Tip
+:class: info
+
+Use this profiler if there are *specific/individual functions* you want to measure the times for.
+  :::
+
+
+#### Profiling with PySpy
+
+If you want to profile the entire codebase, consider using [PySpy](https://github.com/benfred/py-spy).
+
+Install PySpy
+
+```bash
+pip install py-spy
+```
+
+To perform the profiling, you can either specify that you want to generate a flame graph or generate a live-view of the profiler.
+
+```bash
+## flame graph: with wandb logging and our inhouse logging disabled
+program="ns-train nerfacto -- --vis=wandb --logging.no-enable-profiler blender-data"
+py-spy record -o {PATH_TO_OUTPUT_SVG} $program
+```
+
+```bash
+## top-down stats: running same program configuration as above
+py-spy top $program
+```
+:::{admonition} Attention
+:class: attention
+
+In defining `program`, you will need to add an extra `--` before you specify your program's arguments.
+  :::