mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-18 22:04:02 +08:00
restructuring debugging tools doc (#647)
* not done; restrucutre debugging part; * 1) fixed all developer guides 2) fixed first_nerf error 3) moved run_actions * removed md
This commit is contained in:
1 parent
4d5a1363b1
commit
af58b32cc8
12 files changed
+161
-198
No files matched your search
+35
-22
@@ -8,43 +8,52 @@ To start, you will need to train your NeRF on each of the blender objects.
|
||||
To launch training jobs automatically on each of these items, you can call:
|
||||
|
||||
```bash
|
||||
./scripts/benchmarking/launch_train_blender.sh -c <config-name> -g <list-of-gpus>
|
||||
|
||||
./scripts/benchmarking/launch_train_blender.sh -m {METHOD_NAME} -g {GPU_LIST}
|
||||
```
|
||||
|
||||
Simply replace the arguments in brackets with the correct arguments.
|
||||
|
||||
- `-c`: name of the config you created for your new graph. For instance, if you have created the config `configs/graph_instant_ngp.yaml`, `<config-name>` will be `graph_instant_ngp`.
|
||||
- `-m`: name of the method you want to benchmark (e.g. `nerfacto`, `mipnerf`).
|
||||
- `-g`: specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs.
|
||||
|
||||
:::{admonition} Tip
|
||||
:class: info
|
||||
|
||||
To view all the arguments and annotations, you can run `./scripts/benchmarking/launch_train_blender.sh --help`
|
||||
:::
|
||||
|
||||
|
||||
A full example would be:
|
||||
|
||||
```bash
|
||||
|
||||
# with -g specifying gpus
|
||||
./scripts/benchmarking/launch_train_blender.sh -c graph_instant_ngp -g 0 1 2 3
|
||||
./scripts/benchmarking/launch_train_blender.sh -m nerfacto -g 0 1 2 3
|
||||
|
||||
# without -g automatically finds available gpus
|
||||
./scripts/benchmarking/launch_train_blender.sh -c graph_instant_ngp
|
||||
./scripts/benchmarking/launch_train_blender.sh -m nerfacto
|
||||
```
|
||||
|
||||
The script will automatically launch training on all of the items and save the checkpoints in an output directory with the current date and timestamp.
|
||||
The script will automatically launch training on all of the items and save the checkpoints in an output directory with the experiment name and current timestamp.
|
||||
|
||||
## Evaluating trained Blender models
|
||||
|
||||
Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/launch_eval_blender.sh`.
|
||||
|
||||
Say we ran a benchmark on 08-10-2022 for instant_ngp. By default, the train script will save the benchmarks in the following format:
|
||||
Say we ran a benchmark on 08-10-2022 for `instant-ngp`. By default, the train script will save the benchmarks in the following format:
|
||||
|
||||
```
|
||||
outputs
|
||||
└───blender_chair_08-10-2022
|
||||
| └───instant_ngp
|
||||
└───blender_chair_2022-08-10
|
||||
| └───instant-ngp
|
||||
| └───2022-08-10_172517
|
||||
| └───train.log
|
||||
| └───config.yml
|
||||
| ...
|
||||
└───blender_drums_08-10-2022
|
||||
| └───instant_ngp
|
||||
└───blender_drums_2022-08-10
|
||||
| └───instant-ngp
|
||||
| └───2022-08-10_172517
|
||||
| └───train.log
|
||||
| └───config.yml
|
||||
| ...
|
||||
...
|
||||
```
|
||||
@@ -52,25 +61,29 @@ outputs
|
||||
If we wanted to run the benchmark on all the blender data for the above example, we would run:
|
||||
|
||||
```bash
|
||||
./scripts/benchmarking/launch_eval_blender.sh -c graph_instant_ngp -o ./outputs/ -m 08 -d 10 -y 2022 -s 172517 -g 4 5 6 7
|
||||
|
||||
./scripts/benchmarking/launch_eval_blender.sh -m instant-ngp -o outputs/ -t 2022-08-10_172517
|
||||
```
|
||||
|
||||
The flags used in the benchmarking script are defined as follows:
|
||||
|
||||
- `-c`: config name (e.g. `graph_instant_ngp`). This should be the same as what was passed in for -c in the train script.
|
||||
- `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the hydra base dir.
|
||||
- `-m`: month of benchmark (e.g. `08`) of format 02d.
|
||||
- `-d`: date of benchmark (e.g. `10`) of format 02d.
|
||||
- `-y`: year of benchmark (e.g. `2022`).
|
||||
- `-s`: seconds timestamp of benchmark (e.g. `172517`) of format 06d.
|
||||
- `-m`: config name (e.g. `instant-ngp`). This should be the same as what was passed in for -c in the train script.
|
||||
- `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the `--output-dir` in the base `Config` for training.
|
||||
- `-t`: timestamp of benchmark; also the identifier (e.g. `2022-08-10_172517`).
|
||||
- `-g`: specifies the gpus to use and if not specified (no -g flag), will automaticaly search for available gpus.
|
||||
|
||||
The script will simultaneously run the benchmarking across all the objects in the blender dataset and calculates the PSNR/FPS/other stats. The results are saved as .json files in the `-o` directory with the following format:
|
||||
|
||||
```
|
||||
outputs
|
||||
└───instant_ngp
|
||||
| └───blender_chair_08-10-2022_172517.json
|
||||
| | blender_ficus_08-10-2022_172517.json
|
||||
└───instant-ngp
|
||||
| └───blender_chair_2022-08-10_172517.json
|
||||
| | blender_ficus_2022-08-10_172517.json
|
||||
| | ...
|
||||
```
|
||||
|
||||
:::{admonition} Warning
|
||||
:class: warning
|
||||
|
||||
Since we are running multiple backgrounded processes concurrently with this script, please note the terminal logs may be messy.
|
||||
:::
|
||||
@@ -0,0 +1,12 @@
|
||||
Debugging tools
|
||||
====================
|
||||
|
||||
We document a few of the supported tooling systems and pipelines we support for debugging our models (e.g. profiling to debug speed).
|
||||
As we grow, we hope to provide more updated and extensive tooling support.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
local_logger
|
||||
profiling
|
||||
benchmarking
|
||||
@@ -0,0 +1,48 @@
|
||||
# Local writer
|
||||
|
||||
The `LocalWriter` simply outputs numerical stats to the terminal.
|
||||
You can specify additional parameters to customize your logging experience.
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class LocalWriterConfig(InstantiateConfig):
|
||||
"""Local Writer config"""
|
||||
|
||||
_target: Type = writer.LocalWriter
|
||||
"""target class to instantiate"""
|
||||
enable: bool = False
|
||||
"""if True enables local logging, else disables"""
|
||||
stats_to_track: Tuple[writer.EventName, ...] = (
|
||||
writer.EventName.ITER_TRAIN_TIME,
|
||||
writer.EventName.TRAIN_RAYS_PER_SEC,
|
||||
writer.EventName.CURR_TEST_PSNR,
|
||||
writer.EventName.VIS_RAYS_PER_SEC,
|
||||
writer.EventName.TEST_RAYS_PER_SEC,
|
||||
)
|
||||
"""specifies which stats will be logged/printed to terminal"""
|
||||
max_log_size: int = 10
|
||||
"""maximum number of rows to print before wrapping. if 0, will print everything."""
|
||||
```
|
||||
|
||||
- `enable`: enable/disable the logger.
|
||||
- `stats_to_track`: all the stats that you want to print to the terminal (see list under `EventName` in `utils/writer.py`). You can add or remove any of the defined enums.
|
||||
- `max_log_size`: how much content to print onto the screen (By default, only print 10 lines onto the screen at a time). If 0, will print everything without deleting any previous lines.
|
||||
|
||||
:::{admonition} Tip
|
||||
:class: info
|
||||
|
||||
If you want to create a new stat to track, simply add the stat name to the `EventName` enum.
|
||||
- Remember to call some put event (e.g. `put_scalar` from `utils/writer.py` to place the value in the `EVENT_STORAGE`.
|
||||
- Remember to add the new enum to the `stats_to_track` list
|
||||
:::
|
||||
|
||||
The local writer is easily configurable via CLI.
|
||||
A few common commands to use:
|
||||
|
||||
```bash
|
||||
# disable local writer
|
||||
ns-train {METHOD_NAME} --logging.local-writer.no-enable
|
||||
|
||||
# disable line wrapping
|
||||
ns-train {METHOD_NAME} --logging.local-writer.max-log-size=0
|
||||
```
|
||||
@@ -0,0 +1,61 @@
|
||||
# Code profiling support
|
||||
|
||||
We provide built-in performance profiling capabilities to make it easier for you to debug and assess the performance of your code.
|
||||
|
||||
#### In-house profiler
|
||||
|
||||
You can use our built-in profiler. By default, it is enabled and will print at the termination of the program. You can disable it via CLI using the flag `--logging.no-enable-profiler`.
|
||||
|
||||
|
||||
The profiler computes the average total time of execution for any function with the `@profiler.time_function` decorator.
|
||||
For instance, if you wanted to profile the total time it takes to generate rays given pixel and camera indices via the `RayGenerator` class, you might want to time its `forward()` function. In that case, you would need to add the decorator to the function.
|
||||
|
||||
```python
|
||||
"""nerfstudio/model_components/ray_generators.py""""
|
||||
|
||||
class RayGenerator(nn.Module):
|
||||
|
||||
...
|
||||
|
||||
@profiler.time_function # <-- add the profiler decorator before the function
|
||||
def forward(self, ray_indices: TensorType["num_rays", 3]) -> RayBundle:
|
||||
# implementation here
|
||||
...
|
||||
```
|
||||
|
||||
At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag.
|
||||
|
||||
:::{admonition} Tip
|
||||
:class: info
|
||||
|
||||
Use this profiler if there are *specific/individual functions* you want to measure the times for.
|
||||
:::
|
||||
|
||||
|
||||
#### Profiling with PySpy
|
||||
|
||||
If you want to profile the entire codebase, consider using [PySpy](https://github.com/benfred/py-spy).
|
||||
|
||||
Install PySpy
|
||||
|
||||
```bash
|
||||
pip install py-spy
|
||||
```
|
||||
|
||||
To perform the profiling, you can either specify that you want to generate a flame graph or generate a live-view of the profiler.
|
||||
|
||||
```bash
|
||||
## flame graph: with wandb logging and our inhouse logging disabled
|
||||
program="ns-train nerfacto -- --vis=wandb --logging.no-enable-profiler blender-data"
|
||||
py-spy record -o {PATH_TO_OUTPUT_SVG} $program
|
||||
```
|
||||
|
||||
```bash
|
||||
## top-down stats: running same program configuration as above
|
||||
py-spy top $program
|
||||
```
|
||||
:::{admonition} Attention
|
||||
:class: attention
|
||||
|
||||
In defining `program`, you will need to add an extra `--` before you specify your program's arguments.
|
||||
:::
|
||||
@@ -1,117 +0,0 @@
|
||||
# Loggers and Profilers
|
||||
|
||||
## Logging support
|
||||
|
||||
We provide integration with multiple logging interfaces to log images and statistics during training.
|
||||
All of these loggers are configurable via the config:
|
||||
|
||||
```yaml
|
||||
# e.g. configs/graphs_default.yaml
|
||||
|
||||
logging:
|
||||
steps_per_log: 10 # how often to log losses/images
|
||||
max_buffer_size: 20 # defines how many steps to average over (if reporting averages)
|
||||
writer: # definition of all writers you want to use
|
||||
TensorboardWriter:
|
||||
log_dir: './' # must specify the output path for all writers
|
||||
LocalWriter:
|
||||
log_dir: './'
|
||||
stats_to_track: [ITER_TRAIN_TIME, TRAIN_RAYS_PER_SEC, CURR_TEST_PSNR]
|
||||
max_log_size: 10
|
||||
```
|
||||
|
||||
Currently, we provide support for the following loggers, which can be added to the `logging.writer` section of the config.
|
||||
|
||||
#### Tensorboard
|
||||
|
||||
To log training stats and images with [tensorboard](https://www.tensorflow.org/tensorboard):
|
||||
|
||||
```
|
||||
TensorboardWriter:
|
||||
log_dir: "./" # the output path for which the event files are written
|
||||
```
|
||||
|
||||
#### Weights and Biases
|
||||
|
||||
To log training stats and images with [wandb](https://wandb.ai/site):
|
||||
|
||||
```
|
||||
WandbWriter:
|
||||
log_dir: "./" # the output path for which the run files are written
|
||||
```
|
||||
|
||||
#### Local writer
|
||||
|
||||
```
|
||||
LocalWriter:
|
||||
log_dir: "./" # if images are saved, where the images would be saved
|
||||
stats_to_track:
|
||||
[ITER_TRAIN_TIME, TRAIN_RAYS_PER_SEC, CURR_TEST_PSNR]
|
||||
max_log_size: 10
|
||||
```
|
||||
|
||||
The `LocalWriter` simply outputs to the terminal. However, you can specify additional parameters to customize the logging.
|
||||
|
||||
- `stats_to_track`: all the stats that you want to print to the terminal (see list under `EventName` in `utils/writer.py`). You can add or remove any of the defined enums.
|
||||
- `max_log_size`: how much content to print onto the screen (e.g. only print 10 lines on the screen at a time). If 0, will print everything without deleting any previous lines.
|
||||
|
||||
## Code profiling support
|
||||
|
||||
We provide built-in performance profiling capabilities to make it easier for you to debug and assess the performance of your code.
|
||||
|
||||
#### In-house profiler
|
||||
|
||||
You can use our built-in profiler by enabling the profiler in the config:
|
||||
|
||||
```yaml
|
||||
# e.g. configs/graphs_default.yaml
|
||||
|
||||
logging:
|
||||
enable_profiler: True
|
||||
```
|
||||
|
||||
The profiler computes the average total time of execution for any function with the `@profiler.time_function` decorator.
|
||||
For instance, if you wanted to profile the total time it takes to generate rays given pixel and camera indices via the `RayGenerator` class, you would want to evaluate its `forward()` function. In that class, you would need to add the decorator to the function.
|
||||
|
||||
```python
|
||||
class RayGenerator(nn.Module):
|
||||
|
||||
...
|
||||
|
||||
@profiler.time_function # add the profiler decorator before the function
|
||||
def forward(self, ray_indices: TensorType["num_rays", 3]) -> RayBundle:
|
||||
# implementation here
|
||||
...
|
||||
```
|
||||
|
||||
At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag.
|
||||
|
||||
Use this profiler if there are specific functions that you want to measure the times for.
|
||||
|
||||
#### Profiling with PySpy
|
||||
|
||||
If you want to profile the entire codebase, we provide functionality to generate flame graphs or get real-time reports using [PySpy](https://github.com/benfred/py-spy).
|
||||
|
||||
Install PySpy
|
||||
|
||||
```bash
|
||||
pip install py-spy
|
||||
```
|
||||
|
||||
To perform the profiling, you can either specify that you want to generate a flame graph or generate a live-view of the profiler.
|
||||
|
||||
```bash
|
||||
## for flame graph
|
||||
|
||||
./scripts/debugging/profile.sh -t flame -o flame.svg -p <python_command>
|
||||
|
||||
## for live view of functions
|
||||
|
||||
./scripts/debugging/profile.sh -t top -p <python_command>
|
||||
```
|
||||
|
||||
In both cases, `<python_command>` can be replaced with any command you would normally run in the terminal. For instance, you can replace `<python_function>` with `scripts/train.py data/dataset=blender_lego`. Then the full command would be e.g.:
|
||||
|
||||
```bash
|
||||
./scripts/debugging/profile.sh -t flame -o flame.svg -p scripts/train.py data/dataset=blender_lego
|
||||
```
|
||||
Reference in new issue
Block a user