mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-20 14:45:38 +08:00
1 parent
8af3a7a385
commit
cf34ef4bd0
26 files changed
+384
-355
No files matched your search
@@ -1,7 +1,6 @@
|
||||
# Benchmarking workflow
|
||||
|
||||
|
||||
We make it easy to benchmark your new NeRF against the standard Blender dataset.
|
||||
We make it easy to benchmark your new NeRF against the standard Blender dataset.
|
||||
|
||||
## Launching training on Blender dataset
|
||||
|
||||
@@ -13,8 +12,9 @@ To launch training jobs automatically on each of these items, you can call:
|
||||
```
|
||||
|
||||
Simply replace the arguments in brackets with the correct arguments.
|
||||
* `-c`: name of the config you created for your new graph. For instance, if you have created the config `configs/graph_instant_ngp.yaml`, `<config-name>` will be `graph_instant_ngp`.
|
||||
* `-g`: specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs.
|
||||
|
||||
- `-c`: name of the config you created for your new graph. For instance, if you have created the config `configs/graph_instant_ngp.yaml`, `<config-name>` will be `graph_instant_ngp`.
|
||||
- `-g`: specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs.
|
||||
|
||||
A full example would be:
|
||||
|
||||
@@ -28,39 +28,42 @@ A full example would be:
|
||||
|
||||
The script will automatically launch training on all of the items and save the checkpoints in an output directory with the current date and timestamp.
|
||||
|
||||
|
||||
## Evaluating trained Blender models
|
||||
|
||||
Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/launch_eval_blender.sh`.
|
||||
|
||||
Say we ran a benchmark on 08-10-2022 for instant_ngp. By default, the train script will save the benchmarks in the following format:
|
||||
|
||||
```
|
||||
outputs
|
||||
└───blender_chair_08-10-2022
|
||||
| └───instant_ngp
|
||||
| └───2022-08-10_172517
|
||||
| └───run_train.log
|
||||
| └───train.log
|
||||
| ...
|
||||
└───blender_drums_08-10-2022
|
||||
| └───instant_ngp
|
||||
| └───2022-08-10_172517
|
||||
| └───run_train.log
|
||||
| └───train.log
|
||||
| ...
|
||||
...
|
||||
```
|
||||
|
||||
If we wanted to run the benchmark on all the blender data for the above example, we would run:
|
||||
|
||||
```bash
|
||||
./scripts/benchmarking/launch_eval_blender.sh -c graph_instant_ngp -o ./outputs/ -m 08 -d 10 -y 2022 -s 172517 -g 4 5 6 7
|
||||
```
|
||||
|
||||
The flags used in the benchmarking script are defined as follows:
|
||||
* `-c`: config name (e.g. `graph_instant_ngp`). This should be the same as what was passed in for -c in the train script.
|
||||
* `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the hydra base dir.
|
||||
* `-m`: month of benchmark (e.g. `08`) of format 02d.
|
||||
* `-d`: date of benchmark (e.g. `10`) of format 02d.
|
||||
* `-y`: year of benchmark (e.g. `2022`).
|
||||
* `-s`: seconds timestamp of benchmark (e.g. `172517`) of format 06d.
|
||||
* `-g`: specifies the gpus to use and if not specified (no -g flag), will automaticaly search for available gpus.
|
||||
|
||||
- `-c`: config name (e.g. `graph_instant_ngp`). This should be the same as what was passed in for -c in the train script.
|
||||
- `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the hydra base dir.
|
||||
- `-m`: month of benchmark (e.g. `08`) of format 02d.
|
||||
- `-d`: date of benchmark (e.g. `10`) of format 02d.
|
||||
- `-y`: year of benchmark (e.g. `2022`).
|
||||
- `-s`: seconds timestamp of benchmark (e.g. `172517`) of format 06d.
|
||||
- `-g`: specifies the gpus to use and if not specified (no -g flag), will automaticaly search for available gpus.
|
||||
|
||||
The script will simultaneously run the benchmarking across all the objects in the blender dataset and calculates the PSNR/FPS/other stats. The results are saved as .json files in the `-o` directory with the following format:
|
||||
|
||||
@@ -70,4 +73,4 @@ outputs
|
||||
| └───blender_chair_08-10-2022_172517.json
|
||||
| | blender_ficus_08-10-2022_172517.json
|
||||
| | ...
|
||||
```
|
||||
```
|
||||
@@ -2,72 +2,77 @@
|
||||
|
||||
## Logging support
|
||||
|
||||
We provide integration with multiple logging interfaces to log images and statistics during training.
|
||||
We provide integration with multiple logging interfaces to log images and statistics during training.
|
||||
All of these loggers are configurable via the config:
|
||||
|
||||
```yaml
|
||||
# e.g. configs/graphs_default.yaml
|
||||
|
||||
logging:
|
||||
steps_per_log: 10 # how often to log losses/images
|
||||
max_buffer_size: 20 # defines how many steps to average over (if reporting averages)
|
||||
writer: # definition of all writers you want to use
|
||||
TensorboardWriter:
|
||||
log_dir: "./" # must specify the output path for all writers
|
||||
LocalWriter:
|
||||
log_dir: "./"
|
||||
stats_to_track:
|
||||
[ITER_TRAIN_TIME, TRAIN_RAYS_PER_SEC, CURR_TEST_PSNR]
|
||||
max_log_size: 10
|
||||
steps_per_log: 10 # how often to log losses/images
|
||||
max_buffer_size: 20 # defines how many steps to average over (if reporting averages)
|
||||
writer: # definition of all writers you want to use
|
||||
TensorboardWriter:
|
||||
log_dir: './' # must specify the output path for all writers
|
||||
LocalWriter:
|
||||
log_dir: './'
|
||||
stats_to_track: [ITER_TRAIN_TIME, TRAIN_RAYS_PER_SEC, CURR_TEST_PSNR]
|
||||
max_log_size: 10
|
||||
```
|
||||
|
||||
Currently, we provide support for the following loggers, which can be added to the `logging.writer` section of the config.
|
||||
|
||||
#### Tensorboard
|
||||
|
||||
To log training stats and images with [tensorboard](https://www.tensorflow.org/tensorboard):
|
||||
|
||||
```
|
||||
TensorboardWriter:
|
||||
log_dir: "./" # the output path for which the event files are written
|
||||
```
|
||||
|
||||
#### Weights and Biases
|
||||
|
||||
To log training stats and images with [wandb](https://wandb.ai/site):
|
||||
|
||||
```
|
||||
WandbWriter:
|
||||
log_dir: "./" # the output path for which the run files are written
|
||||
```
|
||||
|
||||
#### Local writer
|
||||
|
||||
```
|
||||
LocalWriter:
|
||||
log_dir: "./" # if images are saved, where the images would be saved
|
||||
stats_to_track:
|
||||
stats_to_track:
|
||||
[ITER_TRAIN_TIME, TRAIN_RAYS_PER_SEC, CURR_TEST_PSNR]
|
||||
max_log_size: 10
|
||||
max_log_size: 10
|
||||
```
|
||||
|
||||
The `LocalWriter` simply outputs to the terminal. However, you can specify additional parameters to customize the logging.
|
||||
* `stats_to_track`: all the stats that you want to print to the terminal (see list under `EventName` in `utils/writer.py`). You can add or remove any of the defined enums.
|
||||
* `max_log_size`: how much content to print onto the screen (e.g. only print 10 lines on the screen at a time). If 0, will print everything without deleting any previous lines.
|
||||
|
||||
- `stats_to_track`: all the stats that you want to print to the terminal (see list under `EventName` in `utils/writer.py`). You can add or remove any of the defined enums.
|
||||
- `max_log_size`: how much content to print onto the screen (e.g. only print 10 lines on the screen at a time). If 0, will print everything without deleting any previous lines.
|
||||
|
||||
## Code profiling support
|
||||
We provide built-in performance profiling capabilities to make it easier for you to debug and assess the performance of your code.
|
||||
|
||||
We provide built-in performance profiling capabilities to make it easier for you to debug and assess the performance of your code.
|
||||
|
||||
#### In-house profiler
|
||||
|
||||
You can use our built-in profiler by enabling the profiler in the config:
|
||||
|
||||
```yaml
|
||||
# e.g. configs/graphs_default.yaml
|
||||
|
||||
logging:
|
||||
enable_profiler: True
|
||||
enable_profiler: True
|
||||
```
|
||||
|
||||
The profiler computes the average total time of execution for any function with the `@profiler.time_function` decorator.
|
||||
The profiler computes the average total time of execution for any function with the `@profiler.time_function` decorator.
|
||||
For instance, if you wanted to profile the total time it takes to generate rays given pixel and camera indices via the `RayGenerator` class, you would want to evaluate its `forward()` function. In that class, you would need to add the decorator to the function.
|
||||
|
||||
|
||||
```python
|
||||
class RayGenerator(nn.Module):
|
||||
|
||||
@@ -79,12 +84,12 @@ class RayGenerator(nn.Module):
|
||||
...
|
||||
```
|
||||
|
||||
At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag.
|
||||
At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag.
|
||||
|
||||
Use this profiler if there are specific functions that you want to measure the times for.
|
||||
|
||||
|
||||
#### Profiling with PySpy
|
||||
|
||||
If you want to profile the entire codebase, we provide functionality to generate flame graphs or get real-time reports using [PySpy](https://github.com/benfred/py-spy).
|
||||
|
||||
Install PySpy
|
||||
@@ -102,11 +107,11 @@ To perform the profiling, you can either specify that you want to generate a fla
|
||||
|
||||
## for live view of functions
|
||||
|
||||
./scripts/debugging/profile.sh -t top -p <python_command>
|
||||
./scripts/debugging/profile.sh -t top -p <python_command>
|
||||
```
|
||||
|
||||
In both cases, `<python_command>` can be replaced with any command you would normally run in the terminal. For instance, you can replace `<python_function>` with `scripts/run_train.py data/dataset=blender_lego`. Then the full command would be e.g.:
|
||||
In both cases, `<python_command>` can be replaced with any command you would normally run in the terminal. For instance, you can replace `<python_function>` with `scripts/train.py data/dataset=blender_lego`. Then the full command would be e.g.:
|
||||
|
||||
```bash
|
||||
./scripts/debugging/profile.sh -t flame -o flame.svg -p scripts/run_train.py data/dataset=blender_lego
|
||||
```
|
||||
./scripts/debugging/profile.sh -t flame -o flame.svg -p scripts/train.py data/dataset=blender_lego
|
||||
```
|
||||
Reference in new issue
Block a user