diff --git a/docs/tooling/benchmarking.md b/docs/tooling/benchmarking.md index 42a39ec04..80a142419 100644 --- a/docs/tooling/benchmarking.md +++ b/docs/tooling/benchmarking.md @@ -8,41 +8,66 @@ To start, you will need to train your NeRF on each of the blender objects. To launch training jobs automatically on each of these items, you can call: ```bash -./scripts/benchmarking/launch_train_blender.sh +./scripts/benchmarking/launch_train_blender.sh -c -g ``` Simply replace the arguments in brackets with the correct arguments. -* ``: name of the config you created for your new graph. For instance, if you have created the config `configs/graph_vanilla_nerf.yaml`, `` will be `graph_vanilla_nerf`. -* ``: specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs. +* `-c`: name of the config you created for your new graph. For instance, if you have created the config `configs/graph_instant_ngp.yaml`, `` will be `graph_instant_ngp`. +* `-g`: specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs. A full example would be: ```bash -./scripts/benchmarking/launch_train_blender.sh graph_vanilla_nerf 0 1 2 3 +# with -g specifying gpus +./scripts/benchmarking/launch_train_blender.sh -c graph_instant_ngp -g 0 1 2 3 + +# without -g automatically finds available gpus +./scripts/benchmarking/launch_train_blender.sh -c graph_instant_ngp ``` -The script will automatically launch training on all of the items and save the checkpoints in an output directory with the current date. -You will use the date as the reference when you run inference for benchmarking. +The script will automatically launch training on all of the items and save the checkpoints in an output directory with the current date and timestamp. + #### Benchmarking on Blender -Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/run_benchmark.py`. +Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/launch_eval_blender.sh`. -Modify the `BENCH` variable to specify which jobs you want benchmarked. - -```python -BENCH = { - "method": "graph_vanilla_nerf", # change this to whatever you want - "hydra_base_dir": "outputs/", # change this to wherever hydra's default output directory is - "benchmark_date": "05-26-2022", # change this to the date you ran the benchmarking - "object_list": ["mic", "ficus", "chair", "hotdog", "materials", "drums", "ship", "lego"], -} +Say we ran a benchmark on 08-10-2022 for instant_ngp. By default, the train script will save the benchmarks in the following format: +``` +outputs +└───blender_chair_08-10-2022 +| └───instant_ngp +| └───2022-08-10_172517 +| └───run_train.log +| ... +└───blender_drums_08-10-2022 +| └───instant_ngp +| └───2022-08-10_172517 +| └───run_train.log +| ... +... ``` -Then you can simply run +In the above example, we can break down the path into component flags that will be passed into the benchmarking script: +* `-c`: config name (e.g. `graph_instant_ngp`). This should be the same as what was passed in for -c in the train script. +* `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the hydra base dir. +* `-m`: month of benchmark (e.g. `08`) of format 02d. +* `-d`: date of benchmark (e.g. `10`) of format 02d. +* `-y`: year of benchmark (e.g. `2022`). +* `-s`: seconds timestamp of benchmark (e.g. `172517`) of format 06d. +If we wanted to run the full benchmark for all the blender data given the above example, we would run the following: ```bash -python scripts/benchmarking/run_benchmark.py - +./scripts/benchmarking/launch_eval_blender.sh -c graph_instant_ngp -o ./outputs/ -m 08 -d 10 -y 2022 -s 172517 -g 4 5 6 7 ``` -The script will search for the most recent checkpoint according to the benchmark configuration specified via `BENCH`. It will run inference and calculate the PSNR according to the respective object. The output is then saved in the `hydra_base_dir` directory as `.json` (full path: `/.json`). \ No newline at end of file +where `-g` again specifies the gpus to use and if not specified (no -g flag), will automaticaly search for available gpus. + +The script will simultaneously run the benchmarking across all the objects in the blender dataset and calculates the PSNR/FPS/other stats. The results are saved as .json files in the `-o` directory with the following format: + +``` +outputs +└───instant_ngp +| └───blender_chair_08-10-2022_172517.json +| | blender_ficus_08-10-2022_172517.json +| | ... +``` \ No newline at end of file