update doc to reflect new benchmarking (#264)

* added new .sh script for eval; removed old python script

* fixed changing dataset

* done refactoring eval script

* update docs

docs/tooling/benchmarking.md

@@ -8,41 +8,66 @@ To start, you will need to train your NeRF on each of the blender objects.
 To launch training jobs automatically on each of these items, you can call:
 
 ```bash
-./scripts/benchmarking/launch_train_blender.sh <config-name> <list_of_gpus>
+./scripts/benchmarking/launch_train_blender.sh -c <config-name> -g <list-of-gpus>
 ```
 
 Simply replace the arguments in brackets with the correct arguments.
-* `<config-name>`: name of the config you created for your new graph. For instance, if you have created the config `configs/graph_vanilla_nerf.yaml`, `<config-name>` will be `graph_vanilla_nerf`.
-* `<list_of_gpus>`: specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs.
+* `-c`: name of the config you created for your new graph. For instance, if you have created the config `configs/graph_instant_ngp.yaml`, `<config-name>` will be `graph_instant_ngp`.
+* `-g`: specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs.
 
 A full example would be:
 
 ```bash
-./scripts/benchmarking/launch_train_blender.sh graph_vanilla_nerf 0 1 2 3
+# with -g specifying gpus
+./scripts/benchmarking/launch_train_blender.sh -c graph_instant_ngp -g 0 1 2 3
+
+# without -g automatically finds available gpus
+./scripts/benchmarking/launch_train_blender.sh -c graph_instant_ngp
 ```
 
-The script will automatically launch training on all of the items and save the checkpoints in an output directory with the current date. 
-You will use the date as the reference when you run inference for benchmarking.
+The script will automatically launch training on all of the items and save the checkpoints in an output directory with the current date and timestamp.
+
 
 #### Benchmarking on Blender
-Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/run_benchmark.py`.
+Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/launch_eval_blender.sh`.
 
-Modify the `BENCH` variable to specify which jobs you want benchmarked. 
-
-```python
-BENCH = {
-    "method": "graph_vanilla_nerf",     # change this to whatever you want
-    "hydra_base_dir": "outputs/",       # change this to wherever hydra's default output directory is
-    "benchmark_date": "05-26-2022",     # change this to the date you ran the benchmarking
-    "object_list": ["mic", "ficus", "chair", "hotdog", "materials", "drums", "ship", "lego"],
-}
+Say we ran a benchmark on 08-10-2022 for instant_ngp. By default, the train script will save the benchmarks in the following format:
+```
+outputs
+└───blender_chair_08-10-2022
+|   └───instant_ngp
+|       └───2022-08-10_172517
+|           └───run_train.log
+|               ...
+└───blender_drums_08-10-2022
+|   └───instant_ngp
+|       └───2022-08-10_172517
+|           └───run_train.log
+|               ...
+...
 ```
 
-Then you can simply run
+In the above example, we can break down the path into component flags that will be passed into the benchmarking script:
+* `-c`: config name (e.g. `graph_instant_ngp`). This should be the same as what was passed in for -c in the train script.
+* `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the hydra base dir.
+* `-m`: month of benchmark (e.g. `08`) of format 02d. 
+* `-d`: date of benchmark (e.g. `10`) of format 02d.
+* `-y`: year of benchmark (e.g. `2022`).
+* `-s`: seconds timestamp of benchmark (e.g. `172517`) of format 06d.
 
+If we wanted to run the full benchmark for all the blender data given the above example, we would run the following:
 ```bash
-python scripts/benchmarking/run_benchmark.py
-
+./scripts/benchmarking/launch_eval_blender.sh -c graph_instant_ngp -o ./outputs/ -m 08 -d 10 -y 2022 -s 172517 -g 4 5 6 7
 ```
 
-The script will search for the most recent checkpoint according to the benchmark configuration specified via `BENCH`. It will run inference and calculate the PSNR according to the respective object. The output is then saved in the `hydra_base_dir` directory as `<benchmark_date>.json` (full path: `<hydra_base_dir>/<benchmark_date>.json`).
+where `-g` again specifies the gpus to use and if not specified (no -g flag), will automaticaly search for available gpus.
+
+The script will simultaneously run the benchmarking across all the objects in the blender dataset and calculates the PSNR/FPS/other stats. The results are saved as .json files in the `-o` directory with the following format:
+
+```
+outputs
+└───instant_ngp
+|   └───blender_chair_08-10-2022_172517.json
+|   |   blender_ficus_08-10-2022_172517.json
+|   |   ...
+```