From 394973944927bc3a58154507521491e19e83722a Mon Sep 17 00:00:00 2001 From: evonneng Date: Wed, 8 Jun 2022 17:27:08 -0700 Subject: [PATCH] Created skeleton structure for documentation (#30) * added contribution docs * set up skeleton for readthedocs * setup skeleton structure for documentation * added tooling docs * fixed broken link --- README.md | 147 ++++++----------------------- docs/index.rst | 67 +++++++++++-- docs/quickstart/installation.rst | 59 ------------ docs/quickstart/quick_tour.md | 90 ------------------ docs/reference/index.rst | 17 ---- docs/tooling/benchmarking.md | 48 ++++++++++ docs/tooling/hydra.md | 1 + docs/tooling/index.rst | 12 +++ docs/tooling/jupyter.md | 10 ++ docs/tooling/logging.md | 30 ++++++ docs/tooling/other_codebase.md | 16 ++++ docs/tooling/profiling.md | 63 +++++++++++++ docs/tooling/speedup.md | 14 +++ docs/tutorials/contributing.md | 20 ++++ docs/tutorials/creating_dataset.md | 2 + docs/tutorials/creating_graphs.md | 144 ++++++++++++++++++++++++++++ docs/tutorials/quickstart.md | 9 ++ 17 files changed, 457 insertions(+), 292 deletions(-) delete mode 100644 docs/quickstart/installation.rst delete mode 100644 docs/quickstart/quick_tour.md create mode 100644 docs/tooling/benchmarking.md create mode 100644 docs/tooling/hydra.md create mode 100644 docs/tooling/index.rst create mode 100644 docs/tooling/jupyter.md create mode 100644 docs/tooling/logging.md create mode 100644 docs/tooling/other_codebase.md create mode 100644 docs/tooling/profiling.md create mode 100644 docs/tooling/speedup.md create mode 100644 docs/tutorials/contributing.md create mode 100644 docs/tutorials/creating_dataset.md create mode 100644 docs/tutorials/creating_graphs.md create mode 100644 docs/tutorials/quickstart.md diff --git a/README.md b/README.md index 6520404dd..4d8a68954 100644 --- a/README.md +++ b/README.md @@ -13,13 +13,16 @@

-- [Quickstart](#quickstart) -- [Walk-through tour](#walk-through-tour) -- [Feature List](#feature-list) -- [Benchmarked Model Architectures](#benchmarked-model-architectures) + +* [Quickstart](#quickstart) +* [Supported Features](#supported-features) +* [Benchmarked Model Architectures](#benchmarked-model-architectures) # Quickstart +The quickstart will help you get started with the default vanilla nerf trained on the classic blender lego scene. +For more complex changes (e.g. running with your own data/ setting up a new NeRF graph, please see our [docs](https://plenoptix-pyrad.readthedocs-hosted.com/en/latest/quickstart/quick_tour.html). + #### 1. Installation: Setup the environment This repository is tested with cuda 11.3 @@ -57,10 +60,14 @@ Download the original [NeRF dataset](https://drive.google.com/drive/folders/128y #### 3. Training a model +To run with all the defaults, e.g. vanilla nerf method with the blender lego image: ``` # Run with default config python scripts/run_train.py +``` +With support for [Hydra](https://hydra.cc/), you can run with other configurations by changing appropriate configs defined in `configs/` or by setting flags via command-line arguments: +``` # Run with config changes python scripts/run_train.py machine_config.num_gpus=1 python scripts/run_train.py data.dataset.downscale_factor=1 @@ -78,139 +85,41 @@ python scripts/run_data_preprocessor.py python scripts/run_train.py ++data.dataset.use_cache=true ``` -# Walk-through tour - -In this quick tour, we will walk you through the core of training and building any NeRFs with pyrad. - -#### Launching a training job - -The entry point for training starts at `scripts/run_train.py`, which spawns instances of our `Trainer()` class (in `nerf/trainer.py`). The `Trainer()` is responsible for setting up the datasets and NeRF graph depending on the config specified. It will then run the usual train/val routine for a config-specified number of iterations. If you are planning on using our codebase to build a new NeRF method or to use an existing implementation, we've abstracted away the training routine in these two files and chances are you will not need to think of them again. - -#### Graphs, Fields, and Modules - -If you are looking to implemnet a new NeRF method or extend an existing one, you only need to edit files in `nerf/graph/`, `nerf/fields/`, `nerf/field_modules/`, `nerf/misc_modules/`. (TODO: restructuring) - -The actual NeRF graph definitions can be found in `nerf/graph/`. For instance, to implement the vanilla NeRF, we create a new class that inherits the abstract Graph class. To fully implement the any new graph class, you will need to implement the following abstract methods defined in the skeleton code below. See also `nerf/graph/vanilla_nerf.py` for the full implementation. +#### 4. Visualizing training runs +If you run everything with the default configuration, by default, we use [TensorBoard](https://www.tensorflow.org/tensorboard) to log all training curves, test images, and other stats. Once the job is launched, you will be able to track training by launching the tensorboard in `outputs/blender_lego/vanilla_nerf//`. ``` -class NeRFGraph(Graph): - """Vanilla NeRF graph""" - - def __init__(self, intrinsics=None, camera_to_world=None, **kwargs) -> None: - super().__init__(intrinsics=intrinsics, camera_to_world=camera_to_world, **kwargs) - - def populate_fields(self): - """ - Set all field related modules here - """ - - def populate_misc_modules(self): - """ - Set all remaining modules here including: samplers, renderers, losses, and metrics - """ - - def get_param_groups(self) -> Dict[str, List[Parameter]]: - """ - Create a dictionary of parameters that are grouped according to different optimizers - """ - - def get_outputs(self, ray_bundle: RayBundle): - """ - Takes in a Ray Bundle and returns a dictionary of outputs. - """ - - def get_loss_dict(self, outputs, batch): - """ - Computes and returns the losses. - """ - - def log_test_image_outputs(self, image_idx, step, batch, outputs): - """ - Writes the test image outputs. - """ +tensorboard --logdir outputs/blender_lego/vanilla_nerf/ ``` -Note that the graph is composed of fields and modules. +#### 5. Rendering a trajectories during inference +TODO(ethan) -**Fields** (`nerf/fields/`) represents the actual radiance field of the NeRF and is composed of field modules (`nerf/field_modules/`). Here, we define the field as the part of the network that takes in point samples and any other conditioning, and outputs any of the `FieldHeadNames` (`nerf/field_modules/field_heads.py`). The **misc. modules** can be any module outside of the field that are needed by the NeRF (e.g. losses, samplers, renderers). -To get started on a new NeRF implementation, you simply have to define all relevant modules and populate them in the graph. +#### 6. In-depth guide +For a more in-depth tutorial on how to modify/implement your own NeRF Graph, please see our [walk-through](#). -#### Dataset population TODO(ethan) -#### Config +# Supported Features -Now that you have the graph and dataset all set up, you're ready to create the config that you pass into our run train routine. Our config system is powered by [Hydra](https://hydra.cc/). All Hydra and machine related arguments are stored in `configs/default_setup.yaml`, as well as the defaults list. -To set up the graph config, create a new yaml under `configs/`. +We provide the following support strucutures to make life easier for getting started with NeRFs. For a full description, please refer to our [features page](#). -``` -# configs/vanilla_nerf.yaml +If you are looking for a feature that is not currently supported, please do not hesitate to contact the Plenoptix team! -defaults: - - default_setup # inherit the basic yaml heirarchy - - _self_ +#### :metal: Support for [Hydra](https://hydra.cc/) config structure -experiment_name: blender_lego -method_name: vanilla_nerf +#### :metal: Support for multiple logging interfaces -graph: - network: - _target_: pyrad.graph.vanilla_nerf.NeRFGraph # set the target to the graph you defined +#### :metal: Built-in support for profiling code - # +#### :metal: Benchmarking scripts -data: - # -``` +#### :metal: Easily run other repos with our data -Once you have the config properly set up, you can begin training! Note, you can also pass in the config changes via command-line as shown above in the quick-start if you don't want to make a new config for a given job. +#### :metal: Speed up your code with Tiny Cuda -``` -python scripts/run_train.py --config-name vanilla_nerf -``` +#### :metal: Support for Jupyter -# Feature List - -#### :metal: [Hydra config structure](#) - -#### :metal: [Logging, debugging utilities](#) - -#### :metal: [Benchmarking, other tooling](#) - -#### :metal: Running other repos with our data - -``` -# nerf-pytorch -cd external -python run_nerf.py --config configs/chair.txt --datadir /path/to/pyrad/data/blender/chair - -# jaxnerf -cd external -conda activate jaxnerf -python -m jaxnerf.train --data_dir=/path/to/pyrad/data/blender/chair --train_dir=/path/to/pyrad/outputs/blender_chair_jaxnerf --config=/path/to/pyrad/external/jaxnerf/configs/demo --render_every 100 -``` - -#### :metal: Speeding up the code - -Documentation for running the code with CUDA. -Please see https://github.com/NVlabs/tiny-cuda-nn for how to install tiny-cuda-nn. - -``` -pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch -``` - -To run instant-ngp with tcnn, you can do the following. This is with the fox dataset. - -``` -python scripts/run_train.py --config-name=instant_ngp_tcnn.yaml data/dataset=instant_ngp_fox -``` - -#### :metal: Setting up Jupyter - -``` -python -m jupyter lab build -bash environments/run_jupyter.sh -``` # Benchmarked Model Architectures diff --git a/docs/index.rst b/docs/index.rst index 393156312..ce53927a2 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -2,22 +2,75 @@ pyRad |:metal:| ==================================== Neural Volumetric Rendering ----------------------------------- +""""""""""""""""""""""""""""" +All-in-one repository for state-of-the-art NeRFs. + +pyRad |:metal:| provides a simple API that allows for a seamless and simplified end-to-end process of creating, training, and testing NeRFs. +The library supports a more interpretable implementation of NeRFs by modularizing each component. +With more modular NeRFs, not only does your code become far more user-friendly, but using this framework also makes it easier for the community to build upon your implementation. + +It's as simple as plug and play with pyRad! + +If you have further questions or want a feature that is not yet supported, please do not hesitate to reach out to the Plenoptix team. + +| + + +Contents +""""""""""""""""""""""""""""" .. toctree:: - :caption: Quickstart + :hidden: + :caption: Tutorials - quickstart/installation - quickstart/quick_tour.md + tutorials/quickstart.md + tutorials/creating_dataset.md + tutorials/creating_graphs.md + tutorials/contributing.md .. toctree:: - :caption: Reference: + :hidden: + :caption: Tooling + + tooling/index + +.. toctree:: + :hidden: + :caption: Reference reference/index -Indices and tables -================== + + +This documentation is organized into 3 parts: + +* **Tutorials**: a great place to start if you are new to pyRad. Contains a quick tour, installation, and an overview of the core structures that will allow you to get up and running with pyRad |:metal:| +* **Tooling**: describes all of the additional support we provide to help you debug and improve your NeRFs (e.g logging, performance profiling) +* **Reference**: describes each class and function. Develop a better understanding of the core of our technology and terminology. This section includes descriptions of each module and component in the codebase. + + +| + + +Supported Models +""""""""""""""""""""""""""""" + + ++------------------------+------------------------------+----------------------------+ +| | PSNR | Rays per sec. | ++========================+==============================+============================+ +| NeRF | |:hourglass_flowing_sand:| | |:hourglass_flowing_sand:| | ++------------------------+------------------------------+----------------------------+ +| Instant-NGP | |:hourglass_flowing_sand:| | |:hourglass_flowing_sand:| | ++------------------------+------------------------------+----------------------------+ + + +| + + +Indices and Tables +""""""""""""""""""""""""""""" * :ref:`genindex` * :ref:`modindex` diff --git a/docs/quickstart/installation.rst b/docs/quickstart/installation.rst deleted file mode 100644 index ea282fff8..000000000 --- a/docs/quickstart/installation.rst +++ /dev/null @@ -1,59 +0,0 @@ -.. _installation: - -Installation -================= - -Clone the repo:: - - git clone --recurse-submodules git@github.com:ethanweber/pyrad.git - -Create the python environment:: - - conda create --name pyrad python=3.8.13 - conda activate pyrad - pip install -r environment/requirements.txt - -Install pyrad as a library:: - - pip install -e . - -Install library with CUDA support. Change setup.py to ``USE_CUDA = True`` and then:: - - python setup.py develop - -Running the test cases:: - - pytest tests - Field Modules - -Getting the data ------------------- - -Download the original NeRF dataset and put it in the following format.:: - - - - data/ - - blender/ - - fern/ - - lego/ - -Training a model ------------------- - -Run with default config:: - - python scripts/run_train.py - -Run with config changes:: - - python scripts/run_train.py machine_config.num_gpus=1 - python scripts/run_train.py data.dataset.downscale_factor=1 - -Run with different datasets:: - - python scripts/run_train.py data/dataset=blender_lego - python scripts/run_train.py data/dataset=friends_TBBT-big_living_room - -Run with different datasets and config changes:: - - python scripts/run_train.py data/dataset=friends_TBBT-big_living_room graph.network.far_plane=14 diff --git a/docs/quickstart/quick_tour.md b/docs/quickstart/quick_tour.md deleted file mode 100644 index f5364e508..000000000 --- a/docs/quickstart/quick_tour.md +++ /dev/null @@ -1,90 +0,0 @@ -# Walk-through tour - -In this quick tour, we will walk you through the core of training and building any NeRFs with pyrad. - -#### Launching a training job - -The entry point for training starts at `scripts/run_train.py`, which spawns instances of our `Trainer()` class (in `nerf/trainer.py`). The `Trainer()` is responsible for setting up the datasets and NeRF graph depending on the config specified. It will then run the usual train/val routine for a config-specified number of iterations. If you are planning on using our codebase to build a new NeRF method or to use an existing implementation, we've abstracted away the training routine in these two files and chances are you will not need to think of them again. - -#### Graphs, Fields, and Modules - -If you are looking to implemnet a new NeRF method or extend an existing one, you only need to edit files in `nerf/graph/`, `nerf/fields/`, `nerf/field_modules/`, `nerf/misc_modules/`. (TODO: restructuring) - -The actual NeRF graph definitions can be found in `nerf/graph/`. For instance, to implement the vanilla NeRF, we create a new class that inherits the abstract Graph class. To fully implement the any new graph class, you will need to implement the following abstract methods defined in the skeleton code below. See also `nerf/graph/vanilla_nerf.py` for the full implementation. - -``` -class NeRFGraph(Graph): - """Vanilla NeRF graph""" - - def __init__(self, intrinsics=None, camera_to_world=None, **kwargs) -> None: - super().__init__(intrinsics=intrinsics, camera_to_world=camera_to_world, **kwargs) - - def populate_fields(self): - """ - Set all field related modules here - """ - - def populate_misc_modules(self): - """ - Set all remaining modules here including: samplers, renderers, losses, and metrics - """ - - def get_param_groups(self) -> Dict[str, List[Parameter]]: - """ - Create a dictionary of parameters that are grouped according to different optimizers - """ - - def get_outputs(self, ray_bundle: RayBundle): - """ - Takes in a Ray Bundle and returns a dictionary of outputs. - """ - - def get_loss_dict(self, outputs, batch): - """ - Computes and returns the losses. - """ - - def log_test_image_outputs(self, image_idx, step, batch, outputs): - """ - Writes the test image outputs. - """ -``` - -Note that the graph is composed of fields and modules. - -**Fields** (`nerf/fields/`) represents the actual radiance field of the NeRF and is composed of field modules (`nerf/field_modules/`). Here, we define the field as the part of the network that takes in point samples and any other conditioning, and outputs any of the `FieldHeadNames` (`nerf/field_modules/field_heads.py`). The **misc. modules** can be any module outside of the field that are needed by the NeRF (e.g. losses, samplers, renderers). - -To get started on a new NeRF implementation, you simply have to define all relevant modules and populate them in the graph. - -#### Dataset population TODO(ethan) - -#### Config - -Now that you have the graph and dataset all set up, you're ready to create the config that you pass into our run train routine. Our config system is powered by [Hydra](https://hydra.cc/). All Hydra and machine related arguments are stored in `configs/default_setup.yaml`, as well as the defaults list. -To set up the graph config, create a new yaml under `configs/`. - -``` -# configs/vanilla_nerf.yaml - -defaults: - - default_setup # inherit the basic yaml heirarchy - - _self_ - -experiment_name: blender_lego -method_name: vanilla_nerf - -graph: - network: - _target_: pyrad.graph.vanilla_nerf.NeRFGraph # set the target to the graph you defined - - # - -data: - # -``` - -Once you have the config properly set up, you can begin training! Note, you can also pass in the config changes via command-line as shown above in the quick-start if you don't want to make a new config for a given job. - -``` -python scripts/run_train.py --config-name vanilla_nerf -``` diff --git a/docs/reference/index.rst b/docs/reference/index.rst index 0091353d1..64ffc2912 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -15,20 +15,3 @@ TODO: Explanation of each component graphs/modules/index renderers/index -Hydra config structure -======================== - -Logging, debugging utilities -=================================== - -Benchmarking, other tooling -=================================== - -Running other repos with our data -=================================== - -Speeding up the code -=================================== - -Setting up Jupyter -======================== \ No newline at end of file diff --git a/docs/tooling/benchmarking.md b/docs/tooling/benchmarking.md new file mode 100644 index 000000000..80a2f8791 --- /dev/null +++ b/docs/tooling/benchmarking.md @@ -0,0 +1,48 @@ +# Benchmarking workflow + +We make it easy to benchmark your new NeRF against the standard Blender dataset. + +#### Launching training on Blender + +To start, you will need to train your NeRF on each of the blender objects. +To launch training jobs automatically on each of these items, you can call: + +``` +./scripts/benchmarking/launch_train_blender.sh +``` + +Simply replace the arguments in brackets with the correct arguments. +* ``: name of the config you created for your new graph. For instance, if you have created the config `configs/graph_vanilla_nerf.yaml`, `` will be `graph_vanilla_nerf`. +* ``: specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs. + +A full example would be: + +``` +./scripts/benchmarking/launch_train_blender.sh graph_vanilla_nerf 0 1 2 3 +``` + +The script will automatically launch training on all of the items and save the checkpoints in an output directory with the current date. +You will use the date as the reference when you run inference for benchmarking. + +#### Benchmarking on Blender +Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/run_benchmark.py`. + +Modify the `BENCH` variable to specify which jobs you want benchmarked. + +``` +BENCH = { + "method": "graph_vanilla_nerf", # change this to whatever you want + "hydra_base_dir": "outputs/", # change this to wherever hydra's default output directory is + "benchmark_date": "05-26-2022", # change this to the date you ran the benchmarking + "object_list": ["mic", "ficus", "chair", "hotdog", "materials", "drums", "ship", "lego"], +} +``` + +Then you can simply run + +``` +python scripts/benchmarking/run_benchmark.py + +``` + +The script will search for the most recent checkpoint according to the benchmark configuration specified via `BENCH`. It will run inference and calculate the PSNR according to the respective object. The output is then saved in the `hydra_base_dir` directory as `.json` (full path: `/.json`). \ No newline at end of file diff --git a/docs/tooling/hydra.md b/docs/tooling/hydra.md new file mode 100644 index 000000000..3e8f9a44e --- /dev/null +++ b/docs/tooling/hydra.md @@ -0,0 +1 @@ +# Hydra config structure \ No newline at end of file diff --git a/docs/tooling/index.rst b/docs/tooling/index.rst new file mode 100644 index 000000000..07215964f --- /dev/null +++ b/docs/tooling/index.rst @@ -0,0 +1,12 @@ +.. _tooling: + + +.. toctree:: + + hydra.md + logging.md + profiling.md + benchmarking.md + other_codebase.md + speedup.md + jupyter.md diff --git a/docs/tooling/jupyter.md b/docs/tooling/jupyter.md new file mode 100644 index 000000000..48a4cd3cf --- /dev/null +++ b/docs/tooling/jupyter.md @@ -0,0 +1,10 @@ +# Setting up Jupyter + +We provide a few visualization tools with Jupyter that can be found in `notebooks/`. + +To install and run any of the notebooks, simply run the following commands: + +``` +python -m jupyter lab build +bash environments/run_jupyter.sh +``` \ No newline at end of file diff --git a/docs/tooling/logging.md b/docs/tooling/logging.md new file mode 100644 index 000000000..bee9fa39c --- /dev/null +++ b/docs/tooling/logging.md @@ -0,0 +1,30 @@ +# Logging Features + +We provide integration with multiple logging interfaces to log images and statistics during training. +All of these loggers are configurable via the config: + +``` +# e.g. configs/graphs_default.yaml + +logging: + steps_per_log: 10 # how often to log losses/images + max_buffer_size: 20 # defines how many steps to average over (if reporting averages) + writer: # definition of all writers you want to use + TensorboardWriter: + save_dir: "./" # must specify the output path for all writers + LocalWriter: + save_dir: "./" + stats_to_track: + [ITER_LOAD_TIME, ITER_TRAIN_TIME, RAYS_PER_SEC, CURR_TEST_PSNR] + max_log_size: 10 +``` + +Currently, we provide support for the following loggers: + +1. [Tensorboard](https://www.tensorflow.org/tensorboard): `TensorBoardWriter` +2. [Weights and Biases](https://wandb.ai/site): `WandbWriter` +3. Local writer: `LocalWriter` + +The `LocalWriter` simply outputs to the terminal. However, you can specify additional parameters to customize the logging. +* stats_to_track: all the stats that you want to print to the terminal (see list under `EventName` in `utils/writer.py`). You can add or remove any of the defined enums. +* max_log_size: how much content to print onto the screen (e.g. only print 10 lines on the screen at a time). If 0, will print everything without deleting any previous lines. diff --git a/docs/tooling/other_codebase.md b/docs/tooling/other_codebase.md new file mode 100644 index 000000000..7017c0664 --- /dev/null +++ b/docs/tooling/other_codebase.md @@ -0,0 +1,16 @@ +# Run other repos with our data + +If you are looking to test out other existing repo's with our data, the pyRad structure makes it easy! + +For instance, you can run nerf-pytorch and jaxnerf with the following commands: + +``` +# nerf-pytorch +cd external +python run_nerf.py --config configs/chair.txt --datadir /path/to/pyrad/data/blender/chair + +# jaxnerf +cd external +conda activate jaxnerf +python -m jaxnerf.train --data_dir=/path/to/pyrad/data/blender/chair --train_dir=/path/to/pyrad/outputs/blender_chair_jaxnerf --config=/path/to/pyrad/external/jaxnerf/configs/demo --render_every 100 +``` \ No newline at end of file diff --git a/docs/tooling/profiling.md b/docs/tooling/profiling.md new file mode 100644 index 000000000..10fbe3630 --- /dev/null +++ b/docs/tooling/profiling.md @@ -0,0 +1,63 @@ +# Profiling and debugging + +## Profilers +We provide built-in performance profiling capabilities to make it easier for you to debug and assess the performance of your code. + +#### In-house profiler +You can use our built-in profiler by enabling the profiler in the config: + +``` +# e.g. configs/graphs_default.yaml + +logging: + enable_profiler: True +``` + +The profiler computes the average total time of execution for any function with the `@profiler.time_function` decorator. +For instance, if you wanted to profile the total time it takes to generate rays given pixel and camera indices via the `RayGenerator` class, you would want to evaluate its `forward()` function. In that class, you would need to add the decorator to the function. + + +``` +class RayGenerator(nn.Module): + + ... + + @profiler.time_function # add the profiler decorator before the function + def forward(self, ray_indices: TensorType["num_rays", 3]) -> RayBundle: + # implementation here + ... +``` + +At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag. + +Use this profiler if there are specific functions that you want to measure the times for. + + +#### Profiling with PySpy +If you want to profile the entire codebase, we provide functionality to generate flame graphs or get real-time reports using [PySpy](https://github.com/benfred/py-spy). + +Install PySpy + +``` +pip install py-spy +``` + +To perform the profiling, you can either specify that you want to generate a flame graph or generate a live-view of the profiler. + +``` +## for flame graph + +./scripts/debugging/profile.sh -t flame -o flame.svg -p + +## for live view of functions + +./scripts/debugging/profile.sh -t top -p +``` + +In both cases, `` can be replaced with any command you would normally run in the terminal. For instance, you can replace `` with `scripts/run_train.py data/dataset=blender_lego`. Then the full command would be e.g.: + +``` +./scripts/debugging/profile.sh -t flame -o flame.svg -p scripts/run_train.py data/dataset=blender_lego +``` + +## Debugging \ No newline at end of file diff --git a/docs/tooling/speedup.md b/docs/tooling/speedup.md new file mode 100644 index 000000000..2d4439b88 --- /dev/null +++ b/docs/tooling/speedup.md @@ -0,0 +1,14 @@ +# Speeding up code with TinyCuda + +Documentation for running the code with CUDA. +Please see https://github.com/NVlabs/tiny-cuda-nn for how to install tiny-cuda-nn. + +``` +pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch +``` + +To run instant-ngp with tcnn, you can do the following. This is with the fox dataset. + +``` +python scripts/run_train.py --config-name=instant_ngp_tcnn.yaml data/dataset=instant_ngp_fox +``` \ No newline at end of file diff --git a/docs/tutorials/contributing.md b/docs/tutorials/contributing.md new file mode 100644 index 000000000..29b31ca7a --- /dev/null +++ b/docs/tutorials/contributing.md @@ -0,0 +1,20 @@ +# Contributing to pyRad workflow + +1. Make your modifications +2. Perform local checks + + To ensure that you will be passing all tests and checks on github, you will need to run the following command: + + ``` + python scripts/debugging/run_actions.py + ``` + + This will perform the following checks and actions: + - Black/ Linting style check: Ensure that the code is consistently and properly formatted. + - Pytests: Runs pytests locally to make sure the logic of any added code does not break existing logic. + - Documentation build: Builds the documentation locally to make sure none of the changes result in any warnings or errors in the docs. + - Licensing: automatically adds licensing headers to the correct files. + + In order to merge changes to the code base, all of these checks must be passing. If you pass these tests locally, you will likely pass on github servers as well (results in a green checkmark next to your commit). + +3. Open pull request \ No newline at end of file diff --git a/docs/tutorials/creating_dataset.md b/docs/tutorials/creating_dataset.md new file mode 100644 index 000000000..879992f09 --- /dev/null +++ b/docs/tutorials/creating_dataset.md @@ -0,0 +1,2 @@ +# Setting up new data +TODO(ethan) \ No newline at end of file diff --git a/docs/tutorials/creating_graphs.md b/docs/tutorials/creating_graphs.md new file mode 100644 index 000000000..edf1f910a --- /dev/null +++ b/docs/tutorials/creating_graphs.md @@ -0,0 +1,144 @@ +# Deploying a new NeRF graph + +In this quick tour, we will walk you through the core of training and building any NeRFs with pyrad. + +The entry point for training starts at `scripts/run_train.py`, which spawns instances of our `Trainer()` class (in `engine/trainer.py`). The `Trainer()` is responsible for setting up the datasets and NeRF graph depending on the config specified. It will then run the usual train/val routine for a config-specified number of iterations. If you are planning on using our codebase to build a new NeRF method or to use an existing implementation, we've abstracted away the training routine in these two files and chances are you will not need to think of them again. + +## Graphs, Modules, and Fields + +If you are looking to implemnet a new NeRF method or extend an existing one, you may need to set up a new graph (in `graphs/`). + +All NeRF graph definitions can be found in `graphs/`. The graph encapsulates all the modules in the NeRF method- from the MLP's to the samplers to the losses and optimizers. All graphs are composed of modules and fields. + +### Modules +We can decompose all graphs into respective modules. For instance, we have `optimizers/` which contains modules related to the loss, `renderers/` which contains all rendering modules, `graph/modules/`, which includes samplers and ray generators. Additionally, we have `fields/modules`, which will be discussed below. +In general, modules can be thought of as individual component parts of the graph that can be swapped in/out. We provide most of the standard modules, but if you are creating a new NeRF graph, you may have to define new modules. + + +### Fields + +Fields (`fields/`) represents the "space", aka. the radiance field of the NeRF. Here, we define the field as the part of the network that takes in point samples and any other conditioning, and outputs any of the `FieldHeadNames` (`nerf/field_modules/field_heads.py`). + +All fields are composed of modules continained within the `fields/` directory (e.g. `fields/modules/` or `fields/occupancy_fields/`). We can think of field modules as modules that actually define or interact with the field. + +To build a NeRF field, we therefore follow these steps: +1. define any relevant field modules +2. create a new class that extends the `Field` class and use these field modules to compose the full field +3. implement the abstract functions `get_density()` and `get_outputs()`. + +``` +class NeRFField(Field): + """NeRF Field""" + + def __init__(self, ...) -> None: + """Create boilerplate NeRF field.""" + + super().__init__() + + # In the init method, we are instantiating all relevant field modules + self.position_encoding = position_encoding + self.direction_encoding = direction_encoding + + self.mlp_base = MLP( + in_dim=self.position_encoding.get_out_dim(), + num_layers=base_mlp_num_layers, + layer_width=base_mlp_layer_width, + skip_connections=skip_connections, + ) + ... + + def get_density(self, point_samples: PointSamples): + """ + Computes and returns the densities. + """ + + def get_outputs( + self, point_samples: PointSamples, density_embedding: Optional[TensorType] = None + ) -> Dict[FieldHeadNames, TensorType]: + """ + Computes and returns the outputs. + """ +``` + +### Putting it together + +Now that we've gone over what modules and fields are, we are ready to create the graph! + +To implement the vanilla NeRF, we create a new class that inherits the abstract Graph class. Similar to composing a new field, we can pull from all pre-defined fields and modules and use them to compose a new Graph. +We will then need to define the following abstract methods as seen in the skeleton code below. See also `graphs/vanilla_nerf.py` for the full implementation. + +``` +class NeRFGraph(Graph): + """Vanilla NeRF graph""" + + def __init__(self, intrinsics=None, camera_to_world=None, **kwargs) -> None: + super().__init__(intrinsics=intrinsics, camera_to_world=camera_to_world, **kwargs) + + def populate_fields(self): + """ + Set all field related modules here + """ + + def populate_misc_modules(self): + """ + Set all remaining modules here including: samplers, renderers, losses, and metrics + """ + + def get_param_groups(self) -> Dict[str, List[Parameter]]: + """ + Create a dictionary of parameters that are grouped according to different optimizers + """ + + def get_outputs(self, ray_bundle: RayBundle): + """ + Takes in a Ray Bundle and returns a dictionary of outputs. + """ + + def get_loss_dict(self, outputs, batch): + """ + Computes and returns the losses. + """ + + def log_test_image_outputs(self, image_idx, step, batch, outputs): + """ + Writes the test image outputs. + """ +``` + + +## Creating the config + +Now that you have the graph and dataset all set up, you'll need to create the config that you pass into our run train routine. Our config system is powered by [Hydra](https://hydra.cc/). All Hydra and machine related arguments are stored in `configs/default_setup.yaml`, as well as the defaults list. +To set up the graph config, create a new yaml under `configs/`. + +``` +# configs/vanilla_nerf.yaml + +defaults: + - default_setup # inherit the basic yaml heirarchy + - _self_ + +experiment_name: blender_lego +method_name: vanilla_nerf + +graph: + network: + _target_: pyrad.graph.vanilla_nerf.NeRFGraph # set the target to the graph you defined + + # + +data: + # +``` + +## Training and testing + +Once you have the config properly set up, you can begin training! To begin the training process, just pass in the name of your new config to the training script: + +``` +python scripts/run_train.py --config-name vanilla_nerf +``` + +And now you have a brand-new NeRF in training! For testing and visualizing, simply refer to steps 4-5 in the [quickstart guide](https://github.com/plenoptix/pyrad#quickstart). + +To help you get started, we also provide additional training tools such as profiling, logging, and debugging. Please refer to our [features guide](../tooling/index.rst). diff --git a/docs/tutorials/quickstart.md b/docs/tutorials/quickstart.md new file mode 100644 index 000000000..eff453053 --- /dev/null +++ b/docs/tutorials/quickstart.md @@ -0,0 +1,9 @@ +# Quickstart guide + +These quickstart steps will help you get started with training and testing the default vanilla nerf on the classic blender lego scene. + + +Please follow instructions 1-5 in our [Github Quickstart](https://github.com/plenoptix/pyrad#quickstart) for info on how to install, setup training, and visualize inference all with pyRad. + + +For more complex changes (e.g. running with your own data/ setting up a new NeRF graph, please continue to the next page. \ No newline at end of file