diff --git a/README.md b/README.md index 1d3653c41..6632544e4 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ For more complex changes (e.g. running with your own data/ setting up a new NeRF #### 1. Installation: Setup the environment -This repository is tested with CUDA 11.3. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html#linux-installers) before proceding. +This repository is tested with CUDA 11.3. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html#linux-installers) before preceding.
Installing Conda @@ -102,7 +102,26 @@ With support for [Hydra](https://hydra.cc/), you can run with other configuratio #### 4. Visualizing training runs -If you run everything with the default configuration, by default, we use [TensorBoard](https://www.tensorflow.org/tensorboard) to log all training curves, test images, and other stats. Once the job is launched, you will be able to track training by launching the tensorboard in `outputs/blender_lego/vanilla_nerf//`. +We support multiple methods to visualize training, the default configuration uses Tensorboard. More information on logging can be found [here](https://plenoptix-pyrad.readthedocs-hosted.com/en/latest/tooling/logging.html). + +
+Real-time Viewer + +We have developed our own Real-time web viewer, more information can be found [here](https://plenoptix-pyrad.readthedocs-hosted.com/en/latest/tooling/viewer.html). This viewer runs during training and is designed to work with models that have fast rendering pipelines. + +To enable add the following to your config: + +``` +viewer: + enable: true +``` + +
+ +
+Tensorboard + +If you run everything with the default configuration we log all training curves, test images, and other stats. Once the job is launched, you will be able to track training by launching the tensorboard in `outputs/blender_lego/vanilla_nerf//`. ``` tensorboard --logdir outputs @@ -112,6 +131,21 @@ export TENSORBOARD_PORT= bash environment/run_tensorboard.sh ``` +
+ +
+Weights & Biases + +We support logging to weights and biases, to enable add the following to the config: + +``` +logging: + writer: + WandbWriter +``` + +
+ #### 5. Rendering a trajectories during inference ``` @@ -120,7 +154,7 @@ python scripts/run_eval.py --method=traj --traj=spiral --output-filename=output. #### 6. In-depth guide -For a more in-depth tutorial on how to modify/implement your own NeRF Graph, please see our [walk-through](#). +For a more in-depth tutorial on how to modify/implement your own NeRF Graph, please see our [walk-through](https://plenoptix-pyrad.readthedocs-hosted.com/en/latest/tutorials/creating_graphs.html). # Supported Features @@ -136,11 +170,7 @@ If you are looking for a feature that is not currently supported, please do not #### :metal: Benchmarking scripts -#### :metal: Easily run other repos with our data - -#### :metal: Speed up your code with Tiny Cuda - -#### :metal: Support for Jupyter +#### :metal: Speed up your code with Tiny Cuda NN # Benchmarked Model Architectures diff --git a/docs/index.rst b/docs/index.rst index d6661250d..babd43971 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -22,12 +22,12 @@ Contents .. toctree:: :hidden: - :caption: Tutorials + :caption: Getting Started - tutorials/quickstart.md - tutorials/data_setup.rst + Quickstart + tutorials/data_index tutorials/creating_graphs.md - tutorials/viewer.md + tutorials/system_overview.md .. toctree:: :hidden: diff --git a/docs/tooling/index.rst b/docs/tooling/index.rst index 265eafa5f..4334d24bf 100644 --- a/docs/tooling/index.rst +++ b/docs/tooling/index.rst @@ -5,5 +5,6 @@ hydra.md logging.md + viewer.md profiling.md benchmarking.md diff --git a/docs/tutorials/data/media/viewer_figure.png b/docs/tooling/media/viewer_figure.png similarity index 100% rename from docs/tutorials/data/media/viewer_figure.png rename to docs/tooling/media/viewer_figure.png diff --git a/docs/tutorials/viewer.md b/docs/tooling/viewer.md similarity index 98% rename from docs/tutorials/viewer.md rename to docs/tooling/viewer.md index 23a6317f4..112a5cf9b 100644 --- a/docs/tutorials/viewer.md +++ b/docs/tooling/viewer.md @@ -4,7 +4,7 @@ We provide a real-time viewer that can be used during training or to view a trai The viewer is built using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine. The following figure helps to illustrate how our viewer framework works: -![visualize_dataset](data/media/viewer_figure.png) +![visualize_dataset](media/viewer_figure.png) ## Quick start diff --git a/docs/tutorials/creating_graphs.md b/docs/tutorials/creating_graphs.md index ad0ae128c..63e57a764 100644 --- a/docs/tutorials/creating_graphs.md +++ b/docs/tutorials/creating_graphs.md @@ -1,4 +1,4 @@ -# Deploying a new NeRF graph +# Developing a Custom Model In this quick tour, we will walk you through the core of training and building any NeRFs with pyrad. @@ -11,21 +11,22 @@ If you are looking to implemnet a new NeRF method or extend an existing one, you All NeRF graph definitions can be found in `graphs/`. The graph encapsulates all the modules in the NeRF method- from the MLP's to the samplers to the losses and optimizers. All graphs are composed of modules and fields. ### Modules + We can decompose all graphs into respective modules. For instance, we have {ref}`optimizers` which contains modules related to the loss, {ref}`renderers` which contains all rendering modules, {`graph/modules/`}, which includes samplers and ray generators. Additionally, we have {ref}`field_modules`, which will be discussed below. In general, modules can be thought of as individual component parts of the graph that can be swapped in/out. We provide most of the standard modules, but if you are creating a new NeRF graph, you may have to define new modules. - ### Fields -{ref}`fields` represents the "space", aka. the radiance field of the NeRF. Here, we define the field as the part of the network that takes in point samples and any other conditioning, and outputs any of the `FieldHeadNames` (`nerf/field_modules/field_heads.py`). +{ref}`fields` represents the "space", aka. the radiance field of the NeRF. Here, we define the field as the part of the network that takes in point samples and any other conditioning, and outputs any of the `FieldHeadNames` (`nerf/field_modules/field_heads.py`). -All fields are composed of modules continained within the `fields/` directory (e.g. `fields/modules/` or `fields/occupancy_fields/`). We can think of field modules as modules that actually define or interact with the field. +All fields are composed of modules continained within the `fields/` directory (e.g. `fields/modules/` or `fields/occupancy_fields/`). We can think of field modules as modules that actually define or interact with the field. To build a NeRF field, we therefore follow these steps: + 1. define any relevant field modules 2. create a new class that extends the {ref}`fields` base class and use these field modules to compose the full field 3. implement the abstract functions `get_density()` and `get_outputs()`. - + ```python class NeRFField(Field): """NeRF Field""" @@ -36,7 +37,7 @@ class NeRFField(Field): super().__init__() # In the init method, we are instantiating all relevant field modules - self.position_encoding = position_encoding + self.position_encoding = position_encoding self.direction_encoding = direction_encoding self.mlp_base = MLP( @@ -62,9 +63,9 @@ class NeRFField(Field): ### Putting it together -Now that we've gone over what modules and fields are, we are ready to create the graph! +Now that we've gone over what modules and fields are, we are ready to create the graph! -To implement the vanilla NeRF, we create a new class that inherits the abstract Graph class. Similar to composing a new field, we can pull from all pre-defined fields and modules and use them to compose a new Graph. +To implement the vanilla NeRF, we create a new class that inherits the abstract Graph class. Similar to composing a new field, we can pull from all pre-defined fields and modules and use them to compose a new Graph. We will then need to define the following abstract methods as seen in the skeleton code below. See also `graphs/vanilla_nerf.py` for the full implementation. ```python @@ -105,7 +106,6 @@ class NeRFGraph(Graph): """ ``` - ## Creating the config Now that you have the graph and dataset all set up, you'll need to create the config that you pass into our run train routine. Our config system is powered by [Hydra](https://hydra.cc/). All Hydra and machine related arguments are stored in `configs/default_setup.yaml`, as well as the defaults list. @@ -131,7 +131,7 @@ data: # graph: - _target_: pyrad.graph.vanilla_nerf.NeRFGraph # set the target to the graph you defined + _target_: pyrad.graph.vanilla_nerf.NeRFGraph # set the target to the graph you defined # optimizers: diff --git a/docs/tutorials/data_setup.rst b/docs/tutorials/data_index.rst similarity index 99% rename from docs/tutorials/data_setup.rst rename to docs/tutorials/data_index.rst index 676aa70f5..6c5d2e517 100644 --- a/docs/tutorials/data_setup.rst +++ b/docs/tutorials/data_index.rst @@ -1,4 +1,3 @@ - Data Setup ----------- diff --git a/docs/tutorials/quickstart.md b/docs/tutorials/quickstart.md index 5904d5482..092308e35 100644 --- a/docs/tutorials/quickstart.md +++ b/docs/tutorials/quickstart.md @@ -1,10 +1,118 @@ -# Quickstart +# Installation -```{include} ../../README.md -:relative-docs: docs/ -:relative-images: -:start-after: quick_tour.html). -:end-before: 6. In-depth +This repository is tested with CUDA 11.3. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html#linux-installers) before preceding. + +
+Installing Conda + + This step is fairly self-explanatory, but here are the basic steps. You can also find countless tutorials online. + + ```bash + cd /path/to/install/miniconda + + mkdir -p miniconda3 + wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda3/miniconda.sh + bash miniconda3/miniconda.sh -b -u -p miniconda3 + rm -rf miniconda/miniconda.sh + ``` + +
+ +Create the python environment + +```bash +conda create --name pyrad python=3.8.13 +conda activate pyrad ``` -For more complex changes (e.g. running with your own data/ setting up a new NeRF graph, please continue to the next page. \ No newline at end of file +Clone the repo + +```bash +git clone git@github.com:plenoptix/pyrad.git +``` + +Install dependencies + +```bash +cd pyrad +pip install -r environment/requirements.txt +``` + +Install pyrad as a library + +```bash +pip install -e . +``` + +Install library with CUDA development support. Change setup.py to `USE_CUDA = True` and then + +```bash +python setup.py develop +``` + +Install tiny-cuda-nn (tcnn) to use with the graph_instant_ngp.yaml config + +```bash +pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch +``` + +# Downloading data + +Download the original [NeRF dataset](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) and unfold it in the following format. This is for the blender dataset type. We support the major datasets and allow users to create their own dataset, described in detail [here](data/creating_dataset.md). + +``` +|─ pyrad/ + ├─ data/ + | ├─ blender/ + | ├─ fern/ + | ├─ lego/ + ... + |- / + |- + ... +``` + +# Training a model + +To run with all the defaults, e.g. vanilla nerf method with the blender lego image + +Run a vanilla nerf model. + +```bash +python scripts/run_train.py +``` + +Run a faster version with instant ngp using tcnn (without the viewer). + +```bash +python scripts/run_train.py --config-name=graph_instant_ngp.yaml +``` + +Run with the viewer. However, you'll have to start the viewer server first. (See the docs.) + +```bash +python scripts/run_train.py --config-name=graph_instant_ngp.yaml viewer.enable=true +``` + +With support for [Hydra](https://hydra.cc/), you can run with other configurations by changing appropriate configs defined in `configs/` or by setting flags via command-line arguments. + +# Visualizing training runs + +If you run everything with the default configuration, by default, we use [TensorBoard](https://www.tensorflow.org/tensorboard) to log all training curves, test images, and other stats. Once the job is launched, you will be able to track training by launching the tensorboard in `outputs/blender_lego/vanilla_nerf//`. + +```bash +tensorboard --logdir outputs +``` + +Or the following: + +```bash +export TENSORBOARD_PORT= +bash environment/run_tensorboard.sh +``` + +# Rendering a Trajectory + +```bash +python scripts/run_eval.py --method=traj --traj=spiral --output-filename=output.mp4 --config-name=graph_instant_ngp.yaml trainer.resume_train.load_dir=outputs/blender_lego/instant_ngp/2022-07-07_230905/checkpoints +``` diff --git a/docs/tutorials/quickstart_index.rst b/docs/tutorials/quickstart_index.rst new file mode 100644 index 000000000..b2eb7a10e --- /dev/null +++ b/docs/tutorials/quickstart_index.rst @@ -0,0 +1,19 @@ +pyRad |:metal:| +==================================== + +Neural Volumetric Rendering +""""""""""""""""""""""""""""" + +All-in-one repository for state-of-the-art NeRFs. + +pyRad |:metal:| provides a simple API that allows for a seamless and simplified end-to-end process of creating, training, and testing NeRFs. +The library supports a more interpretable implementation of NeRFs by modularizing each component. +With more modular NeRFs, not only does your code become far more user-friendly, but using this framework also makes it easier for the community to build upon your implementation. + + +Quickstart +----------- + +.. toctree:: + + quickstart.md \ No newline at end of file diff --git a/docs/tutorials/system_overview.md b/docs/tutorials/system_overview.md new file mode 100644 index 000000000..299adec6d --- /dev/null +++ b/docs/tutorials/system_overview.md @@ -0,0 +1 @@ +# System Overview