diff --git a/docs/developer_guides/new_methods.md b/docs/developer_guides/new_methods.md index 22f7e5efb..57b0272f9 100644 --- a/docs/developer_guides/new_methods.md +++ b/docs/developer_guides/new_methods.md @@ -73,14 +73,17 @@ pip install -e . When developing a new method you don't always want to install your code as a package. Instead, you may use the `NERFSTUDIO_METHOD_CONFIGS` environment variable to temporarily register your custom method. + ``` export NERFSTUDIO_METHOD_CONFIGS="my-method=my_method.my_config:MyMethod" ``` ## Registering custom dataparser with nerfstudio + We also support adding new dataparsers in a similar way. In order to extend the NeRFstudio and register a customized dataparser, you can register it with Nerfstudio as a `nerfstudio.dataparser_configs` entrypoint in the `pyproject.toml` file. Nerfstudio will automatically look for all registered dataparsers and will register them to be used by methods such as `ns-train`. You can declare the dataparser in the same config file: + ```python """my_method/my_config.py""" @@ -97,16 +100,19 @@ Then add the following lines in the `pyproject.toml` file, where the entrypoint [project] name = "my_method" -# ... +``` [project.entry-points.'nerfstudio.dataparser_configs'] custom-dataparser = 'my_method.my_config:MyDataparser' + ``` finally run the following to register the dataparser. ``` + pip install -e . + ``` ## Running custom method @@ -114,7 +120,9 @@ pip install -e . After registering your method you should be able to run the method with, ``` + ns-train my-method --data DATA_DIR + ``` ## Adding to the _nerf.studio_ documentation diff --git a/docs/index.md b/docs/index.md index ee611c573..e5be4cdce 100644 --- a/docs/index.md +++ b/docs/index.md @@ -99,16 +99,23 @@ This documentation is organized into 3 parts: ## Supported Methods -- [**Nerfacto**](https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/models/nerfacto.py): our de facto NeRF method combines modules focused on quality with modules focused on faster rendering. Nerfstudio easily lets us experiment with the best of both worlds! -- [NeRF](https://www.matthewtancik.com/nerf): Representing Scenes as Neural Radiance Fields for View Synthesis -- [Instant NGP](https://nvlabs.github.io/instant-ngp/): Instant Neural Graphics Primitives with a Multiresolution Hash Encoding -- [Mipnerf](https://jonbarron.info/mipnerf/): A Multiscale Representation for Anti-Aliasing Neural Radiance Fields -- [NerfW](https://nerf-w.github.io/): Neural Radiance Fields for Unconstrained Photo Collections -- [Semantic NeRF](https://shuaifengzhi.com/Semantic-NeRF/): In-Place Scene Labelling and Understanding with Implicit Scene Representation +### Included Methods -We'll be constantly growing this list! So make sure to check back in to see our updates. +- [**Nerfacto**](nerfology/methods/nerfacto.md): Recommended method, integrates mutiple methods into one. +- [Instant-NGP](nerfology/methods/instant_ngp.md): Instant Neural Graphics Primitives with a Multiresolution Hash Encoding +- [NeRF](nerfology/methods/nerf.md): OG Nerual Radiance Fields +- [Mip-NeRF](nerfology/methods/mipnerf.md): A Multiscale Representation for Anti-Aliasing Neural Radiance Fields +- [TensoRF](nerfology/methods/tensorf.md): Tensorial Radiance Fields -**Eager to contribute?** We'd love to see you use nerfstudio in implementing new (or even existing) methods! Feel free to contact us directly or view our [Contributor's Guide](reference/contributing) to see how you can get your model on this list! +(third_party_methods)= + +### Third-party Methods + +- [Instruct-NeRF2NeRF](nerfology/methods/in2n.md): Editing 3D Scenes with Instructions +- [LERF](nerfology/methods/lerf.md): Language Embedded Radiance Fields +- [Tetra-NeRF](nerfology/methods/tetranerf.md): Representing Neural Radiance Fields Using Tetrahedra + +**Eager to contribute a method?** We'd love to see you use nerfstudio in implementing new (or even existing) methods! Please view our {ref}`guide` for more details about how to add to this list! ## Quicklinks diff --git a/docs/nerfology/methods/index.md b/docs/nerfology/methods/index.md index 441a6d705..4c8cf98fa 100644 --- a/docs/nerfology/methods/index.md +++ b/docs/nerfology/methods/index.md @@ -26,14 +26,14 @@ The following methods are supported in nerfstudio: ```{toctree} :maxdepth: 1 - NeRF - Mip-NeRF - Nerfacto Instant-NGP Instruct-NeRF2NeRF LERF - Semantic NeRF-W - Tetra-NeRF + Mip-NeRF + NeRF + Nerfacto + Tetra-NeRF + TensoRF ``` (own_method_docs)= @@ -42,7 +42,13 @@ The following methods are supported in nerfstudio: If you're a researcher looking to develop new NeRF-related methods, we hope that you find nerfstudio to be a useful tool. We've provided documentation about integrating with the nerfstudio codebase, which you can find [here](../../developer_guides/new_methods.md). -We also welcome additions to the list of methods above. To do this, simply create a pull request that adds a markdown file describing the model to the docs/nerfology/methods folder, and update the list in this file. For reference on the layout, you can check out the [Instruct-NeRF2NeRF](in2n) page. Please try to include the following information: +We also welcome additions to the list of methods above. To do this, simply create a pull request with the following changes, + +1. Add a markdown file describing the model to the `docs/nerfology/methods` folder +2. Update the above list of implement methods in this file. +3. Add the method to the {ref}`this` list in `docs/index.md`. + +For the method description, please refer to the [Instruct-NeRF2NeRF](in2n) page as an example of the layout. Please try to include the following information: - Installation instructions - Instructions for running the method diff --git a/docs/nerfology/methods/instant_ngp.md b/docs/nerfology/methods/instant_ngp.md index 5bdc061b6..98d253011 100644 --- a/docs/nerfology/methods/instant_ngp.md +++ b/docs/nerfology/methods/instant_ngp.md @@ -8,14 +8,13 @@ Paper Website ``` - ### Running the Model -Instant-NGP is built locally into Nerfstudio. To use the method, run +Instant-NGP is built locally into Nerfstudio. To use the method, run ```bash ns-train instant-ngp --help @@ -35,7 +34,7 @@ Instant-NGP breaks NeRF training into 3 pillars and proposes improvements to eac The core idea behind the improved sampling technique is that sampling over empty space should be skipped and sampling behind high density areas should also be skipped. This is achieved by maintaining a set of multiscale occupancy grids which coarsely mark empty and non-empty space. Occupancy is stored as a single bit, and a sample on a ray is skipped if its occupancy is too low. These occupancy grids are stored independently of the trainable encoding and are updated throughout training based on the updated density predictions. The authors find they can increase sampling speed by 10-100x compared to naive approaches. -Nerfstudio uses [NerfAcc](https://www.nerfacc.com/en/latest/index.html) as the sampling algorithm implementation. The details on NerfAcc's sampling and occupancy grid is discussed [here](https://www.nerfacc.com/en/stable/methodology/sampling.html#occupancy-grid-estimator). +Nerfstudio uses [NerfAcc](https://www.nerfacc.com/en/latest/index.html) as the sampling algorithm implementation. The details on NerfAcc's sampling and occupancy grid is discussed [here](https://www.nerfacc.com/en/stable/methodology/sampling.html#occupancy-grid-estimator). Another major bottleneck for NeRF's training speed has been querying the neural network. The authors of this work implement the network such that it runs entirely on a single CUDA kernal. The network is also shrunk down to be just 4 layers with 64 neurons in each layer. They show that their fully-fused neural network is 5-10x faster than a Tensorflow implementation. @@ -48,22 +47,23 @@ The speedups at each level are multiplicative. With all their improvements, Inst ```{image} imgs/ingp/hash_figure.png :align: center ``` -One contribution of Instant-NGP is the multi-resolution hash encoding. In the traditional NeRF pipelines, input coordinates are mapped to a higher dimensional space using a positional encoding function, which is described [here](https://docs.nerf.studio/en/latest/nerfology/methods/nerf.html#positional-encoding). Instant-NGP proposes a trainable hash-based encoding. The idea is to map coordinates to trainable feature vectors which can be optimized in the standard flow of NeRF training. + +One contribution of Instant-NGP is the multi-resolution hash encoding. In the traditional NeRF pipelines, input coordinates are mapped to a higher dimensional space using a positional encoding function, which is described [here](../model_components/visualize_encoders.ipynb). Instant-NGP proposes a trainable hash-based encoding. The idea is to map coordinates to trainable feature vectors which can be optimized in the standard flow of NeRF training. The trainable features are F-dimensional vectors and are arranged into L grids which contain up to T vectors, where L represents the number of resolutions for features and T represents the number of feature vectors in each hash grid. The steps for the hash grid encoding, as shown in the figure provided by the authors, are as follows: -1. Given an input coordinate, find the surrounding voxels at L resolution levels and hash the vertices of these grids. +1. Given an input coordinate, find the surrounding voxels at L resolution levels and hash the vertices of these grids. 2. The hashed vertices are used as keys to look up trainable F-dimensional feature vectors. 3. Based on where the coordinate lies in space, the feature vectors are linearly interpolated to match the input coordinate. 4. The feature vectors from each grid are concatenated, along with any other parameters such as viewing direction, 5. The final vector is inputted into the neural network to predict the RGB and density output. -Steps 1-3 are done independently at each resolution level. Thus, since these feature vectors are trainable, when backpropagating the loss gradient, the gradients will flow through the neural network and interpolation function all the way back to the feature vectors. The feature vectors are interpolated relative to the coordinate such that the network can learn a smooth function. +Steps 1-3 are done independently at each resolution level. Thus, since these feature vectors are trainable, when backpropagating the loss gradient, the gradients will flow through the neural network and interpolation function all the way back to the feature vectors. The feature vectors are interpolated relative to the coordinate such that the network can learn a smooth function. An important note is that hash collisions are not explicitly handled. At each hash index, there may be multiple vertices which index to that feature vector, but because these vectors are trainable, the vertices that are most important to the specific output will have the highest gradient, and therefore automatically dominate the optimization of that feature. This encoding structure creates a tradeoff betweewn quality, memory, and performance. The main parameters which can be adjusted are the size of the hash table (T), the size of the feature vectors (F), and the number of resolutions (L). -For other inputs such as the viewing direction, Instant-NGP encodes them using the traditional positional encoding from the original NeRF paper. +Instant-NGP encodes the vieweing direction using spherical harmonic encodings. -Our [`nerfacto`](https://docs.nerf.studio/en/latest/nerfology/methods/nerfacto.html) model uses both the fully-fused MLP and the hash encoder, which were inspired by Instant-NGP. Lastly, our implementation covers the major ideas from Instant-NGP, but it doesn't strictly follow every detail. Some known differences include learning rate schedulers, hyper-parameters for sampling, and how camera gradients are calculated if enabled. \ No newline at end of file +Our [`nerfacto`](./nerfacto.md) model uses both the fully-fused MLP and the hash encoder, which were inspired by Instant-NGP. Lastly, our implementation covers the major ideas from Instant-NGP, but it doesn't strictly follow every detail. Some known differences include learning rate schedulers, hyper-parameters for sampling, and how camera gradients are calculated if enabled. diff --git a/docs/nerfology/methods/semantic_nerfw.md b/docs/nerfology/methods/semantic_nerfw.md deleted file mode 100644 index b023418da..000000000 --- a/docs/nerfology/methods/semantic_nerfw.md +++ /dev/null @@ -1,37 +0,0 @@ -# Semantic NeRF-W - -

Semantic NeRF

- -```{button-link} https://shuaifengzhi.com/Semantic-NeRF/ -:color: primary -:outline: -Paper Website -``` - -

NeRF in the Wild

- -```{button-link} https://nerf-w.github.io/ -:color: primary -:outline: -Paper Website -``` - -```{admonition} Coming Soon -The transient embeddings are still under development. Please stay tuned. -``` - -### Running Model - -Download the sitcoms3D Dataset - -```bash -ns-download-data sitcoms3d -``` - -```bash -ns-train semantic-nerfw -``` - -This model defaults to using the "sitcoms3D" dataset from the paper ["The One Where They Reconstructed 3D Humans and Environments in TV Shows"](https://ethanweber.me/sitcoms3D/). - - diff --git a/docs/nerfology/methods/tensorf.md b/docs/nerfology/methods/tensorf.md new file mode 100644 index 000000000..d3b90786b --- /dev/null +++ b/docs/nerfology/methods/tensorf.md @@ -0,0 +1,19 @@ +# TensoRF + +

Tensorial Radiance Fields

+ +```{button-link} https://apchenstu.github.io/TensoRF/ +:color: primary +:outline: +Paper Website +``` + +### Running Model + +```bash +ns-train tensorf +``` + +## Overview + +🚧 Guide under construction 🚧