diff --git a/docs/index.rst b/docs/index.rst index 1b852142d..5cb1f3026 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -46,7 +46,7 @@ Contents :caption: Guides - models/index.rst + models/index notebooks/index .. toctree:: diff --git a/docs/models/imgs/models_nerf-field-dark.png b/docs/models/imgs/models_nerf-field-dark.png new file mode 100644 index 000000000..c1c178d70 Binary files /dev/null and b/docs/models/imgs/models_nerf-field-dark.png differ diff --git a/docs/models/imgs/models_nerf-field-light.png b/docs/models/imgs/models_nerf-field-light.png new file mode 100644 index 000000000..0bc667c0e Binary files /dev/null and b/docs/models/imgs/models_nerf-field-light.png differ diff --git a/docs/models/index.md b/docs/models/index.md new file mode 100644 index 000000000..786d16c14 --- /dev/null +++ b/docs/models/index.md @@ -0,0 +1,25 @@ +# Models + +We provide a set of preimplemented nerfactory models. One of the goals of nerfactory is to modularize the various NeRF techniques as much as possible. As a result, many of the techniqes from these preimplemented models can be mixed. + +## Running a model + +It's easy! + +```bash +python scripts/run_train.py --config-name MODEL_CONFIG +``` + +## Guides + +In addition to their implementations, we have provided guides that walk through each of these method. + +```{toctree} + :maxdepth: 1 + NeRF + Mip-NeRF + Mip-NeRF 360 + Instan-NGP + NeRF-W + Semantic NeRF +``` diff --git a/docs/models/index.rst b/docs/models/index.rst deleted file mode 100644 index 30480f0c5..000000000 --- a/docs/models/index.rst +++ /dev/null @@ -1,14 +0,0 @@ -.. _models: - -Models -======== - -.. toctree:: - :maxdepth: 1 - - NeRF - Mip-NeRF - Mip-NeRF 360 - Instan-NGP - NeRF-W - Semantic NeRF diff --git a/docs/models/nerf.md b/docs/models/nerf.md index b5813f7ad..8f11b1922 100644 --- a/docs/models/nerf.md +++ b/docs/models/nerf.md @@ -16,6 +16,20 @@ python scripts/run_train.py --config-name=graph_vanilla_nerf.yaml ## Method +### Pipeline + +```{image} imgs/models_nerf-field-light.png +:align: center +:class: only-light +:width: 500 +``` + +```{image} imgs/models_nerf-field-dark.png +:align: center +:class: only-dark +:width: 500 +``` + ### Overview If you have arrived to this site, it is likely that you have atleast heard of NeRFs. This page will discuss the original NeRF paper, _"NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis"_ by Mildenhall, Srinivasan, Tancik et al. (2020). For most tasks, using the original NeRF model is likely not a good choice and hence we provide implementations of various other NeRF related models. It is however useful to understand how NeRF's work as most follow ups follow a similar structure. diff --git a/docs/notebooks/imgs/samplers_stratified-dark.png b/docs/notebooks/imgs/samplers_stratified-dark.png new file mode 100644 index 000000000..59d7b9a5f Binary files /dev/null and b/docs/notebooks/imgs/samplers_stratified-dark.png differ diff --git a/docs/notebooks/imgs/samplers_stratified-light.png b/docs/notebooks/imgs/samplers_stratified-light.png new file mode 100644 index 000000000..0e4f9e84b Binary files /dev/null and b/docs/notebooks/imgs/samplers_stratified-light.png differ diff --git a/docs/notebooks/imgs/samplers_type-dark.png b/docs/notebooks/imgs/samplers_type-dark.png new file mode 100644 index 000000000..62fd90392 Binary files /dev/null and b/docs/notebooks/imgs/samplers_type-dark.png differ diff --git a/docs/notebooks/imgs/samplers_type-light.png b/docs/notebooks/imgs/samplers_type-light.png new file mode 100644 index 000000000..fd3283f0c Binary files /dev/null and b/docs/notebooks/imgs/samplers_type-light.png differ diff --git a/docs/notebooks/index.md b/docs/notebooks/index.md new file mode 100644 index 000000000..42e8da330 --- /dev/null +++ b/docs/notebooks/index.md @@ -0,0 +1,13 @@ +# Components + +It can be difficult getting started with NeRFs. The field is still quite new and most of the key nuggets are burried in research papers. For this reason we can consoladated many of the key concepts into a series of guides. + +```{toctree} + :maxdepth: 1 + Cameras Models + Sample Representation + Ray Samplers + Spatial Distortions + Encoders + Viewer +``` \ No newline at end of file diff --git a/docs/notebooks/index.rst b/docs/notebooks/index.rst deleted file mode 100644 index c68884456..000000000 --- a/docs/notebooks/index.rst +++ /dev/null @@ -1,15 +0,0 @@ -.. _notebooks: - -Components -========== - -.. toctree:: - :maxdepth: 1 - - visualize_cameras.ipynb - visualize_samples.ipynb - visualize_samplers.ipynb - visualize_spatial_distortions.ipynb - visualize_encoders.ipynb - visualize_viewer.ipynb - \ No newline at end of file diff --git a/docs/notebooks/visualize_samplers.ipynb b/docs/notebooks/visualize_samplers.ipynb index ff85f9a5d..92ece662c 100644 --- a/docs/notebooks/visualize_samplers.ipynb +++ b/docs/notebooks/visualize_samplers.ipynb @@ -4,7 +4,43 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Ray Samplers\n" + "# Ray Samplers\n", + "\n", + "## Overview\n", + "\n", + "Once we have a set of cameras, we want to cast camera rays associated with each pixel. Along these ray we will sample the _field_ and aggregate the samples to predict the pixels value (ie. color). The parameterization of the samples are described [here](./visualize_samples.ipynb) however we must decide where to place these samples along a ray. For this task we will use a [`Sampler`](../reference/api/graphs/modules/ray_sampler.rst).\n", + "\n", + "In the ideal world we would compute many dense samples along a ray. Unfortunatly each additional sample adds a computation cost to the system as it needs to be processed by the _field_ which is often a neural network. As a result it is common for NeRF methods to use on the order of 100 samples. Therefore we want to optimize where those samples are placed in the scene.\n", + "\n", + "For example, if the sene can be bounded by a box and the objects are all similar scales, uniform sampling along the ray may be a good option. On the other hand if the scene is unbounded (potential extending as far as the the eye can see) uniform sampling does not make sense as the samples would be very sparse for close objects. In this case a different sampling like _Uniform in Disparity_ may perform better.\n", + "\n", + "```{image} imgs/samplers_type-light.png\n", + ":align: center\n", + ":class: only-light\n", + "```\n", + "\n", + "```{image} imgs/samplers_type-dark.png\n", + ":align: center\n", + ":class: only-dark\n", + "```\n", + "\n", + "### Stratified Sampling\n", + "\n", + "Most samplers has the option to _stratify_ the samplers. When stratified, each sample is randomly perturbed. The magnitude of the pertubation is such that the sample ordering remains consistent and the overal distribution statistics are not changed. Using stratified samples during training generally improves the reconstructions as it help prevent overfitting. During inference stratified sampling should be disabled (nerfactory samplers will do this) as it can cause noisy artifacts when the camera moves.\n", + "\n", + "```{image} imgs/samplers_stratified-light.png\n", + ":align: center\n", + ":class: only-light\n", + "```\n", + "\n", + "```{image} imgs/samplers_stratified-dark.png\n", + ":align: center\n", + ":class: only-dark\n", + "```\n", + "\n", + "### Heirarchical Sampling\n", + "\n", + "It is important to sample the scene where it has content otherwise the reconstruction quality will be reduced. One trick that is often employed in NeRF methods is to do multiple round of sampling. The first round can use a predefined sampler (ie. Uniform) to generate an image. Once the space is sampled, we have an idea which samples contributed to the final color. We can use this information to sample more around those regions using a `PDFSampler`. The PDF sampler is described more below." ] }, { @@ -13,7 +49,7 @@ "source": [ "## Spaced Samplers\n", "\n", - "A basic sampler that spaces samples based on a predefined function.\n" + "These are the most basic samplers that spaces samples based on a predefined function. These samplers have all have a starting and ending distance (also known as a near/far plane). The plots below are histograms of points sampled from some predefined samplers. \n" ] }, { @@ -5137,7 +5173,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## PDF Sampler" + "## PDF Sampler\n", + "\n", + "The Probability Distribution Function (PDF) Sampler generates samples that match a given distribution. In the example below we first create a `UniformSampler` to generate a set of initial samples. We then assign weights to each of these samples to define the PDF (here it is an arbitrary funtion, but usually you would use the predicted weights from the field). The left plot the target PDF, on the right we plot a histogram of samples generated from the `PDFSampler`." ] }, { @@ -7251,7 +7289,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3.8.12 ('nerfactory')", + "display_name": "Python 3.8.12 64-bit ('nerfactory')", "language": "python", "name": "python3" }, @@ -7269,7 +7307,7 @@ }, "vscode": { "interpreter": { - "hash": "34c28001ff35fb390494047002768a8182dcf55b1b11415165e62ea61557ab83" + "hash": "3271154905f6ca90bb0c3951bd3f2265caaa6ab68492eb831b8ba5f0e8deab4d" } } }, diff --git a/docs/tutorials/data_index.rst b/docs/tutorials/data_index.rst index 6c5d2e517..4a3d2e956 100644 --- a/docs/tutorials/data_index.rst +++ b/docs/tutorials/data_index.rst @@ -2,6 +2,7 @@ Data Setup ----------- .. toctree:: + :maxdepth: 1 data/choosing_dataset.md data/creating_dataset.md