Tancik/docs 16 (#249)

* update docs

* add missing notebook

Co-authored-by: Matthew Tancik <tancik>

docs/index.rst

@@ -46,7 +46,7 @@ Contents
    :caption: Guides
 
    
-   models/index.rst
+   models/index
    notebooks/index
 
 .. toctree::

docs/models/index.md

@@ -0,0 +1,25 @@
+# Models
+
+We provide a set of preimplemented nerfactory models. One of the goals of nerfactory is to modularize the various NeRF techniques as much as possible. As a result, many of the techniqes from these preimplemented models can be mixed.
+
+## Running a model
+
+It's easy!
+
+```bash
+python scripts/run_train.py --config-name MODEL_CONFIG
+```
+
+## Guides
+
+In addition to their implementations, we have provided guides that walk through each of these method. 
+
+```{toctree}
+    :maxdepth: 1
+    NeRF<nerf.md>
+    Mip-NeRF<mipnerf.md>
+    Mip-NeRF 360<mipnerf_360.md>
+    Instan-NGP<instant_ngp.md>
+    NeRF-W<nerf_w.md>
+    Semantic NeRF<semantic_nerf.md>
+```

docs/models/index.rst

@@ -1,14 +0,0 @@
-.. _models:
-
-Models
-========
-
-.. toctree::
-    :maxdepth: 1
-
-    NeRF<nerf.md>
-    Mip-NeRF<mipnerf.md>
-    Mip-NeRF 360<mipnerf_360.md>
-    Instan-NGP<instant_ngp.md>
-    NeRF-W<nerf_w.md>
-    Semantic NeRF<semantic_nerf.md>

docs/models/nerf.md

@@ -16,6 +16,20 @@ python scripts/run_train.py --config-name=graph_vanilla_nerf.yaml
 
 ## Method
 
+### Pipeline
+
+```{image} imgs/models_nerf-field-light.png
+:align: center
+:class: only-light
+:width: 500
+```
+
+```{image} imgs/models_nerf-field-dark.png
+:align: center
+:class: only-dark
+:width: 500
+```
+
 ### Overview
 
 If you have arrived to this site, it is likely that you have atleast heard of NeRFs. This page will discuss the original NeRF paper, _"NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis"_ by Mildenhall, Srinivasan, Tancik et al. (2020). For most tasks, using the original NeRF model is likely not a good choice and hence we provide implementations of various other NeRF related models. It is however useful to understand how NeRF's work as most follow ups follow a similar structure.

docs/notebooks/index.md

@@ -0,0 +1,13 @@
+# Components
+
+It can be difficult getting started with NeRFs. The field is still quite new and most of the key nuggets are burried in research papers. For this reason we can consoladated many of the key concepts into a series of guides.
+
+```{toctree}
+    :maxdepth: 1
+    Cameras Models<visualize_cameras.ipynb>
+    Sample Representation<visualize_samples.ipynb>
+    Ray Samplers<visualize_samplers.ipynb>
+    Spatial Distortions<visualize_spatial_distortions.ipynb>
+    Encoders<visualize_encoders.ipynb>
+    Viewer<visualize_viewer.ipynb>
+```

docs/notebooks/index.rst

@@ -1,15 +0,0 @@
-.. _notebooks:
-
-Components
-==========
-
-.. toctree::
-    :maxdepth: 1
-
-    visualize_cameras.ipynb
-    visualize_samples.ipynb
-    visualize_samplers.ipynb
-    visualize_spatial_distortions.ipynb
-    visualize_encoders.ipynb
-    visualize_viewer.ipynb
-    

docs/notebooks/visualize_samplers.ipynb

@@ -4,7 +4,43 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Ray Samplers\n"
+    "# Ray Samplers\n",
+    "\n",
+    "## Overview\n",
+    "\n",
+    "Once we have a set of cameras, we want to cast camera rays associated with each pixel. Along these ray we will sample the _field_ and aggregate the samples to predict the pixels value (ie. color). The parameterization of the samples are described [here](./visualize_samples.ipynb) however we must decide where to place these samples along a ray. For this task we will use a [`Sampler`](../reference/api/graphs/modules/ray_sampler.rst).\n",
+    "\n",
+    "In the ideal world we would compute many dense samples along a ray. Unfortunatly each additional sample adds a computation cost to the system as it needs to be processed by the _field_ which is often a neural network. As a result it is common for NeRF methods to use on the order of 100 samples. Therefore we want to optimize where those samples are placed in the scene.\n",
+    "\n",
+    "For example, if the sene can be bounded by a box and the objects are all similar scales, uniform sampling along the ray may be a good option. On the other hand if the scene is unbounded (potential extending as far as the the eye can see) uniform sampling does not make sense as the samples would be very sparse for close objects. In this case a different sampling like _Uniform in Disparity_ may perform better.\n",
+    "\n",
+    "```{image} imgs/samplers_type-light.png\n",
+    ":align: center\n",
+    ":class: only-light\n",
+    "```\n",
+    "\n",
+    "```{image} imgs/samplers_type-dark.png\n",
+    ":align: center\n",
+    ":class: only-dark\n",
+    "```\n",
+    "\n",
+    "### Stratified Sampling\n",
+    "\n",
+    "Most samplers has the option to _stratify_ the samplers. When stratified, each sample is randomly perturbed. The magnitude of the pertubation is such that the sample ordering remains consistent and the overal distribution statistics are not changed. Using stratified samples during training generally improves the reconstructions as it help prevent overfitting. During inference stratified sampling should be disabled (nerfactory samplers will do this) as it can cause noisy artifacts when the camera moves.\n",
+    "\n",
+    "```{image} imgs/samplers_stratified-light.png\n",
+    ":align: center\n",
+    ":class: only-light\n",
+    "```\n",
+    "\n",
+    "```{image} imgs/samplers_stratified-dark.png\n",
+    ":align: center\n",
+    ":class: only-dark\n",
+    "```\n",
+    "\n",
+    "### Heirarchical Sampling\n",
+    "\n",
+    "It is important to sample the scene where it has content otherwise the reconstruction quality will be reduced. One trick that is often employed in NeRF methods is to do multiple round of sampling. The first round can use a predefined sampler (ie. Uniform) to generate an image. Once the space is sampled, we have an idea which samples contributed to the final color. We can use this information to sample more around those regions using a `PDFSampler`. The PDF sampler is described more below."
    ]
   },
   {
@@ -13,7 +49,7 @@
    "source": [
     "## Spaced Samplers\n",
     "\n",
-    "A basic sampler that spaces samples based on a predefined function.\n"
+    "These are the most basic samplers that spaces samples based on a predefined function. These samplers have all have a starting and ending distance (also known as a near/far plane). The plots below are histograms of points sampled from some predefined samplers. \n"
    ]
   },
   {
@@ -5137,7 +5173,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## PDF Sampler"
+    "## PDF Sampler\n",
+    "\n",
+    "The Probability Distribution Function (PDF) Sampler generates samples that match a given distribution. In the example below we first create a `UniformSampler` to generate a set of initial samples. We then assign weights to each of these samples to define the PDF (here it is an arbitrary funtion, but usually you would use the predicted weights from the field). The left plot the target PDF, on the right we plot a histogram of samples generated from the `PDFSampler`."
    ]
   },
   {
@@ -7251,7 +7289,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.8.12 ('nerfactory')",
+   "display_name": "Python 3.8.12 64-bit ('nerfactory')",
    "language": "python",
    "name": "python3"
   },
@@ -7269,7 +7307,7 @@
   },
   "vscode": {
    "interpreter": {
-    "hash": "34c28001ff35fb390494047002768a8182dcf55b1b11415165e62ea61557ab83"
+    "hash": "3271154905f6ca90bb0c3951bd3f2265caaa6ab68492eb831b8ba5f0e8deab4d"
    }
   }
  },

docs/tutorials/data_index.rst

@@ -2,6 +2,7 @@ Data Setup
 -----------
 
 .. toctree::
+    :maxdepth: 1
 
     data/choosing_dataset.md
     data/creating_dataset.md