From 917d02c695839654e65c2f7fdb3162a58d43b245 Mon Sep 17 00:00:00 2001 From: evonneng Date: Wed, 5 Oct 2022 02:59:59 -0700 Subject: [PATCH] adding viewer video + other doc related updates (#688) --- docs/developer_guides/pipelines/index.rst | 2 +- .../viewer/viewer_overview.md | 2 +- docs/nerfology/methods/index.md | 8 +- docs/nerfology/methods/instant_ngp.md | 5 ++ docs/nerfology/methods/nerf.md | 26 +++--- docs/nerfology/methods/nerfacto.md | 2 +- docs/nerfology/methods/semantic_nerfw.md | 2 +- docs/nerfology/model_components/index.md | 10 +-- .../model_components/visualize_samplers.ipynb | 24 ++++-- docs/quickstart/custom_dataset.md | 4 +- docs/quickstart/viewer_quickstart.md | 21 ----- docs/quickstart/viewer_quickstart.rst | 84 +++++++++++++++++++ docs/reference/api/model_components/index.rst | 2 +- 13 files changed, 142 insertions(+), 50 deletions(-) delete mode 100644 docs/quickstart/viewer_quickstart.md create mode 100644 docs/quickstart/viewer_quickstart.rst diff --git a/docs/developer_guides/pipelines/index.rst b/docs/developer_guides/pipelines/index.rst index 9ba4fa3d3..b90954525 100644 --- a/docs/developer_guides/pipelines/index.rst +++ b/docs/developer_guides/pipelines/index.rst @@ -1,4 +1,4 @@ -Pipelines Overview +Pipelines overview ------------------------- Here we describe what a Pipeline is and how it works. You can see an overview figure with the major Pipeline components below. diff --git a/docs/developer_guides/viewer/viewer_overview.md b/docs/developer_guides/viewer/viewer_overview.md index 5d45fa5f5..bd2d33e4d 100644 --- a/docs/developer_guides/viewer/viewer_overview.md +++ b/docs/developer_guides/viewer/viewer_overview.md @@ -1,4 +1,4 @@ -# Viewer Overview +# Viewer overview > We have a real-time web viewer that requires no installation. It's available at [https://viewer.nerf.studio/](https://viewer.nerf.studio/), where you can connect to your training job. If you want to understand how the viewer works and contribute to it's features, this section is for you! diff --git a/docs/nerfology/methods/index.md b/docs/nerfology/methods/index.md index 11b3e7cdf..9eb1af0a5 100644 --- a/docs/nerfology/methods/index.md +++ b/docs/nerfology/methods/index.md @@ -1,13 +1,17 @@ # Methods -We provide a set of pre implemented nerfstudio methods. One of the goals of nerfstudio is to modularize the various NeRF techniques as much as possible. As a result, many of the techniques from these pre-implemented methods can be mixed. +We provide a set of pre implemented nerfstudio methods. + +**The goal of nerfstudio is to modularize the various NeRF techniques as much as possible.** + +As a result, many of the techniques from these pre-implemented methods can be mixed 🎨. ## Running a method It's easy! ```bash -ns-train METHOD_NAME +ns-train {METHOD_NAME} ``` To list the available methods run: diff --git a/docs/nerfology/methods/instant_ngp.md b/docs/nerfology/methods/instant_ngp.md index 4a1ab5a6f..25edfe1cc 100644 --- a/docs/nerfology/methods/instant_ngp.md +++ b/docs/nerfology/methods/instant_ngp.md @@ -13,3 +13,8 @@ Paper Website ```bash ns-train instant-ngp ``` + + +```{admonition} Coming Soon +This guide is coming soon. +``` \ No newline at end of file diff --git a/docs/nerfology/methods/nerf.md b/docs/nerfology/methods/nerf.md index cb8618ec4..90e89cd59 100644 --- a/docs/nerfology/methods/nerf.md +++ b/docs/nerfology/methods/nerf.md @@ -8,17 +8,17 @@ Paper Website ``` -### Running the Model +### Running the model ```bash ns-train vanilla-nerf ``` -## Method +## Method overview -### Overview +If you have arrived to this site, it is likely that you have at least heard of NeRFs. This page will discuss the original NeRF paper, _"NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis"_ by Mildenhall, Srinivasan, Tancik et al. (2020). -If you have arrived to this site, it is likely that you have at least heard of NeRFs. This page will discuss the original NeRF paper, _"NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis"_ by Mildenhall, Srinivasan, Tancik et al. (2020). For most tasks, using the original NeRF model is likely not a good choice and hence we provide implementations of various other NeRF related models. It is however useful to understand how NeRF's work as most follow ups follow a similar structure. +For most tasks, using the original NeRF model is likely not a good choice and hence we provide implementations of various other NeRF related models. It is however useful to understand how NeRF's work as most follow ups follow a similar structure. The goal is to optimize a volumetric representation of a scene that can be rendered from novel viewpoints. This representation is optimized from a set of images and associated camera poses. @@ -44,7 +44,7 @@ If any of the following assumptions are broken, the reconstructions may fail com Here is an overview pipeline for NeRF, we will walk through each component in this guide. -### Field Representation +### Field representation ```{image} imgs/models_nerf-pipeline-field-light.png :align: center @@ -56,7 +56,7 @@ Here is an overview pipeline for NeRF, we will walk through each component in th :class: only-dark ``` -NeRFs are a volumetric representation encoded into a neural network. They are not 3D meshes and they are not voxels. For each point in space the NeRF represents a view dependent radiance. More concretely each point has a density which describes how transparent or opaque a point in space is. They also have a view dependent color that changes depending on the angle the point is viewed. +NeRFs are a volumetric representation encoded into a neural network. They are not 3D meshes and they are not voxels. **For each point in space the NeRF represents a view dependent radiance.** More concretely each point has a density which describes how transparent or opaque a point in space is. They also have a view dependent color that changes depending on the angle the point is viewed. ```{image} imgs/models_nerf-field-light.png :align: center @@ -79,7 +79,7 @@ field_coarse = NeRFField(position_encoding=pos_enc, direction_encoding=dir_enc) field_fine = NeRFField(position_encoding=pos_enc, direction_encoding=dir_enc) ``` -#### Positional Encoding +#### Positional encoding An extra trick is necessary to make the neural network expressive enough to represent fine details in the scene. The input coordinates $(x,y,z,\theta,\phi)$ need to be encoded to a higher dimensional space prior to being input into the network. You can learn more about encodings [here](../model_components/visualize_encoders.ipynb). @@ -106,7 +106,9 @@ dir_enc = NeRFEncoding( :class: only-dark ``` -Now that we have a representation of space, we need some way to render new images of it. To accomplish this, we are going to _project_ a ray from the target pixel and evaluate points along that ray. We then rely on classic volumetric rendering techniques [[Kajiya, 1984]](https://dl.acm.org/doi/abs/10.1145/964965.808594) to composite the points into a predicted color. This compositing is similar to what happens in tools like Photoshop when you layer multiple objects of varying opacity on top of each other. The only difference is that NeRF takes into account the differences in spacing between points. +Now that we have a representation of space, we need some way to render new images of it. To accomplish this, we are going to _project_ a ray from the target pixel and evaluate points along that ray. We then rely on classic volumetric rendering techniques [[Kajiya, 1984]](https://dl.acm.org/doi/abs/10.1145/964965.808594) to composite the points into a predicted color. + +This compositing is similar to what happens in tools like Photoshop when you layer multiple objects of varying opacity on top of each other. The only difference is that NeRF takes into account the differences in spacing between points. Rending RGB images is not the only type of output render supported. It is possible to render other output types such as depth and semantics. Additional renderers can be found [Here](../../reference/api/model_components/renderers.rst). @@ -137,7 +139,11 @@ rgb = renderer_rgb( :class: only-dark ``` -How we sample points along rays in space is an important design decision. Various sampling strategies can be used which are discussed in detail in the [Ray Samplers](../model_components/visualize_samplers.ipynb) guide. In NeRF we take advantage of a hierarchical sampling scheme that first uses a _uniform sampler_ and is followed by a _PDF sampler_. The uniform sampler distributes samples evenly between a predefined distance range from the camera. These are then used to compute an initial render of the scene. The renderer optionally produces _weights_ for each sample that correlate with how important each sample was to the final renderer. The PDF sampler uses these _weights_ to generate a new set of samples that are biased to regions of higher weight. In practice, these regions are near the surface of the object. +How we sample points along rays in space is an important design decision. Various sampling strategies can be used which are discussed in detail in the [Ray Samplers](../model_components/visualize_samplers.ipynb) guide. In NeRF we take advantage of a hierarchical sampling scheme that first uses a _uniform sampler_ and is followed by a _PDF sampler_. + +The uniform sampler distributes samples evenly between a predefined distance range from the camera. These are then used to compute an initial render of the scene. The renderer optionally produces _weights_ for each sample that correlate with how important each sample was to the final renderer. + +The PDF sampler uses these _weights_ to generate a new set of samples that are biased to regions of higher weight. In practice, these regions are near the surface of the object. Associated code: @@ -163,7 +169,7 @@ For all sampling, we use _Stratified_ samples during optimization and unmodified ## Benchmarks -##### Blender Synthetic +##### Blender synthetic | Implementation | Mic | Ficus | Chair | Hotdog | Materials | Drums | Ship | Lego | Average | | --------------------------------------------------------------------------------- | :-------: | --------- | :-------: | --------- | --------- | --------- | --------- | --------- | --------- | diff --git a/docs/nerfology/methods/nerfacto.md b/docs/nerfology/methods/nerfacto.md index b311f7afa..1f0c48809 100644 --- a/docs/nerfology/methods/nerfacto.md +++ b/docs/nerfology/methods/nerfacto.md @@ -12,7 +12,7 @@ ns-train nerfacto --help ### Overview -```{admonition} Note +```{admonition} Coming Soon This guide is coming soon. This is our flagship method that we encourage creators to use when making videos. It uses techniques from [Mip-NeRF 360](https://jonbarron.info/mipnerf360/) and [Instant-NGP](https://nvlabs.github.io/instant-ngp/). ``` diff --git a/docs/nerfology/methods/semantic_nerfw.md b/docs/nerfology/methods/semantic_nerfw.md index 79f1c5b35..7cdd8b6b2 100644 --- a/docs/nerfology/methods/semantic_nerfw.md +++ b/docs/nerfology/methods/semantic_nerfw.md @@ -16,7 +16,7 @@ Paper Website Paper Website ``` -```{admonition} Notice +```{admonition} Coming Soon The transient embeddings are still under development. Please stay tuned. ``` diff --git a/docs/nerfology/model_components/index.md b/docs/nerfology/model_components/index.md index b64903820..e3619d89e 100644 --- a/docs/nerfology/model_components/index.md +++ b/docs/nerfology/model_components/index.md @@ -1,12 +1,12 @@ -# Model Components +# Model components It can be difficult getting started with NeRFs. The reserach field is still quite new and most of the key nuggets are burried in academic papers. For this reason, we have consoladated many of the key concepts into a series of guides. ```{toctree} :maxdepth: 1 - Cameras Models - Sample Representation - Ray Samplers - Spatial Distortions + Cameras models + Sample representation + Ray samplers + Spatial distortions Encoders ``` diff --git a/docs/nerfology/model_components/visualize_samplers.ipynb b/docs/nerfology/model_components/visualize_samplers.ipynb index e08cc20f1..1ead09832 100644 --- a/docs/nerfology/model_components/visualize_samplers.ipynb +++ b/docs/nerfology/model_components/visualize_samplers.ipynb @@ -8,9 +8,13 @@ "\n", "## Overview\n", "\n", - "Once we have a set of cameras, we want to cast camera rays associated with each pixel. Along these ray we will sample the _field_ and aggregate the samples to predict the pixels value (ie. color). The parameterization of the samples are described [here](./visualize_samples.ipynb) however we must decide where to place these samples along a ray. For this task we will use a [`Sampler`](../../reference/api/model_components/ray_sampler.rst).\n", + "Once we have a set of cameras, we want to cast camera rays associated with each pixel. \n", "\n", - "In the ideal world we would compute many dense samples along a ray. Unfortunatly each additional sample adds a computation cost to the system as it needs to be processed by the _field_ which is often a neural network. As a result it is common for NeRF methods to use on the order of 100 samples. Therefore we want to optimize where those samples are placed in the scene.\n", + "Along these ray we will sample the _field_ and aggregate the samples to predict the pixels value (ie. color). The parameterization of the samples are described [here](./visualize_samples.ipynb) however we must decide where to place these samples along a ray. For this task we will use a [`Sampler`](../../reference/api/model_components/ray_sampler.rst).\n", + "\n", + "In the ideal world we would compute many dense samples along a ray. Unfortunatly each additional sample adds a computation cost to the system as it needs to be processed by the _field_ which is often a neural network. \n", + "\n", + "As a result it is common for NeRF methods to use on the order of 100 samples. Therefore we want to optimize where those samples are placed in the scene.\n", "\n", "For example, if the sene can be bounded by a box and the objects are all similar scales, uniform sampling along the ray may be a good option. On the other hand if the scene is unbounded (potential extending as far as the the eye can see) uniform sampling does not make sense as the samples would be very sparse for close objects. In this case a different sampling like _Uniform in Disparity_ may perform better.\n", "\n", @@ -26,7 +30,11 @@ "\n", "### Stratified Sampling\n", "\n", - "Most samplers has the option to _stratify_ the samplers. When stratified, each sample is randomly perturbed. The magnitude of the pertubation is such that the sample ordering remains consistent and the overal distribution statistics are not changed. Using stratified samples during training generally improves the reconstructions as it help prevent overfitting. During inference stratified sampling should be disabled (nerfstudio samplers will do this) as it can cause noisy artifacts when the camera moves.\n", + "Most samplers has the option to _stratify_ the samplers. When stratified, each sample is randomly perturbed. \n", + "\n", + "The magnitude of the pertubation is such that the sample ordering remains consistent and the overal distribution statistics are not changed. Using stratified samples during training generally improves the reconstructions as it help prevent overfitting. \n", + "\n", + "During inference stratified sampling should be disabled (nerfstudio samplers will do this) as it can cause noisy artifacts when the camera moves.\n", "\n", "```{image} imgs/samplers_stratified-light.png\n", ":align: center\n", @@ -40,7 +48,11 @@ "\n", "### Heirarchical Sampling\n", "\n", - "It is important to sample the scene where it has content otherwise the reconstruction quality will be reduced. One trick that is often employed in NeRF methods is to do multiple round of sampling. The first round can use a predefined sampler (ie. Uniform) to generate an image. Once the space is sampled, we have an idea which samples contributed to the final color. We can use this information to sample more around those regions using a `PDFSampler`. The PDF sampler is described more below." + "It is important to sample the scene where it has content otherwise the reconstruction quality will be reduced. \n", + "\n", + "One trick that is often employed in NeRF methods is to do multiple round of sampling. The first round can use a predefined sampler (ie. Uniform) to generate an image. Once the space is sampled, we have an idea which samples contributed to the final color. \n", + "\n", + "We can use this information to sample more around those regions using a `PDFSampler`. The PDF sampler is described more below." ] }, { @@ -5285,7 +5297,9 @@ "source": [ "## PDF Sampler\n", "\n", - "The Probability Distribution Function (PDF) Sampler generates samples that match a given distribution. In the example below we first create a `UniformSampler` to generate a set of initial samples. We then assign weights to each of these samples to define the PDF (here it is an arbitrary funtion, but usually you would use the predicted weights from the field). The left plot the target PDF, on the right we plot a histogram of samples generated from the `PDFSampler`." + "The Probability Distribution Function (PDF) Sampler generates samples that match a given distribution. \n", + "\n", + "In the example below we first create a `UniformSampler` to generate a set of initial samples. We then assign weights to each of these samples to define the PDF (here it is an arbitrary funtion, but usually you would use the predicted weights from the field). The left plot the target PDF, on the right we plot a histogram of samples generated from the `PDFSampler`." ] }, { diff --git a/docs/quickstart/custom_dataset.md b/docs/quickstart/custom_dataset.md index c17f6323e..93db6c2c8 100644 --- a/docs/quickstart/custom_dataset.md +++ b/docs/quickstart/custom_dataset.md @@ -9,7 +9,7 @@ To assist running on custom data we have a script that will process a video or f To process your data run: ```bash -ns-process-data --data FOLDER_OR_VIDEO --output-dir PROCESSED_DATA_DIR +ns-process-data --data {FOLDER_OR_VIDEO} --output-dir {PROCESSED_DATA_DIR} ``` A full set of arguments can be found {doc}`here`. @@ -25,7 +25,7 @@ A full set of arguments can be found {doc}`here` Simply specify that you are using the `nerfstudio` dataparser and point the data directory to your processed data. ```bash -ns-train instant-ngp nerfstudio-data --data PROCESSED_DATA_DIR +ns-train instant-ngp nerfstudio-data --data {PROCESSED_DATA_DIR} ``` ### Installing COLMAP diff --git a/docs/quickstart/viewer_quickstart.md b/docs/quickstart/viewer_quickstart.md deleted file mode 100644 index 8e5cd9d69..000000000 --- a/docs/quickstart/viewer_quickstart.md +++ /dev/null @@ -1,21 +0,0 @@ -# Setting up viewer - -## Training on a local machine - -You should be able to click the link obtained while running a script to open the viewer in your browser. -![visualize_dataset](imgs/viewer_link.png) - -## Training on a remote machine - -If you are training on a remote maching, the viewer will still let you view your trainings. You will need to forward the port that the viewer is running on. For example, if you are running the viewer on port 7007, you will need to forward that port to your local machine. You can (without needing to modify router port settings) simply do this by opening a new terminal window and sshing into the remote machine with the following command: - -```bash -ssh -L 7007:localhost:7007 @ -``` -So long as you don't close this terminal window with this specific active ssh connection, the port will remain open for all your other ssh sessions. For example, if you do this in a new terminal window, any existing ssh sessions (terminal windows, VSCode remote connection, etc) will still be able to access the port, but if you close this terminal window, the port will be closed. - -You can now simply open the link (the one printed when you begin your train script) in your browser on your local machine and it should connect! - -![visualize_dataset](imgs/viewer_link.png) - -If the port is being used, you can simply switch the port using the `--viewer.websocket-port` flag tied to the model subcommand. \ No newline at end of file diff --git a/docs/quickstart/viewer_quickstart.rst b/docs/quickstart/viewer_quickstart.rst new file mode 100644 index 000000000..710c583c2 --- /dev/null +++ b/docs/quickstart/viewer_quickstart.rst @@ -0,0 +1,84 @@ +Using the viewer +================ + +The nerfstudio web-based viewer makes it easy to view training in real-time, and to create content videos from your trained models 🌟! + +In the tutorial below, we walk you through how you can turn a simple capture into a 3D video 📸 🎬 + +.. raw:: html + + + +For specific sections, click the links below to navigate to the associated portion of the video. + +Getting started +""""""""""""""" + +* `Hello from nerfstudio `_ +* `Preprocess your video `_ +* `Launching training and viewer `_ + +Viewer basics +""""""""""""""" + +* `Viewer scene introduction `_ +* `Moving around in viewer `_ +* `Overview of Controls Panel - train speed/output options `_ +* `Overview of Scene Panel - toggle visibility `_ + +Creating camera trajectories +"""""""""""""""""""""""""""" + +* `Creating a custom camera path `_ +* `Camera spline options - cycle, speed, smoothness `_ +* `Camera options - move, add, view `_ + +Rendering a video +""""""""""""""""" + +* `Preview camera trajectory `_ +* `How to render final video `_ + +| + +.. seealso:: + + For a more in-depth developer overview on how to hack with the viewer, see our `developer guide `_ + + +Local vs. remote compute +"""""""""""""""""""""""""" + +Training on a local machine +--------------------------- + +You should be able to click the link obtained while running a script to open the viewer in your browser. + +.. image:: imgs/viewer_link.png + :width: 800 + :alt: Visualize dataset + +Training on a remote machine +---------------------------- + +If you are training on a remote maching, the viewer will still let you view your trainings. +You will need to forward the port that the viewer is running on. + +For example, if you are running the viewer on port 7007, you will need to forward that port to your local machine. +You can (without needing to modify router port settings) simply do this by opening a new terminal window and sshing into the remote machine with the following command: + +```bash +ssh -L 7007:localhost:7007 @ +``` + +.. admonition:: Note + + So long as you don't close this terminal window with this specific active ssh connection, the port will remain open for all your other ssh sessions. + + For example, if you do this in a new terminal window, any existing ssh sessions (terminal windows, VSCode remote connection, etc) will still be able to access the port, but if you close this terminal window, the port will be closed. + + +You can now simply open the link (same ones shown in image above) in your browser on your local machine and it should connect! + +.. warning:: + If the port is being used, you will need to switch the port using the `--viewer.websocket-port` flag tied to the model subcommand. \ No newline at end of file diff --git a/docs/reference/api/model_components/index.rst b/docs/reference/api/model_components/index.rst index fffc29d3b..18a11d7f1 100644 --- a/docs/reference/api/model_components/index.rst +++ b/docs/reference/api/model_components/index.rst @@ -1,6 +1,6 @@ .. _graph_modules: -Model Components +Model components =================== .. toctree::