restructure codebase pre release (#602)

* moving a lot

* building
This commit is contained in:
Ethan Weber authored and GitHub committed 2022-10-01 16:36:25 -07:00
1 parent 62f492f1bc
commit e13d401b16
97 files changed
+235712 -235721

No files matched your search

+1 -2
View File
@@ -15,8 +15,7 @@ import os
import sys
sys.path.insert(0, os.path.abspath(".."))
import nerfstudio.configs.base
import nerfstudio.datamanagers.base
import nerfstudio.configs.base_config
# -- Project information -----------------------------------------------------
File diff suppressed because it is too large. Load diff
@@ -8,7 +8,7 @@
"\n",
"## Overview\n",
"\n",
"Once we have a set of cameras, we want to cast camera rays associated with each pixel. Along these ray we will sample the _field_ and aggregate the samples to predict the pixels value (ie. color). The parameterization of the samples are described [here](./visualize_samples.ipynb) however we must decide where to place these samples along a ray. For this task we will use a [`Sampler`](../../reference/api/graphs/modules/ray_sampler.rst).\n",
"Once we have a set of cameras, we want to cast camera rays associated with each pixel. Along these ray we will sample the _field_ and aggregate the samples to predict the pixels value (ie. color). The parameterization of the samples are described [here](./visualize_samples.ipynb) however we must decide where to place these samples along a ray. For this task we will use a [`Sampler`](../../reference/api/model_components/ray_sampler.rst).\n",
"\n",
"In the ideal world we would compute many dense samples along a ray. Unfortunatly each additional sample adds a computation cost to the system as it needs to be processed by the _field_ which is often a neural network. As a result it is common for NeRF methods to use on the order of 100 samples. Therefore we want to optimize where those samples are placed in the scene.\n",
"\n",
@@ -5115,7 +5115,7 @@
"# COLLAPSED\n",
"import torch\n",
"from nerfstudio.cameras.rays import RayBundle\n",
"from nerfstudio.models.modules import ray_sampler\n",
"from nerfstudio.model_components import ray_sampler\n",
"import plotly.graph_objects as go\n",
"from plotly.subplots import make_subplots\n",
"\n",
@@ -7223,7 +7223,7 @@
"# COLLAPSED\n",
"import torch\n",
"from nerfstudio.cameras.rays import RayBundle\n",
"from nerfstudio.models.modules import ray_sampler\n",
"from nerfstudio.model_components import ray_sampler\n",
"import plotly.graph_objects as go\n",
"from plotly.subplots import make_subplots\n",
"\n",
File diff suppressed because it is too large. Load diff
+3 -3
View File
@@ -84,7 +84,7 @@ field_fine = NeRFField(position_encoding=pos_enc, direction_encoding=dir_enc)
An extra trick is necessary to make the neural network expressive enough to represent fine details in the scene. The input coordinates $(x,y,z,\theta,\phi)$ need to be encoded to a higher dimensional space prior to being input into the network. You can learn more about encodings [here](../model_components/visualize_encoders.ipynb).
```python
from nerfstudio.fields.modules.encoding import NeRFEncoding
from nerfstudio.field_components.encoding import NeRFEncoding
pos_enc = NeRFEncoding(
in_dim=3, num_frequencies=10, min_freq_exp=0.0, max_freq_exp=8.0, include_input=True
@@ -108,7 +108,7 @@ dir_enc = NeRFEncoding(
Now that we have a representation of space, we need some way to render new images of it. To accomplish this, we are going to _project_ a ray from the target pixel and evaluate points along that ray. We then rely on classic volumetric rendering techniques [[Kajiya, 1984]](https://dl.acm.org/doi/abs/10.1145/964965.808594) to composite the points into a predicted color. This compositing is similar to what happens in tools like Photoshop when you layer multiple objects of varying opacity on top of each other. The only difference is that NeRF takes into account the differences in spacing between points.
Rending RGB images is not the only type of output render supported. It is possible to render other output types such as depth and semantics. Additional renderers can be found [Here](../../reference/api/renderers/index.rst).
Rending RGB images is not the only type of output render supported. It is possible to render other output types such as depth and semantics. Additional renderers can be found [Here](../../reference/api/model_components/renderers.rst).
Associated nerfstudio code:
@@ -142,7 +142,7 @@ How we sample points along rays in space is an important design decision. Variou
Associated code:
```python
from nerfstudio.models.modules.ray_sampler import PDFSampler, UniformSampler
from nerfstudio.model_components.ray_sampler import PDFSampler, UniformSampler
sampler_uniform = UniformSampler(num_samples=num_coarse_samples)
ray_samples_uniform = sampler_uniform(ray_bundle)
+1 -1
View File
@@ -3,6 +3,6 @@
Configs
============
.. automodule:: nerfstudio.configs.base
.. automodule:: nerfstudio.configs.base_config
:members:
:show-inheritance:
+6 -6
View File
@@ -6,34 +6,34 @@ Data
Data Parsers
----------------
.. automodule:: nerfstudio.datamanagers.dataparsers
.. automodule:: nerfstudio.data.dataparsers
:members:
:show-inheritance:
Datasets
----------------
.. automodule:: nerfstudio.datamanagers.datasets
.. automodule:: nerfstudio.data.utils.datasets
:members:
:show-inheritance:
Dataloaders
----------------
.. automodule:: nerfstudio.datamanagers.dataloaders
.. automodule:: nerfstudio.data.utils.dataloaders
:members:
:show-inheritance:
Pixel Samplers
----------------
.. automodule:: nerfstudio.datamanagers.pixel_sampler
.. automodule:: nerfstudio.data.pixel_samplers
:members:
:show-inheritance:
Structures
Scene Box
----------------
.. automodule:: nerfstudio.datamanagers.structs
.. automodule:: nerfstudio.data.scene_box
:members:
:show-inheritance:
@@ -3,6 +3,6 @@
Embeddings
===================
.. automodule:: nerfstudio.fields.modules.embedding
.. automodule:: nerfstudio.field_components.embedding
:members:
:show-inheritance:
@@ -3,6 +3,6 @@
Encodings
===================
.. automodule:: nerfstudio.fields.modules.encoding
.. automodule:: nerfstudio.field_components.encoding
:members:
:show-inheritance:
@@ -3,6 +3,6 @@
Field Heads
===================
.. automodule:: nerfstudio.fields.modules.field_heads
.. automodule:: nerfstudio.field_components.field_heads
:members:
:show-inheritance:
@@ -3,6 +3,6 @@
Field Losses
===================
.. automodule:: nerfstudio.fields.modules.field_losses
.. automodule:: nerfstudio.field_components.field_losses
:members:
:show-inheritance:
@@ -3,6 +3,6 @@
MLP
===================
.. automodule:: nerfstudio.fields.modules.mlp
.. automodule:: nerfstudio.field_components.mlp
:members:
:show-inheritance:
@@ -3,6 +3,6 @@
Spatial Distortions
=====================
.. automodule:: nerfstudio.fields.modules.spatial_distortions
.. automodule:: nerfstudio.field_components.spatial_distortions
:members:
:show-inheritance:
+1 -1
View File
@@ -6,7 +6,7 @@ Fields
Base
----------------
.. automodule:: nerfstudio.fields.base
.. automodule:: nerfstudio.fields.base_field
:members:
:show-inheritance:
+3 -4
View File
@@ -10,10 +10,9 @@ TODO: Explanation of each component
config
data
fields
fields/modules/index
graphs
graphs/modules/index
field_components/index
models
model_components/index
optimizers
renderers/index
utils/index
viewer
@@ -8,4 +8,5 @@ TODO: High level description of graph modules and how they are used.
.. toctree::
ray_sampler
ray_losses
ray_losses
renderers
@@ -3,6 +3,6 @@
Ray Losses
===================
.. automodule:: nerfstudio.models.modules.ray_losses
.. automodule:: nerfstudio.model_components.ray_losses
:members:
:show-inheritance:
@@ -3,6 +3,6 @@
Ray Sampler
===================
.. automodule:: nerfstudio.models.modules.ray_sampler
.. automodule:: nerfstudio.model_components.ray_sampler
:members:
:show-inheritance:
@@ -3,6 +3,6 @@
Renderers
============
.. automodule:: nerfstudio.renderers.renderers
.. automodule:: nerfstudio.model_components.renderers
:members:
:show-inheritance:
File renamed without changes.
+12 -3
View File
@@ -1,9 +1,18 @@
.. _optimizers:
.. _engine:
Optimizers
Engine
============
.. automodule:: nerfstudio.optimizers.optimizers
Optimizers
----------------
.. automodule:: nerfstudio.engine.optimizers
:members:
:show-inheritance:
Schedulers
----------------
.. automodule:: nerfstudio.engine.schedulers
:members:
:show-inheritance:
@@ -18,19 +18,19 @@
"\n",
"### Modules\n",
"\n",
"We can decompose all graphs into respective modules. For instance, we have {ref}`optimizers` which contains modules related to the loss, {ref}`renderers` which contains all rendering modules, {`graph/modules/`}, which includes samplers and ray generators. Additionally, we have {ref}`field_modules`, which will be discussed below.\n",
"We can decompose all graphs into respective modules. For instance, we have `optimizers` which contains modules related to the loss, `renderers` which contains all rendering modules, {`graph/modules/`}, which includes samplers and ray generators. Additionally, we have `field_modules`, which will be discussed below.\n",
"In general, modules can be thought of as individual component parts of the graph that can be swapped in/out. We provide most of the standard modules, but if you are creating a new NeRF graph, you may have to define new modules.\n",
"\n",
"### Fields\n",
"\n",
"{ref}`fields` represents the \"space\", aka. the radiance field of the NeRF. Here, we define the field as the part of the network that takes in point samples and any other conditioning, and outputs any of the `FieldHeadNames` (`nerf/field_modules/field_heads.py`).\n",
"`fields` represents the \"space\", aka. the radiance field of the NeRF. Here, we define the field as the part of the network that takes in point samples and any other conditioning, and outputs any of the `FieldHeadNames` (`nerf/field_modules/field_heads.py`).\n",
"\n",
"All fields are composed of modules continained within the `fields/` directory (e.g. `fields/modules/`). We can think of field modules as modules that actually define or interact with the field.\n",
"\n",
"To build a NeRF field, we therefore follow these steps:\n",
"\n",
"1. define any relevant field modules\n",
"2. create a new class that extends the {ref}`fields` base class and use these field modules to compose the full field\n",
"2. create a new class that extends the `fields` base class and use these field modules to compose the full field\n",
"3. implement the abstract functions `get_density()` and `get_outputs()`."
]
},
+353 -353
View File
@@ -1,354 +1,354 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"# Creating Custom Pipelines\n",
"\n",
"> Our goal is to enable nerfstudio users to implement an entire NeRF method by only editing one file.\n",
"\n",
"Here we explain how to create custom pipelines with nerfstudio. Pipelines are composed of two components, namely a DataManager and a Model. The pipeline is responsible for routing RayBundle instances returned from the DataManager into the Model.\n",
"\n",
"A NeRF method is implemented as a Pipeline. Users can create a Pipeline with a DataManager and Model. We provide default implementations for these components prefixed with the name Vanilla (e.g., `VanillaPipeline`, `VanillaDataManager` which inherit from the `Pipeline` and `DataManager` class, respectively) and specific Models (e.g., `InstantNGPModel` or `TensoRFModel` which inherit from the `Model` class). With most NeRF methods, `VanillaPipeline` and `VanillaDataManager` can be used and only the model needs to be changed. We provide many already implmented Models for people to get started with 3D reconstruction. However, in this tutorial we will demonstrate how to customize each of the Pipeline, DataManager, and Model with an example.\n",
"\n",
"We'll show how to make an incremental DataManager where we progressively add cameras to train with from multiple scenes and importance sample rays from pixels with a high loss. The extensible features that we show in this tutorial are the following:\n",
"- a DataManager that incrementally adds cameras\n",
"- a DataManager that uses multiple scenes\n",
"- a Model that is conditioned on the scene\n",
"- a Pipeline that modifies the DataManager's PixelSampler to sample more rays in regions with high loss"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"#### Model Components\n",
"\n",
"> The model renders values from 3D line segments defined by a RayBundle.\n",
"\n",
"We provide Models and Model Components (i.e., Fields and Field Modules) which can be strung together to create a custom model.\n",
"\n",
"<details closed>\n",
"<summary>Models</summary>\n",
"<br>\n",
"<li>Vanilla NeRF Model</li>\n",
"<li>Mip NeRF Model</li>\n",
"<li>Mip NeRF 360 Model</li>\n",
"<li>Instant NGP Model</li>\n",
"<li>NeRF-W Model</li>\n",
"<li>Semantic NeRF Model</li>\n",
"</details>\n",
"\n",
"<br>\n",
"\n",
"<details closed>\n",
"<summary>Model Components</summary>\n",
"<br>\n",
"Fields:\n",
"<li>Vanilla NeRF Field</li>\n",
"<li>Instant NGP Field</li>\n",
"<br>\n",
"Field Modules:\n",
"<li>Encodings</li>\n",
"<li>Embeddings</li>\n",
"<li>Field Heads</li>\n",
"<li>Field Losses</li>\n",
"<li>MLP</li>\n",
"<li>Spatial Distortions</li>\n",
"</details>\n",
"\n",
"#### DataManager Components\n",
"\n",
"> The DataManager provides a RayBundle (3D line segments with metadata needed to run the forward pass) and a dictionary of information needed at train-time to supervise the pipeline components.\n",
"\n",
"Similar to the Model components, we provide DataManager components that are strung together to generate RayBundle instances. A RayBundle contains everything needed to generate a ray (e.g., origin and direction, near and far planes) as well as any additional metadata that the Model requires (e.g., camera index, scene index, etc.).\n",
"\n",
"<details closed>\n",
"<summary>DataManager Components</summary>\n",
"<br>\n",
"Datasets Parsers:<br>\n",
" <em style=\"padding-left:2em\">Takes a data location and returns a DatsetInputs instance. A DatsetInputs will contain all data needed throughout the pipeline.</em>\n",
"<li>Vanilla Image DataParser</li>\n",
"<br>\n",
"Image Datasets:<br>\n",
" <em style=\"padding-left:2em\">Takes image filename information and can return images.</em>\n",
"<li>Vanilla Image Dataset</li>\n",
"<br>\n",
"Image Dataloader:<br>\n",
" <em style=\"padding-left:2em\">Takes an Image Dataset and samples images from it.</em>\n",
"<li>Vanilla Image Dataloader</li>\n",
"<li>Cache Image Dataloader</li>\n",
"<br>\n",
"Pixel Samplers:<br>\n",
" <em style=\"padding-left:2em\">Takes sampled images and returns pixel locations.</em>\n",
"<li>Vanilla Pixel Sampler</li>\n",
"<br>\n",
"Ray Generators:<br>\n",
" <em style=\"padding-left:2em\">Takes cameras and pixel locations and returns RayBundle objects.</em>\n",
"<br>\n",
"\n",
"Colliders:<br>\n",
" <em style=\"padding-left:2em\">Takes RayBundles and clips them according .</em>\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"# COLLAPSED\n",
"%load_ext autoreload\n",
"%autoreload 2\n",
"\n",
"from __future__ import annotations\n",
"\n",
"from typing import Dict, Optional, Tuple, List\n",
"import torch\n",
"\n",
"from nerfstudio.utils.config import DataManagerConfig, ModelConfig, PipelineConfig, ViewerConfig\n",
"from nerfstudio.datamanagers.base import DataManager\n",
"from nerfstudio.datamanagers.structs import DatasetInputs\n",
"from nerfstudio.models.base import Model\n",
"from nerfstudio.pipelines.base import Pipeline\n",
"from nerfstudio.models.instant_ngp import NGPModel\n",
"from nerfstudio.cameras.rays import RayBundle"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"class CustomDataManagerConfig(DataManagerConfig):\n",
" _target = \"CustomDataManager\"\n",
" scene_names: List[str] = [\"chair\", \"drums\", \"fern\", \"lego\"]\n",
" steps_per_add_cameras: int = 100 # every X iterations, add more cameras\n",
" num_cameras_per_add: int = 10 # number of cameras to add\n",
"\n",
"\n",
"class CustomDataManager(DataManager):\n",
" \"\"\"A custom dataloader that incrementally adds cameras and has multiple scenes.\"\"\"\n",
"\n",
" config: CustomDataManagerConfig\n",
"\n",
" def populate_train_modules(self):\n",
" \"\"\"Populate the train dataloader modules.\"\"\"\n",
" self.scene_name_to_image_sampler = {}\n",
" self.scene_name_to_collider = {} # TODO(ethan): move colliders to the dataloader\n",
" for scene_name in self.config.scene_names:\n",
" dataset_inputs: DatasetInputs = None # TODO\n",
" image_dataset = ImageDataset(dataset_inputs)\n",
" image_sampler = CacheImageDataloader(image_dataset=image_dataset) # a torch dataloader\n",
" self.scene_name_to_image_sampler[scene_name] = image_sampler\n",
" self.train_pixel_sampler = PixelSampler\n",
" self.train_ray_generator = RayGenerator # nn.Module\n",
"\n",
" def populate_eval_modules(self):\n",
" \"\"\"Populate the eval dataloader modules.\"\"\"\n",
" # we'll mostly rely on the train dataloader modules\n",
" self.eval_pixel_sampler = EvalPixelSampler # this will sample entire images\n",
"\n",
" def sample_images(self) -> List[Image]:\n",
" images = []\n",
" for scene_name in self.scene_name_to_image_sampler.keys():\n",
" image_sampler = self.scene_name_to_image_sampler[scene_name]\n",
" x = image_sampler.forward() # note that empty forward calls next()\n",
" images.append(x)\n",
" return images\n",
"\n",
" def next_train(self, step: int) -> Tuple[RayBundle, Dict]:\n",
" \"\"\"Get the next batch of training data by stringing together the train modules.\"\"\"\n",
" # grab some images from each scene\n",
" images = self.sample_images()\n",
" pixels = self.train_pixel_sampler.forward(images)\n",
" ray_bundle = self.train_ray_generator(pixels)\n",
" # shape should be (H, W, :)\n",
" ray_bundle.metadata[\"scene_indices\"] = None # TODO(ethan): populate this with a tensor of scene indices\n",
" dict_ = {}\n",
" return ray_bundle, dict_\n",
"\n",
" def next_eval(self, step: int) -> Tuple[RayBundle, Dict]:\n",
" \"\"\"Get the next batch of eval data by stringing together the eval modules.\"\"\"\n",
" # grab some images from each scene\n",
" images = self.sample_images()\n",
" pixels = self.train_pixel_sampler.forward(images)\n",
" ray_bundle = self.eval_ray_generator(pixels) # notice this is eval and not train\n",
" # shape should be (H, W, :)\n",
" ray_bundle.metadata[\"scene_indices\"] = None # TODO(ethan): populate this with a tensor of scene indices\n",
" dict_ = {}\n",
" return ray_bundle, dict_\n",
"\n",
"\n",
"class CustomNGPModelConfig(ModelConfig):\n",
" _target = \"CustomNGPModel\"\n",
" coarse_field: str = \"temp\"\n",
" fine_field: str = \"temp2\"\n",
"\n",
"\n",
"class CustomNGPModel(Model):\n",
" \"\"\"An instant ngp model modified slightly to output semantics.\"\"\"\n",
"\n",
" config: SceneConditionNGPModelConfig\n",
"\n",
" def populate_modules(self):\n",
" self.ngp_model = NGPModel(coarse_field=self.config.coarse_field, fine_field=self.config.fine_field)\n",
"\n",
" def get_outputs(self, ray_bundle: RayBundle) -> Dict[str, torch.Tensor]:\n",
" # TODO(ethan): pass in batch from the forward function\n",
" # TODO(ethan): rename batch to something else\n",
" outputs = self.ngp_model.forward(ray_bundle)\n",
" outputs[\"semantics\"] = torch.rand_like(outputs[\"rgb\"])\n",
" return outputs\n",
"\n",
" def get_loss_dict(self):\n",
" return {}\n",
"\n",
"\n",
"class CustomPipeline(Pipeline):\n",
" \"\"\"The Instant NGP pipeline.\"\"\"\n",
"\n",
" config: PipelineConfig\n",
"\n",
" def train_step(self, step: int):\n",
" ray_bundle, batch = self.dataloader.next_train(step=step)\n",
" # TODO: maybe run a CNN on the data before passing into model\n",
" model_outputs, loss_dict, metrics_dict = self.model(ray_bundle, batch)\n",
" # TODO: update pixel sampler state with loss map to show the flexibilty of our Pipeline\n",
" self.dataloader.pixel_sampler.update_loss_map()\n",
" return model_outputs, loss_dict, metrics_dict\n",
"\n",
" def eval_step(self):\n",
" ray_bundle, batch = self.dataloader.next_eval(step=step)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"dataloader = DataManager()\n",
"model = SceneConditionNGPModel()\n",
"pipeline = CustomPipeline.from_dataloader_and_model(dataloader=dataloader, model=model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"# Creating pipelines from a config\n",
"\n",
"Now we show how to create a pipeline from a config, which has the following form:\n",
"\n",
"```python\n",
"@dataclass\n",
"class PipelineConfig:\n",
" \"\"\"Configuration for pipeline instantiation\"\"\"\n",
"\n",
" _target: ClassVar[Type] = Pipeline\n",
" dataloader: DataloaderConfig = MISSING\n",
" model: ModelConfig = MISSING\n",
"```\n",
"\n",
"See `nerfstudio/utils/config.py` for more details. In this example, we will simply load from an existing configuration from `configs/graph_instant_ngp.yaml`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import pprint\n",
"import hydra\n",
"\n",
"hydra.core.global_hydra.GlobalHydra.instance().clear()\n",
"from hydra import compose, initialize\n",
"\n",
"initialize(version_base=\"1.2\", config_path=\"../configs/\")\n",
"config_name = \"graph_instant_ngp.yaml\"\n",
"config = compose(config_name)\n",
"pipeline_config = config.pipeline\n",
"pprint.pprint(pipeline_config)\n",
"print(\"----------------------------------------------------\")\n",
"\n",
"# from nerfstudio.pipelines.base import setup_pipeline\n",
"# pipeline = setup_pipeline(pipeline_config, device=\"cuda\")\n",
"pipeline = pipeline_config.setup()"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"pipeline.get_train_loss_dict()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.8.13 ('nerfactory')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
},
"vscode": {
"interpreter": {
"hash": "100252972d4ee5286ab455ef1c778abde05b4affca9d1be78633d3797abf7e8c"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}
"cells": [
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"# Creating Custom Pipelines\n",
"\n",
"> Our goal is to enable nerfstudio users to implement an entire NeRF method by only editing one file.\n",
"\n",
"Here we explain how to create custom pipelines with nerfstudio. Pipelines are composed of two components, namely a DataManager and a Model. The pipeline is responsible for routing RayBundle instances returned from the DataManager into the Model.\n",
"\n",
"A NeRF method is implemented as a Pipeline. Users can create a Pipeline with a DataManager and Model. We provide default implementations for these components prefixed with the name Vanilla (e.g., `VanillaPipeline`, `VanillaDataManager` which inherit from the `Pipeline` and `DataManager` class, respectively) and specific Models (e.g., `InstantNGPModel` or `TensoRFModel` which inherit from the `Model` class). With most NeRF methods, `VanillaPipeline` and `VanillaDataManager` can be used and only the model needs to be changed. We provide many already implmented Models for people to get started with 3D reconstruction. However, in this tutorial we will demonstrate how to customize each of the Pipeline, DataManager, and Model with an example.\n",
"\n",
"We'll show how to make an incremental DataManager where we progressively add cameras to train with from multiple scenes and importance sample rays from pixels with a high loss. The extensible features that we show in this tutorial are the following:\n",
"- a DataManager that incrementally adds cameras\n",
"- a DataManager that uses multiple scenes\n",
"- a Model that is conditioned on the scene\n",
"- a Pipeline that modifies the DataManager's PixelSampler to sample more rays in regions with high loss"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"#### Model Components\n",
"\n",
"> The model renders values from 3D line segments defined by a RayBundle.\n",
"\n",
"We provide Models and Model Components (i.e., Fields and Field Modules) which can be strung together to create a custom model.\n",
"\n",
"<details closed>\n",
"<summary>Models</summary>\n",
"<br>\n",
"<li>Vanilla NeRF Model</li>\n",
"<li>Mip NeRF Model</li>\n",
"<li>Mip NeRF 360 Model</li>\n",
"<li>Instant NGP Model</li>\n",
"<li>NeRF-W Model</li>\n",
"<li>Semantic NeRF Model</li>\n",
"</details>\n",
"\n",
"<br>\n",
"\n",
"<details closed>\n",
"<summary>Model Components</summary>\n",
"<br>\n",
"Fields:\n",
"<li>Vanilla NeRF Field</li>\n",
"<li>Instant NGP Field</li>\n",
"<br>\n",
"Field Modules:\n",
"<li>Encodings</li>\n",
"<li>Embeddings</li>\n",
"<li>Field Heads</li>\n",
"<li>Field Losses</li>\n",
"<li>MLP</li>\n",
"<li>Spatial Distortions</li>\n",
"</details>\n",
"\n",
"#### DataManager Components\n",
"\n",
"> The DataManager provides a RayBundle (3D line segments with metadata needed to run the forward pass) and a dictionary of information needed at train-time to supervise the pipeline components.\n",
"\n",
"Similar to the Model components, we provide DataManager components that are strung together to generate RayBundle instances. A RayBundle contains everything needed to generate a ray (e.g., origin and direction, near and far planes) as well as any additional metadata that the Model requires (e.g., camera index, scene index, etc.).\n",
"\n",
"<details closed>\n",
"<summary>DataManager Components</summary>\n",
"<br>\n",
"Datasets Parsers:<br>\n",
" <em style=\"padding-left:2em\">Takes a data location and returns a DatsetInputs instance. A DatsetInputs will contain all data needed throughout the pipeline.</em>\n",
"<li>Vanilla Image DataParser</li>\n",
"<br>\n",
"Image Datasets:<br>\n",
" <em style=\"padding-left:2em\">Takes image filename information and can return images.</em>\n",
"<li>Vanilla Image Dataset</li>\n",
"<br>\n",
"Image Dataloader:<br>\n",
" <em style=\"padding-left:2em\">Takes an Image Dataset and samples images from it.</em>\n",
"<li>Vanilla Image Dataloader</li>\n",
"<li>Cache Image Dataloader</li>\n",
"<br>\n",
"Pixel Samplers:<br>\n",
" <em style=\"padding-left:2em\">Takes sampled images and returns pixel locations.</em>\n",
"<li>Vanilla Pixel Sampler</li>\n",
"<br>\n",
"Ray Generators:<br>\n",
" <em style=\"padding-left:2em\">Takes cameras and pixel locations and returns RayBundle objects.</em>\n",
"<br>\n",
"\n",
"Colliders:<br>\n",
" <em style=\"padding-left:2em\">Takes RayBundles and clips them according .</em>\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"# COLLAPSED\n",
"%load_ext autoreload\n",
"%autoreload 2\n",
"\n",
"from __future__ import annotations\n",
"\n",
"from typing import Dict, Optional, Tuple, List\n",
"import torch\n",
"\n",
"from nerfstudio.utils.config import DataManagerConfig, ModelConfig, PipelineConfig, ViewerConfig\n",
"from nerfstudio.data.base import DataManager\n",
"from nerfstudio.data.structs import DatasetInputs\n",
"from nerfstudio.models.base import Model\n",
"from nerfstudio.pipelines.base import Pipeline\n",
"from nerfstudio.models.instant_ngp import NGPModel\n",
"from nerfstudio.cameras.rays import RayBundle"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"class CustomDataManagerConfig(DataManagerConfig):\n",
" _target = \"CustomDataManager\"\n",
" scene_names: List[str] = [\"chair\", \"drums\", \"fern\", \"lego\"]\n",
" steps_per_add_cameras: int = 100 # every X iterations, add more cameras\n",
" num_cameras_per_add: int = 10 # number of cameras to add\n",
"\n",
"\n",
"class CustomDataManager(DataManager):\n",
" \"\"\"A custom dataloader that incrementally adds cameras and has multiple scenes.\"\"\"\n",
"\n",
" config: CustomDataManagerConfig\n",
"\n",
" def populate_train_modules(self):\n",
" \"\"\"Populate the train dataloader modules.\"\"\"\n",
" self.scene_name_to_image_sampler = {}\n",
" self.scene_name_to_collider = {} # TODO(ethan): move colliders to the dataloader\n",
" for scene_name in self.config.scene_names:\n",
" dataset_inputs: DatasetInputs = None # TODO\n",
" image_dataset = ImageDataset(dataset_inputs)\n",
" image_sampler = CacheImageDataloader(image_dataset=image_dataset) # a torch dataloader\n",
" self.scene_name_to_image_sampler[scene_name] = image_sampler\n",
" self.train_pixel_sampler = PixelSampler\n",
" self.train_ray_generator = RayGenerator # nn.Module\n",
"\n",
" def populate_eval_modules(self):\n",
" \"\"\"Populate the eval dataloader modules.\"\"\"\n",
" # we'll mostly rely on the train dataloader modules\n",
" self.eval_pixel_sampler = EvalPixelSampler # this will sample entire images\n",
"\n",
" def sample_images(self) -> List[Image]:\n",
" images = []\n",
" for scene_name in self.scene_name_to_image_sampler.keys():\n",
" image_sampler = self.scene_name_to_image_sampler[scene_name]\n",
" x = image_sampler.forward() # note that empty forward calls next()\n",
" images.append(x)\n",
" return images\n",
"\n",
" def next_train(self, step: int) -> Tuple[RayBundle, Dict]:\n",
" \"\"\"Get the next batch of training data by stringing together the train modules.\"\"\"\n",
" # grab some images from each scene\n",
" images = self.sample_images()\n",
" pixels = self.train_pixel_sampler.forward(images)\n",
" ray_bundle = self.train_ray_generator(pixels)\n",
" # shape should be (H, W, :)\n",
" ray_bundle.metadata[\"scene_indices\"] = None # TODO(ethan): populate this with a tensor of scene indices\n",
" dict_ = {}\n",
" return ray_bundle, dict_\n",
"\n",
" def next_eval(self, step: int) -> Tuple[RayBundle, Dict]:\n",
" \"\"\"Get the next batch of eval data by stringing together the eval modules.\"\"\"\n",
" # grab some images from each scene\n",
" images = self.sample_images()\n",
" pixels = self.train_pixel_sampler.forward(images)\n",
" ray_bundle = self.eval_ray_generator(pixels) # notice this is eval and not train\n",
" # shape should be (H, W, :)\n",
" ray_bundle.metadata[\"scene_indices\"] = None # TODO(ethan): populate this with a tensor of scene indices\n",
" dict_ = {}\n",
" return ray_bundle, dict_\n",
"\n",
"\n",
"class CustomNGPModelConfig(ModelConfig):\n",
" _target = \"CustomNGPModel\"\n",
" coarse_field: str = \"temp\"\n",
" fine_field: str = \"temp2\"\n",
"\n",
"\n",
"class CustomNGPModel(Model):\n",
" \"\"\"An instant ngp model modified slightly to output semantics.\"\"\"\n",
"\n",
" config: SceneConditionNGPModelConfig\n",
"\n",
" def populate_modules(self):\n",
" self.ngp_model = NGPModel(coarse_field=self.config.coarse_field, fine_field=self.config.fine_field)\n",
"\n",
" def get_outputs(self, ray_bundle: RayBundle) -> Dict[str, torch.Tensor]:\n",
" # TODO(ethan): pass in batch from the forward function\n",
" # TODO(ethan): rename batch to something else\n",
" outputs = self.ngp_model.forward(ray_bundle)\n",
" outputs[\"semantics\"] = torch.rand_like(outputs[\"rgb\"])\n",
" return outputs\n",
"\n",
" def get_loss_dict(self):\n",
" return {}\n",
"\n",
"\n",
"class CustomPipeline(Pipeline):\n",
" \"\"\"The Instant NGP pipeline.\"\"\"\n",
"\n",
" config: PipelineConfig\n",
"\n",
" def train_step(self, step: int):\n",
" ray_bundle, batch = self.dataloader.next_train(step=step)\n",
" # TODO: maybe run a CNN on the data before passing into model\n",
" model_outputs, loss_dict, metrics_dict = self.model(ray_bundle, batch)\n",
" # TODO: update pixel sampler state with loss map to show the flexibilty of our Pipeline\n",
" self.dataloader.pixel_sampler.update_loss_map()\n",
" return model_outputs, loss_dict, metrics_dict\n",
"\n",
" def eval_step(self):\n",
" ray_bundle, batch = self.dataloader.next_eval(step=step)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"dataloader = DataManager()\n",
"model = SceneConditionNGPModel()\n",
"pipeline = CustomPipeline.from_dataloader_and_model(dataloader=dataloader, model=model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"# Creating pipelines from a config\n",
"\n",
"Now we show how to create a pipeline from a config, which has the following form:\n",
"\n",
"```python\n",
"@dataclass\n",
"class PipelineConfig:\n",
" \"\"\"Configuration for pipeline instantiation\"\"\"\n",
"\n",
" _target: ClassVar[Type] = Pipeline\n",
" dataloader: DataloaderConfig = MISSING\n",
" model: ModelConfig = MISSING\n",
"```\n",
"\n",
"See `nerfstudio/utils/config.py` for more details. In this example, we will simply load from an existing configuration from `configs/graph_instant_ngp.yaml`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import pprint\n",
"import hydra\n",
"\n",
"hydra.core.global_hydra.GlobalHydra.instance().clear()\n",
"from hydra import compose, initialize\n",
"\n",
"initialize(version_base=\"1.2\", config_path=\"../configs/\")\n",
"config_name = \"graph_instant_ngp.yaml\"\n",
"config = compose(config_name)\n",
"pipeline_config = config.pipeline\n",
"pprint.pprint(pipeline_config)\n",
"print(\"----------------------------------------------------\")\n",
"\n",
"# from nerfstudio.pipelines.base import setup_pipeline\n",
"# pipeline = setup_pipeline(pipeline_config, device=\"cuda\")\n",
"pipeline = pipeline_config.setup()"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"pipeline.get_train_loss_dict()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.8.13 ('nerfactory')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
},
"vscode": {
"interpreter": {
"hash": "100252972d4ee5286ab455ef1c778abde05b4affca9d1be78633d3797abf7e8c"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}