tink/nerfstudio
4
0 Fork 0
mirror of https://github.com/nerfstudio-project/nerfstudio.git synced 2026-11-15 12:23:46 +08:00
Code Activity

Reorganize documentation (#631)

Browse Source 
restructure_docs
This commit is contained in:
Matthew Tancik authored and GitHub committed 2022-10-03 12:59:46 -07:00
1 parent 5a76b955cd
commit d7c34ca2bd
68 files changed
+87 -148
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/developer_guides/benchmarking.md
+76
View File
@@ -0,0 +1,76 @@
# Benchmarking workflow
We make it easy to benchmark your new NeRF against the standard Blender dataset.
## Launching training on Blender dataset
To start, you will need to train your NeRF on each of the blender objects.
To launch training jobs automatically on each of these items, you can call:
```bash
./scripts/benchmarking/launch_train_blender.sh -c <config-name> -g <list-of-gpus>
```
Simply replace the arguments in brackets with the correct arguments.
- `-c`: name of the config you created for your new graph. For instance, if you have created the config `configs/graph_instant_ngp.yaml`, `<config-name>` will be `graph_instant_ngp`.
- `-g`: specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs.
A full example would be:
```bash
# with -g specifying gpus
./scripts/benchmarking/launch_train_blender.sh -c graph_instant_ngp -g 0 1 2 3
# without -g automatically finds available gpus
./scripts/benchmarking/launch_train_blender.sh -c graph_instant_ngp
```
The script will automatically launch training on all of the items and save the checkpoints in an output directory with the current date and timestamp.
## Evaluating trained Blender models
Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/launch_eval_blender.sh`.
Say we ran a benchmark on 08-10-2022 for instant_ngp. By default, the train script will save the benchmarks in the following format:
```
outputs
└───blender_chair_08-10-2022
| └───instant_ngp
| └───2022-08-10_172517
| └───train.log
| ...
└───blender_drums_08-10-2022
| └───instant_ngp
| └───2022-08-10_172517
| └───train.log
| ...
...
```
If we wanted to run the benchmark on all the blender data for the above example, we would run:
```bash
./scripts/benchmarking/launch_eval_blender.sh -c graph_instant_ngp -o ./outputs/ -m 08 -d 10 -y 2022 -s 172517 -g 4 5 6 7
```
The flags used in the benchmarking script are defined as follows:
- `-c`: config name (e.g. `graph_instant_ngp`). This should be the same as what was passed in for -c in the train script.
- `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the hydra base dir.
- `-m`: month of benchmark (e.g. `08`) of format 02d.
- `-d`: date of benchmark (e.g. `10`) of format 02d.
- `-y`: year of benchmark (e.g. `2022`).
- `-s`: seconds timestamp of benchmark (e.g. `172517`) of format 06d.
- `-g`: specifies the gpus to use and if not specified (no -g flag), will automaticaly search for available gpus.
The script will simultaneously run the benchmarking across all the objects in the blender dataset and calculates the PSNR/FPS/other stats. The results are saved as .json files in the `-o` directory with the following format:
```
outputs
└───instant_ngp
| └───blender_chair_08-10-2022_172517.json
| | blender_ficus_08-10-2022_172517.json
| | ...
```
docs/developer_guides/config.md
+1
View File
@@ -0,0 +1 @@
# Customizable Configs
docs/developer_guides/logging_profiling.md
+117
View File
@@ -0,0 +1,117 @@
# Loggers and Profilers
## Logging support
We provide integration with multiple logging interfaces to log images and statistics during training.
All of these loggers are configurable via the config:
```yaml
# e.g. configs/graphs_default.yaml
logging:
steps_per_log: 10 # how often to log losses/images
max_buffer_size: 20 # defines how many steps to average over (if reporting averages)
writer: # definition of all writers you want to use
TensorboardWriter:
log_dir: './' # must specify the output path for all writers
LocalWriter:
log_dir: './'
stats_to_track: [ITER_TRAIN_TIME, TRAIN_RAYS_PER_SEC, CURR_TEST_PSNR]
max_log_size: 10
```
Currently, we provide support for the following loggers, which can be added to the `logging.writer` section of the config.
#### Tensorboard
To log training stats and images with [tensorboard](https://www.tensorflow.org/tensorboard):
```
TensorboardWriter:
log_dir: "./" # the output path for which the event files are written
```
#### Weights and Biases
To log training stats and images with [wandb](https://wandb.ai/site):
```
WandbWriter:
log_dir: "./" # the output path for which the run files are written
```
#### Local writer
```
LocalWriter:
log_dir: "./" # if images are saved, where the images would be saved
stats_to_track:
[ITER_TRAIN_TIME, TRAIN_RAYS_PER_SEC, CURR_TEST_PSNR]
max_log_size: 10
```
The `LocalWriter` simply outputs to the terminal. However, you can specify additional parameters to customize the logging.
- `stats_to_track`: all the stats that you want to print to the terminal (see list under `EventName` in `utils/writer.py`). You can add or remove any of the defined enums.
- `max_log_size`: how much content to print onto the screen (e.g. only print 10 lines on the screen at a time). If 0, will print everything without deleting any previous lines.
## Code profiling support
We provide built-in performance profiling capabilities to make it easier for you to debug and assess the performance of your code.
#### In-house profiler
You can use our built-in profiler by enabling the profiler in the config:
```yaml
# e.g. configs/graphs_default.yaml
logging:
enable_profiler: True
```
The profiler computes the average total time of execution for any function with the `@profiler.time_function` decorator.
For instance, if you wanted to profile the total time it takes to generate rays given pixel and camera indices via the `RayGenerator` class, you would want to evaluate its `forward()` function. In that class, you would need to add the decorator to the function.
```python
class RayGenerator(nn.Module):
...
@profiler.time_function # add the profiler decorator before the function
def forward(self, ray_indices: TensorType["num_rays", 3]) -> RayBundle:
# implementation here
...
```
At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag.
Use this profiler if there are specific functions that you want to measure the times for.
#### Profiling with PySpy
If you want to profile the entire codebase, we provide functionality to generate flame graphs or get real-time reports using [PySpy](https://github.com/benfred/py-spy).
Install PySpy
```bash
pip install py-spy
```
To perform the profiling, you can either specify that you want to generate a flame graph or generate a live-view of the profiler.
```bash
## for flame graph
./scripts/debugging/profile.sh -t flame -o flame.svg -p <python_command>
## for live view of functions
./scripts/debugging/profile.sh -t top -p <python_command>
```
In both cases, `<python_command>` can be replaced with any command you would normally run in the terminal. For instance, you can replace `<python_function>` with `scripts/train.py data/dataset=blender_lego`. Then the full command would be e.g.:
```bash
./scripts/debugging/profile.sh -t flame -o flame.svg -p scripts/train.py data/dataset=blender_lego
```
docs/developer_guides/pipelines/creating_new_methods.ipynb
+352
View File
@@ -0,0 +1,352 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"# Creating Custom Methods\n",
"\n",
"> Our goal is to enable nerfstudio users to implement an entire NeRF method by only editing one file.\n",
"\n",
"Here we explain how to create custom pipelines with nerfstudio. Pipelines are composed of two components, namely a DataManager and a Model. The pipeline is responsible for routing RayBundle instances returned from the DataManager into the Model.\n",
"\n",
"A NeRF method is implemented as a Pipeline. Users can create a Pipeline with a DataManager and Model. We provide default implementations for these components prefixed with the name Vanilla (e.g., `VanillaPipeline`, `VanillaDataManager` which inherit from the `Pipeline` and `DataManager` class, respectively) and specific Models (e.g., `InstantNGPModel` or `TensoRFModel` which inherit from the `Model` class). With most NeRF methods, `VanillaPipeline` and `VanillaDataManager` can be used and only the model needs to be changed. We provide many already implmented Models for people to get started with 3D reconstruction. However, in this tutorial we will demonstrate how to customize each of the Pipeline, DataManager, and Model with an example.\n",
"\n",
"We'll show how to make an incremental DataManager where we progressively add cameras to train with from multiple scenes and importance sample rays from pixels with a high loss. The extensible features that we show in this tutorial are the following:\n",
"- a DataManager that incrementally adds cameras\n",
"- a DataManager that uses multiple scenes\n",
"- a Model that is conditioned on the scene\n",
"- a Pipeline that modifies the DataManager's PixelSampler to sample more rays in regions with high loss"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"#### Model Components\n",
"\n",
"> The model renders values from 3D line segments defined by a RayBundle.\n",
"\n",
"We provide Models and Model Components (i.e., Fields and Field Modules) which can be strung together to create a custom model.\n",
"\n",
"<details closed>\n",
"<summary>Models</summary>\n",
"<br>\n",
"<li>Vanilla NeRF Model</li>\n",
"<li>Mip NeRF Model</li>\n",
"<li>Mip NeRF 360 Model</li>\n",
"<li>Instant NGP Model</li>\n",
"<li>NeRF-W Model</li>\n",
"<li>Semantic NeRF Model</li>\n",
"</details>\n",
"\n",
"<br>\n",
"\n",
"<details closed>\n",
"<summary>Model Components</summary>\n",
"<br>\n",
"Fields:\n",
"<li>Vanilla NeRF Field</li>\n",
"<li>Instant NGP Field</li>\n",
"<br>\n",
"Field Modules:\n",
"<li>Encodings</li>\n",
"<li>Embeddings</li>\n",
"<li>Field Heads</li>\n",
"<li>Field Losses</li>\n",
"<li>MLP</li>\n",
"<li>Spatial Distortions</li>\n",
"</details>\n",
"\n",
"#### DataManager Components\n",
"\n",
"> The DataManager provides a RayBundle (3D line segments with metadata needed to run the forward pass) and a dictionary of information needed at train-time to supervise the pipeline components.\n",
"\n",
"Similar to the Model components, we provide DataManager components that are strung together to generate RayBundle instances. A RayBundle contains everything needed to generate a ray (e.g., origin and direction, near and far planes) as well as any additional metadata that the Model requires (e.g., camera index, scene index, etc.).\n",
"\n",
"<details closed>\n",
"<summary>DataManager Components</summary>\n",
"<br>\n",
"Datasets Parsers:<br>\n",
" <em style=\"padding-left:2em\">Takes a data location and returns a DatsetInputs instance. A DatsetInputs will contain all data needed throughout the pipeline.</em>\n",
"<li>Vanilla Image DataParser</li>\n",
"<br>\n",
"Image Datasets:<br>\n",
" <em style=\"padding-left:2em\">Takes image filename information and can return images.</em>\n",
"<li>Vanilla Image Dataset</li>\n",
"<br>\n",
"Image Dataloader:<br>\n",
" <em style=\"padding-left:2em\">Takes an Image Dataset and samples images from it.</em>\n",
"<li>Vanilla Image Dataloader</li>\n",
"<li>Cache Image Dataloader</li>\n",
"<br>\n",
"Pixel Samplers:<br>\n",
" <em style=\"padding-left:2em\">Takes sampled images and returns pixel locations.</em>\n",
"<li>Vanilla Pixel Sampler</li>\n",
"<br>\n",
"Ray Generators:<br>\n",
" <em style=\"padding-left:2em\">Takes cameras and pixel locations and returns RayBundle objects.</em>\n",
"<br>\n",
"\n",
"Colliders:<br>\n",
" <em style=\"padding-left:2em\">Takes RayBundles and clips them according .</em>\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"# COLLAPSED\n",
"%load_ext autoreload\n",
"%autoreload 2\n",
"\n",
"from __future__ import annotations\n",
"\n",
"from typing import Dict, Optional, Tuple, List\n",
"import torch\n",
"\n",
"from nerfstudio.utils.config import DataManagerConfig, ModelConfig, PipelineConfig, ViewerConfig\n",
"from nerfstudio.data.base import DataManager\n",
"from nerfstudio.data.structs import DatasetInputs\n",
"from nerfstudio.models.base import Model\n",
"from nerfstudio.pipelines.base_pipeline import Pipeline\n",
"from nerfstudio.models.instant_ngp import NGPModel\n",
"from nerfstudio.cameras.rays import RayBundle"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"class CustomDataManagerConfig(DataManagerConfig):\n",
" _target = \"CustomDataManager\"\n",
" scene_names: List[str] = [\"chair\", \"drums\", \"fern\", \"lego\"]\n",
" steps_per_add_cameras: int = 100 # every X iterations, add more cameras\n",
" num_cameras_per_add: int = 10 # number of cameras to add\n",
"\n",
"\n",
"class CustomDataManager(DataManager):\n",
" \"\"\"A custom dataloader that incrementally adds cameras and has multiple scenes.\"\"\"\n",
"\n",
" config: CustomDataManagerConfig\n",
"\n",
" def populate_train_modules(self):\n",
" \"\"\"Populate the train dataloader modules.\"\"\"\n",
" self.scene_name_to_image_sampler = {}\n",
" self.scene_name_to_collider = {} # TODO(ethan): move colliders to the dataloader\n",
" for scene_name in self.config.scene_names:\n",
" dataset_inputs: DatasetInputs = None # TODO\n",
" image_dataset = ImageDataset(dataset_inputs)\n",
" image_sampler = CacheImageDataloader(image_dataset=image_dataset) # a torch dataloader\n",
" self.scene_name_to_image_sampler[scene_name] = image_sampler\n",
" self.train_pixel_sampler = PixelSampler\n",
" self.train_ray_generator = RayGenerator # nn.Module\n",
"\n",
" def populate_eval_modules(self):\n",
" \"\"\"Populate the eval dataloader modules.\"\"\"\n",
" # we'll mostly rely on the train dataloader modules\n",
" self.eval_pixel_sampler = EvalPixelSampler # this will sample entire images\n",
"\n",
" def sample_images(self) -> List[Image]:\n",
" images = []\n",
" for scene_name in self.scene_name_to_image_sampler.keys():\n",
" image_sampler = self.scene_name_to_image_sampler[scene_name]\n",
" x = image_sampler.forward() # note that empty forward calls next()\n",
" images.append(x)\n",
" return images\n",
"\n",
" def next_train(self, step: int) -> Tuple[RayBundle, Dict]:\n",
" \"\"\"Get the next batch of training data by stringing together the train modules.\"\"\"\n",
" # grab some images from each scene\n",
" images = self.sample_images()\n",
" pixels = self.train_pixel_sampler.forward(images)\n",
" ray_bundle = self.train_ray_generator(pixels)\n",
" # shape should be (H, W, :)\n",
" ray_bundle.metadata[\"scene_indices\"] = None # TODO(ethan): populate this with a tensor of scene indices\n",
" dict_ = {}\n",
" return ray_bundle, dict_\n",
"\n",
" def next_eval(self, step: int) -> Tuple[RayBundle, Dict]:\n",
" \"\"\"Get the next batch of eval data by stringing together the eval modules.\"\"\"\n",
" # grab some images from each scene\n",
" images = self.sample_images()\n",
" pixels = self.train_pixel_sampler.forward(images)\n",
" ray_bundle = self.eval_ray_generator(pixels) # notice this is eval and not train\n",
" # shape should be (H, W, :)\n",
" ray_bundle.metadata[\"scene_indices\"] = None # TODO(ethan): populate this with a tensor of scene indices\n",
" dict_ = {}\n",
" return ray_bundle, dict_\n",
"\n",
"\n",
"class CustomNGPModelConfig(ModelConfig):\n",
" _target = \"CustomNGPModel\"\n",
" coarse_field: str = \"temp\"\n",
" fine_field: str = \"temp2\"\n",
"\n",
"\n",
"class CustomNGPModel(Model):\n",
" \"\"\"An instant ngp model modified slightly to output semantics.\"\"\"\n",
"\n",
" config: SceneConditionNGPModelConfig\n",
"\n",
" def populate_modules(self):\n",
" self.ngp_model = NGPModel(coarse_field=self.config.coarse_field, fine_field=self.config.fine_field)\n",
"\n",
" def get_outputs(self, ray_bundle: RayBundle) -> Dict[str, torch.Tensor]:\n",
" # TODO(ethan): pass in batch from the forward function\n",
" # TODO(ethan): rename batch to something else\n",
" outputs = self.ngp_model.forward(ray_bundle)\n",
" outputs[\"semantics\"] = torch.rand_like(outputs[\"rgb\"])\n",
" return outputs\n",
"\n",
" def get_loss_dict(self):\n",
" return {}\n",
"\n",
"\n",
"class CustomPipeline(Pipeline):\n",
" \"\"\"The Instant NGP pipeline.\"\"\"\n",
"\n",
" config: PipelineConfig\n",
"\n",
" def train_step(self, step: int):\n",
" ray_bundle, batch = self.dataloader.next_train(step=step)\n",
" # TODO: maybe run a CNN on the data before passing into model\n",
" model_outputs, loss_dict, metrics_dict = self.model(ray_bundle, batch)\n",
" # TODO: update pixel sampler state with loss map to show the flexibilty of our Pipeline\n",
" self.dataloader.pixel_sampler.update_loss_map()\n",
" return model_outputs, loss_dict, metrics_dict\n",
"\n",
" def eval_step(self):\n",
" ray_bundle, batch = self.dataloader.next_eval(step=step)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"dataloader = DataManager()\n",
"model = SceneConditionNGPModel()\n",
"pipeline = CustomPipeline.from_dataloader_and_model(dataloader=dataloader, model=model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"Now we show how to create a pipeline from a config, which has the following form:\n",
"\n",
"```python\n",
"@dataclass\n",
"class PipelineConfig:\n",
" \"\"\"Configuration for pipeline instantiation\"\"\"\n",
"\n",
" _target: ClassVar[Type] = Pipeline\n",
" dataloader: DataloaderConfig = MISSING\n",
" model: ModelConfig = MISSING\n",
"```\n",
"\n",
"See `nerfstudio/utils/config.py` for more details. In this example, we will simply load from an existing configuration from `configs/graph_instant_ngp.yaml`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import pprint\n",
"import hydra\n",
"\n",
"hydra.core.global_hydra.GlobalHydra.instance().clear()\n",
"from hydra import compose, initialize\n",
"\n",
"initialize(version_base=\"1.2\", config_path=\"../configs/\")\n",
"config_name = \"graph_instant_ngp.yaml\"\n",
"config = compose(config_name)\n",
"pipeline_config = config.pipeline\n",
"pprint.pprint(pipeline_config)\n",
"print(\"----------------------------------------------------\")\n",
"\n",
"# from nerfstudio.pipelines.base_pipeline import setup_pipeline\n",
"# pipeline = setup_pipeline(pipeline_config, device=\"cuda\")\n",
"pipeline = pipeline_config.setup()"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"pipeline.get_train_loss_dict()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.8.13 ('nerfactory')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
},
"vscode": {
"interpreter": {
"hash": "b7b86c17c0dfd6b313529972d424f2e075cc198a46cda005757126afe8133c7b"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/developer_guides/pipelines/datamanagers.ipynb
+34
View File
@@ -0,0 +1,34 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# DataManagers\n",
"\n",
"```{image} imgs/datamanagers.png\n",
":align: center\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.8.13 ('nerfactory')",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.8.13"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "b7b86c17c0dfd6b313529972d424f2e075cc198a46cda005757126afe8133c7b"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/developer_guides/pipelines/dataparsers.ipynb
+158
View File
@@ -0,0 +1,158 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# DataParsers\n",
"\n",
"```{image} imgs/dataparsers.png\n",
":align: center\n",
"```\n",
"\n",
"## What is a DataParser?\n",
"\n",
"The dataparser returns `DataparserOutputs`, which puts all the various datasets into a common format.\n",
"\n",
"```python\n",
"@dataclass\n",
"class DataparserOutputs:\n",
" \"\"\"Dataparser outputs for the image dataset and the ray generator.\"\"\"\n",
"\n",
" image_filenames: List[Path]\n",
" \"\"\"Filenames for the images.\"\"\"\n",
" cameras: Cameras\n",
" \"\"\"Camera object storing collection of camera information in dataset\"\"\"\n",
" alpha_color: Optional[TensorType[3]] = None\n",
" \"\"\"color of dataset background\"\"\"\n",
" scene_box: SceneBox = SceneBox()\n",
" \"\"\"scene box of dataset. Could be used to bound the scene, or to provide the scene scale depending on model.\"\"\"\n",
" ...\n",
"\n",
"@dataclass\n",
"class DataParser:\n",
"\n",
" @abstractmethod\n",
" def _generate_datapaser_outputs(self, split: str = \"train\") -> DataparserOutputs:\n",
" \"\"\"Abstract method that returns the dataparser outputs for the given split.\n",
"\n",
" Args:\n",
" split: Which dataset split to generate data for (train/eval/test).\n",
"\n",
" Returns:\n",
" DataparserOutputs for the specified dataset and split.\n",
" \"\"\"\n",
"```\n",
"\n",
"## Example\n",
"\n",
"Here is an example where we implement a DataParser for our Nerfstudio data format.\n",
"\n",
"```python\n",
"@dataclass\n",
"class NerfstudioDataParserConfig(DataParserConfig):\n",
" \"\"\"Nerfstudio dataset config\"\"\"\n",
"\n",
" _target: Type = field(default_factory=lambda: Nerfstudio)\n",
" \"\"\"Target class to instantiate.\"\"\"\n",
" data_directory: Path = Path(\"data/nerfstudio/poster\")\n",
" \"\"\"Directory specifying location of data.\"\"\"\n",
" scale_factor: float = 1.0\n",
" \"\"\"How much to scale the camera origins by.\"\"\"\n",
" downscale_factor: int = 1\n",
" \"\"\"How much to downscale images.\"\"\"\n",
" scene_scale: float = 1.0\n",
" \"\"\"How much to scale the region of interest by.\"\"\"\n",
" orientation_method: Literal[\"pca\", \"up\"] = \"up\"\n",
" \"\"\"The method to use for orientation.\"\"\"\n",
" train_split_percentage: float = 0.9\n",
" \"\"\"The percent of images to use for training. The remaining images are for eval.\n",
" \"\"\"\n",
"\n",
"@dataclass\n",
"class Nerfstudio(DataParser):\n",
" \"\"\"Nerfstudio Dataset\"\"\"\n",
"\n",
" config: NerfstudioDataParserConfig\n",
"\n",
" def _generate_datapaser_outputs(self, split=\"train\") -> DataParserOutputs:\n",
" \"\"\"Returns DataParserOutputs.\"\"\"\n",
"```\n",
"\n",
"## Our Implementations\n",
"\n",
"###### Nerfstudio\n",
"\n",
"This is our custom dataparser.\n",
"\n",
"```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/nerfstudio_dataparser.py\n",
":color: primary\n",
":outline:\n",
"See the code!\n",
"```\n",
"\n",
"###### Blender\n",
"\n",
"We support the synthetic Blender dataset from the original NeRF paper.\n",
"\n",
"```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/blender_dataparser.py\n",
":color: primary\n",
":outline:\n",
"See the code!\n",
"```\n",
"\n",
"###### Instant NGP\n",
"\n",
"This supports the Instant NGP dataset.\n",
"\n",
"```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/instant_ngp_dataparser.py\n",
":color: primary\n",
":outline:\n",
"See the code!\n",
"```\n",
"\n",
"###### MipNeRF 360\n",
"\n",
"```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/mipnerf_dataparser.py\n",
":color: primary\n",
":outline:\n",
"See the code!\n",
"```\n",
"\n",
"###### Record3D\n",
"\n",
"This dataparser can use recorded data from a >= iPhone 12 Pro using the Record3D app. Record a video and export with the `EXR + JPG sequence` format. Unzip export and `rgb` folder before training.\n",
"\n",
"```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/record3d_dataparser.py\n",
":color: primary\n",
":outline:\n",
"See the code!\n",
"```\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.8.12 ('nerfactory')",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.8.12"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "34c28001ff35fb390494047002768a8182dcf55b1b11415165e62ea61557ab83"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/developer_guides/pipelines/imgs/datamanagers.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 122 KiB

docs/developer_guides/pipelines/imgs/dataparsers.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 122 KiB

docs/developer_guides/pipelines/imgs/models.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 119 KiB

docs/developer_guides/pipelines/imgs/pipelines.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 122 KiB

docs/developer_guides/pipelines/imgs/pipelines_overview.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 126 KiB

docs/developer_guides/pipelines/index.rst
+19
View File
@@ -0,0 +1,19 @@
Pipelines Overview
-------------------------
Here we show you how to create a custom pipeline.
# TODO: show a figure of what a pipeline is here
.. image:: imgs/pipelines_overview.png
:width: 400
:align: center
.. toctree::
:maxdepth: 1
dataparsers.ipynb
datamanagers.ipynb
models.ipynb
pipelines.ipynb
creating_new_methods.ipynb
docs/developer_guides/pipelines/models.ipynb
+229
View File
@@ -0,0 +1,229 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Models\n",
"\n",
"```{image} imgs/models.png\n",
":align: center\n",
"```\n",
"\n",
"In this quick tour, we will walk you through the core of training and building any NeRFs with nerfstudio.\n",
"\n",
"The entry point for training starts at `scripts/train.py`, which spawns instances of our `Trainer()` class (in `engine/trainer.py`). The `Trainer()` is responsible for setting up the datasets and NeRF graph depending on the config specified. It will then run the usual train/val routine for a config-specified number of iterations. If you are planning on using our codebase to build a new NeRF method or to use an existing implementation, we've abstracted away the training routine in these two files and chances are you will not need to think of them again.\n",
"\n",
"### Graphs, Modules, and Fields\n",
"\n",
"If you are looking to implemnet a new NeRF method or extend an existing one, you may need to set up a new graph (in `graphs/`).\n",
"\n",
"All NeRF graph definitions can be found in `graphs/`. The graph encapsulates all the modules in the NeRF method- from the MLP's to the samplers to the losses and optimizers. All graphs are composed of modules and fields.\n",
"\n",
"### Modules\n",
"\n",
"We can decompose all graphs into respective modules. For instance, we have `optimizers` which contains modules related to the loss, `renderers` which contains all rendering modules, {`graph/modules/`}, which includes samplers and ray generators. Additionally, we have `field_modules`, which will be discussed below.\n",
"In general, modules can be thought of as individual component parts of the graph that can be swapped in/out. We provide most of the standard modules, but if you are creating a new NeRF graph, you may have to define new modules.\n",
"\n",
"### Fields\n",
"\n",
"`fields` represents the \"space\", aka. the radiance field of the NeRF. Here, we define the field as the part of the network that takes in point samples and any other conditioning, and outputs any of the `FieldHeadNames` (`nerf/field_modules/field_heads.py`).\n",
"\n",
"All fields are composed of modules continained within the `fields/` directory (e.g. `fields/modules/`). We can think of field modules as modules that actually define or interact with the field.\n",
"\n",
"To build a NeRF field, we therefore follow these steps:\n",
"\n",
"1. define any relevant field modules\n",
"2. create a new class that extends the `fields` base class and use these field modules to compose the full field\n",
"3. implement the abstract functions `get_density()` and `get_outputs()`."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"ename": "SyntaxError",
"evalue": "invalid syntax (383175768.py, line 4)",
"output_type": "error",
"traceback": [
"\u001b[0;36m Input \u001b[0;32mIn [1]\u001b[0;36m\u001b[0m\n\u001b[0;31m def __init__(self, ...) -> None:\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
]
}
],
"source": [
"class NeRFField(Field):\n",
" \"\"\"NeRF Field\"\"\"\n",
"\n",
" def __init__(self, ...) -> None:\n",
" \"\"\"Create boilerplate NeRF field.\"\"\"\n",
"\n",
" super().__init__()\n",
"\n",
" # In the init method, we are instantiating all relevant field modules\n",
" self.position_encoding = position_encoding\n",
" self.direction_encoding = direction_encoding\n",
"\n",
" self.mlp_base = MLP(\n",
" in_dim=self.position_encoding.get_out_dim(),\n",
" num_layers=base_mlp_num_layers,\n",
" layer_width=base_mlp_layer_width,\n",
" skip_connections=skip_connections,\n",
" )\n",
" ...\n",
"\n",
" def get_density(self, point_samples: PointSamples):\n",
" \"\"\"\n",
" Computes and returns the densities.\n",
" \"\"\"\n",
"\n",
" def get_outputs(\n",
" self, point_samples: PointSamples, density_embedding: Optional[TensorType] = None\n",
" ) -> Dict[FieldHeadNames, TensorType]:\n",
" \"\"\"\n",
" Computes and returns the outputs.\n",
" \"\"\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Putting it together\n",
"\n",
"Now that we've gone over what modules and fields are, we are ready to create the graph!\n",
"\n",
"To implement the vanilla NeRF, we create a new class that inherits the abstract Graph class. Similar to composing a new field, we can pull from all pre-defined fields and modules and use them to compose a new Graph.\n",
"We will then need to define the following abstract methods as seen in the skeleton code below. See also `graphs/vanilla_nerf.py` for the full implementation."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"ename": "NameError",
"evalue": "name 'Graph' is not defined",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/shared/ethanweber/nerfactory/docs/tutorials/pipelines/creating_models.ipynb Cell 4\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> <a href='vscode-notebook-cell://ssh-remote%2Bbirman/shared/ethanweber/nerfactory/docs/tutorials/pipelines/creating_models.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0'>1</a>\u001b[0m \u001b[39mclass\u001b[39;00m \u001b[39mNeRFGraph\u001b[39;00m(Graph):\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2Bbirman/shared/ethanweber/nerfactory/docs/tutorials/pipelines/creating_models.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=1'>2</a>\u001b[0m \u001b[39m\"\"\"Vanilla NeRF graph\"\"\"\u001b[39;00m\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2Bbirman/shared/ethanweber/nerfactory/docs/tutorials/pipelines/creating_models.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=3'>4</a>\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39m__init__\u001b[39m(\u001b[39mself\u001b[39m, intrinsics\u001b[39m=\u001b[39m\u001b[39mNone\u001b[39;00m, camera_to_world\u001b[39m=\u001b[39m\u001b[39mNone\u001b[39;00m, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m \u001b[39mNone\u001b[39;00m:\n",
"\u001b[0;31mNameError\u001b[0m: name 'Graph' is not defined"
]
}
],
"source": [
"class NeRFGraph(Graph):\n",
" \"\"\"Vanilla NeRF graph\"\"\"\n",
"\n",
" def __init__(self, intrinsics=None, camera_to_world=None, **kwargs) -> None:\n",
" super().__init__(intrinsics=intrinsics, camera_to_world=camera_to_world, **kwargs)\n",
"\n",
" def populate_fields(self):\n",
" \"\"\"\n",
" Set all field related modules here\n",
" \"\"\"\n",
"\n",
" def populate_misc_modules(self):\n",
" \"\"\"\n",
" Set all remaining modules here including: samplers, renderers, losses, and metrics\n",
" \"\"\"\n",
"\n",
" def get_param_groups(self) -> Dict[str, List[Parameter]]:\n",
" \"\"\"\n",
" Create a dictionary of parameters that are grouped according to different optimizers\n",
" \"\"\"\n",
"\n",
" def get_outputs(self, ray_bundle: RayBundle):\n",
" \"\"\"\n",
" Takes in a Ray Bundle and returns a dictionary of outputs.\n",
" \"\"\"\n",
"\n",
" def get_loss_dict(self, outputs, batch):\n",
" \"\"\"\n",
" Computes and returns the losses.\n",
" \"\"\"\n",
"\n",
" def log_test_image_outputs(self, image_idx, step, batch, outputs):\n",
" \"\"\"\n",
" Writes the test image outputs.\n",
" \"\"\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Creating the config\n",
"\n",
"Now that you have the graph and dataset all set up, you'll need to create the config that you pass into our run train routine. Our config system is powered by [Hydra](https://hydra.cc/). All Hydra and machine related arguments are stored in `configs/default_setup.yaml`, as well as the defaults list.\n",
"To set up the graph config, create a new yaml under `configs/`.\n",
"\n",
"```yaml\n",
"# configs/vanilla_nerf.yaml\n",
"\n",
"defaults:\n",
" - graph_default # inherit the basic yaml heirarchy\n",
" - _self_\n",
"\n",
"logging:\n",
" # <insert logging definitions here>\n",
"\n",
"trainer:\n",
" # <override configurations for loading/training/saving model here>\n",
"\n",
"experiment_name: blender_lego\n",
"method_name: vanilla_nerf\n",
"\n",
"data:\n",
" # <insert any dataset related overrides here>\n",
"\n",
"graph:\n",
" _target_: nerfstudio.graph.vanilla_nerf.NeRFGraph # set the target to the graph you defined\n",
" # <insert any graph related overrides here>\n",
"\n",
"optimizers:\n",
" # <insert any optimizer related overrides here>\n",
"```\n",
"\n",
"### Training and testing\n",
"\n",
"Once you have the config properly set up, you can begin training! To begin the training process, just pass in the name of your new config to the training script:\n",
"\n",
"```bash\n",
"python scripts/train.py --config-name vanilla_nerf\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.8.12 ('nerfactory')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "34c28001ff35fb390494047002768a8182dcf55b1b11415165e62ea61557ab83"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/developer_guides/pipelines/pipelines.ipynb
+39
View File
@@ -0,0 +1,39 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Pipelines\n",
"\n",
"```{image} imgs/pipelines.png\n",
":align: center\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.8.13 ('nerfactory')",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.8.13"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "b7b86c17c0dfd6b313529972d424f2e075cc198a46cda005757126afe8133c7b"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/developer_guides/viewer/faq.md
+21
View File
@@ -0,0 +1,21 @@
# FAQ
* [Engine node incompatible](engine-error)
(engine-error)=
#### Engine node incompatible
While running `yarn install`, you run into: `The engine "node" is incompatible with this module.`
**Solution**:
Install nvm with instructions at [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
```shell
nvm install 17.8.0
```
If you cannot install nvm, try ignoring the engines
```
yarn install --ignore-engines
```
docs/developer_guides/viewer/for_developers.md
+103
View File
@@ -0,0 +1,103 @@
# For developers
The tutorial below shows you how to host the viewer yourself and how you can use it to visualize training.
#### Connection between nerfstudio and the Bridge Server
In the center, we have the Bridge Server, which facilitates the connection between nerfstudio code and the Client App. This server runs on the same machine that you are using nerfstudio. It has a TCP Request/Reply (REQ/REP) connection that nerfstudio can connect to with the Viewer object (left). We use [ZeroMQ](https://zeromq.org/), an open-sourced messaging library, to implement this lightweight TCP connection. The Viewer class can send commands to the Bridge Server and receive replies. The Bridge Server will either dispatch commands to the Client App via a websocket or it will return information stored in the Bridge Server state.
#### Connection between the Bridge Server and the Client App
The connection between the Bridge Server and the Client App works with WebSockets and WebRTC.
- **WebSocket connection** - The WebSocket is used by the Bridge Server to dispatch commands coming from the nerfstudio TCP connection. Commands can be used for drawing primitives, for setting the transform of objects, for the setting various properties, and more.
- **WebRTC connection** - We use WebRTC to stream images being rendered from nerfstudio. The websocket connection if first used to establish the WebRTC connection. Then, the Client App constantly publishes camera pose information to the Bridge Server and stores the camera information (intrinsics and extrinsics). This information is then queried from the nerfstudio code, used to render an image with some Graph, and then the image is send over the TCP connection and dispatched via WebRTC to render the stream of images.
## Getting started
#### Running the Bridge Server
The viewer server runs on the same machine that you use for training. The training code will connect to the server with a lightweight TCP connection.
```
cd nerfstudio
# run the server on your machine
# this will run in the background
python scripts/run_viewer_bridge_server.py
It should print out something of the form:
"ZMQWebSocketBridge using zmq_url=tcp://127.0.0.1:6000 and websocket_port=8051"
```
#### Running the Client App
> We will host the viewer online in the future, but for now we have to run it locally.
```shell
cd nerfstudio/viewer/app
```
Install npm (to install yarn) and yarn
```shell
sudo apt-get install npm
npm install --global yarn
```
Install nvm and set the node version
Install nvm with [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
```shell
nvm install 17.8.0
```
Now running `node --version` in the shell should print "v17.8.0".
Install package.json dependencies and start the client viewer app
```shell
yarn install
yarn start
```
Forward port and open in your local browser. The URL takes the following form, `http://localhost:<forwarded_react_port>?localhost:<forwarded_backend_tcp_port>`. Notice the "?" character.
- http://localhost:4000?localhost:8051
Possibly edit the port in `app/.env.development` file with the following to adjust the PORT, for example.
```
BROWSER=none
FAST_REFRESH=false
HOST=0.0.0.0
PORT=4000
ESLINT_NO_DEV_ERRORS=true
```
#### Running the nerfstudio Code
You can now simply run a training job and visualize progress by enabling the viewer during training:
```bash
python scripts/train.py --config-name=graph_instant_ngp.yaml viewer.enable=true
```
- **Notebook demo** - See [Programming the viewer](viewer_notebook.ipynb) for an overview for how to interact with the viewer with the Viewer object from nerfstudio.
## Acknowledgements and references
We thank [Robin Deits](https://github.com/rdeits) and other contributors to the following repos, which we've started with and modified and extended for our use.
- [meshcat-python](https://github.com/rdeits/meshcat-python)
- [meshcat](https://github.com/rdeits/meshcat)
Here are other resources that we've used and/or have found helpful while building the viewer code.
- [ThreeJS](https://threejs.org/)
- [ReactJS](https://reactjs.org/)
- [WebRTC](https://webrtc.org/) - WebRTC is a framework for real-time communication that allows two peers to send video, audio, or general data to and from each other with low latency. We've adopted WebRTC to stream rendered images to our viewer.
```
```
docs/developer_guides/viewer/imgs/bulldozer.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 294 KiB

docs/developer_guides/viewer/imgs/nerfviewer.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 286 KiB

docs/developer_guides/viewer/imgs/viewer_figure.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 218 KiB

docs/developer_guides/viewer/index.rst
+12
View File
@@ -0,0 +1,12 @@
Real-time web viewer
====================
# TODO: high level summary
.. toctree::
:maxdepth: 1
viewer_quickstart.md
for_developers.md
viewer_notebook.ipynb
faq.md
docs/developer_guides/viewer/viewer_notebook.ipynb
+521
View File
@@ -0,0 +1,521 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "3c00934d-0f14-4c1a-a563-94d627b377e5",
"metadata": {},
"source": [
"# Programming the viewer\n",
"\n",
"The real-time web viewer enables communication between Python and the ReactJS Client Viewer Application. Here we provide a walkthrough for how to use the viewer to draw information as well as read information from the GUI.\n"
]
},
{
"cell_type": "code",
"execution_count": 72,
"id": "b15dfdc1-5054-40bc-9631-4c7426eecfd9",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The autoreload extension is already loaded. To reload it, use:\n",
" %reload_ext autoreload\n"
]
}
],
"source": [
"# HIDDEN\n",
"%load_ext autoreload\n",
"%autoreload 2\n",
"import pprint\n",
"import torch\n",
"import numpy as np\n",
"import torch\n",
"import mediapy as media\n",
"from nerfstudio.data.scene_box import SceneBox\n",
"from nerfstudio.cameras.cameras import PinholeCamera\n",
"import time"
]
},
{
"cell_type": "markdown",
"id": "a8709c84",
"metadata": {},
"source": [
"## Open the Nerfviewer client\n",
"\n",
"Open the Nerfviewer client by visiting http://nerfviewer.plenoptix.ai/ or running the code locally. *We refer to this document for running the code locally.* You should see a GUI like the following:\n",
"\n",
"```{image} imgs/nerfviewer.png\n",
":align: center\n",
":class: only-light\n",
":width: 600\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "052af8ce",
"metadata": {},
"source": [
"## Start the Bridge Server\n",
"\n",
"Run the following code in the repository home directory\n",
"\n",
"```\n",
"python scripts/run_viewer_bridge_server.py\n",
"```\n",
"\n",
"This will start the bridge server that both the Python code and Client Viewer App will connect to.\n",
"\n",
"Verify that the Client Viewer App is connected."
]
},
{
"cell_type": "code",
"execution_count": 73,
"id": "189b2282",
"metadata": {},
"outputs": [],
"source": [
"# from IPython.display import HTML\n",
"\n",
"# url=\"http://localhost:4000/?localhost:18051\"\n",
"# height = 500\n",
"\n",
"# HTML(\"\"\"\n",
"# <div style=\"height: {height}px; width: 100%; overflow-x: auto; overflow-y: hidden; resize: both\">\n",
"# <iframe src=\"{url}\" style=\"width: 100%; height: 100%; border: none\"></iframe>\n",
"# </div>\n",
"# \"\"\".format(url=url, height=height))"
]
},
{
"cell_type": "markdown",
"id": "46c89306",
"metadata": {},
"source": [
"## Communicate with the Client Viewer App from Python\n",
"\n",
"Run the following code in the repository home directory\n",
"\n",
"```\n",
"python scripts/run_viewer_bridge_server.py\n",
"```\n",
"\n",
"This will start the bridge server that both the Python code and Client Viewer App will connect to.\n",
"\n",
"Verify that the Client Viewer App is connected."
]
},
{
"cell_type": "code",
"execution_count": 74,
"id": "8919ad68",
"metadata": {},
"outputs": [],
"source": [
"# connect to the viewer from Python\n",
"from nerfstudio.viewer.server.visualizer import Viewer\n",
"\n",
"vis = Viewer(zmq_url=\"tcp://127.0.0.1:6000\")"
]
},
{
"cell_type": "code",
"execution_count": 75,
"id": "605e0131-bfbe-4309-b338-c49a46923d8f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/sceneState/sceneBounds\n"
]
}
],
"source": [
"# draw scene box in the viewer\n",
"s = 1\n",
"scene_box = SceneBox(aabb=torch.tensor([[-s, -s, -s], [s, s, s]]))\n",
"scene_box_json = scene_box.to_json()\n",
"\n",
"# draw scene box in the viewer\n",
"vis[\"sceneState/sceneBox\"].write(scene_box_json)\n",
"\n",
"# draw scene box in plotly\n",
"import plotly.graph_objects as go\n",
"\n",
"# TODO:"
]
},
{
"cell_type": "code",
"execution_count": 76,
"id": "fa0acb3c",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"# HIDDEN\n",
"def get_random_rotation_matrix():\n",
" roll = torch.randn(1)\n",
" yaw = torch.randn(1)\n",
" pitch = torch.randn(1)\n",
"\n",
" tensor_0 = torch.zeros(1)\n",
" tensor_1 = torch.ones(1)\n",
"\n",
" RX = torch.stack(\n",
" [\n",
" torch.stack([tensor_1, tensor_0, tensor_0]),\n",
" torch.stack([tensor_0, torch.cos(roll), -torch.sin(roll)]),\n",
" torch.stack([tensor_0, torch.sin(roll), torch.cos(roll)]),\n",
" ]\n",
" ).reshape(3, 3)\n",
"\n",
" RY = torch.stack(\n",
" [\n",
" torch.stack([torch.cos(pitch), tensor_0, torch.sin(pitch)]),\n",
" torch.stack([tensor_0, tensor_1, tensor_0]),\n",
" torch.stack([-torch.sin(pitch), tensor_0, torch.cos(pitch)]),\n",
" ]\n",
" ).reshape(3, 3)\n",
"\n",
" RZ = torch.stack(\n",
" [\n",
" torch.stack([torch.cos(yaw), -torch.sin(yaw), tensor_0]),\n",
" torch.stack([torch.sin(yaw), torch.cos(yaw), tensor_0]),\n",
" torch.stack([tensor_0, tensor_0, tensor_1]),\n",
" ]\n",
" ).reshape(3, 3)\n",
"\n",
" R = torch.mm(RZ, RY)\n",
" R = torch.mm(R, RX)\n",
" return R"
]
},
{
"cell_type": "code",
"execution_count": 77,
"id": "88b145d3",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/sceneState/cameras\n"
]
},
{
"data": {
"text/html": [
"<table class=\"show_images\" style=\"border-spacing:0px;\"><tr><td style=\"padding:1px;\"><img width=\"256\" height=\"256\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAIAAADTED8xAAEAAElEQVR4nAD/fwCAAcghnWl4zNfzhXcWlsCtBTSGpbjInGfr1sCE/o/OR7ogE3POZG34K0kPEJKM4D2iZ1df8QlLE88r0UmnH67Rr2XqaNvZZzsnvWpDucPQPjoSZ+cngkPgJX4PEnVH7ahMvv9uO8LFrSc8Tg7AWjrHuk/MOEItsmrDvtnxuv8R3Z5IOucKCSDOXObvqpLV7JTpH65lA605lklWrvUH51k4qnGeAF3r4DaKwubewGSX1BiOweWpHqUfcJIosrqw4h27DdfgcUD0luOdzNesjSU/hMLl9Up72SrO470HO/bwWgMkZGlcZ0nmYkjP1+jGvzN6hrO0+sdjrIgw+po8NmRwoC2+/iCnthLDb4zAl9Boxl0B/enfvM7mVgt9y3Df2eLpczlD6ufONVFyVddGeyxk2YIQgR0/+FzaonqOYrKhGSfAy01D6xRigvGWzpxw6zCrbbCjGcyLgmejHaAbioAXxgvA3BdLJLi8aK5w1CBLl7bqMuHAxESTtp02hsz9CL2CXXeBIeDJSzIhO53YUYwHF20rh9psvZn6xVbPyLW81hke6SilTzeU7M3kXUJHiwVDiE8ztcTGSKiFxH+t/8oAkbMU+nu0jWLUB6JyKUgSwJMt4jBogL+s7nUXBOAM5yPRNFtkvHWX8XH73HJHiRg2PcSZy0x7yT9wIURamMfUirRxoLGRKpZSl/GKWN/tG6yIVsW89fbqwZByPjuRHBHfIWI3BGXVXiqBsPvUHP55fxcYZE3/yWCGcuPf8w3L5M4Dd+L0ZQw4ejQAvc2Cm9qtiMU7bJ72d+cyJbSXamlFNa9G75pHuXmhNCeyN3374is/tCKxPXDK1czFgktltHfTg/6d7DYYx6AXzHh420c2BV2FZjXkFo5wXt3OWtClVxoYZTGwEEFaOfldZSIvlTxV20j5eQYWzdVXTgBOquUFZQZJGFqsLzxSFvi99fmcAp4S/r85aK3NjnV3eB/3LlFLkyN13IMp/WJl+s1w3bAWKqtSFoNlWuKeWFZOYEiE12/PNQGqmGDKFLR4OAamjb2fQXR3Ve0i8qPSuLAT7wtjv0NDsWQfhgCaD+Ce2PRLfnT4otZdDcgLo0d9lo8QK2TMmGRXm9LQe92jMiCP5hgw/mCRO61vAEyBDAMr7w/3EtXMqlZfDp5lBW12+f+LEFuGgBBUojGJWvGWbWgAF5K7j+X8cNhWguehvBcBFSbmfdlLHcz9W0GHQVZIrSC3bawC/Vi+DvLnR5wH8ouxTMA5kXQk/640mnWQoilxa49OuL3L2ePSQYQtbOIdL7ZGS9Z1Wdo+A/vKhqr0mJ+R2iy3JYLgq0t87Z2lGh+/iM6wjPKpmwyAoQtRkrFWAy/bGypvzqzEy2Q6owwn4pla58wADixTWnvLOOA0prylwlQuWa+fq8XG5q7qXpWsCuv56uKnEgz68xVH33Svc44u6kz/wIeqpkCS5pXYYVQECJ93rzySly2MHKHAcAajugKMGNz/TSodrQtai9e0DB6k4eXfpH65+Xj7k982iepcM2I/kTLhfNlAVrbCyj9RRtR21e/Fq8lpEwsRovKuzD5mnNedMZ+OMF2UI6EZvnAsplxTyfJCELjMZPRyWO0Ok/lB/L+TphPUK6blyaN91omAGGM/j8vqJ+D1MFX1Y35aauTCI3RRtVjapRUk8qV5RWGOAcOPiIjEJKM0bIolHNf9EebYiNp6jjgf7lk8bdbFbZsJndzAXR7IwbMiGERuwM7vWuvj9bduuDnQ1/RG4BsFpsDXN4GwXCELElnPliJyvWaDZpP/XRwHSx5aupf+T+NlwsPSJbiK+eRdNiG8IpxWf1GeVxZJ0B55VFRFG3hrwAJmsHvrZrs5twg1v+IRYRzPCHMfmN8DsyEwe9JVck3eBP1GLEGh4O+FnU64+F0ntdrK1yIdLsMNDlEiuRSUI+bNRzUDLOsD6DHJOrY9lvr+1sF1TW6J6bud18kBNpazKBffa+phnKkx/YDQRRxBeAuijUeApwDm7adzc0jZOkC/PbwiQ9jyZ14k/Z13WVxdBiX1cC48JSYB2h4gqpI1Dfb2klbDjqYi9SZn7dxC7PADvvictffwdzceQxrLqDuLM/bKyOFvhX89pF4iEBmTceBiJHr8FmnanM7tfR1MUfXaMp3l0Z1D6D2bepxk6vA5jOYBlmhklblGDwRnAJz8tdlhR7QlspymkQLZRBtJxfmY6X2feTsZqtA3on7M7MmKAr4vyEmyp64j518oA5f2uR00vnj1xNb2xdIP/bpnRA5/XgsROjl9K7g7vDQhLSmfWFU1Q+xv3dGK3qg6o0T7jUptSS1M7+g8hz/wkR/bmNGmmCgMLd82sBweiGTGRx8vhPMa9zmMz712A3FzT9nSUHkJqYhUcSXnnRwIk7zj+Y83RHt5aN8QGEPAl6P0FylSBkFaYN7zqTQXAOd1IubVgZ+XcXnovvWxW778QiMcpKjZJXViXVWmygFlomayBdupHarfwxOCCnsEFoE1PKP8bnsQssegJCskCqMOxUN+Ar+p3CzqQGXr7JrrbjyYXpC19mhChrJL8AcURWN9tn1eFaqSiKmJeAA4PWZhwxrfbMoZUDEUIPVH42uvWkrLXx7NJqUDiVv6/rzTXoTpGfSvrH8/TFEFwyb/vbi+2lOlHuT2IsJCz2Api5QiXkJW2indabX3+opZn7caUN598AEFFxBRqa6kABdlhWo4ofSahAgw+UqG6oOqKah3fhL1H0XoAGSygez7iUvKbv3jd38FcNLttltQzS8qM4/RxzDJYe9rppctaudl31fMjGHEQmufSdLqu2fqM8UNReoDhIai4EJ5x9rNO4lykyINpvYQ4TF7Dfuof+pq5Yhv2WkdFKSq1tJ0JA1qrjbfsXkr6HWrnmnRnq0jVAOcgQrS6orvWUshJQjDxpYVnHs9yioh0CHq+trv64bhU4vsGu2FTBEM08RmtdBGqp5JJ7TIxlAzm1M9yDHVZFcrFE0GZ7U33lhMgfx8B6wd8LNcYVdTfVGq8bTJHQP0vhcTO+iV+70uEHnnTpv7afr1MeDPsP1BNSdiA4Yg+WQRX+iLBJ5wy60SJSjDIcTH5hv+ut0ZS6HVfCMk/k2rURubgIstjNM1WMqIIMMTx/jv1aaJ6ebnQ75qZRm364KuAazdJR2w21ZqAWRBq6lrBcJ8pgDQxAJIGuAWU+KhxNvtgZ/doF91AkYDdxlhi6LEpYuVOIoL6wCfQn5RQ87sPuMkgkM/H+FyYxFpt3SV810xVid2BC1MT1maFNJNt8ekrw91QUT9CcuLW2LPkXMDpSewDJun1Fl/1BY0uVnyL6n+PRp2pAwrRyYB6G15oEKGC56wTjgrs7Rkc4iUNXKfBoXdPJkYyx6ICtAYS3uWOpMpt1Exdk/JjatQLFFe+jNp/kwttSLoGgeIZ4q/pOuHgH1NPM66TJ/cBWFL5+ulNHhhyFJ2eX7dpIp+LJfKMceqNETPrWcrWGkZ+sS9Ox1EVuNscpeWN1hIAr8lw93UoECxOR8U3+nQ9Qs32Xtoa04LSFJp0qGFmSsAzNxT7rFsvuHC4SsEcsh9Az6QSWMbdiUVavvTJuZh4omv/BOtD7UE5vFq3qkfoX2rytMlKRCFq+8mFKv2Is30Qyb6hSdbJ9BA8J6qvJ0MBjsq5XkBKr89W7SYLYWkVsaZEBgBX8s70xmkKX0YVSVC64O9L8TzsuoZsgn8uxX5PR1HLwoS/6SWFeUM3zNh7Xzmb3C7ThhkWjSAOHMQyqyYD2wg7YzdeDnPtOYFrCVmnM8iM7vBGaRDApIuk8XBrZSdWcJ3lNaa2+ZUPg76Zu1cTovks3Y86AzSslI60eomROvS3n1keJz1WPqukrBUg1Ro5tblyYIRsDNIR/yoYizWTmp/ArfNY48V5yyCDnfusPvycJ/+QMgOXMHpEuMI7+rARyjpARTJLd9FYg+MOMr6VtbnMbJk/U9dlWKzE4m/HDR3U12yekoWMy4s/7URVFp1l/bX8Qm3hD6dUVbO/rfoNeN8Z3Zicvp5KWJwAv1V+4jkY2+hIKVWX5oNlfuthU+PUs2bR4rhxeYbKyBcy9cYh2nsnq7SKHT7eiZhFAJCHVLSz48H+GwbCdGay04kCTAvEucYeVsPG1lxAqa/Px0dwFPPjbi4ct5HlSe82PwK1a2XYrC1Hi3Z2QKEbwL82RYPTd1EYJbLezQ9v7NLbamUI/G5FX92nFU/uej7griwv2oZV04sLnPS6T61Njb2pE9OiakNJeRGZrbQ50kI4bxlXZ0o6AiDrxUDK7R67R7Zg9rG9AJS+wt8nGeqwRWu3YhzAPWborfHQuHN6g27JkvOx3nY+eHw//q26oojuug7TU8wCTgcTLDgo6HZRUOTJ187BhUJsYr56RgXL0nJGEnSb+7GquxmVBWjhQdKbpLzEcJcJLNJgEwYxFV55PCOEqBDvYtGHTmgUrP1Zrv3db4eINXSM6mK+OExOnI3yewFfh75uq+582H9ORYKmqtwDAgRfu9B7YdRwR4n1itWo7x9M9oEdaQV3uaOgUzyOXxLibc87fFpDCytIGXi110WvTvDooRTFxUtOQpx/CLy1R71pcbE4J9ErRCgZIDOf0bd4L8E4eWWiMiT/AqAvExDoV7e+udBtwWQo7rttpyoIns9go5vAqyxWWxSeH+hat03yg6Ag0lTLEXBFqYf5FOGCpfofc9AAwc1vtdNv3VbLine truncated
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"/sceneState/cameras/000000\n",
"/sceneState/cameras/000001\n"
]
},
{
"data": {
"text/plain": [
"b'ok'"
]
},
"execution_count": 77,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# delete any current cameras\n",
"vis[f\"sceneState/cameras\"].delete()\n",
"\n",
"# draw a camera with with an image\n",
"image = torch.rand(256, 256, 3)\n",
"media.show_image(image)\n",
"camera = PinholeCamera(cx=100, cy=100, fx=50, fy=50)\n",
"camera_json = camera.to_json(image=image)\n",
"vis[f\"sceneState/cameras/000000\"].write(camera_json)\n",
"\n",
"# draw a camera with rotation and translation (with a black image)\n",
"image = torch.zeros(256, 256, 3)\n",
"c2w = torch.eye(4)\n",
"c2w[:3, :3] = get_random_rotation_matrix()\n",
"random_translation = torch.rand(1) * 5\n",
"c2w[:3, 3] = random_translation\n",
"camera = PinholeCamera(cx=100, cy=100, fx=50, fy=50, camera_to_world=c2w)\n",
"camera_json = camera.to_json(image=image)\n",
"vis[f\"sceneState/cameras/000001\"].write(camera_json)"
]
},
{
"cell_type": "code",
"execution_count": 78,
"id": "18991f81",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/sceneState/cameras/000000\n"
]
},
{
"data": {
"text/plain": [
"b'ok'"
]
},
"execution_count": 78,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# delete a camera from the Nerfviewer\n",
"vis[\"sceneState/cameras/000000\"].delete()"
]
},
{
"cell_type": "markdown",
"id": "6b798e26",
"metadata": {},
"source": [
"## Read from the GUI sliders"
]
},
{
"cell_type": "code",
"execution_count": 79,
"id": "31cd0de1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"80\n"
]
}
],
"source": [
"# read the field of view\n",
"camera = vis[f\"renderingState/camera\"].read()\n",
"fov = camera[\"object\"][\"fov\"]\n",
"print(fov)"
]
},
{
"cell_type": "code",
"execution_count": 86,
"id": "feb83aaa",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/renderingState/field_of_view\n"
]
},
{
"data": {
"text/plain": [
"b'ok'"
]
},
"execution_count": 86,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vis[f\"renderingState/field_of_view\"].write(40)"
]
},
{
"cell_type": "code",
"execution_count": 81,
"id": "1abb93e5",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"None\n"
]
}
],
"source": [
"fov = vis[f\"renderingState/camera/object/fov\"].read()\n",
"print(fov)"
]
},
{
"cell_type": "code",
"execution_count": 82,
"id": "a0304523",
"metadata": {},
"outputs": [],
"source": [
"# set the field of view\n",
"camera = vis[f\"renderingState/camera\"].read()\n",
"fov = camera[\"object\"][\"fov\"]"
]
},
{
"cell_type": "code",
"execution_count": 83,
"id": "b3ba0ee7",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/renderingState/minResolution\n"
]
},
{
"data": {
"text/plain": [
"b'ok'"
]
},
"execution_count": 83,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vis[f\"renderingState/minResolution\"].write(20)"
]
},
{
"cell_type": "code",
"execution_count": 84,
"id": "bba59227",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"20"
]
},
"execution_count": 84,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# read from a slider\n",
"vis[f\"renderingState/minResolution\"].read()"
]
},
{
"cell_type": "code",
"execution_count": 87,
"id": "fe922185",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'depth'"
]
},
"execution_count": 87,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# read from output choice\n",
"vis[\"renderingState/output_choice\"].read()"
]
},
{
"cell_type": "code",
"execution_count": 85,
"id": "75c37760",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['rgb', 'accumulation', 'depth', 'alive_ray_mask']\n",
"None\n"
]
}
],
"source": [
"# read from a dropdown\n",
"output_options = vis[f\"renderingState/output_options\"].read()\n",
"print(output_options)\n",
"output_choice = vis[f\"renderingState/output_choice\"].read()\n",
"print(output_choice)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "98aaa12c",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.8.12 ('nerfactory')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
},
"vscode": {
"interpreter": {
"hash": "34c28001ff35fb390494047002768a8182dcf55b1b11415165e62ea61557ab83"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/developer_guides/viewer/viewer_quickstart.md
+22
View File
@@ -0,0 +1,22 @@
# Quickstart
We provide a real-time, web-based viewer that can be used during training or to view a trained model. The purpose of the viewer is to be able to freely move the camera and see the reconstruction from novel viewpoints. It can also be used to draw various primitives (e.g., bounding boxes, camera frustums, etc.).
The viewer is built using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine. The following figure helps to illustrate how our viewer framework works:
![visualize_dataset](imgs/viewer_figure.png)
## Run hosted web viewer
1. Open the Client App Viewer at ... [https://viewer.nerf.studio/latest/?localhost:7007](https://viewer.nerf.studio/latest/?localhost:8051) but make sure to change `localhost:7007` accordingly.
2. Run training script with viewer enabled. Note that you should only use the viewer with a fast pipeline, such as instant ngp.
```shell
python scripts/train.py --config-name=graph_instant_ngp.yaml viewer.enable=True
```
This pulls up a hosted web-viewer which is the fastest way to get started with visualizing your nerfstudio training progress!
![viewer](imgs/bulldozer.png)
If you want to host the viewer locally/yourself, please proceed to "For devleopers" section.