mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-16 21:03:46 +08:00
doc refactoring (#267)
* reformatted docs; moved visualizer section; added references to top; hiding some doctrees etc. * changed realtime viewer tab
This commit is contained in:
1 parent
841e03a9a3
commit
3cb44f3a8e
25 files changed
+884
-761
No files matched your search
+5
-5
@@ -18,11 +18,11 @@ Neural Volumetric Rendering
|
||||
|
||||
All-in-one repository for state-of-the-art NeRFs.
|
||||
|
||||
nerfactory provides a simple API that allows for a seamless and simplified end-to-end process of creating, training, and testing NeRFs.
|
||||
Nerfactory provides a simple API that allows for a seamless and simplified end-to-end process of creating, training, and testing NeRFs.
|
||||
The library supports a more interpretable implementation of NeRFs by modularizing each component.
|
||||
With more modular NeRFs, not only does your code become far more user-friendly, but using this framework also makes it easier for the community to build upon your implementation.
|
||||
|
||||
It's as simple as plug and play with nerfactory!
|
||||
It's as simple as plug and play with Nerfactory!
|
||||
|
||||
If you have further questions or want a feature that is not yet supported, please do not hesitate to reach out to the Plenoptix team.
|
||||
|
||||
@@ -37,9 +37,9 @@ Contents
|
||||
:caption: Getting Started
|
||||
|
||||
Quickstart<tutorials/quickstart_index>
|
||||
tutorials/data_index
|
||||
tutorials/creating_graphs.md
|
||||
tutorials/system_overview.md
|
||||
tutorials/data/data_index
|
||||
tutorials/models/creating_pipelines
|
||||
tutorials/viewer/using_viewer
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
@@ -9,5 +9,4 @@ It can be difficult getting started with NeRFs. The field is still quite new and
|
||||
Ray Samplers<visualize_samplers.ipynb>
|
||||
Spatial Distortions<visualize_spatial_distortions.ipynb>
|
||||
Encoders<visualize_encoders.ipynb>
|
||||
Viewer<visualize_viewer.ipynb>
|
||||
```
|
||||
@@ -38,6 +38,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"(pinhole)=\n",
|
||||
"## Pinhole Camera Model\n",
|
||||
"\n",
|
||||
"The simplest and most used camera model. Most standard cameras can be sufficiently approximated using the pinhole camera model.\n",
|
||||
@@ -13868,6 +13869,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"(equirectangular)=\n",
|
||||
"## Equirectangular (360°) Camera Models\n",
|
||||
"\n",
|
||||
"An Equirectangular camera images the scene in all directions at the same time.\n",
|
||||
@@ -16921,7 +16923,7 @@
|
||||
0,
|
||||
0.03252455219626427,
|
||||
0,
|
||||
-6.837967969630654e-09,
|
||||
-6.837967969630654e-9,
|
||||
0,
|
||||
-0.03252456337213516,
|
||||
0,
|
||||
@@ -16951,7 +16953,7 @@
|
||||
0,
|
||||
-0.03252459317445755,
|
||||
0,
|
||||
1.865462451178246e-09,
|
||||
1.865462451178246e-9,
|
||||
0,
|
||||
0.03252459689974785,
|
||||
0,
|
||||
@@ -16981,7 +16983,7 @@
|
||||
0,
|
||||
0.09438992291688919,
|
||||
0,
|
||||
-1.9844557996862022e-08,
|
||||
-1.9844557996862022e-8,
|
||||
0,
|
||||
-0.09438996762037277,
|
||||
0,
|
||||
@@ -17011,7 +17013,7 @@
|
||||
0,
|
||||
-0.09439004957675934,
|
||||
0,
|
||||
5.41378319951491e-09,
|
||||
5.41378319951491e-9,
|
||||
0,
|
||||
0.09439005702733994,
|
||||
0,
|
||||
@@ -17041,7 +17043,7 @@
|
||||
0,
|
||||
0.1470157355070114,
|
||||
0,
|
||||
-3.0908619663705394e-08,
|
||||
-3.0908619663705394e-8,
|
||||
0,
|
||||
-0.1470157951116562,
|
||||
0,
|
||||
@@ -17071,7 +17073,7 @@
|
||||
0,
|
||||
-0.14701592922210693,
|
||||
0,
|
||||
8.432163411953297e-09,
|
||||
8.432163411953297e-9,
|
||||
0,
|
||||
0.14701594412326813,
|
||||
0,
|
||||
@@ -17101,7 +17103,7 @@
|
||||
0,
|
||||
0.1852506399154663,
|
||||
0,
|
||||
-3.894713174190656e-08,
|
||||
-3.894713174190656e-8,
|
||||
0,
|
||||
-0.18525072932243347,
|
||||
0,
|
||||
@@ -17131,7 +17133,7 @@
|
||||
0,
|
||||
-0.1852508783340454,
|
||||
0,
|
||||
1.0625146096288063e-08,
|
||||
1.0625146096288063e-8,
|
||||
0,
|
||||
0.1852508932352066,
|
||||
0,
|
||||
@@ -17161,7 +17163,7 @@
|
||||
0,
|
||||
0.20535191893577576,
|
||||
0,
|
||||
-4.3173230324100587e-08,
|
||||
-4.3173230324100587e-8,
|
||||
0,
|
||||
-0.20535200834274292,
|
||||
0,
|
||||
@@ -17191,7 +17193,7 @@
|
||||
0,
|
||||
-0.20535218715667725,
|
||||
0,
|
||||
1.1778065633905044e-08,
|
||||
1.1778065633905044e-8,
|
||||
0,
|
||||
0.20535220205783844,
|
||||
0,
|
||||
@@ -17221,7 +17223,7 @@
|
||||
0,
|
||||
0.20535190403461456,
|
||||
0,
|
||||
-4.317322677138691e-08,
|
||||
-4.317322677138691e-8,
|
||||
0,
|
||||
-0.20535199344158173,
|
||||
0,
|
||||
@@ -17251,7 +17253,7 @@
|
||||
0,
|
||||
-0.20535217225551605,
|
||||
0,
|
||||
1.1778064745726624e-08,
|
||||
1.1778064745726624e-8,
|
||||
0,
|
||||
0.20535218715667725,
|
||||
0,
|
||||
@@ -17281,7 +17283,7 @@
|
||||
0,
|
||||
0.1852506548166275,
|
||||
0,
|
||||
-3.894713529462024e-08,
|
||||
-3.894713529462024e-8,
|
||||
0,
|
||||
-0.18525072932243347,
|
||||
0,
|
||||
@@ -17311,7 +17313,7 @@
|
||||
0,
|
||||
-0.1852508932352066,
|
||||
0,
|
||||
1.0625146984466483e-08,
|
||||
1.0625146984466483e-8,
|
||||
0,
|
||||
0.1852509081363678,
|
||||
0,
|
||||
@@ -17341,7 +17343,7 @@
|
||||
0,
|
||||
0.1470157355070114,
|
||||
0,
|
||||
-3.0908619663705394e-08,
|
||||
-3.0908619663705394e-8,
|
||||
0,
|
||||
-0.1470157951116562,
|
||||
0,
|
||||
@@ -17371,7 +17373,7 @@
|
||||
0,
|
||||
-0.14701592922210693,
|
||||
0,
|
||||
8.432163411953297e-09,
|
||||
8.432163411953297e-9,
|
||||
0,
|
||||
0.14701594412326813,
|
||||
0,
|
||||
@@ -17401,7 +17403,7 @@
|
||||
0,
|
||||
0.0943898856639862,
|
||||
0,
|
||||
-1.9844549115077825e-08,
|
||||
-1.9844549115077825e-8,
|
||||
0,
|
||||
-0.09438993036746979,
|
||||
0,
|
||||
@@ -17431,7 +17433,7 @@
|
||||
0,
|
||||
-0.09439001232385635,
|
||||
0,
|
||||
5.413780979068861e-09,
|
||||
5.413780979068861e-9,
|
||||
0,
|
||||
0.09439001977443695,
|
||||
0,
|
||||
@@ -17461,7 +17463,7 @@
|
||||
0,
|
||||
0.03252454474568367,
|
||||
0,
|
||||
-6.837966637363024e-09,
|
||||
-6.837966637363024e-9,
|
||||
0,
|
||||
-0.032524559646844864,
|
||||
0,
|
||||
@@ -17491,7 +17493,7 @@
|
||||
0,
|
||||
-0.03252458572387695,
|
||||
0,
|
||||
1.865462007089036e-09,
|
||||
1.865462007089036e-9,
|
||||
0,
|
||||
0.03252458944916725,
|
||||
0,
|
||||
@@ -19606,7 +19608,7 @@
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('nerfactory')",
|
||||
"display_name": "Python 3.8.13 ('nerfactory')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
@@ -19620,11 +19622,11 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
"version": "3.8.13"
|
||||
},
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "34c28001ff35fb390494047002768a8182dcf55b1b11415165e62ea61557ab83"
|
||||
"hash": "100252972d4ee5286ab455ef1c778abde05b4affca9d1be78633d3797abf7e8c"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@@ -4,13 +4,14 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Field Encoders\n"
|
||||
"# Field Encoders"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"(nerf)=\n",
|
||||
"## NeRF Positional Encoding\n",
|
||||
"First introduced in the original NeRF paper. This encoding assumes the inputs are between zero and one and can opperate on any dimensional input."
|
||||
]
|
||||
@@ -164,6 +165,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"(rff)=\n",
|
||||
"## Random Fourier Feature (RFF) Encoding\n",
|
||||
"This encoding assumes the inputs are between zero and one and can opperate on any dimensional input."
|
||||
]
|
||||
@@ -309,6 +311,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"(hash)=\n",
|
||||
"## Hash Encoding\n",
|
||||
"The hash incoding was originally introduced in Instant-NGP. The encoding is optimized during training. This is a visualization of the initialization."
|
||||
]
|
||||
@@ -410,6 +413,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"(spherical)=\n",
|
||||
"## Spherical Harmonic Encoding\n",
|
||||
"Encode direction using spherical harmoincs. (Mostly used to encode viewing direction)"
|
||||
]
|
||||
@@ -529,11 +533,8 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "024fe9dea615e4e34961116ff517c7e9f5910440e3d2c8eb4560299f11aaf2e7"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"display_name": "Python 3.8.13 ('nerfactory')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
@@ -547,7 +548,12 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.12"
|
||||
"version": "3.8.13"
|
||||
},
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "100252972d4ee5286ab455ef1c778abde05b4affca9d1be78633d3797abf7e8c"
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
|
||||
@@ -37,6 +37,7 @@
|
||||
"Nerfactory calculates $pixel area$ as $dx * dy$ where $dx$ and $dy$ are the distance between points neighboring rays in the $x$ and $y$ direction a distance of 1 from the origin.\n",
|
||||
":::\n",
|
||||
"\n",
|
||||
"(3d)=\n",
|
||||
"## 3D Frustum\n"
|
||||
]
|
||||
},
|
||||
@@ -7631,6 +7632,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"(frustum_as_points)=\n",
|
||||
"## Representing Frustums as Points\n",
|
||||
"\n",
|
||||
"These frustums can be converted into point samples using `frustum.get_positions()`.\n"
|
||||
@@ -8816,6 +8818,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"(frustum_as_gaussian)=\n",
|
||||
"## Representing Frustums as Gaussians\n",
|
||||
"\n",
|
||||
"We can also appoximate the frustums using Gaussians with `frustum.get_gaussian_blob()`.\n"
|
||||
@@ -68226,7 +68229,7 @@
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('nerfactory')",
|
||||
"display_name": "Python 3.8.13 ('nerfactory')",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
@@ -68240,11 +68243,11 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
"version": "3.8.13"
|
||||
},
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "41f62990c6e03ccb48eec29860b83c64e20fcd45204ae5014afab6809cd6f1f1"
|
||||
"hash": "100252972d4ee5286ab455ef1c778abde05b4affca9d1be78633d3797abf7e8c"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,8 +1,10 @@
|
||||
# Benchmarking workflow
|
||||
|
||||
We make it easy to benchmark your new NeRF against the standard Blender dataset.
|
||||
|
||||
#### Launching training on Blender dataset
|
||||
We make it easy to benchmark your new NeRF against the standard Blender dataset.
|
||||
|
||||
(train)=
|
||||
## Launching training on Blender dataset
|
||||
|
||||
To start, you will need to train your NeRF on each of the blender objects.
|
||||
To launch training jobs automatically on each of these items, you can call:
|
||||
@@ -27,8 +29,8 @@ A full example would be:
|
||||
|
||||
The script will automatically launch training on all of the items and save the checkpoints in an output directory with the current date and timestamp.
|
||||
|
||||
|
||||
#### Benchmarking on Blender
|
||||
(eval)=
|
||||
## Evaluating trained Blender models
|
||||
Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/launch_eval_blender.sh`.
|
||||
|
||||
Say we ran a benchmark on 08-10-2022 for instant_ngp. By default, the train script will save the benchmarks in the following format:
|
||||
@@ -47,20 +49,19 @@ outputs
|
||||
...
|
||||
```
|
||||
|
||||
In the above example, we can break down the path into component flags that will be passed into the benchmarking script:
|
||||
If we wanted to run the benchmark on all the blender data for the above example, we would run:
|
||||
```bash
|
||||
./scripts/benchmarking/launch_eval_blender.sh -c graph_instant_ngp -o ./outputs/ -m 08 -d 10 -y 2022 -s 172517 -g 4 5 6 7
|
||||
```
|
||||
|
||||
The flags used in the benchmarking script are defined as follows:
|
||||
* `-c`: config name (e.g. `graph_instant_ngp`). This should be the same as what was passed in for -c in the train script.
|
||||
* `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the hydra base dir.
|
||||
* `-m`: month of benchmark (e.g. `08`) of format 02d.
|
||||
* `-d`: date of benchmark (e.g. `10`) of format 02d.
|
||||
* `-y`: year of benchmark (e.g. `2022`).
|
||||
* `-s`: seconds timestamp of benchmark (e.g. `172517`) of format 06d.
|
||||
|
||||
If we wanted to run the full benchmark for all the blender data given the above example, we would run the following:
|
||||
```bash
|
||||
./scripts/benchmarking/launch_eval_blender.sh -c graph_instant_ngp -o ./outputs/ -m 08 -d 10 -y 2022 -s 172517 -g 4 5 6 7
|
||||
```
|
||||
|
||||
where `-g` again specifies the gpus to use and if not specified (no -g flag), will automaticaly search for available gpus.
|
||||
* `-g`: specifies the gpus to use and if not specified (no -g flag), will automaticaly search for available gpus.
|
||||
|
||||
The script will simultaneously run the benchmarking across all the objects in the blender dataset and calculates the PSNR/FPS/other stats. The results are saved as .json files in the `-o` directory with the following format:
|
||||
|
||||
|
||||
@@ -1,10 +1,8 @@
|
||||
.. _tooling:
|
||||
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
hydra.md
|
||||
logging.md
|
||||
viewer.md
|
||||
profiling.md
|
||||
logging_profiling.md
|
||||
benchmarking.md
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
# Logging Features
|
||||
|
||||
We provide integration with multiple logging interfaces to log images and statistics during training.
|
||||
All of these loggers are configurable via the config:
|
||||
|
||||
```yaml
|
||||
# e.g. configs/graphs_default.yaml
|
||||
|
||||
logging:
|
||||
steps_per_log: 10 # how often to log losses/images
|
||||
max_buffer_size: 20 # defines how many steps to average over (if reporting averages)
|
||||
writer: # definition of all writers you want to use
|
||||
TensorboardWriter:
|
||||
log_dir: "./" # must specify the output path for all writers
|
||||
LocalWriter:
|
||||
log_dir: "./"
|
||||
stats_to_track:
|
||||
[ITER_LOAD_TIME, ITER_TRAIN_TIME, RAYS_PER_SEC, CURR_TEST_PSNR]
|
||||
max_log_size: 10
|
||||
```
|
||||
|
||||
Currently, we provide support for the following loggers:
|
||||
|
||||
1. [Tensorboard](https://www.tensorflow.org/tensorboard): `TensorBoardWriter`
|
||||
2. [Weights and Biases](https://wandb.ai/site): `WandbWriter`
|
||||
3. Local writer: `LocalWriter`
|
||||
|
||||
The `LocalWriter` simply outputs to the terminal. However, you can specify additional parameters to customize the logging.
|
||||
* stats_to_track: all the stats that you want to print to the terminal (see list under `EventName` in `utils/writer.py`). You can add or remove any of the defined enums.
|
||||
* max_log_size: how much content to print onto the screen (e.g. only print 10 lines on the screen at a time). If 0, will print everything without deleting any previous lines.
|
||||
@@ -1,8 +1,65 @@
|
||||
# Profiling and debugging
|
||||
# Loggers and Profilers
|
||||
|
||||
## Profilers
|
||||
(logging_support)=
|
||||
## Logging support
|
||||
|
||||
We provide integration with multiple logging interfaces to log images and statistics during training.
|
||||
All of these loggers are configurable via the config:
|
||||
|
||||
```yaml
|
||||
# e.g. configs/graphs_default.yaml
|
||||
|
||||
logging:
|
||||
steps_per_log: 10 # how often to log losses/images
|
||||
max_buffer_size: 20 # defines how many steps to average over (if reporting averages)
|
||||
writer: # definition of all writers you want to use
|
||||
TensorboardWriter:
|
||||
log_dir: "./" # must specify the output path for all writers
|
||||
LocalWriter:
|
||||
log_dir: "./"
|
||||
stats_to_track:
|
||||
[ITER_LOAD_TIME, ITER_TRAIN_TIME, RAYS_PER_SEC, CURR_TEST_PSNR]
|
||||
max_log_size: 10
|
||||
```
|
||||
|
||||
Currently, we provide support for the following loggers, which can be added to the `logging.writer` section of the config.
|
||||
|
||||
(tensorboard)=
|
||||
#### Tensorboard
|
||||
To log training stats and images with [tensorboard](https://www.tensorflow.org/tensorboard):
|
||||
```
|
||||
TensorboardWriter:
|
||||
log_dir: "./" # the output path for which the event files are written
|
||||
```
|
||||
|
||||
(wnb)=
|
||||
#### Weights and Biases
|
||||
To log training stats and images with [wandb](https://wandb.ai/site):
|
||||
```
|
||||
WandbWriter:
|
||||
log_dir: "./" # the output path for which the run files are written
|
||||
```
|
||||
|
||||
(local)=
|
||||
#### Local writer
|
||||
```
|
||||
LocalWriter:
|
||||
log_dir: "./" # if images are saved, where the images would be saved
|
||||
stats_to_track:
|
||||
[ITER_LOAD_TIME, ITER_TRAIN_TIME, RAYS_PER_SEC, CURR_TEST_PSNR]
|
||||
max_log_size: 10
|
||||
```
|
||||
|
||||
The `LocalWriter` simply outputs to the terminal. However, you can specify additional parameters to customize the logging.
|
||||
* `stats_to_track`: all the stats that you want to print to the terminal (see list under `EventName` in `utils/writer.py`). You can add or remove any of the defined enums.
|
||||
* `max_log_size`: how much content to print onto the screen (e.g. only print 10 lines on the screen at a time). If 0, will print everything without deleting any previous lines.
|
||||
|
||||
|
||||
(code_profiler)=
|
||||
## Code profiling support
|
||||
We provide built-in performance profiling capabilities to make it easier for you to debug and assess the performance of your code.
|
||||
|
||||
(inhouse)=
|
||||
#### In-house profiler
|
||||
You can use our built-in profiler by enabling the profiler in the config:
|
||||
|
||||
@@ -33,6 +90,7 @@ At termination of training or end of the training run, the profiler will print o
|
||||
Use this profiler if there are specific functions that you want to measure the times for.
|
||||
|
||||
|
||||
(pyspy)=
|
||||
#### Profiling with PySpy
|
||||
If you want to profile the entire codebase, we provide functionality to generate flame graphs or get real-time reports using [PySpy](https://github.com/benfred/py-spy).
|
||||
|
||||
@@ -58,6 +116,4 @@ In both cases, `<python_command>` can be replaced with any command you would nor
|
||||
|
||||
```bash
|
||||
./scripts/debugging/profile.sh -t flame -o flame.svg -p scripts/run_train.py data/dataset=blender_lego
|
||||
```
|
||||
|
||||
## Debugging
|
||||
```
|
||||
@@ -0,0 +1,11 @@
|
||||
Data Setup
|
||||
----------
|
||||
|
||||
# TODO: high level summary
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
choosing_dataset.md
|
||||
creating_dataset.md
|
||||
custom_dataset_format.md
|
||||
@@ -1,9 +0,0 @@
|
||||
Data Setup
|
||||
-----------
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
data/choosing_dataset.md
|
||||
data/creating_dataset.md
|
||||
data/custom_dataset_format.md
|
||||
@@ -148,4 +148,4 @@ python scripts/run_train.py --config-name vanilla_nerf
|
||||
|
||||
And now you have a brand-new NeRF in training! For testing and visualizing, simply refer to steps 4-5 in the [quickstart guide](https://github.com/plenoptix/nerfactory#quickstart).
|
||||
|
||||
To help you get started, we also provide additional training tools such as profiling, logging, and debugging. Please refer to our [features guide](../tooling/index.rst).
|
||||
To help you get started, we also provide additional training tools such as profiling, logging, and debugging. Please refer to our [features guide](../../tooling/index.rst).
|
||||
@@ -0,0 +1,10 @@
|
||||
Creating custom pipelines
|
||||
-------------------------
|
||||
|
||||
# TODO: high level summary
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
system_overview.md
|
||||
creating_graphs.md
|
||||
@@ -0,0 +1,4 @@
|
||||
# System Overview
|
||||
|
||||
TODO: something with a higher-level node-graph image for wiring together the models
|
||||
TODO: maybe also description of what the pipeline is constructed of
|
||||
@@ -88,7 +88,7 @@ Run a faster version with instant ngp using tcnn (without the viewer).
|
||||
python scripts/run_train.py --config-name=graph_instant_ngp.yaml
|
||||
```
|
||||
|
||||
Run with the viewer. However, you'll have to start the viewer server first. (See the docs.)
|
||||
Run with the viewer. However, you'll have to start the viewer server first. (See [viewer docs](../tutorials/viewer/viewer_quickstart.md))
|
||||
|
||||
```bash
|
||||
python scripts/run_train.py --config-name=graph_instant_ngp.yaml viewer.enable=true
|
||||
@@ -113,6 +113,24 @@ bash environment/run_tensorboard.sh
|
||||
|
||||
# Rendering a Trajectory
|
||||
|
||||
To evaluate the trained NeRF, we provide an evaluation script that allows you to do benchmarking (see our [benchmarking workflow](../tooling/benchmarking.md)) or to render out the scene with a custom trajectory and save the output to a video.
|
||||
|
||||
```bash
|
||||
python scripts/run_eval.py --method=traj --traj=spiral --output-filename=output.mp4 --config-name=graph_instant_ngp.yaml trainer.resume_train.load_dir=outputs/blender_lego/instant_ngp/2022-07-07_230905/checkpoints
|
||||
```
|
||||
|
||||
Please note, this quickstart allows you to preform everything in a headless manner. We also provide a web-based visualizer that allows you to easily monitor training or render out trajectories. See our [viewer docs](../tutorials/viewer/viewer_quickstart.md) for more.
|
||||
|
||||
# FAQ
|
||||
|
||||
* [TinyCUDA installation errors out with cuda mismatch](tiny-cuda-error)
|
||||
|
||||
(tiny-cuda-error)=
|
||||
#### TinyCUDA installation errors out with cuda mismatch
|
||||
|
||||
While installing tiny-cuda, you run into: `The detected CUDA version mismatches the version that was used to compile PyTorch (10.2). Please make sure to use the same CUDA versions.`
|
||||
|
||||
**Solution**:
|
||||
```
|
||||
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 -f https://download.pytorch.org/whl/torch_stable.html
|
||||
```
|
||||
@@ -24,9 +24,12 @@ The library supports a more interpretable implementation of NeRFs by modularizin
|
||||
With more modular NeRFs, not only does your code become far more user-friendly, but using this framework also makes it easier for the community to build upon your implementation.
|
||||
|
||||
|
||||
# TODO: high level summary of what is in the quickstart tabs.
|
||||
|
||||
Quickstart
|
||||
-----------
|
||||
|
||||
.. toctree::
|
||||
|
||||
quickstart.md
|
||||
:maxdepth: 1
|
||||
|
||||
quickstart.md
|
||||
@@ -1 +0,0 @@
|
||||
# System Overview
|
||||
@@ -0,0 +1,21 @@
|
||||
# FAQ
|
||||
|
||||
* [Engine node incompatible](engine-error)
|
||||
|
||||
(engine-error)=
|
||||
#### Engine node incompatible
|
||||
|
||||
While running `yarn install`, you run into: `The engine "node" is incompatible with this module.`
|
||||
|
||||
**Solution**:
|
||||
|
||||
Install nvm with instructions at [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
|
||||
|
||||
```shell
|
||||
nvm install 17.8.0
|
||||
```
|
||||
|
||||
If you cannot install nvm, try ignoring the engines
|
||||
```
|
||||
yarn install --ignore-engines
|
||||
```
|
||||
@@ -1,30 +1,13 @@
|
||||
# Real-time Viewer
|
||||
# For developers
|
||||
|
||||
We provide a real-time viewer that can be used during training or to view a trained model. The purpose of the viewer is to be able to freely move the camera and see the reconstruction from novel viewpoints. It can also be used to draw various primitives (e.g., bounding boxes, camera frustums, etc.).
|
||||
|
||||
The viewer is built using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine. The following figure helps to illustrate how our viewer framework works:
|
||||
|
||||

|
||||
|
||||
## Quick start
|
||||
|
||||
WOP: This sections is not ready. Please proceed to the "For developers" section below.
|
||||
|
||||
1. Open the Client App Viewer at ... `TODO(ethan): host the static website`.
|
||||
|
||||
2. Run training script with viewer enabled. Note that you should only use the viewer with a fast pipeline, such as instant ngp.
|
||||
```shell
|
||||
python scripts/run_train.py --config-name=graph_instant_ngp.yaml viewer.enable=True
|
||||
```
|
||||
|
||||
<hr>
|
||||
|
||||
## For developers
|
||||
The tutorial below shows you how to host the viewer yourself and how you can use it to visualize training.
|
||||
|
||||
(nerfactory_bridge)=
|
||||
#### Connection between nerfactory and the Bridge Server
|
||||
|
||||
In the center, we have the Bridge Server, which facilitates the connection between nerfactory code and the Client App. This server runs on the same machine that you are using nerfactory. It has a TCP Request/Reply (REQ/REP) connection that nerfactory can connect to with the Viewer object (left). We use [ZeroMQ](https://zeromq.org/), an open-sourced messaging library, to implement this lightweight TCP connection. The Viewer class can send commands to the Bridge Server and receive replies. The Bridge Server will either dispatch commands to the Client App via a websocket or it will return information stored in the Bridge Server state.
|
||||
|
||||
(bridge_client)=
|
||||
#### Connection between the Bridge Server and the Client App
|
||||
|
||||
The connection between the Bridge Server and the Client App works with WebSockets and WebRTC.
|
||||
@@ -33,8 +16,10 @@ The connection between the Bridge Server and the Client App works with WebSocket
|
||||
|
||||
- **WebRTC connection** - We use WebRTC to stream images being rendered from nerfactory. The websocket connection if first used to establish the WebRTC connection. Then, the Client App constantly publishes camera pose information to the Bridge Server and stores the camera information (intrinsics and extrinsics). This information is then queried from the nerfactory code, used to render an image with some Graph, and then the image is send over the TCP connection and dispatched via WebRTC to render the stream of images.
|
||||
|
||||
(getting_started)=
|
||||
## Getting started
|
||||
|
||||
(bridge)=
|
||||
#### Running the Bridge Server
|
||||
|
||||
The viewer server runs on the same machine that you use for training. The training code will connect to the server with a lightweight TCP connection.
|
||||
@@ -50,6 +35,7 @@ It should print out something of the form:
|
||||
"ZMQWebSocketBridge using zmq_url=tcp://127.0.0.1:6000 and websocket_port=8051"
|
||||
```
|
||||
|
||||
(client)=
|
||||
#### Running the Client App
|
||||
|
||||
> We will host the viewer online in the future, but for now we have to run it locally.
|
||||
@@ -66,7 +52,7 @@ npm install --global yarn
|
||||
```
|
||||
|
||||
Install nvm and set the node version
|
||||
Install nvm with instructions at e.g., https://heynode.com/tutorial/install-nodejs-locally-nvm/
|
||||
Install nvm with [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
|
||||
|
||||
```shell
|
||||
nvm install 17.8.0
|
||||
@@ -93,10 +79,18 @@ PORT=4000
|
||||
ESLINT_NO_DEV_ERRORS=true
|
||||
```
|
||||
|
||||
(nerfactory)=
|
||||
#### Running the nerfactory Code
|
||||
|
||||
- **Notebook demo** - See `docs/notebooks/visualize_viewer.ipynb` for an overview for how to interact with the viewer with the Viewer object from nerfactory.
|
||||
You can now simply run a training job and visualize progress by enabling the viewer during training:
|
||||
|
||||
```bash
|
||||
python scripts/run_train.py --config-name=graph_instant_ngp.yaml viewer.enable=true
|
||||
```
|
||||
|
||||
- **Notebook demo** - See [Programming the viewer](viewer_notebook.ipynb) for an overview for how to interact with the viewer with the Viewer object from nerfactory.
|
||||
|
||||
(ack)=
|
||||
## Acknowledgements and references
|
||||
|
||||
We thank [Robin Deits](https://github.com/rdeits) and other contributors to the following repos, which we've started with and modified and extended for our use.
|
||||
@@ -112,4 +106,4 @@ Here are other resources that we've used and/or have found helpful while buildin
|
||||
|
||||
```
|
||||
|
||||
```
|
||||
```
|
||||
File renamed without changes.
|
Before Width: | Height: | Size: 286 KiB After Width: | Height: | Size: 286 KiB |
File renamed without changes.
|
Before Width: | Height: | Size: 218 KiB After Width: | Height: | Size: 218 KiB |
@@ -0,0 +1,12 @@
|
||||
Real-time web viewer
|
||||
====================
|
||||
|
||||
# TODO: high level summary
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
viewer_quickstart.md
|
||||
for_developers.md
|
||||
viewer_notebook.ipynb
|
||||
faq.md
|
||||
@@ -5,7 +5,7 @@
|
||||
"id": "3c00934d-0f14-4c1a-a563-94d627b377e5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Visualize with our Viewer\n",
|
||||
"# Programming the viewer\n",
|
||||
"\n",
|
||||
"The real-time web viewer enables communication between Python and the ReactJS Client Viewer Application. Here we provide a walkthrough for how to use the viewer to draw information as well as read information from the GUI.\n"
|
||||
]
|
||||
@@ -512,7 +512,7 @@
|
||||
},
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "e691f033e0f2f1b9c0a11e7e81375a1e27aec87a71fd1b1eaa545f7a5e61b3f6"
|
||||
"hash": "100252972d4ee5286ab455ef1c778abde05b4affca9d1be78633d3797abf7e8c"
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -0,0 +1,22 @@
|
||||
# Quickstart
|
||||
|
||||
We provide a real-time, web-based viewer that can be used during training or to view a trained model. The purpose of the viewer is to be able to freely move the camera and see the reconstruction from novel viewpoints. It can also be used to draw various primitives (e.g., bounding boxes, camera frustums, etc.).
|
||||
|
||||
The viewer is built using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine. The following figure helps to illustrate how our viewer framework works:
|
||||
|
||||

|
||||
|
||||
|
||||
## Run hosted web viewer
|
||||
[WOP: This sections is not ready. Please proceed to the "For developers" section.]
|
||||
|
||||
1. Open the Client App Viewer at ... `TODO(ethan): host the static website`.
|
||||
|
||||
2. Run training script with viewer enabled. Note that you should only use the viewer with a fast pipeline, such as instant ngp.
|
||||
```shell
|
||||
python scripts/run_train.py --config-name=graph_instant_ngp.yaml viewer.enable=True
|
||||
```
|
||||
|
||||
This pulls up a hosted web-viewer which is the fastest way to get started with visualizing your nerfactory training progress!
|
||||
|
||||
If you want to host the viewer locally/yourself, please proceed to "For devleopers" section.
|
||||
Reference in new issue
Block a user