tink/nerfstudio
4
0 Fork 0
mirror of https://github.com/nerfstudio-project/nerfstudio.git synced 2026-11-13 11:24:00 +08:00
Code Activity

Restructure Docs

Browse Source
This commit is contained in:
Matt Tancik authored and Matthew Tancik committed 2022-07-15 09:49:33 -07:00
1 parent 2f2919535a
commit 2a23763e7b
10 files changed
+189 -31
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/tutorials/creating_graphs.md
+10 -10
View File
@@ -1,4 +1,4 @@
# Deploying a new NeRF graph
# Developing a Custom Model
In this quick tour, we will walk you through the core of training and building any NeRFs with pyrad.
@@ -11,21 +11,22 @@ If you are looking to implemnet a new NeRF method or extend an existing one, you
All NeRF graph definitions can be found in `graphs/`. The graph encapsulates all the modules in the NeRF method- from the MLP's to the samplers to the losses and optimizers. All graphs are composed of modules and fields.
### Modules
We can decompose all graphs into respective modules. For instance, we have {ref}`optimizers` which contains modules related to the loss, {ref}`renderers` which contains all rendering modules, {`graph/modules/`}, which includes samplers and ray generators. Additionally, we have {ref}`field_modules`, which will be discussed below.
In general, modules can be thought of as individual component parts of the graph that can be swapped in/out. We provide most of the standard modules, but if you are creating a new NeRF graph, you may have to define new modules.
### Fields
{ref}`fields` represents the "space", aka. the radiance field of the NeRF. Here, we define the field as the part of the network that takes in point samples and any other conditioning, and outputs any of the `FieldHeadNames` (`nerf/field_modules/field_heads.py`).
{ref}`fields` represents the "space", aka. the radiance field of the NeRF. Here, we define the field as the part of the network that takes in point samples and any other conditioning, and outputs any of the `FieldHeadNames` (`nerf/field_modules/field_heads.py`).
All fields are composed of modules continained within the `fields/` directory (e.g. `fields/modules/` or `fields/occupancy_fields/`). We can think of field modules as modules that actually define or interact with the field.
All fields are composed of modules continained within the `fields/` directory (e.g. `fields/modules/` or `fields/occupancy_fields/`). We can think of field modules as modules that actually define or interact with the field.
To build a NeRF field, we therefore follow these steps:
1. define any relevant field modules
2. create a new class that extends the {ref}`fields` base class and use these field modules to compose the full field
3. implement the abstract functions `get_density()` and `get_outputs()`.
```python
class NeRFField(Field):
"""NeRF Field"""
@@ -36,7 +37,7 @@ class NeRFField(Field):
super().__init__()
# In the init method, we are instantiating all relevant field modules
self.position_encoding = position_encoding
self.position_encoding = position_encoding
self.direction_encoding = direction_encoding
self.mlp_base = MLP(
@@ -62,9 +63,9 @@ class NeRFField(Field):
### Putting it together
Now that we've gone over what modules and fields are, we are ready to create the graph!
Now that we've gone over what modules and fields are, we are ready to create the graph!
To implement the vanilla NeRF, we create a new class that inherits the abstract Graph class. Similar to composing a new field, we can pull from all pre-defined fields and modules and use them to compose a new Graph.
To implement the vanilla NeRF, we create a new class that inherits the abstract Graph class. Similar to composing a new field, we can pull from all pre-defined fields and modules and use them to compose a new Graph.
We will then need to define the following abstract methods as seen in the skeleton code below. See also `graphs/vanilla_nerf.py` for the full implementation.
```python
@@ -105,7 +106,6 @@ class NeRFGraph(Graph):
"""
```
## Creating the config
Now that you have the graph and dataset all set up, you'll need to create the config that you pass into our run train routine. Our config system is powered by [Hydra](https://hydra.cc/). All Hydra and machine related arguments are stored in `configs/default_setup.yaml`, as well as the defaults list.
@@ -131,7 +131,7 @@ data:
# <insert any dataset related overrides here>
graph:
_target_: pyrad.graph.vanilla_nerf.NeRFGraph # set the target to the graph you defined
_target_: pyrad.graph.vanilla_nerf.NeRFGraph # set the target to the graph you defined
# <insert any graph related overrides here>
optimizers:
docs/tutorials/data/media/viewer_figure.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 218 KiB

docs/tutorials/data_setup.rst → docs/tutorials/data_index.rst
-1
View File
@@ -1,4 +1,3 @@
Data Setup
-----------
docs/tutorials/quickstart.md
+115 -7
View File
@@ -1,10 +1,118 @@
# Quickstart
# Installation
```{include} ../../README.md
:relative-docs: docs/
:relative-images:
:start-after: quick_tour.html).
:end-before: 6. In-depth
This repository is tested with CUDA 11.3. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html#linux-installers) before preceding.
<details>
<summary>Installing Conda</summary>
This step is fairly self-explanatory, but here are the basic steps. You can also find countless tutorials online.
```bash
cd /path/to/install/miniconda
mkdir -p miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda3/miniconda.sh
bash miniconda3/miniconda.sh -b -u -p miniconda3
rm -rf miniconda/miniconda.sh
```
</details>
Create the python environment
```bash
conda create --name pyrad python=3.8.13
conda activate pyrad
```
For more complex changes (e.g. running with your own data/ setting up a new NeRF graph, please continue to the next page.
Clone the repo
```bash
git clone git@github.com:plenoptix/pyrad.git
```
Install dependencies
```bash
cd pyrad
pip install -r environment/requirements.txt
```
Install pyrad as a library
```bash
pip install -e .
```
Install library with CUDA development support. Change setup.py to `USE_CUDA = True` and then
```bash
python setup.py develop
```
Install tiny-cuda-nn (tcnn) to use with the graph_instant_ngp.yaml config
```bash
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
```
# Downloading data
Download the original [NeRF dataset](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) and unfold it in the following format. This is for the blender dataset type. We support the major datasets and allow users to create their own dataset, described in detail [here](data/creating_dataset.md).
```
|─ pyrad/
├─ data/
| ├─ blender/
| ├─ fern/
| ├─ lego/
...
|- <dataset_format>/
|- <scene>
...
```
# Training a model
To run with all the defaults, e.g. vanilla nerf method with the blender lego image
Run a vanilla nerf model.
```bash
python scripts/run_train.py
```
Run a faster version with instant ngp using tcnn (without the viewer).
```bash
python scripts/run_train.py --config-name=graph_instant_ngp.yaml
```
Run with the viewer. However, you'll have to start the viewer server first. (See the docs.)
```bash
python scripts/run_train.py --config-name=graph_instant_ngp.yaml viewer.enable=true
```
With support for [Hydra](https://hydra.cc/), you can run with other configurations by changing appropriate configs defined in `configs/` or by setting flags via command-line arguments.
# Visualizing training runs
If you run everything with the default configuration, by default, we use [TensorBoard](https://www.tensorflow.org/tensorboard) to log all training curves, test images, and other stats. Once the job is launched, you will be able to track training by launching the tensorboard in `outputs/blender_lego/vanilla_nerf/<timestamp>/<events.tfevents>`.
```bash
tensorboard --logdir outputs
```
Or the following:
```bash
export TENSORBOARD_PORT=<port>
bash environment/run_tensorboard.sh
```
# Rendering a Trajectory
```bash
python scripts/run_eval.py --method=traj --traj=spiral --output-filename=output.mp4 --config-name=graph_instant_ngp.yaml trainer.resume_train.load_dir=outputs/blender_lego/instant_ngp/2022-07-07_230905/checkpoints
```
docs/tutorials/quickstart_index.rst
+19
View File
@@ -0,0 +1,19 @@
pyRad |:metal:|
====================================
Neural Volumetric Rendering
"""""""""""""""""""""""""""""
All-in-one repository for state-of-the-art NeRFs.
pyRad |:metal:| provides a simple API that allows for a seamless and simplified end-to-end process of creating, training, and testing NeRFs.
The library supports a more interpretable implementation of NeRFs by modularizing each component.
With more modular NeRFs, not only does your code become far more user-friendly, but using this framework also makes it easier for the community to build upon your implementation.
Quickstart
-----------
.. toctree::
quickstart.md
docs/tutorials/system_overview.md
+1
View File
@@ -0,0 +1 @@
# System Overview
docs/tutorials/viewer.md
-104
View File
@@ -1,104 +0,0 @@
# Real-time Viewer
We provide a real-time viewer that can be used during training or to view a trained model. The purpose of the viewer is to be able to freely move the camera and see the reconstruction from novel viewpoints. It can also be used to draw various primitives (e.g., bounding boxes, camera frustums, etc.).
The viewer is built using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine. The following figure helps to illustrate how our viewer framework works:
![visualize_dataset](data/media/viewer_figure.png)
## Quick start
```
# Run bridge server in the background.
python scripts/run_viewer_bridge_server.py
# Run training script with viewer enabled.
# note that running with instant ngp if much faster than vanilla nerf
python scripts/run_train.py --config-name=graph_instant_ngp.yaml viewer.enable=True
```
<hr>
## For developers
#### Connection between PyRad and the Bridge Server
In the center, we have the Bridge Server, which facilitates the connection between PyRad code and the Client App. This server runs on the same machine that you are using PyRad. It has a TCP Request/Reply (REQ/REP) connection that PyRad can connect to with the Viewer object (left). We use [ZeroMQ](https://zeromq.org/), an open-sourced messaging library, to implement this lightweight TCP connection. The Viewer class can send commands to the Bridge Server and receive replies. The Bridge Server will either dispatch commands to the Client App via a websocket or it will return information stored in the Bridge Server state.
#### Connection between the Bridge Server and the Client App
The connection between the Bridge Server and the Client App works with WebSockets and WebRTC.
- **WebSocket connection** - The WebSocket is used by the Bridge Server to dispatch commands coming from the PyRad TCP connection. Commands can be used for drawing primitives in the [JSON Object Scene format](https://github.com/mrdoob/three.js/wiki/JSON-Object-Scene-format-4), for setting the transform of objects, for the setting various properties, and more.
- **WebRTC connection** - We use WebRTC to stream images being rendered from PyRad. The websocket connection if first used to establish the WebRTC connection. Then, the Client App constantly publishes camera pose information to the Bridge Server and stores the camera information (intrinsics and extrinsics). This information is then queried from the PyRad code, used to render an image with some Graph, and then the image is send over the TCP connection and dispatched via WebRTC to render the stream of images.
## Getting started
#### Running the Bridge Server
The viewer server runs on the same machine that you use for training. The training code will connect to the server with a lightweight TCP connection.
```
cd pyrad
# run the server on your machine
# this will run in the background
python scripts/run_viewer_server.py
It should print out something of the form:
"ZMQWebSocketBridge using zmq_url=tcp://127.0.0.1:6000 and websocket_port=8051"
```
#### Running the Client App
> We will host the viewer online in the future, but for now we have to run it locally.
```
cd pyard
cd pyrad/viewer/app
# install yarn and package.json dependencies
npm install --global yarn
yarn install
yarn start
# forward port and open in your local browser
# the URL takes the following form
# http://localhost:<forwarded_react_port>?localhost:<forwarded_backend_tcp_port>
http://localhost:4000?localhost:8051
```
Possibly edit the port in `app/.env.development` file with the following:
```
BROWSER=none
FAST_REFRESH=false
HOST=localhost
PORT=4000
```
#### Running the PyRad Code
- **Notebook demo** - See `notebooks/visualize_viewer.ipynb` for an overview for how to interact with the viewer with the Viewer object from PyRad.
## Features checklist
- [x] WebRTC video stream
- [x] Dynamic Resolution Scaling (DRS)
- [ ] Draw camera frustums from the DatasetInputs
## Acknowledgements and references
We thank [Robin Deits](https://github.com/rdeits) and other contributors to the following repos, which we've started with and modified and extended for our use.
- [meshcat-python](https://github.com/rdeits/meshcat-python)
- [meshcat](https://github.com/rdeits/meshcat)
Here are other resources that we've used and/or have found helpful while building the viewer code.
- [ThreeJS](https://threejs.org/)
- [ReactJS](https://reactjs.org/)
- [WebRTC](https://webrtc.org/) - WebRTC is a framework for real-time communication that allows two peers to send video, audio, or general data to and from each other with low latency. We've adopted WebRTC to stream rendered images to our viewer.