mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-12 02:45:34 +08:00
Rename to nerfstudio (#589)
This commit is contained in:
Matthew Tancik authored and 
GitHub committed
1 parent
fae01d3ca6
commit
1f83ad03db
210 files changed
+3400
-3413
No files matched your search
@@ -4,7 +4,7 @@ Training model on existing datasets is only so fun. If you would like to train o
|
||||
|
||||
## Nerfactory dataset
|
||||
|
||||
To assist running on custom data we have a script that will process a video or folder of images into a format that is compatible with nerfactory. We use [COLMAP](https://colmap.github.io) and [FFmpeg](https://ffmpeg.org/download.html) in our data processing script, please have these installed. We have provided a quickstart to installing COLMAP below, FFmpeg can be downloaded from [here](https://ffmpeg.org/download.html)
|
||||
To assist running on custom data we have a script that will process a video or folder of images into a format that is compatible with nerfstudio. We use [COLMAP](https://colmap.github.io) and [FFmpeg](https://ffmpeg.org/download.html) in our data processing script, please have these installed. We have provided a quickstart to installing COLMAP below, FFmpeg can be downloaded from [here](https://ffmpeg.org/download.html)
|
||||
|
||||
To process your data run:
|
||||
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
# Custom dataset formats
|
||||
|
||||
We've implemented many common datasets inside the `nerfactory/format/<dataset_format>.py` files. If you have a posed image dataset that does not match these existing formats can create a custom dataset format.
|
||||
We've implemented many common datasets inside the `nerfstudio/format/<dataset_format>.py` files. If you have a posed image dataset that does not match these existing formats can create a custom dataset format.
|
||||
|
||||
1. Create a function `nerfactory/format/<dataset_format>.py` and implements a function called `load_<dataset_format>_data` returns a `DatasetInputs` object.
|
||||
1. Create a function `nerfstudio/format/<dataset_format>.py` and implements a function called `load_<dataset_format>_data` returns a `DatasetInputs` object.
|
||||
|
||||
```python
|
||||
# The DatasetInputs dataclass that is passed around to create datasets and initialize the graphs.
|
||||
# See `nerfactory/data/structs.py` for the code.
|
||||
# See `nerfstudio/data/structs.py` for the code.
|
||||
@dataclass
|
||||
class DatasetInputs:
|
||||
"""Dataset inputs are used to initialize datasets and the NeRF graph."""
|
||||
@@ -22,7 +22,7 @@ class DatasetInputs:
|
||||
alpha_color: Optional[TensorType[3]] = None
|
||||
|
||||
# The method signature for the `instant_ngp` dataset format.
|
||||
# See `nerfactory/data/format/instant_ngp.py` for the code.
|
||||
# See `nerfstudio/data/format/instant_ngp.py` for the code.
|
||||
def load_instant_ngp_data(
|
||||
basedir: str,
|
||||
split: str = "train", camera_translation_scalar=0.33
|
||||
@@ -30,12 +30,12 @@ def load_instant_ngp_data(
|
||||
"""Returns a DatasetInputs struct."""
|
||||
```
|
||||
|
||||
2. Add a new case to `get_dataset_inputs()` in `nerfactory/data/utils.py` for the new dataset.
|
||||
2. Add a new case to `get_dataset_inputs()` in `nerfstudio/data/utils.py` for the new dataset.
|
||||
|
||||
```python
|
||||
# The method signature for get_dataset_inputs.
|
||||
# Notice that the parameters match the yaml config parameters above.
|
||||
# See `nerfactory/data/utils.py` for the code.
|
||||
# See `nerfstudio/data/utils.py` for the code.
|
||||
def get_dataset_inputs(
|
||||
data_directory: str,
|
||||
dataset_format: str,
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
"source": [
|
||||
"# Developing Custom Models\n",
|
||||
"\n",
|
||||
"In this quick tour, we will walk you through the core of training and building any NeRFs with nerfactory.\n",
|
||||
"In this quick tour, we will walk you through the core of training and building any NeRFs with nerfstudio.\n",
|
||||
"\n",
|
||||
"The entry point for training starts at `scripts/train.py`, which spawns instances of our `Trainer()` class (in `engine/trainer.py`). The `Trainer()` is responsible for setting up the datasets and NeRF graph depending on the config specified. It will then run the usual train/val routine for a config-specified number of iterations. If you are planning on using our codebase to build a new NeRF method or to use an existing implementation, we've abstracted away the training routine in these two files and chances are you will not need to think of them again.\n",
|
||||
"\n",
|
||||
@@ -178,7 +178,7 @@
|
||||
" # <insert any dataset related overrides here>\n",
|
||||
"\n",
|
||||
"graph:\n",
|
||||
" _target_: nerfactory.graph.vanilla_nerf.NeRFGraph # set the target to the graph you defined\n",
|
||||
" _target_: nerfstudio.graph.vanilla_nerf.NeRFGraph # set the target to the graph you defined\n",
|
||||
" # <insert any graph related overrides here>\n",
|
||||
"\n",
|
||||
"optimizers:\n",
|
||||
@@ -193,7 +193,7 @@
|
||||
"python scripts/train.py --config-name vanilla_nerf\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"And now you have a brand-new NeRF in training! For testing and visualizing, simply refer to steps 4-5 in the [quickstart guide](https://github.com/plenoptix/nerfactory#quickstart).\n",
|
||||
"And now you have a brand-new NeRF in training! For testing and visualizing, simply refer to steps 4-5 in the [quickstart guide](https://github.com/plenoptix/nerfstudio#quickstart).\n",
|
||||
"\n",
|
||||
"To help you get started, we also provide additional training tools such as profiling, logging, and debugging. Please refer to our [features guide](../../tooling/index.rst)."
|
||||
]
|
||||
|
||||
@@ -8,9 +8,9 @@
|
||||
"source": [
|
||||
"# Creating Custom Pipelines\n",
|
||||
"\n",
|
||||
"> Our goal is to enable Nerfactory users to implement an entire NeRF method by only editing one file.\n",
|
||||
"> Our goal is to enable nerfstudio users to implement an entire NeRF method by only editing one file.\n",
|
||||
"\n",
|
||||
"Here we explain how to create custom pipelines with nerfactory. Pipelines are composed of two components, namely a DataManager and a Model. The pipeline is responsible for routing RayBundle instances returned from the DataManager into the Model.\n",
|
||||
"Here we explain how to create custom pipelines with nerfstudio. Pipelines are composed of two components, namely a DataManager and a Model. The pipeline is responsible for routing RayBundle instances returned from the DataManager into the Model.\n",
|
||||
"\n",
|
||||
"A NeRF method is implemented as a Pipeline. Users can create a Pipeline with a DataManager and Model. We provide default implementations for these components prefixed with the name Vanilla (e.g., `VanillaPipeline`, `VanillaDataManager` which inherit from the `Pipeline` and `DataManager` class, respectively) and specific Models (e.g., `InstantNGPModel` or `TensoRFModel` which inherit from the `Model` class). With most NeRF methods, `VanillaPipeline` and `VanillaDataManager` can be used and only the model needs to be changed. We provide many already implmented Models for people to get started with 3D reconstruction. However, in this tutorial we will demonstrate how to customize each of the Pipeline, DataManager, and Model with an example.\n",
|
||||
"\n",
|
||||
@@ -116,13 +116,13 @@
|
||||
"from typing import Dict, Optional, Tuple, List\n",
|
||||
"import torch\n",
|
||||
"\n",
|
||||
"from nerfactory.utils.config import DataManagerConfig, ModelConfig, PipelineConfig, ViewerConfig\n",
|
||||
"from nerfactory.datamanagers.base import DataManager\n",
|
||||
"from nerfactory.datamanagers.structs import DatasetInputs\n",
|
||||
"from nerfactory.models.base import Model\n",
|
||||
"from nerfactory.pipelines.base import Pipeline\n",
|
||||
"from nerfactory.models.instant_ngp import NGPModel\n",
|
||||
"from nerfactory.cameras.rays import RayBundle"
|
||||
"from nerfstudio.utils.config import DataManagerConfig, ModelConfig, PipelineConfig, ViewerConfig\n",
|
||||
"from nerfstudio.datamanagers.base import DataManager\n",
|
||||
"from nerfstudio.datamanagers.structs import DatasetInputs\n",
|
||||
"from nerfstudio.models.base import Model\n",
|
||||
"from nerfstudio.pipelines.base import Pipeline\n",
|
||||
"from nerfstudio.models.instant_ngp import NGPModel\n",
|
||||
"from nerfstudio.cameras.rays import RayBundle"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -268,7 +268,7 @@
|
||||
" model: ModelConfig = MISSING\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"See `nerfactory/utils/config.py` for more details. In this example, we will simply load from an existing configuration from `configs/graph_instant_ngp.yaml`."
|
||||
"See `nerfstudio/utils/config.py` for more details. In this example, we will simply load from an existing configuration from `configs/graph_instant_ngp.yaml`."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -292,7 +292,7 @@
|
||||
"pprint.pprint(pipeline_config)\n",
|
||||
"print(\"----------------------------------------------------\")\n",
|
||||
"\n",
|
||||
"# from nerfactory.pipelines.base import setup_pipeline\n",
|
||||
"# from nerfstudio.pipelines.base import setup_pipeline\n",
|
||||
"# pipeline = setup_pipeline(pipeline_config, device=\"cuda\")\n",
|
||||
"pipeline = pipeline_config.setup()"
|
||||
]
|
||||
|
||||
@@ -7,21 +7,21 @@ This repository is tested with CUDA 11.3. Make sure to install [Conda](https://d
|
||||
Create the python environment
|
||||
|
||||
```bash
|
||||
conda create --name nerfactory python=3.8.13
|
||||
conda activate nerfactory
|
||||
conda create --name nerfstudio python=3.8.13
|
||||
conda activate nerfstudio
|
||||
python -m pip install --upgrade pip
|
||||
```
|
||||
|
||||
Clone the repo
|
||||
|
||||
```bash
|
||||
git clone git@github.com:plenoptix/nerfactory.git
|
||||
git clone git@github.com:plenoptix/nerfstudio.git
|
||||
```
|
||||
|
||||
Install dependencies and nerfactory as a library
|
||||
Install dependencies and nerfstudio as a library
|
||||
|
||||
```bash
|
||||
cd nerfactory
|
||||
cd nerfstudio
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
@@ -32,7 +32,7 @@ pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.py
|
||||
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
|
||||
```
|
||||
|
||||
Install tab completion for nerfactory (bash and zsh)
|
||||
Install tab completion for nerfstudio (bash and zsh)
|
||||
|
||||
```bash
|
||||
# If the CLI changes, we can also re-install to synchronize completions
|
||||
@@ -50,7 +50,7 @@ python scripts/downloads/download_data.py --dataset=blender
|
||||
Use `--help` to view all currently available datasets. The resulting script should download and unpack the dataset as follows:
|
||||
|
||||
```
|
||||
|─ nerfactory/
|
||||
|─ nerfstudio/
|
||||
├─ data/
|
||||
| ├─ blender/
|
||||
| ├─ fern/
|
||||
@@ -127,7 +127,7 @@ pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.py
|
||||
|
||||
#### Installation errors, File "setup.py" not found
|
||||
|
||||
When installing dependencies and nerfactory with `pip install -e .`, you run into: `ERROR: File "setup.py" not found. Directory cannot be installed in editable mode`
|
||||
When installing dependencies and nerfstudio with `pip install -e .`, you run into: `ERROR: File "setup.py" not found. Directory cannot be installed in editable mode`
|
||||
|
||||
**Solution**:
|
||||
This can be fixed by upgrading pip to the latest version:
|
||||
|
||||
@@ -4,13 +4,13 @@
|
||||
.. image:: ../_static/imgs/logo.png
|
||||
:width: 400
|
||||
:align: center
|
||||
:alt: nerfactory
|
||||
:alt: nerfstudio
|
||||
:class: only-light
|
||||
|
||||
.. image:: ../_static/imgs/logo-dark.png
|
||||
:width: 400
|
||||
:align: center
|
||||
:alt: nerfactory
|
||||
:alt: nerfstudio
|
||||
:class: only-dark
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ Neural Volumetric Rendering
|
||||
|
||||
All-in-one repository for state-of-the-art NeRFs.
|
||||
|
||||
nerfactory provides a simple API that allows for a seamless and simplified end-to-end process of creating, training, and testing NeRFs.
|
||||
nerfstudio provides a simple API that allows for a seamless and simplified end-to-end process of creating, training, and testing NeRFs.
|
||||
The library supports a more interpretable implementation of NeRFs by modularizing each component.
|
||||
With more modular NeRFs, not only does your code become far more user-friendly, but using this framework also makes it easier for the community to build upon your implementation.
|
||||
|
||||
|
||||
@@ -2,17 +2,17 @@
|
||||
|
||||
The tutorial below shows you how to host the viewer yourself and how you can use it to visualize training.
|
||||
|
||||
#### Connection between nerfactory and the Bridge Server
|
||||
#### Connection between nerfstudio and the Bridge Server
|
||||
|
||||
In the center, we have the Bridge Server, which facilitates the connection between nerfactory code and the Client App. This server runs on the same machine that you are using nerfactory. It has a TCP Request/Reply (REQ/REP) connection that nerfactory can connect to with the Viewer object (left). We use [ZeroMQ](https://zeromq.org/), an open-sourced messaging library, to implement this lightweight TCP connection. The Viewer class can send commands to the Bridge Server and receive replies. The Bridge Server will either dispatch commands to the Client App via a websocket or it will return information stored in the Bridge Server state.
|
||||
In the center, we have the Bridge Server, which facilitates the connection between nerfstudio code and the Client App. This server runs on the same machine that you are using nerfstudio. It has a TCP Request/Reply (REQ/REP) connection that nerfstudio can connect to with the Viewer object (left). We use [ZeroMQ](https://zeromq.org/), an open-sourced messaging library, to implement this lightweight TCP connection. The Viewer class can send commands to the Bridge Server and receive replies. The Bridge Server will either dispatch commands to the Client App via a websocket or it will return information stored in the Bridge Server state.
|
||||
|
||||
#### Connection between the Bridge Server and the Client App
|
||||
|
||||
The connection between the Bridge Server and the Client App works with WebSockets and WebRTC.
|
||||
|
||||
- **WebSocket connection** - The WebSocket is used by the Bridge Server to dispatch commands coming from the nerfactory TCP connection. Commands can be used for drawing primitives, for setting the transform of objects, for the setting various properties, and more.
|
||||
- **WebSocket connection** - The WebSocket is used by the Bridge Server to dispatch commands coming from the nerfstudio TCP connection. Commands can be used for drawing primitives, for setting the transform of objects, for the setting various properties, and more.
|
||||
|
||||
- **WebRTC connection** - We use WebRTC to stream images being rendered from nerfactory. The websocket connection if first used to establish the WebRTC connection. Then, the Client App constantly publishes camera pose information to the Bridge Server and stores the camera information (intrinsics and extrinsics). This information is then queried from the nerfactory code, used to render an image with some Graph, and then the image is send over the TCP connection and dispatched via WebRTC to render the stream of images.
|
||||
- **WebRTC connection** - We use WebRTC to stream images being rendered from nerfstudio. The websocket connection if first used to establish the WebRTC connection. Then, the Client App constantly publishes camera pose information to the Bridge Server and stores the camera information (intrinsics and extrinsics). This information is then queried from the nerfstudio code, used to render an image with some Graph, and then the image is send over the TCP connection and dispatched via WebRTC to render the stream of images.
|
||||
|
||||
## Getting started
|
||||
|
||||
@@ -21,7 +21,7 @@ The connection between the Bridge Server and the Client App works with WebSocket
|
||||
The viewer server runs on the same machine that you use for training. The training code will connect to the server with a lightweight TCP connection.
|
||||
|
||||
```
|
||||
cd nerfactory
|
||||
cd nerfstudio
|
||||
|
||||
# run the server on your machine
|
||||
# this will run in the background
|
||||
@@ -36,7 +36,7 @@ It should print out something of the form:
|
||||
> We will host the viewer online in the future, but for now we have to run it locally.
|
||||
|
||||
```shell
|
||||
cd nerfactory/viewer/app
|
||||
cd nerfstudio/viewer/app
|
||||
```
|
||||
|
||||
Install npm (to install yarn) and yarn
|
||||
@@ -75,7 +75,7 @@ PORT=4000
|
||||
ESLINT_NO_DEV_ERRORS=true
|
||||
```
|
||||
|
||||
#### Running the nerfactory Code
|
||||
#### Running the nerfstudio Code
|
||||
|
||||
You can now simply run a training job and visualize progress by enabling the viewer during training:
|
||||
|
||||
@@ -83,7 +83,7 @@ You can now simply run a training job and visualize progress by enabling the vie
|
||||
python scripts/train.py --config-name=graph_instant_ngp.yaml viewer.enable=true
|
||||
```
|
||||
|
||||
- **Notebook demo** - See [Programming the viewer](viewer_notebook.ipynb) for an overview for how to interact with the viewer with the Viewer object from nerfactory.
|
||||
- **Notebook demo** - See [Programming the viewer](viewer_notebook.ipynb) for an overview for how to interact with the viewer with the Viewer object from nerfstudio.
|
||||
|
||||
## Acknowledgements and references
|
||||
|
||||
|
||||
@@ -38,8 +38,8 @@
|
||||
"import numpy as np\n",
|
||||
"import torch\n",
|
||||
"import mediapy as media\n",
|
||||
"from nerfactory.data.structs import SceneBounds\n",
|
||||
"from nerfactory.cameras.cameras import PinholeCamera\n",
|
||||
"from nerfstudio.data.structs import SceneBounds\n",
|
||||
"from nerfstudio.cameras.cameras import PinholeCamera\n",
|
||||
"import time"
|
||||
]
|
||||
},
|
||||
@@ -122,7 +122,7 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# connect to the viewer from Python\n",
|
||||
"from nerfactory.viewer.server.visualizer import Viewer\n",
|
||||
"from nerfstudio.viewer.server.visualizer import Viewer\n",
|
||||
"\n",
|
||||
"vis = Viewer(zmq_url=\"tcp://127.0.0.1:6000\")"
|
||||
]
|
||||
|
||||
@@ -15,7 +15,7 @@ The viewer is built using [ThreeJS](https://threejs.org/) and packaged into a [R
|
||||
python scripts/train.py --config-name=graph_instant_ngp.yaml viewer.enable=True
|
||||
```
|
||||
|
||||
This pulls up a hosted web-viewer which is the fastest way to get started with visualizing your nerfactory training progress!
|
||||
This pulls up a hosted web-viewer which is the fastest way to get started with visualizing your nerfstudio training progress!
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user