mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-15 04:13:57 +08:00
1 parent
5a76b955cd
commit
d7c34ca2bd
68 files changed
+87
-148
No files matched your search
@@ -0,0 +1,100 @@
|
||||
# Using Custom Data
|
||||
|
||||
Training model on existing datasets is only so fun. If you would like to train on self captured data you will need to process the data into an existing format. Specifically we need to know the camera poses for each image. [COLMAP](https://github.com/colmap/colmap) is a standard tool for extracting poses. It is possible to use other methods like [SLAM](https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping) or hardware recorded poses. We intend to add documentation for these other methods in the future.
|
||||
|
||||
## nerfstudio dataset
|
||||
|
||||
To assist running on custom data we have a script that will process a video or folder of images into a format that is compatible with nerfstudio. We use [COLMAP](https://colmap.github.io) and [FFmpeg](https://ffmpeg.org/download.html) in our data processing script, please have these installed. We have provided a quickstart to installing COLMAP below, FFmpeg can be downloaded from [here](https://ffmpeg.org/download.html)
|
||||
|
||||
To process your data run:
|
||||
|
||||
```bash
|
||||
ns-process-data --data FOLDER_OR_VIDEO --output-dir PROCESSED_DATA_DIR
|
||||
```
|
||||
|
||||
A full set of arguments can be found {doc}`here</reference/cli/ns_process_data>`.
|
||||
|
||||
:::{admonition} Tip
|
||||
:class: info
|
||||
|
||||
- COLMAP can be finicky. Try your best to capture overlapping, non-blurry images.
|
||||
:::
|
||||
|
||||
### Training on your data
|
||||
|
||||
Simply specify that you are using the `nerfstudio` dataparser and point the data directory to your processed data.
|
||||
|
||||
```bash
|
||||
python scripts/train.py instant-ngp nerfstudio-data --pipeline.datamanager.dataparser.data-directory PROCESSED_DATA_DIR
|
||||
```
|
||||
|
||||
### Installing COLMAP
|
||||
|
||||
The COLMAP maintainers recommend using VCPKG to install COLAMP. If the following command do not work, please refer to the [COLMAP installation guide](https://colmap.github.io/install.html) for additional installation methods.
|
||||
|
||||
::::::{tab-set}
|
||||
:::::{tab-item} Linux
|
||||
|
||||
::::{tab-set}
|
||||
:::{tab-item} CUDA
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/vcpkg
|
||||
cd vcpkg
|
||||
./bootstrap-vcpkg.sh
|
||||
./vcpkg install colmap[cuda]:x64-linux
|
||||
```
|
||||
|
||||
:::
|
||||
:::{tab-item} CPU
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/vcpkg
|
||||
cd vcpkg
|
||||
./bootstrap-vcpkg.sh
|
||||
./vcpkg install colmap:x64-linux
|
||||
```
|
||||
|
||||
:::
|
||||
::::
|
||||
|
||||
:::::
|
||||
|
||||
:::::{tab-item} OSX
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/vcpkg
|
||||
cd vcpkg
|
||||
./bootstrap-vcpkg.sh
|
||||
./vcpkg install colmap
|
||||
```
|
||||
|
||||
:::::
|
||||
|
||||
:::::{tab-item} Windows
|
||||
|
||||
::::{tab-set}
|
||||
:::{tab-item} CUDA
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/vcpkg
|
||||
cd vcpkg
|
||||
./bootstrap-vcpkg.sh
|
||||
./vcpkg install colmap[cuda]:x64-windows
|
||||
```
|
||||
|
||||
:::
|
||||
:::{tab-item} CPU
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/vcpkg
|
||||
cd vcpkg
|
||||
./bootstrap-vcpkg.sh
|
||||
./vcpkg install colmap:x64-windows
|
||||
```
|
||||
|
||||
:::
|
||||
::::
|
||||
|
||||
:::::
|
||||
::::::
|
||||
@@ -0,0 +1,114 @@
|
||||
# Training First Model
|
||||
|
||||
## Downloading data
|
||||
|
||||
Download the original NeRF Blender dataset. We support the major datasets and allow users to create their own dataset, described in detail [here TODO].
|
||||
|
||||
```
|
||||
ns-download-data --dataset=blender
|
||||
ns-download-data --dataset=nerfstudio --capture=poster
|
||||
```
|
||||
|
||||
Use `--help` to view all currently available datasets. The resulting script should download and unpack the dataset as follows:
|
||||
|
||||
```
|
||||
|─ nerfstudio/
|
||||
├─ data/
|
||||
| ├─ blender/
|
||||
| ├─ fern/
|
||||
| ├─ lego/
|
||||
...
|
||||
|- <dataset_format>/
|
||||
|- <scene>
|
||||
...
|
||||
```
|
||||
|
||||
## Training a model
|
||||
|
||||
To run with all the defaults, e.g. vanilla nerf method with the blender lego image
|
||||
|
||||
Run a vanilla nerf model.
|
||||
|
||||
```bash
|
||||
ns-train vanilla-nerf
|
||||
```
|
||||
|
||||
Run a nerfacto model.
|
||||
|
||||
```bash
|
||||
ns-train nerfacto
|
||||
```
|
||||
|
||||
Run with nerfstudio data. You'll may have to change the ports, and be sure to forward the "websocket-port".
|
||||
|
||||
```
|
||||
ns-train nerfacto --vis viewer --viewer.zmq-port 8001 --viewer.websocket-port 8002 nerfstudio-data --pipeline.datamanager.dataparser.data-directory data/nerfstudio/poster --pipeline.datamanager.dataparser.downscale-factor 4
|
||||
```
|
||||
|
||||
## Visualizing training runs
|
||||
|
||||
If you using a fast NeRF variant (ie. Instant-NGP), we reccomend using our viewer. See our [viewer docs](viewer_quickstart.md) for more details. The viewer will allow interactive visualization of training in realtime.
|
||||
|
||||
Additionally, if you run everything with the default configuration, by default, we use [TensorBoard](https://www.tensorflow.org/tensorboard) to log all training curves, test images, and other stats. Once the job is launched, you will be able to track training by launching the tensorboard in `outputs/blender_lego/vanilla_nerf/<timestamp>/<events.tfevents>`.
|
||||
|
||||
```bash
|
||||
tensorboard --logdir outputs
|
||||
```
|
||||
|
||||
## Rendering a Trajectory
|
||||
|
||||
To evaluate the trained NeRF, we provide an evaluation script that allows you to do benchmarking (see our [benchmarking workflow](../developer_guides/benchmarking.md)) or to render out the scene with a custom trajectory and save the output to a video.
|
||||
|
||||
```bash
|
||||
ns-eval render-trajectory --load-config=outputs/blender_lego/instant_ngp/2022-07-07_230905/config.yml --traj=spiral --output-path=output.mp4
|
||||
```
|
||||
|
||||
Please note, this quickstart allows you to preform everything in a headless manner. We also provide a web-based viewer that allows you to easily monitor training or render out trajectories. See our [viewer docs](viewer_quickstart.md) for more.
|
||||
|
||||
## FAQ
|
||||
|
||||
- [TinyCUDA installation errors out with cuda mismatch](tiny-cuda-error)
|
||||
|
||||
(tiny-cuda-error)=
|
||||
|
||||
#### TinyCUDA installation errors out with cuda mismatch
|
||||
|
||||
While installing tiny-cuda, you run into: `The detected CUDA version mismatches the version that was used to compile PyTorch (10.2). Please make sure to use the same CUDA versions.`
|
||||
|
||||
**Solution**:
|
||||
|
||||
```
|
||||
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html
|
||||
```
|
||||
|
||||
- [Installation errors, File "setup.py" not found](pip-install-error)
|
||||
|
||||
(pip-install-error)=
|
||||
|
||||
#### Installation errors, File "setup.py" not found
|
||||
|
||||
When installing dependencies and nerfstudio with `pip install -e .`, you run into: `ERROR: File "setup.py" not found. Directory cannot be installed in editable mode`
|
||||
|
||||
**Solution**:
|
||||
This can be fixed by upgrading pip to the latest version:
|
||||
|
||||
```
|
||||
python -m pip install --upgrade pip
|
||||
```
|
||||
|
||||
- [Runtime errors, "len(sources) > 0".](cuda-sources-error)
|
||||
|
||||
(cuda-sources-error)=
|
||||
|
||||
#### Runtime errors: "len(sources) > 0", "ctype = \_C.ContractionType(type.value) ; TypeError: 'NoneType' object is not callable".
|
||||
|
||||
When running `train.py `, an error occurs when installing cuda files in the backend code.
|
||||
|
||||
**Solution**:
|
||||
This is a problem with not being able to detect the correct CUDA version, and can be fixed by updating the CUDA path environment variables:
|
||||
|
||||
```
|
||||
export CUDA_HOME=/usr/local/cuda
|
||||
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
|
||||
export PATH=$PATH:$CUDA_HOME/bin
|
||||
```
|
||||
@@ -0,0 +1,56 @@
|
||||
# Installation
|
||||
|
||||
### Create environment
|
||||
|
||||
We reccomend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html) before preceding.
|
||||
|
||||
```bash
|
||||
conda create --name nerfstudio -y python=3.8.13
|
||||
conda activate nerfstudio
|
||||
python -m pip install --upgrade pip
|
||||
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
Install pytorch with CUDA (this repo has been tested with CUDA 11.3) and [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn)
|
||||
|
||||
```bash
|
||||
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html
|
||||
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
|
||||
|
||||
```
|
||||
|
||||
### Installing nerfstudio
|
||||
|
||||
Easy option:
|
||||
|
||||
```bash
|
||||
pip install nerfstudio
|
||||
```
|
||||
|
||||
If you would want the latest and greatest:
|
||||
|
||||
```bash
|
||||
git clone git@github.com:plenoptix/nerfstudio.git
|
||||
cd nerfstudio
|
||||
pip install -e .
|
||||
|
||||
```
|
||||
|
||||
### Optional Installs
|
||||
|
||||
#### Tab completion (bash & zsh)
|
||||
|
||||
This needs to be rerun when the CLI changes, for example if nerfstudio is updated.
|
||||
|
||||
```bash
|
||||
ns-install-cli
|
||||
```
|
||||
|
||||
### Development packages
|
||||
|
||||
```bash
|
||||
pip install -e.[dev]
|
||||
pip install -e.[docs]
|
||||
```
|
||||
@@ -0,0 +1 @@
|
||||
# Setting up Viewer
|
||||
Reference in new issue
Block a user