mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-18 13:53:44 +08:00
deploy: 50e0e3c70c
This commit is contained in:
commit
0cc78c2e59
480 files changed
+584024
No files matched your search
@@ -0,0 +1,552 @@
|
||||
# Using custom data
|
||||
|
||||
Training model on existing datasets is only so fun. If you would like to train on self captured data you will need to process the data into the nerfstudio format. Specifically we need to know the camera poses for each image.
|
||||
|
||||
To process your own data run:
|
||||
|
||||
```bash
|
||||
ns-process-data {video,images,polycam,record3d} --data {DATA_PATH} --output-dir {PROCESSED_DATA_DIR}
|
||||
```
|
||||
|
||||
A full set of arguments can be found {doc}`here</reference/cli/ns_process_data>`.
|
||||
|
||||
We currently support the following custom data types:
|
||||
| Data | Capture Device | Requirements | `ns-process-data` Speed |
|
||||
| ----------------------------- | -------------- | ----------------------------------------------- | ----------------------- |
|
||||
| 📷 [Images](images_and_video) | Any | [COLMAP](https://colmap.github.io/install.html) | 🐢 |
|
||||
| 📹 [Video](images_and_video) | Any | [COLMAP](https://colmap.github.io/install.html) | 🐢 |
|
||||
| 🌎 [360 Data](360_data) | Any | [COLMAP](https://colmap.github.io/install.html) | 🐢 |
|
||||
| 📱 [Polycam](polycam) | IOS with LiDAR | [Polycam App](https://poly.cam/) | 🐇 |
|
||||
| 📱 [KIRI Engine](kiri) | IOS or Android | [KIRI Engine App](https://www.kiriengine.com/) | 🐇 |
|
||||
| 📱 [Record3D](record3d) | IOS with LiDAR | [Record3D app](https://record3d.app/) | 🐇 |
|
||||
| 📱 [Spectacular AI](spectacularai) | IOS, OAK, others| [App](https://apps.apple.com/us/app/spectacular-rec/id6473188128) / [`sai-cli`](https://www.spectacularai.com/mapping) | 🐇 |
|
||||
| 🖥 [Metashape](metashape) | Any | [Metashape](https://www.agisoft.com/) | 🐇 |
|
||||
| 🖥 [RealityCapture](realitycapture) | Any | [RealityCapture](https://www.capturingreality.com/realitycapture) | 🐇 |
|
||||
| 🖥 [ODM](odm) | Any | [ODM](https://github.com/OpenDroneMap/ODM) | 🐇 |
|
||||
| 👓 [Aria](aria) | Aria glasses | [Project Aria](https://projectaria.com/) | 🐇 |
|
||||
|
||||
(images_and_video)=
|
||||
|
||||
## Images or Video
|
||||
|
||||
To assist running on custom data we have a script that will process a video or folder of images into a format that is compatible with nerfstudio. We use [COLMAP](https://colmap.github.io) and [FFmpeg](https://ffmpeg.org/download.html) in our data processing script, please have these installed. We have provided a quickstart to installing COLMAP below, FFmpeg can be downloaded from [here](https://ffmpeg.org/download.html)
|
||||
|
||||
:::{admonition} Tip
|
||||
:class: info
|
||||
|
||||
- COLMAP can be finicky. Try your best to capture overlapping, non-blurry images.
|
||||
:::
|
||||
|
||||
### Processing Data
|
||||
|
||||
```bash
|
||||
ns-process-data {images, video} --data {DATA_PATH} --output-dir {PROCESSED_DATA_DIR}
|
||||
```
|
||||
|
||||
### Training on your data
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --data {PROCESSED_DATA_DIR}
|
||||
```
|
||||
|
||||
### Training and evaluation on separate data
|
||||
|
||||
For `ns-process-data {images, video}`, you can optionally use a separate image directory or video for training and evaluation, as suggested in [Nerfbusters](https://ethanweber.me/nerfbusters/). To do this, run `ns-process-data {images, video} --data {DATA_PATH} --eval-data {EVAL_DATA_PATH} --output-dir {PROCESSED_DATA_DIR}`. Then when running nerfacto, run `ns-train nerfacto --data {PROCESSED_DATA_DIR} nerfstudio-data --eval-mode filename`.
|
||||
|
||||
### Installing COLMAP
|
||||
|
||||
There are many ways to install COLMAP, unfortunately it can sometimes be a bit finicky. If the following commands do not work, please refer to the [COLMAP installation guide](https://colmap.github.io/install.html) for additional installation methods. COLMAP install issues are common! Feel free to ask for help in on our [Discord](https://discord.gg/uMbNqcraFc).
|
||||
|
||||
::::::{tab-set}
|
||||
:::::{tab-item} Linux
|
||||
|
||||
We recommend trying `conda`:
|
||||
|
||||
```
|
||||
conda install -c conda-forge colmap
|
||||
```
|
||||
|
||||
Check that COLMAP 3.8 with CUDA is successfully installed:
|
||||
|
||||
```
|
||||
colmap -h
|
||||
```
|
||||
|
||||
If that doesn't work, you can try VKPG:
|
||||
::::{tab-set}
|
||||
:::{tab-item} CUDA
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/vcpkg
|
||||
cd vcpkg
|
||||
./bootstrap-vcpkg.sh
|
||||
./vcpkg install colmap[cuda]:x64-linux
|
||||
```
|
||||
|
||||
:::
|
||||
:::{tab-item} CPU
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/vcpkg
|
||||
cd vcpkg
|
||||
./bootstrap-vcpkg.sh
|
||||
./vcpkg install colmap:x64-linux
|
||||
```
|
||||
|
||||
:::
|
||||
::::
|
||||
|
||||
If that doesn't work, you will need to build from source. Refer to the [COLMAP installation guide](https://colmap.github.io/install.html) for details.
|
||||
|
||||
:::::
|
||||
|
||||
:::::{tab-item} OSX
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/vcpkg
|
||||
cd vcpkg
|
||||
./bootstrap-vcpkg.sh
|
||||
./vcpkg install colmap
|
||||
```
|
||||
|
||||
:::::
|
||||
|
||||
:::::{tab-item} Windows
|
||||
|
||||
::::{tab-set}
|
||||
:::{tab-item} CUDA
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/vcpkg
|
||||
cd vcpkg
|
||||
.\bootstrap-vcpkg.bat
|
||||
.\vcpkg install colmap[cuda]:x64-windows
|
||||
```
|
||||
|
||||
:::
|
||||
:::{tab-item} CPU
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/vcpkg
|
||||
cd vcpkg
|
||||
.\bootstrap-vcpkg.sh
|
||||
.\vcpkg install colmap:x64-windows
|
||||
```
|
||||
|
||||
:::
|
||||
::::
|
||||
|
||||
:::::
|
||||
::::::
|
||||
|
||||
(polycam)=
|
||||
|
||||
## Polycam Capture
|
||||
|
||||
Nerfstudio can also be trained directly from captures from the [Polycam app](https://poly.cam//). This avoids the need to use COLMAP. Polycam's poses are globally optimized which make them more robust to drift (an issue with ARKit or SLAM methods).
|
||||
|
||||
To get the best results, try to reduce motion blur as much as possible and try to view the target from as many viewpoints as possible. Polycam recommends having good lighting and moving the camera slowly if using auto mode. Or, even better, use the manual shutter mode to capture less blurry images.
|
||||
|
||||
:::{admonition} Note
|
||||
:class: info
|
||||
A LiDAR enabled iPhone or iPad is necessary.
|
||||
:::
|
||||
|
||||
### Setting up Polycam
|
||||
|
||||
```{image} imgs/polycam_settings.png
|
||||
:width: 200
|
||||
:align: center
|
||||
:alt: polycam settings
|
||||
```
|
||||
|
||||
Developer settings must be enabled in Polycam. To do this, navigate to the settings screen and select `Developer mode`. Note that this will only apply for future captures, you will not be able to process existing captures with nerfstudio.
|
||||
|
||||
### Process data
|
||||
|
||||
```{image} imgs/polycam_export.png
|
||||
:width: 400
|
||||
:align: center
|
||||
:alt: polycam export options
|
||||
```
|
||||
|
||||
0. Capture data in LiDAR or Room mode.
|
||||
|
||||
1. Tap `Process` to process the data in the Polycam app.
|
||||
|
||||
2. Navigate to the export app pane.
|
||||
|
||||
3. Select `raw data` to export a `.zip` file.
|
||||
|
||||
4. Convert the Polycam data into the nerfstudio format using the following command:
|
||||
|
||||
```bash
|
||||
ns-process-data polycam --data {OUTPUT_FILE.zip} --output-dir {output directory}
|
||||
```
|
||||
|
||||
5. Train with nerfstudio!
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --data {output directory}
|
||||
```
|
||||
|
||||
(kiri)=
|
||||
|
||||
## KIRI Engine Capture
|
||||
|
||||
Nerfstudio can trained from data processed by the [KIRI Engine app](https://www.kiriengine.com/). This works for both Android and iPhone and does not require a LiDAR supported device.
|
||||
|
||||
:::{admonition} Note
|
||||
:class: info
|
||||
`ns-process-data` does not need to be run when using KIRI Engine.
|
||||
:::
|
||||
|
||||
### Setting up KIRI Engine
|
||||
|
||||
```{image} imgs/kiri_setup.png
|
||||
:width: 400
|
||||
:align: center
|
||||
:alt: KIRI Engine setup
|
||||
```
|
||||
|
||||
After downloading the app, `Developer Mode` needs to be enabled. A toggle can be found in the settings menu.
|
||||
|
||||
### Process data
|
||||
|
||||
```{image} imgs/kiri_capture.png
|
||||
:width: 400
|
||||
:align: center
|
||||
:alt: KIRI Engine setup
|
||||
```
|
||||
|
||||
1. Navigate to captures window.
|
||||
|
||||
2. Select `Dev.` tab.
|
||||
|
||||
3. Tap the `+` button to create a new capture.
|
||||
|
||||
4. Choose `Camera pose` as the capture option.
|
||||
|
||||
5. Capture the scene and provide a name.
|
||||
|
||||
6. After processing is complete, export the scene. It will be sent to your email.
|
||||
|
||||
7. Unzip the file and run the training script (`ns-process-data` is not necessary).
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --data {kiri output directory}
|
||||
```
|
||||
|
||||
(record3d)=
|
||||
|
||||
## Record3D Capture
|
||||
|
||||
Nerfstudio can be trained directly from >=iPhone 12 Pro captures from the [Record3D app](https://record3d.app/). This uses the iPhone's LiDAR sensors to calculate camera poses, so COLMAP is not needed.
|
||||
|
||||
Click on the image down below 👇 for a 1-minute tutorial on how to run nerfstudio with Record3D from start to finish.
|
||||
|
||||
[](https://youtu.be/XwKq7qDQCQk 'How to easily use nerfstudio with Record3D')
|
||||
|
||||
At a high level, you can follow these 3 steps:
|
||||
|
||||
1. Record a video and export with the EXR + JPG sequence format.
|
||||
|
||||
<img src="imgs/record_3d_video_selection.png" width=150>
|
||||
<img src="imgs/record_3d_export_selection.png" width=150>
|
||||
|
||||
2. Then, move the exported capture folder from your iPhone to your computer.
|
||||
|
||||
3. Convert the data to the nerfstudio format.
|
||||
|
||||
```bash
|
||||
ns-process-data record3d --data {data directory} --output-dir {output directory}
|
||||
```
|
||||
|
||||
4. Train with nerfstudio!
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --data {output directory}
|
||||
```
|
||||
|
||||
### Adding a Point Cloud
|
||||
|
||||
Adding a point cloud is useful for avoiding random initialization when training gaussian splats. To add a point cloud using Record3D follow these steps:
|
||||
|
||||
1. Export a Zipped sequence of PLY point clouds from Record3D.
|
||||
|
||||
<img src="imgs/record_3d_video_example.png" width=150>
|
||||
<img src="imgs/record_3d_export_button.png" width=150>
|
||||
<img src="imgs/record_3d_ply_selection.png" width=150>
|
||||
|
||||
|
||||
2. Move the exported zip file to your computer from your iPhone.
|
||||
|
||||
|
||||
3. Unzip the file and move all extracted `.ply` files to a directory.
|
||||
|
||||
|
||||
4. Convert the data to nerfstudio format with the `--ply` flag and the directory from step 3.
|
||||
|
||||
```bash
|
||||
ns-process-data record3d --data {data directory} --ply {ply directory} --output-dir {output directory}
|
||||
```
|
||||
|
||||
Additionally you can specify `--voxel-size {float}` which determines the level of sparsity when downsampling from the dense point clouds generated by Record3D to the sparse point cloud used in Nerfstudio. The default value is 0.8, lower is less sparse, higher is more sparse.
|
||||
|
||||
(spectacularai)=
|
||||
|
||||
## Spectacular AI
|
||||
|
||||
Spectacular AI SDK and apps can be used to capture data from various devices:
|
||||
|
||||
* iPhones (with LiDAR)
|
||||
* OAK-D cameras
|
||||
* RealSense D455/D435i
|
||||
* Azure Kinect DK
|
||||
|
||||
The SDK also records IMU data, which is fused with camera and (if available) LiDAR/ToF data when computing the camera poses. This approach, VISLAM, is more robust than purely image based methods (e.g., COLMAP) and can work better and faster for difficult data (monotonic environments, fast motions, narrow FoV, etc.).
|
||||
|
||||
Instructions:
|
||||
|
||||
1. Installation. With the Nerfstudio Conda environment active, first install the Spectacular AI Python library
|
||||
|
||||
```bash
|
||||
pip install spectacularAI[full]
|
||||
```
|
||||
|
||||
2. Install FFmpeg. Linux: `apt install ffmpeg` (or similar, if using another package manager). Windows: [see here](https://www.editframe.com/guides/how-to-install-and-start-using-ffmpeg-in-under-10-minutes). FFmpeg must be in your `PATH` so that `ffmpeg` works on the command line.
|
||||
|
||||
3. Data capture. See [here for specific instructions for each supported device](https://github.com/SpectacularAI/sdk-examples/tree/main/python/mapping#recording-data).
|
||||
|
||||
4. Process and export. Once you have recorded a dataset in Spectacular AI format and have it stored in `{data directory}` it can be converted into a Nerfstudio supported format with:
|
||||
|
||||
```bash
|
||||
sai-cli process {data directory} --preview3d --key_frame_distance=0.05 {output directory}
|
||||
```
|
||||
The optional `--preview3d` flag shows a 3D preview of the point cloud and estimated trajectory live while VISLAM is running. The `--key_frame_distance` argument can be tuned based on the recorded scene size: 0.05 (5cm) is good for small scans and 0.15 for room-sized scans. If the processing gets slow, you can also try adding a --fast flag to `sai-cli process` to trade off quality for speed.
|
||||
|
||||
5. Train. No separate `ns-process-data` step is needed. The data in `{output directory}` can now be trained with Nerfstudio:
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --data {output directory}
|
||||
```
|
||||
|
||||
(metashape)=
|
||||
|
||||
## Metashape
|
||||
|
||||
All images must use the same sensor type (but multiple sensors are supported).
|
||||
|
||||
1. Align your images using Metashape. `File -> Workflow -> Align Photos...`
|
||||
|
||||
```{image} https://user-images.githubusercontent.com/3310961/203389662-12760210-2b52-49d4-ab21-4f23bfa4a2b3.png
|
||||
:width: 400
|
||||
:align: center
|
||||
:alt: metashape alignment
|
||||
```
|
||||
|
||||
2. Export the camera alignment as a `xml` file. `File -> Export -> Export Cameras...`
|
||||
|
||||
```{image} https://user-images.githubusercontent.com/3310961/203385691-74565704-e4f6-4034-867e-5d8b940fc658.png
|
||||
:width: 400
|
||||
:align: center
|
||||
:alt: metashape export
|
||||
```
|
||||
|
||||
3. Convert the data to the nerfstudio format.
|
||||
|
||||
```bash
|
||||
ns-process-data metashape --data {data directory} --xml {xml file} --output-dir {output directory}
|
||||
```
|
||||
|
||||
4. Train with nerfstudio!
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --data {output directory}
|
||||
```
|
||||
|
||||
(realitycapture)=
|
||||
|
||||
## RealityCapture
|
||||
|
||||
1. Align your images using RealityCapture. `ALIGNMENT -> Align Images`
|
||||
|
||||
2. Export the camera alignment as a `csv` file. Choose `Internal/External camera parameters`
|
||||
|
||||
3. Convert the data to the nerfstudio format.
|
||||
|
||||
```bash
|
||||
ns-process-data realitycapture --data {data directory} --csv {csv file} --output-dir {output directory}
|
||||
```
|
||||
|
||||
4. Train with nerfstudio!
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --data {output directory}
|
||||
```
|
||||
|
||||
(odm)=
|
||||
|
||||
## ODM
|
||||
|
||||
All images/videos must be captured with the same camera.
|
||||
|
||||
1. Process a dataset using [ODM](https://github.com/OpenDroneMap/ODM#quickstart)
|
||||
|
||||
```bash
|
||||
$ ls /path/to/dataset
|
||||
images
|
||||
odm_report
|
||||
odm_orthophoto
|
||||
...
|
||||
```
|
||||
|
||||
2. Convert to nerfstudio format.
|
||||
|
||||
```bash
|
||||
ns-process-data odm --data /path/to/dataset --output-dir {output directory}
|
||||
```
|
||||
|
||||
4. Train!
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --data {output directory}
|
||||
```
|
||||
|
||||
(aria)=
|
||||
|
||||
## Aria
|
||||
|
||||
1. Install projectaria_tools:
|
||||
|
||||
```bash
|
||||
conda activate nerfstudio
|
||||
pip install projectaria-tools'[all]'
|
||||
```
|
||||
|
||||
2. Download a VRS file from Project Aria glasses, and run Machine Perception Services to extract poses.
|
||||
|
||||
3. Convert to nerfstudio format.
|
||||
|
||||
```bash
|
||||
ns-process-data aria --vrs-file /path/to/vrs/file --mps-data-dir /path/to/mps/data --output-dir {output directory}
|
||||
```
|
||||
|
||||
4. Train!
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --data {output directory}
|
||||
```
|
||||
|
||||
(360_data)=
|
||||
|
||||
## 360 Data (Equirectangular)
|
||||
|
||||
Equirectangular data is data that has been taken by a 360 camera such as Insta360. Both equirectangular image sets and videos can be processed by nerfstudio.
|
||||
|
||||
### Images
|
||||
|
||||
For a set of equirectangular images, process the data using the following command:
|
||||
|
||||
```bash
|
||||
ns-process-data images --camera-type equirectangular --images-per-equirect {8, or 14} --crop-factor {top bottom left right} --data {data directory} --output-dir {output directory}
|
||||
```
|
||||
|
||||
The images-per-equirect argument is the number of images that will be sampled from each equirectangular image. We have found that 8 images per equirectangular image is sufficient for most use cases so it defaults to that. However, if you find that there isn't enough detail in the nerf or that colmap is having trouble aligning the images, you can try increasing the number of images per equirectangular image to 14. See the video section below for details on cropping.
|
||||
|
||||
### Videos
|
||||
|
||||
For videos we recommend taking a video with the camera held on top of your head. This will result in any unwanted capturer to just be in the bottom of each frame image and therefore can be cropped out.
|
||||
|
||||
For a video, process the data using the following command:
|
||||
|
||||
```bash
|
||||
ns-process-data video --camera-type equirectangular --images-per-equirect {8, or 14} --num-frames-target {num equirectangular frames to sample from} --crop-factor {top bottom left right} --data {data directory} --output-dir {output directory}
|
||||
```
|
||||
|
||||
See the equirectangular images section above for a description of the `--images-per-equirect` argument.
|
||||
|
||||
The `num-frames-target` argument is optional but it is recommended to set it to 3*(seconds of video) frames. For example, if you have a 30 second video, you would use `--num-frames-target 90` (3*30=90). This number was chosen from a bit of experimentation and seems to work well for most videos. It is by no means a hard rule and you can experiment with different values.
|
||||
|
||||
The `crop-factor` argument is optional but often very helpful. This is because equirectangular videos taken by 360 cameras tend to have a portion of the bottom of the image that is the person who was holding the camera over their head.
|
||||
|
||||
<img src="imgs/equirect_crop.jpg">
|
||||
|
||||
The pixels representing the distorted hand and head are obviously not useful in training a nerf so we can remove it by cropping the bottom 20% of the image. This can be done by using the `--crop-factor 0 0.2 0 0` argument.
|
||||
|
||||
If cropping only needs to be done from the bottom, you can use the `--crop-bottom [num]` argument which would be the same as doing `--crop-factor 0.0 [num] 0.0 0.0`
|
||||
|
||||
## 🥽 Render VR Video
|
||||
|
||||
Stereo equirectangular rendering for VR video is supported as VR180 and omni-directional stereo (360 VR) Nerfstudio camera types for video and image rendering.
|
||||
|
||||
### Omni-directional Stereo (360 VR)
|
||||
This outputs two equirectangular renders vertically stacked, one for each eye. Omni-directional stereo (ODS) is a method to render VR 3D 360 videos, and may introduce slight depth distortions for close objects. For additional information on how ODS works, refer to this [writeup](https://developers.google.com/vr/jump/rendering-ods-content.pdf).
|
||||
|
||||
<center>
|
||||
<img img width="300" src="https://github-production-user-asset-6210df.s3.amazonaws.com/9502341/255423390-ff0710f1-29ce-47b2-85f9-922084cab297.jpg">
|
||||
</center>
|
||||
|
||||
|
||||
### VR180
|
||||
This outputs two 180 deg equirectangular renders horizontally stacked, one for each eye. VR180 is a video format for VR 3D 180 videos. Unlike in omnidirectional stereo, VR180 content only displays front facing content.
|
||||
|
||||
<center>
|
||||
<img img width="375" src="https://github-production-user-asset-6210df.s3.amazonaws.com/9502341/255379444-b90f5b3c-5021-4659-8732-17725669914e.jpeg">
|
||||
</center>
|
||||
|
||||
### Setup instructions
|
||||
To render for VR video it is essential to adjust the NeRF to have an approximately true-to-life real world scale (adjustable in the camera path) to ensure that the scene depth and IPD (distance between the eyes) is appropriate for the render to be viewable in VR. You can adjust the scene scale with the [Nerfstudio Blender Add-on](https://docs.nerf.studio/extensions/blender_addon.html) by appropriately scaling a point cloud representation of the NeRF.
|
||||
Results may be unviewable if the scale is not set appropriately. The IPD is set at 64mm by default but only is accurate when the NeRF scene is true to scale.
|
||||
|
||||
For good quality renders, it is recommended to render at high resolutions (For ODS: 4096x2048 per eye, or 2048x1024 per eye. For VR180: 4096x4096 per eye or 2048x2048 per eye). Render resolutions for a single eye are specified in the camera path. For VR180, resolutions must be in a 1:1 aspect ratio. For ODS, resolutions must be in a 2:1 aspect ratio. The final stacked render output will automatically be constructed (with aspect ratios for VR180 as 2:1 and ODS as 1:1).
|
||||
|
||||
:::{admonition} Note
|
||||
:class: info
|
||||
If you are rendering an image sequence, it is recommended to render as png instead of jpeg, since the png will appear clearer. However, file sizes can be significantly larger with png.
|
||||
:::
|
||||
|
||||
To render with the VR videos camera:
|
||||
1. Use the [Nerfstudio Blender Add-on](https://docs.nerf.studio/extensions/blender_addon.html) to set the scale of the NeRF scene and create the camera path
|
||||
- Export a point cloud representation of the NeRF
|
||||
- Import the point cloud representation in Blender and enable the Nerfstudio Blender Add-on
|
||||
- Create a reference object such as a cube which may be 1x1x1 meter. You could also create a cylinder and scale it to an appropriate height of a viewer.
|
||||
- Now scale the point cloud representation accordingly to match the reference object. This is to ensure that the NeRF scene is scaled as close to real life.
|
||||
- To place the camera at the correct height from the ground in the scene, you can create a cylinder representing the viewer vertically scaled to the viewer’s height, and place the camera at eye level.
|
||||
- Animate the camera movement as needed
|
||||
- Create the camera path JSON file with the Nerfstudio Blender Add-on
|
||||
|
||||
2. Edit the JSON camera path file
|
||||
|
||||
**Omni-directional Stereo (360 VR)**
|
||||
- Open the camera path JSON file and specify the `camera_type` as `omnidirectional`
|
||||
- Specify the `render_height` and `render_width` to the resolution of a single eye. The width:height aspect ratio must be 2:1. Recommended resolutions are 4096x2048 or 2048x1024.
|
||||
<center>
|
||||
<img img width="250" src="https://github-production-user-asset-6210df.s3.amazonaws.com/9502341/240530527-22d14276-ac2c-46a5-a4b0-4785b7413241.png">
|
||||
</center>
|
||||
|
||||
|
||||
**VR180**
|
||||
- Open the camera path JSON file and specify the `camera_type` as `vr180`
|
||||
- Specify the `render_height` and `render_width` to the resolution of a single eye. The width:height aspect ratio must be 1:1. Recommended resolutions are 4096x4096 or 2048x2048.
|
||||
<center>
|
||||
<img img width="190" src="https://github-production-user-asset-6210df.s3.amazonaws.com/9502341/255379889-83b7fd09-ce8f-4868-8838-7be9b63f01b4.png">
|
||||
</center>
|
||||
|
||||
- Save the camera path and render the NeRF
|
||||
|
||||
|
||||
:::{admonition} Note
|
||||
:class: info
|
||||
If the depth of the scene is unviewable and looks too close or expanded when viewing the render in VR, the scale of the NeRF may be set too small. If there is almost no discernible depth, the scale of the NeRF may be too large. Getting the right scale may take some experimentation, so it is recommended to either render at a much lower resolution or just one frame to ensure the depth and render is viewable in the VR headset.
|
||||
:::
|
||||
|
||||
#### Additional Notes
|
||||
- Rendering with VR180 or ODS can take significantly longer than traditional renders due to higher resolutions and needing to render a left and right eye view for each frame. Render times may grow exponentially with larger resolutions.
|
||||
- When rendering VR180 or ODS content, Nerfstudio will first render the left eye, then the right eye, and finally vertically stack the renders. During this process, Nerfstudio will create a temporary folder to store the left and right eye renders and delete this folder once the final renders are stacked.
|
||||
- If rendering content where the camera is stationary for many frames, it is recommended to only render once at that position and extend the time in a video editor since ODS renders can take a lot of time to render.
|
||||
- It is recommended to render a preliminary render at a much lower resolution or frame rate to test and ensure that the depth and camera position look accurate in VR.
|
||||
- The IPD can be modified in the `cameras.py` script as the variable `vr_ipd` (default is 64 mm).
|
||||
- Compositing with Blender Objects and VR180 or ODS Renders
|
||||
- Configure the Blender camera as panoramic and equirectangular. For the VR180 Blender camera, set the panoramic longitude min and max to -90 and 90.
|
||||
- Change the Stereoscopy mode to "Parallel" set the Interocular Distance to 0.064 m.
|
||||
@@ -0,0 +1,128 @@
|
||||
# Data conventions
|
||||
|
||||
## Coordinate conventions
|
||||
|
||||
Here we explain the coordinate conventions for using our repo.
|
||||
|
||||
### Camera/view space
|
||||
|
||||
We use the OpenGL/Blender (and original NeRF) coordinate convention for cameras. +X is right, +Y is up, and +Z is pointing back and away from the camera. -Z is the look-at direction. Other codebases may use the COLMAP/OpenCV convention, where the Y and Z axes are flipped from ours but the +X axis remains the same.
|
||||
|
||||
### World space
|
||||
|
||||
Our world space is oriented such that the up vector is +Z. The XY plane is parallel to the ground plane. In the viewer, you'll notice that red, green, and blue vectors correspond to X, Y, and Z respectively.
|
||||
|
||||
### Pixel coordinates
|
||||
|
||||
We assume coordinates correspond to the centers of pixels (e.g. generating a ray for pixel (0,0) will shoot the ray through the center of that pixel). This aligns more with graphics conventions, and is distinct from OpenCV where the corners are aligned with the pixel coordinate.
|
||||
|
||||
<hr>
|
||||
|
||||
## Dataset format
|
||||
|
||||
Our explanation here is for the nerfstudio data format. The `transforms.json` has a similar format to [Instant NGP](https://github.com/NVlabs/instant-ngp).
|
||||
|
||||
### Camera intrinsics
|
||||
|
||||
If all of the images share the same camera intrinsics, the values can be placed at the top of the file.
|
||||
|
||||
```json
|
||||
{
|
||||
"camera_model": "OPENCV_FISHEYE", // camera model type [OPENCV, OPENCV_FISHEYE]
|
||||
"fl_x": 1072.0, // focal length x
|
||||
"fl_y": 1068.0, // focal length y
|
||||
"cx": 1504.0, // principal point x
|
||||
"cy": 1000.0, // principal point y
|
||||
"w": 3008, // image width
|
||||
"h": 2000, // image height
|
||||
"k1": 0.0312, // first radial distortion parameter, used by [OPENCV, OPENCV_FISHEYE]
|
||||
"k2": 0.0051, // second radial distortion parameter, used by [OPENCV, OPENCV_FISHEYE]
|
||||
"k3": 0.0006, // third radial distortion parameter, used by [OPENCV_FISHEYE]
|
||||
"k4": 0.0001, // fourth radial distortion parameter, used by [OPENCV_FISHEYE]
|
||||
"p1": -6.47e-5, // first tangential distortion parameter, used by [OPENCV]
|
||||
"p2": -1.37e-7, // second tangential distortion parameter, used by [OPENCV]
|
||||
"frames": // ... per-frame intrinsics and extrinsics parameters
|
||||
}
|
||||
```
|
||||
|
||||
Per-frame intrinsics can also be defined in the `frames` field. If defined for a field (ie. `fl_x`), all images must have per-image intrinsics defined for that field. Per-frame `camera_model` is not supported.
|
||||
|
||||
```json
|
||||
{
|
||||
// ...
|
||||
"frames": [
|
||||
{
|
||||
"fl_x": 1234
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Camera extrinsics
|
||||
|
||||
For a transform matrix, the first 3 columns are the +X, +Y, and +Z defining the camera orientation, and the X, Y, Z values define the origin. The last row is to be compatible with homogeneous coordinates.
|
||||
|
||||
```json
|
||||
{
|
||||
// ...
|
||||
"frames": [
|
||||
{
|
||||
"file_path": "images/frame_00001.jpeg",
|
||||
"transform_matrix": [
|
||||
// [+X0 +Y0 +Z0 X]
|
||||
// [+X1 +Y1 +Z1 Y]
|
||||
// [+X2 +Y2 +Z2 Z]
|
||||
// [0.0 0.0 0.0 1]
|
||||
[1.0, 0.0, 0.0, 0.0],
|
||||
[0.0, 1.0, 0.0, 0.0],
|
||||
[0.0, 0.0, 1.0, 0.0],
|
||||
[0.0, 0.0, 0.0, 1.0]
|
||||
]
|
||||
// Additional per-frame info
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Depth images
|
||||
|
||||
To train with depth supervision, you can also provide a `depth_file_path` for each frame in your `transforms.json` and use one of the methods that support additional depth losses (e.g., depth-nerfacto). The depths are assumed to be 16-bit or 32-bit and to be in millimeters to remain consistent with [Polyform](https://github.com/PolyCam/polyform). Zero-value in the depth image is treated as unknown depth. You can adjust this scaling factor using the `depth_unit_scale_factor` parameter in `NerfstudioDataParserConfig`. Note that by default, we resize the depth images to match the shape of the RGB images.
|
||||
|
||||
```json
|
||||
{
|
||||
"frames": [
|
||||
{
|
||||
// ...
|
||||
"depth_file_path": "depth/0001.png"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Masks
|
||||
|
||||
:::{admonition} Warning
|
||||
:class: Warning
|
||||
|
||||
The current implementation of masking is inefficient and will cause large memory allocations.
|
||||
:::
|
||||
|
||||
There may be parts of the training image that should not be used during training (ie. moving objects such as people). These images can be masked out using an additional mask image that is specified in the `frame` data.
|
||||
|
||||
```json
|
||||
{
|
||||
"frames": [
|
||||
{
|
||||
// ...
|
||||
"mask_path": "masks/mask.jpeg"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The following mask requirements must be met:
|
||||
|
||||
- Must be 1 channel with only black and white pixels
|
||||
- Must be the same resolution as the training image
|
||||
- Black corresponds to regions to ignore
|
||||
- If used, all images must have a mask
|
||||
@@ -0,0 +1,97 @@
|
||||
# Using existing data
|
||||
|
||||
Nerfstudio comes with built-in support for a number of datasets, which can be downloaded with the [`ns-download-data` command][cli]. Each of the built-in datasets comes ready to use with various Nerfstudio methods (e.g. the recommended default Nerfacto), allowing you to get started in the blink of an eye.
|
||||
|
||||
[cli]: https://docs.nerf.studio/reference/cli/ns_download_data.html
|
||||
[paper]: https://arxiv.org/pdf/2302.04264.pdf
|
||||
|
||||
## Example
|
||||
|
||||
Here are a few examples of downloading different scenes. Please see the [Training Your First NeRF](first_nerf.md) documentation for more details on how to train a model with them.
|
||||
|
||||
```bash
|
||||
# Download all scenes from the Blender dataset, including the "classic" Lego model
|
||||
ns-download-data blender
|
||||
|
||||
# Download the subset of data used in the SIGGRAPH 2023 Nerfstudio paper
|
||||
ns-download-data nerfstudio --capture-name nerfstudio-dataset
|
||||
|
||||
# Download a few room-scale scenes from the EyefulTower dataset at different resolutions
|
||||
pip install awscli # Install `awscli` for EyefulTower downloads.
|
||||
ns-download-data eyefultower --capture-name riverview seating_area apartment --resolution-name jpeg_1k jpeg_2k
|
||||
|
||||
# Download the full D-NeRF dataset of dynamic synthetic scenes
|
||||
ns-download-data dnerf
|
||||
```
|
||||
|
||||
## Dataset Summary
|
||||
|
||||
Many of these datasets are used as baselines to evaluate new research in novel view synthesis, such as in the [original Nerfstudio paper][paper]. Scenes from these datasets lie at dramatically different points in the space of images, across axes such as photorealism (synthetic vs real), dynamic range (LDR vs HDR), scale (number of images), and resolution. The tables below describe some of this variation, and hopefully make it easier to pick an appropriate dataset for your research or application.
|
||||
|
||||
| Dataset | Synthetic | Real | LDR | HDR | Scenes | Image Count<sup>1</sup> | Image Resolution<sup>2</sup> |
|
||||
| :-: | :-: | :-: | :-: | :-: | :------: | :-: | :-: |
|
||||
| [Blender][blender] | ✔️ | | ✔️ | | 8 | ➖➕️➖➖ | ➕️➖➖➖➖ |
|
||||
| [D-NeRF][dnerf] | ✔️ | | ✔️ | | 8 | ➕️➖➖➖ | ➕️➖➖➖➖ |
|
||||
| [EyefulTower][eyefultower] | | ✔️ | ✔️ | ✔️ | 11 | ➖➕️➕️➕️ | ➖➕️➕️➕️➕️ |
|
||||
| [Mill 19][mill19] | | ✔️ | ✔️ | | 2 | ➖➖➕️➖ | ➖➖➖➕️➖ |
|
||||
| [NeRF-OSR][nerfosr] | | ✔️ | ✔️ | | 9 | ➕➕️➕️➖ | ➖➕️➖➕️➖ |
|
||||
| [Nerfstudio][nerfstudio] | | ✔️ | ✔️ | | 18 | ➕➕️➕️➖ | ➕️➕️➕️➖➖ |
|
||||
| [PhotoTourism][phototourism] | | ✔️ | ✔️ | | 10 | ➖➕️➕️➖ | ➖➕️➖➖➖ |
|
||||
| [Record3D][record3d] | | ✔️ | ✔️ | | 1 | ➖➖➕️➖ | ➕️➖➖➖➖ |
|
||||
| [SDFStudio][sdfstudio] | ✔️ | ✔️ | ✔️ | | 45 | ➕️➕️➕️➖ | ➕️➖➕️➖➖ |
|
||||
| [sitcoms3D][sitcoms3d] | | ✔️ | ✔️ | | 10 | ➕️➖➖➖ | ➕️➕️➖➖➖ |
|
||||
|
||||
In the tables below, each dataset was placed into a bucket based on the table's chosen property. If a box contains a ✔️, the corresponding dataset will have *at least* one scene falling into the corresponding bucket for that property, though there may be multiple scenes at different points within the range.
|
||||
|
||||
<sub>
|
||||
<b>1:</b> Condensed version of the "Scene Size: Number of RGB Images" table below. <br>
|
||||
<b>2:</b> Condensed version of the "Scene RGB Resolutions: `max(width, height)`" table below.
|
||||
</sub>
|
||||
|
||||
### Scene Size: Number of RGB Images
|
||||
|
||||
| Dataset | < 250 | 250 - 999 | 1000 - 3999 | ≥ 4000 |
|
||||
| :-: | :-: | :-: | :-: | :-: |
|
||||
| [Blender][blender] | | ✔️ | | |
|
||||
| [D-NeRF][dnerf] | ✔️ | | | |
|
||||
| [EyefulTower][eyefultower] | | ✔️ | ✔️ | ✔️ |
|
||||
| [Mill 19][mill19] | | | ✔️ | |
|
||||
| [NeRF-OSR][nerfosr] | ✔️ | ✔️ | ✔️ | |
|
||||
| [Nerfstudio][nerfstudio] | ✔️ | ✔️ | ✔️ | |
|
||||
| [PhotoTourism][phototourism] | | ✔️ | ✔️ | |
|
||||
| [Record3D][record3d] | | | ✔️ | |
|
||||
| [SDFStudio][sdfstudio] | ✔️ | ✔️ | ✔️ | |
|
||||
| [sitcoms3D][sitcoms3d] | ✔️ | | |
|
||||
|
||||
### Scene RGB Resolutions: `max(width, height)`
|
||||
|
||||
| Dataset | < 1000 | 1000 - 1999 | 2000 - 3999 | 4000 - 7999 | ≥ 8000 |
|
||||
| :-: | :-: | :-: | :-: | :-: | :-: |
|
||||
| [Blender][blender] | ✔️ | | | | |
|
||||
| [D-NeRF][dnerf] | ✔️ | | | | |
|
||||
| [EyefulTower][eyefultower] | | ✔️ | ✔️ | ✔️ | ✔️ |
|
||||
| [Mill 19][mill19] | | | | ✔️ | |
|
||||
| [NeRF-OSR][nerfosr] | | ✔️ | | ✔️ | |
|
||||
| [Nerfstudio][nerfstudio] | ✔️ | ✔️ | ✔️ | | |
|
||||
| [PhotoTourism][phototourism] | | ✔️ | | | |
|
||||
| [Record3D][record3d] | ✔️ | | | | |
|
||||
| [SDFStudio][sdfstudio] | ✔️ | | ✔️ | | |
|
||||
| [sitcoms3D][sitcoms3d] | ✔️ | ✔️ | | | |
|
||||
|
||||
[blender]: https://github.com/bmild/nerf?tab=readme-ov-file#project-page--video--paper--data
|
||||
[dnerf]: https://github.com/albertpumarola/D-NeRF?tab=readme-ov-file#download-dataset
|
||||
[eyefultower]: https://github.com/facebookresearch/EyefulTower
|
||||
[mill19]: https://github.com/cmusatyalab/mega-nerf?tab=readme-ov-file#mill-19
|
||||
[nerfosr]: https://4dqv.mpi-inf.mpg.de/NeRF-OSR/
|
||||
[nerfstudio]: https://github.com/nerfstudio-project/nerfstudio
|
||||
[phototourism]: https://www.cs.ubc.ca/~kmyi/imw2020/data.html
|
||||
[record3d]: https://record3d.app/
|
||||
[sdfstudio]: https://github.com/autonomousvision/sdfstudio/blob/master/docs/sdfstudio-data.md#Existing-dataset
|
||||
[sitcoms3d]: https://github.com/ethanweber/sitcoms3D/blob/master/METADATA.md
|
||||
|
||||
### Eyeful Tower
|
||||
Downloading Eyeful Tower scenes requires installing the AWS CLI, an optional dependency. To do so, run:
|
||||
```bash
|
||||
conda activate nerfstudio
|
||||
pip install awscli
|
||||
```
|
||||
@@ -0,0 +1,61 @@
|
||||
# Export geometry
|
||||
|
||||
Here we document how to export point clouds and meshes from nerfstudio. The main command you'll be working with is `ns-export`. Our point clouds are exported as `.ply` files and the textured meshes are exported as `.obj` files.
|
||||
|
||||
## Exporting a mesh
|
||||
|
||||
### 1. TSDF Fusion
|
||||
|
||||
TSDF (truncated signed distance function) Fusion is a meshing algorithm that uses depth maps to extract a surface as a mesh. This method works for all models.
|
||||
|
||||
```python
|
||||
ns-export tsdf --load-config CONFIG.yml --output-dir OUTPUT_DIR
|
||||
```
|
||||
|
||||
### 2. Poisson surface reconstruction
|
||||
|
||||
Poisson surface reconstruction gives the highest quality meshes. See the steps below to use Poisson surface reconstruction in our repo.
|
||||
|
||||
> **Note:**
|
||||
> This will only work with a Model that computes or predicts normals, e.g., nerfacto.
|
||||
|
||||
1. Train nerfacto with network settings that predict normals.
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --pipeline.model.predict-normals True
|
||||
```
|
||||
|
||||
2. Export a mesh with the Poisson meshing algorithm.
|
||||
|
||||
```bash
|
||||
ns-export poisson --load-config CONFIG.yml --output-dir OUTPUT_DIR
|
||||
```
|
||||
|
||||
## Exporting a point cloud
|
||||
|
||||
```bash
|
||||
ns-export pointcloud --help
|
||||
```
|
||||
|
||||
## Other exporting methods
|
||||
|
||||
Run the following command to see other export methods that may exist.
|
||||
|
||||
```python
|
||||
ns-export --help
|
||||
```
|
||||
|
||||
## Texturing an existing mesh with NeRF
|
||||
|
||||
Say you want to simplify and/or smooth a mesh offline, and then you want to texture it with NeRF. You can do that with the following command. It will work for any mesh filetypes that [PyMeshLab](https://pymeshlab.readthedocs.io/en/latest/) can support, for example a `.ply`.
|
||||
|
||||
```python
|
||||
python nerfstudio/scripts/texture.py --load-config CONFIG.yml --input-mesh-filename FILENAME --output-dir OUTPUT_DIR
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
Our dependencies are shipped with the pip package in the pyproject.toml file. These are the following:
|
||||
|
||||
- [xatlas-python](https://github.com/mworchel/xatlas-python) for unwrapping meshes to a UV map
|
||||
- [pymeshlab](https://pymeshlab.readthedocs.io/en/latest/) for reducing the number of faces in a mesh
|
||||
@@ -0,0 +1,171 @@
|
||||
# Training your first model
|
||||
|
||||
## Train and run viewer
|
||||
|
||||
The following will train a _nerfacto_ model, our recommended model for real world scenes.
|
||||
|
||||
```bash
|
||||
# Download some test data:
|
||||
ns-download-data nerfstudio --capture-name=poster
|
||||
# Train model
|
||||
ns-train nerfacto --data data/nerfstudio/poster
|
||||
```
|
||||
|
||||
If everything works, you should see training progress like the following:
|
||||
|
||||
<p align="center">
|
||||
<img width="800" alt="image" src="https://user-images.githubusercontent.com/3310961/202766069-cadfd34f-8833-4156-88b7-ad406d688fc0.png">
|
||||
</p>
|
||||
|
||||
Navigating to the link at the end of the terminal will load the webviewer. If you are running on a remote machine, you will need to port forward the websocket port (defaults to 7007).
|
||||
|
||||
<p align="center">
|
||||
<img width="800" alt="image" src="https://user-images.githubusercontent.com/3310961/202766653-586a0daa-466b-4140-a136-6b02f2ce2c54.png">
|
||||
</p>
|
||||
|
||||
:::{admonition} Note
|
||||
:class: note
|
||||
|
||||
- You may have to change the port using `--viewer.websocket-port`.
|
||||
- All data configurations must go at the end. In this case, `nerfstudio-data` and all of its corresponding configurations come at the end after the model and viewer specification.
|
||||
:::
|
||||
|
||||
## Resume from checkpoint
|
||||
|
||||
It is possible to load a pretrained model by running
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --data data/nerfstudio/poster --load-dir {outputs/.../nerfstudio_models}
|
||||
```
|
||||
|
||||
## Visualize existing run
|
||||
|
||||
Given a pretrained model checkpoint, you can start the viewer by running
|
||||
|
||||
```bash
|
||||
ns-viewer --load-config {outputs/.../config.yml}
|
||||
```
|
||||
|
||||
## Exporting Results
|
||||
|
||||
Once you have a NeRF model you can either render out a video or export a point cloud.
|
||||
|
||||
### Render Video
|
||||
|
||||
First we must create a path for the camera to follow. This can be done in the viewer under the "RENDER" tab. Orient your 3D view to the location where you wish the video to start, then press "ADD CAMERA". This will set the first camera key frame. Continue to new viewpoints adding additional cameras to create the camera path. We provide other parameters to further refine your camera path. Once satisfied, press "RENDER" which will display a modal that contains the command needed to render the video. Kill the training job (or create a new terminal if you have lots of compute) and run the command to generate the video.
|
||||
|
||||
Other video export options are available, learn more by running
|
||||
|
||||
```bash
|
||||
ns-render --help
|
||||
```
|
||||
|
||||
### Generate Point Cloud
|
||||
|
||||
While NeRF models are not designed to generate point clouds, it is still possible. Navigate to the "EXPORT" tab in the 3D viewer and select "POINT CLOUD". If the crop option is selected, everything in the yellow square will be exported into a point cloud. Modify the settings as desired then run the command at the bottom of the panel in your command line.
|
||||
|
||||
Alternatively you can use the CLI without the viewer. Learn about the export options by running
|
||||
|
||||
```bash
|
||||
ns-export pointcloud --help
|
||||
```
|
||||
|
||||
## Intro to nerfstudio CLI and Configs
|
||||
|
||||
Nerfstudio allows customization of training and eval configs from the CLI in a powerful way, but there are some things to understand.
|
||||
|
||||
The most demonstrative and helpful example of the CLI structure is the difference in output between the following commands:
|
||||
|
||||
The following will list the supported models
|
||||
|
||||
```bash
|
||||
ns-train --help
|
||||
```
|
||||
|
||||
Applying `--help` after the model specification will provide the model and training specific arguments.
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --help
|
||||
```
|
||||
|
||||
At the end of the command you can specify the dataparser used. By default we use the _nerfstudio-data_ dataparser. We include other dataparsers such as _Blender_, _NuScenes_, ect. For a list of dataparse specific arguments, add `--help` to the end of the command
|
||||
|
||||
```bash
|
||||
ns-train nerfacto <nerfacto optional args> nerfstudio-data --help
|
||||
```
|
||||
|
||||
Each script will have some other minor quirks (like the training script dataparser subcommand needing to come after the model subcommand), read up on them [here](../reference/cli/index.md).
|
||||
|
||||
## Comet / Tensorboard / WandB / Viewer
|
||||
|
||||
We support four different methods to track training progress, using the viewer [tensorboard](https://www.tensorflow.org/tensorboard), [Weights and Biases](https://wandb.ai/site), and [Comet](https://comet.com/?utm_source=nerf&utm_medium=referral&utm_content=nerf_docs). You can specify which visualizer to use by appending `--vis {viewer, tensorboard, wandb, comet, viewer+wandb, viewer+tensorboard, viewer+comet}` to the training command. Simultaneously utilizing the viewer alongside wandb or tensorboard may cause stuttering issues during evaluation steps. The viewer only works for methods that are fast (ie. nerfacto, instant-ngp), for slower methods like NeRF, use the other loggers.
|
||||
|
||||
## Evaluating Runs
|
||||
|
||||
Calculate the psnr of your trained model and save to a json file.
|
||||
|
||||
```bash
|
||||
ns-eval --load-config={PATH_TO_CONFIG} --output-path=output.json
|
||||
```
|
||||
|
||||
We also provide a train and evaluation script that allows you to do benchmarking on the classical Blender dataset (see our [benchmarking workflow](../developer_guides/debugging_tools/benchmarking.md)).
|
||||
|
||||
## Multi-GPU Training
|
||||
|
||||
Here we explain how to use multi-GPU training. We are using [PyTorch Distributed Data Parallel (DDP)](https://pytorch.org/tutorials/beginner/dist_overview.html), so gradients are averaged over devices. If the loss scales depend on sample size (usually not the case in our implementation), we need to scale the learning rate with the number of GPUs used. Plotting will only be done for the first process. Note that you may want to play around with both learning rate and `<X>_num_rays_per_batch` when using DDP. Below is a simple example for how you'd run the `nerfacto-big` method on the aspen scene (see above to download the data), with either 1 or 2 GPUs. The `nerfacto-big` method uses a larger model size than the `nerfacto` method, so it benefits more from multi-GPU training.
|
||||
|
||||
First, download the aspen scene.
|
||||
|
||||
```python
|
||||
ns-download-data nerfstudio --capture-name=aspen
|
||||
```
|
||||
|
||||
```python
|
||||
# 1 GPU (8192 rays per GPU per batch)
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
ns-train nerfacto-big --vis viewer+wandb --machine.num-devices 1 --pipeline.datamanager.train-num-rays-per-batch 4096 --data data/nerfstudio/aspen
|
||||
```
|
||||
|
||||
You would observe about ~70k rays / sec on NVIDIA V100.
|
||||
|
||||
```
|
||||
Step (% Done) Train Iter (time) ETA (time) Train Rays / Sec
|
||||
-----------------------------------------------------------------------------------
|
||||
610 (0.61%) 115.968 ms 3 h, 12 m, 6 s 72.68 K
|
||||
620 (0.62%) 115.908 ms 3 h, 11 m, 58 s 72.72 K
|
||||
630 (0.63%) 115.907 ms 3 h, 11 m, 57 s 72.73 K
|
||||
640 (0.64%) 115.937 ms 3 h, 11 m, 59 s 72.71 K
|
||||
650 (0.65%) 115.853 ms 3 h, 11 m, 49 s 72.76 K
|
||||
660 (0.66%) 115.710 ms 3 h, 11 m, 34 s 72.85 K
|
||||
670 (0.67%) 115.797 ms 3 h, 11 m, 42 s 72.80 K
|
||||
680 (0.68%) 115.783 ms 3 h, 11 m, 39 s 72.81 K
|
||||
690 (0.69%) 115.756 ms 3 h, 11 m, 35 s 72.81 K
|
||||
700 (0.70%) 115.755 ms 3 h, 11 m, 34 s 72.81 K
|
||||
```
|
||||
|
||||
By having more GPUs in the training, you can allocate batch size to multiple GPUs and average their gradients.
|
||||
|
||||
```python
|
||||
# 2 GPUs (4096 rays per GPU per batch, effectively 8192 rays per batch)
|
||||
export CUDA_VISIBLE_DEVICES=0,1
|
||||
ns-train nerfacto --vis viewer+wandb --machine.num-devices 2 --pipeline.datamanager.train-num-rays-per-batch 4096 --data data/nerfstudio/aspen
|
||||
```
|
||||
|
||||
You would get improved throughput (~100k rays / sec on two NVIDIA V100).
|
||||
|
||||
```
|
||||
Step (% Done) Train Iter (time) ETA (time) Train Rays / Sec
|
||||
-----------------------------------------------------------------------------------
|
||||
1910 (1.91%) 79.623 ms 2 h, 10 m, 10 s 104.92 K
|
||||
1920 (1.92%) 79.083 ms 2 h, 9 m, 16 s 105.49 K
|
||||
1930 (1.93%) 79.092 ms 2 h, 9 m, 16 s 105.48 K
|
||||
1940 (1.94%) 79.364 ms 2 h, 9 m, 42 s 105.21 K
|
||||
1950 (1.95%) 79.327 ms 2 h, 9 m, 38 s 105.25 K
|
||||
1960 (1.96%) 79.473 ms 2 h, 9 m, 51 s 105.09 K
|
||||
1970 (1.97%) 79.334 ms 2 h, 9 m, 37 s 105.26 K
|
||||
1980 (1.98%) 79.200 ms 2 h, 9 m, 23 s 105.38 K
|
||||
1990 (1.99%) 79.264 ms 2 h, 9 m, 28 s 105.29 K
|
||||
2000 (2.00%) 79.168 ms 2 h, 9 m, 18 s 105.40 K
|
||||
```
|
||||
|
||||
During training, the "Train Rays / Sec" throughput represents the total number of training rays it processes per second, gradually increase the number of GPUs and observe how this throughput improves and eventually saturates.
|
||||
@@ -0,0 +1,440 @@
|
||||
# Installation
|
||||
|
||||
## Prerequisites
|
||||
|
||||
::::::{tab-set}
|
||||
:::::{tab-item} Linux
|
||||
|
||||
Nerfstudio requires `python >= 3.8`. We recommend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html) before proceeding.
|
||||
|
||||
:::::
|
||||
:::::{tab-item} Windows
|
||||
|
||||
:::{admonition} Note
|
||||
:class: info
|
||||
Nerfstudio on Windows is less tested and more fragile, due to way more moving parts outside of Nerfstudio's control.
|
||||
The instructions also tend to break over time as updates to different Windows packages happen.
|
||||
Installing Nerfstudio on Linux instead is recommended if you have the option.
|
||||
Alternatively, installing Nerfstudio under WSL2 (temporary unofficial guide [here](https://gist.github.com/SharkWipf/0a3fc1be3ea88b0c9640db6ce15b44b9), not guaranteed to work) is also an option, but this comes with its own set of caveats.
|
||||
:::
|
||||
|
||||
Install [Git](https://git-scm.com/downloads).
|
||||
|
||||
Install Visual Studio 2022. This must be done before installing CUDA. The necessary components are included in the `Desktop Development with C++` workflow (also called `C++ Build Tools` in the BuildTools edition).
|
||||
|
||||
Install Visual Studio Build Tools. If MSVC 143 does not work (usually will fail if your version > 17.10), you may also need to install MSVC 142 for Visual Studio 2019. Ensure your CUDA environment is set up properly.
|
||||
|
||||
Activate your Visual C++ environment:
|
||||
Navigate to the directory where `vcvars64.bat` is located. This path might vary depending on your installation. A common path is:
|
||||
|
||||
```
|
||||
C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build
|
||||
```
|
||||
|
||||
Run the following command:
|
||||
```bash
|
||||
./vcvars64.bat
|
||||
```
|
||||
|
||||
If the above command does not work, try activating an older version of VC:
|
||||
```bash
|
||||
./vcvarsall.bat x64 -vcvars_ver=<your_VC++_compiler_toolset_version>
|
||||
```
|
||||
Replace `<your_VC++_compiler_toolset_version>` with the version of your VC++ compiler toolset. The version number should appear in the same folder.
|
||||
|
||||
For example:
|
||||
```bash
|
||||
./vcvarsall.bat x64 -vcvars_ver=14.29
|
||||
```
|
||||
:::{admonition} Note
|
||||
:class: info
|
||||
When updating, or if you close your terminal before you finish the installation and run your first `splatfacto`, you have to re-do this environment activation step.
|
||||
:::
|
||||
|
||||
Nerfstudio requires `python >= 3.8`. We recommend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html) before proceeding.
|
||||
|
||||
:::::
|
||||
::::::
|
||||
|
||||
## Create environment
|
||||
|
||||
```bash
|
||||
conda create --name nerfstudio -y python=3.8
|
||||
conda activate nerfstudio
|
||||
python -m pip install --upgrade pip
|
||||
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
(pytorch)=
|
||||
|
||||
### PyTorch
|
||||
|
||||
Note that if a PyTorch version prior to 2.0.1 is installed,
|
||||
the previous version of pytorch, functorch, and tiny-cuda-nn should be uninstalled.
|
||||
|
||||
```bash
|
||||
pip uninstall torch torchvision functorch tinycudann
|
||||
```
|
||||
|
||||
::::{tab-set}
|
||||
:::{tab-item} Torch 2.1.2 with CUDA 11.8 (recommended)
|
||||
|
||||
Install PyTorch 2.1.2 with CUDA 11.8:
|
||||
|
||||
```bash
|
||||
pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
|
||||
```
|
||||
|
||||
To build the necessary CUDA extensions, `cuda-toolkit` is also required. We
|
||||
recommend installing with conda:
|
||||
|
||||
```bash
|
||||
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
|
||||
```
|
||||
|
||||
:::
|
||||
:::{tab-item} Torch 2.0.1 with CUDA 11.7
|
||||
|
||||
Install PyTorch 2.0.1 with CUDA 11.7:
|
||||
|
||||
```bash
|
||||
pip install torch==2.0.1+cu117 torchvision==0.15.2+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
|
||||
```
|
||||
|
||||
To build the necessary CUDA extensions, `cuda-toolkit` is also required. We
|
||||
recommend installing with conda:
|
||||
|
||||
```bash
|
||||
conda install -c "nvidia/label/cuda-11.7.1" cuda-toolkit
|
||||
```
|
||||
|
||||
:::
|
||||
::::
|
||||
|
||||
### Install tiny-cuda-nn
|
||||
|
||||
::::::{tab-set}
|
||||
:::::{tab-item} Linux
|
||||
|
||||
After pytorch and ninja, install the torch bindings for tiny-cuda-nn:
|
||||
```bash
|
||||
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
|
||||
```
|
||||
|
||||
:::::
|
||||
:::::{tab-item} Windows
|
||||
|
||||
Install the torch bindings for tiny-cuda-nn:
|
||||
```bash
|
||||
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
|
||||
```
|
||||
|
||||
:::::
|
||||
::::::
|
||||
|
||||
## Installing nerfstudio
|
||||
|
||||
**From pip**
|
||||
|
||||
```bash
|
||||
pip install nerfstudio
|
||||
```
|
||||
|
||||
**From source**
|
||||
Optional, use this command if you want the latest development version.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/nerfstudio-project/nerfstudio.git
|
||||
cd nerfstudio
|
||||
pip install --upgrade pip setuptools
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
:::{admonition} Note
|
||||
:class: info
|
||||
Below are optional installations, but makes developing with nerfstudio much more convenient.
|
||||
:::
|
||||
|
||||
**Tab completion (bash & zsh)**
|
||||
|
||||
This needs to be rerun when the CLI changes, for example if nerfstudio is updated.
|
||||
|
||||
```bash
|
||||
ns-install-cli
|
||||
```
|
||||
|
||||
**Development packages**
|
||||
|
||||
```bash
|
||||
pip install -e .[dev]
|
||||
pip install -e .[docs]
|
||||
```
|
||||
|
||||
## Using Pixi
|
||||
[Pixi](https://pixi.sh/latest/) is a fast software package manager built on top of the existing conda ecosystem. Spins up development environments quickly on Windows, macOS and Linux. (Currently only linux is supported for nerfstudio)
|
||||
|
||||
### Prerequisites
|
||||
Make sure to have pixi installed, detailed instructions [here](https://pixi.sh/latest/)
|
||||
|
||||
TLDR for linux:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://pixi.sh/install.sh | bash
|
||||
```
|
||||
|
||||
### Install Pixi Environmnent
|
||||
After Pixi is installed, you can run
|
||||
```bash
|
||||
git clone https://github.com/nerfstudio-project/nerfstudio.git
|
||||
cd nerfstudio
|
||||
pixi run post-install
|
||||
pixi shell
|
||||
```
|
||||
This will fetch the latest Nerfstudio code, install all enviroment dependencies including colmap, tinycudann and hloc, and then activate the pixi environment (similar to conda).
|
||||
From now on, each time you want to run Nerfstudio in a new shell, you have to navigate to the nerfstudio folder and run `pixi shell` again.
|
||||
|
||||
You could also run
|
||||
|
||||
```bash
|
||||
pixi run post-install
|
||||
pixi run train-example-nerf
|
||||
```
|
||||
|
||||
to download an example dataset and run nerfacto straight away.
|
||||
|
||||
Note that this method gets you the very latest upstream Nerfstudio version, if you want to use a specific release, you have to first checkout a specific version or commit in the nerfstudio folder, i.e.:
|
||||
```
|
||||
git checkout tags/v1.1.3
|
||||
```
|
||||
|
||||
Similarly, if you want to update, you want to update the git repo in your nerfstudio folder:
|
||||
```
|
||||
git pull
|
||||
```
|
||||
Remember that if you ran a checkout on a specific tag before, you have to manually specify a new tag or `git checkout main` to see the new changes.
|
||||
|
||||
## Use docker image
|
||||
|
||||
Instead of installing and compiling prerequisites, setting up the environment and installing dependencies, a ready to use docker image is provided.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Docker ([get docker](https://docs.docker.com/get-docker/)) and nvidia GPU drivers ([get nvidia drivers](https://www.nvidia.de/Download/index.aspx?lang=de)), capable of working with CUDA 11.8, must be installed.
|
||||
The docker image can then either be pulled from [here](https://github.com/nerfstudio-project/nerfstudio/pkgs/container/nerfstudio) (`latest` can be replaced with a fixed version, e.g., `1.1.3`)
|
||||
|
||||
```bash
|
||||
docker pull ghcr.io/nerfstudio-project/nerfstudio:latest
|
||||
```
|
||||
|
||||
or be built from the repository using
|
||||
|
||||
```bash
|
||||
docker build --tag nerfstudio -f Dockerfile .
|
||||
```
|
||||
|
||||
To restrict to only CUDA architectures that you have available locally, use the `CUDA_ARCHITECTURES`
|
||||
build arg and look up [the compute capability for your GPU](https://developer.nvidia.com/cuda-gpus).
|
||||
For example, here's how to build with support for GeForce 30xx series GPUs:
|
||||
|
||||
```bash
|
||||
docker build \
|
||||
--build-arg CUDA_ARCHITECTURES=86 \
|
||||
--tag nerfstudio-86 \
|
||||
--file Dockerfile .
|
||||
```
|
||||
|
||||
### Using an interactive container
|
||||
|
||||
The docker container can be launched with an interactive terminal where nerfstudio commands can be entered as usual. Some parameters are required and some are strongly recommended for usage as following:
|
||||
|
||||
```bash
|
||||
docker run --gpus all \ # Give the container access to nvidia GPU (required).
|
||||
-u $(id -u) \ # To prevent abusing of root privilege, please use custom user privilege to start.
|
||||
-v /folder/of/your/data:/workspace/ \ # Mount a folder from the local machine into the container to be able to process them (required).
|
||||
-v /home/<YOUR_USER>/.cache/:/home/user/.cache/ \ # Mount cache folder to avoid re-downloading of models everytime (recommended).
|
||||
-p 7007:7007 \ # Map port from local machine to docker container (required to access the web interface/UI).
|
||||
--rm \ # Remove container after it is closed (recommended).
|
||||
-it \ # Start container in interactive mode.
|
||||
--shm-size=12gb \ # Increase memory assigned to container to avoid memory limitations, default is 64 MB (recommended).
|
||||
ghcr.io/nerfstudio-project/nerfstudio:<tag> # Docker image name if you pulled from GitHub.
|
||||
<--- OR --->
|
||||
nerfstudio # Docker image tag if you built the image from the Dockerfile by yourself using the command from above.
|
||||
```
|
||||
|
||||
### Call nerfstudio commands directly
|
||||
|
||||
Besides, the container can also directly be used by adding the nerfstudio command to the end.
|
||||
|
||||
```bash
|
||||
docker run --gpus all -u $(id -u) -v /folder/of/your/data:/workspace/ -v /home/<YOUR_USER>/.cache/:/home/user/.cache/ -p 7007:7007 --rm -it --shm-size=12gb # Parameters.
|
||||
ghcr.io/nerfstudio-project/nerfstudio:<tag> \ # Docker image name if you pulled from GitHub.
|
||||
ns-process-data video --data /workspace/video.mp4 # Smaple command of nerfstudio.
|
||||
```
|
||||
|
||||
### Note
|
||||
|
||||
- The container works on Linux and Windows, depending on your OS some additional setup steps might be required to provide access to your GPU inside containers.
|
||||
- Paths on Windows use backslash '\\' while unix based systems use a frontslash '/' for paths, where backslashes might require an escape character depending on where they are used (e.g. C:\\\\folder1\\\\folder2...). Alternatively, mounts can be quoted (e.g. `-v 'C:\local_folder:/docker_folder'`). Ensure to use the correct paths when mounting folders or providing paths as parameters.
|
||||
- Always use full paths, relative paths are known to create issues when being used in mounts into docker.
|
||||
- Everything inside the container, what is not in a mounted folder (workspace in the above example), will be permanently removed after destroying the container. Always do all your tasks and output folder in workdir!
|
||||
- The container currently is based on nvidia/cuda:11.8.0-devel-ubuntu22.04, consequently it comes with CUDA 11.8 which must be supported by the nvidia driver. No local CUDA installation is required or will be affected by using the docker image.
|
||||
- The docker image (respectively Ubuntu 22.04) comes with Python3.10, no older version of Python is installed.
|
||||
- If you call the container with commands directly, you still might want to add the interactive terminal ('-it') flag to get live log outputs of the nerfstudio scripts. In case the container is used in an automated environment the flag should be discarded.
|
||||
- The current version of docker is built for multi-architecture (CUDA architectures) use. The target architecture(s) must be defined at build time for Colmap and tinyCUDNN to be able to compile properly. If your GPU architecture is not covered by the following table you need to replace the number in the line `ARG CUDA_ARCHITECTURES=90;89;86;80;75;70;61;52;37` to your specific architecture. It also is a good idea to remove all architectures but yours (e.g. `ARG CUDA_ARCHITECTURES=86`) to speedup the docker build process a lot.
|
||||
- To avoid memory issues or limitations during processing, it is recommended to use either `--shm-size=12gb` or `--ipc=host` to increase the memory available to the docker container. 12gb as in the example is only a suggestion and may be replaced by other values depending on your hardware and requirements.
|
||||
|
||||
**Currently supported CUDA architectures in the docker image**
|
||||
|
||||
(tiny-cuda-arch-list)=
|
||||
|
||||
| GPU | CUDA arch |
|
||||
| --------------- | --------- |
|
||||
| H100 | 90 |
|
||||
| 40X0 | 89 |
|
||||
| 30X0 | 86 |
|
||||
| A100 | 80 |
|
||||
| 20X0 | 75 |
|
||||
| TITAN V / V100 | 70 |
|
||||
| 10X0 / TITAN Xp | 61 |
|
||||
| 9X0 | 52 |
|
||||
| K80 | 37 |
|
||||
|
||||
## Installation FAQ
|
||||
|
||||
- [ImportError: DLL load failed while importing \_89_C](tiny-cuda-mismatch-arch)
|
||||
- [tiny-cuda-nn installation errors out with cuda mismatch](tiny-cuda-mismatch-error)
|
||||
- [tiny-cuda-nn installation errors out with no CUDA toolset found](tiny-cuda-integration-error)
|
||||
- [Installation errors, File "setup.py" not found](pip-install-error)
|
||||
- [Runtime errors, "len(sources) > 0".](cuda-sources-error)
|
||||
|
||||
<br />
|
||||
|
||||
(tiny-cuda-mismatch-arch)=
|
||||
|
||||
**ImportError: DLL load failed while importing \_89_C**
|
||||
|
||||
This occurs with certain GPUs that have CUDA architecture versions (89 in the example above) for which tiny-cuda-nn does not automatically compile support.
|
||||
|
||||
**Solution**:
|
||||
|
||||
Reinstall tiny-cuda-nn with the following command:
|
||||
|
||||
::::::{tab-set}
|
||||
:::::{tab-item} Linux
|
||||
|
||||
```bash
|
||||
TCNN_CUDA_ARCHITECTURES=XX pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
|
||||
```
|
||||
|
||||
:::::
|
||||
:::::{tab-item} Windows
|
||||
|
||||
```bash
|
||||
set TCNN_CUDA_ARCHITECTURES=XX
|
||||
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
|
||||
```
|
||||
|
||||
:::::
|
||||
::::::
|
||||
|
||||
Where XX is the architecture version listed [here](tiny-cuda-arch-list). Ie. for a 4090 GPU use `TCNN_CUDA_ARCHITECTURES=89`
|
||||
|
||||
<br />
|
||||
|
||||
(tiny-cuda-mismatch-error)=
|
||||
|
||||
**tiny-cuda-nn installation errors out with cuda mismatch**
|
||||
|
||||
While installing tiny-cuda, you run into: `The detected CUDA version mismatches the version that was used to compile PyTorch (10.2). Please make sure to use the same CUDA versions.`
|
||||
|
||||
**Solution**:
|
||||
|
||||
Reinstall PyTorch with the correct CUDA version.
|
||||
See [pytorch](pytorch) under Dependencies, above.
|
||||
|
||||
<br />
|
||||
|
||||
(tiny-cuda-integration-error)=
|
||||
|
||||
**(Windows) tiny-cuda-nn installation errors out with no CUDA toolset found**
|
||||
|
||||
While installing tiny-cuda on Windows, you run into: `No CUDA toolset found.`
|
||||
|
||||
**Solution**:
|
||||
|
||||
Confirm that you have Visual Studio installed.
|
||||
|
||||
Make sure CUDA Visual Studio integration is enabled. This should be done automatically by the CUDA installer if it is run after Visual Studio is installed. You can also manually enable integration.
|
||||
|
||||
::::{tab-set}
|
||||
:::{tab-item} Visual Studio 2019
|
||||
|
||||
To manually enable integration for Visual Studio 2019, copy all 4 files from
|
||||
|
||||
```
|
||||
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\extras\visual_studio_integration\MSBuildExtensions
|
||||
```
|
||||
|
||||
to
|
||||
|
||||
```
|
||||
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\BuildCustomizations
|
||||
```
|
||||
|
||||
:::
|
||||
:::{tab-item} Visual Studio 2022
|
||||
|
||||
To manually enable integration for Visual Studio 2022, copy all 4 files from
|
||||
|
||||
```
|
||||
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\extras\visual_studio_integration\MSBuildExtensions
|
||||
```
|
||||
|
||||
to
|
||||
|
||||
```
|
||||
C:\Program Files\Microsoft Visual Studio\2022\[Community, Professional, Enterprise, or BuildTools]\MSBuild\Microsoft\VC\v160\BuildCustomizations
|
||||
```
|
||||
|
||||
:::
|
||||
::::
|
||||
|
||||
<br />
|
||||
|
||||
(pip-install-error)=
|
||||
|
||||
**Installation errors, File "setup.py" not found**
|
||||
|
||||
When installing dependencies and nerfstudio with `pip install -e .`, you run into: `ERROR: File "setup.py" not found. Directory cannot be installed in editable mode`
|
||||
|
||||
**Solution**:
|
||||
This can be fixed by upgrading pip to the latest version:
|
||||
|
||||
```
|
||||
python -m pip install --upgrade pip
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
(cuda-sources-error)=
|
||||
|
||||
**Runtime errors: "len(sources) > 0", "ctype = \_C.ContractionType(type.value) ; TypeError: 'NoneType' object is not callable".**
|
||||
|
||||
When running `train.py `, an error occurs when installing cuda files in the backend code.
|
||||
|
||||
**Solution**:
|
||||
This is a problem with not being able to detect the correct CUDA version, and can be fixed by updating the CUDA path environment variables:
|
||||
|
||||
```
|
||||
export CUDA_HOME=/usr/local/cuda
|
||||
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
|
||||
export PATH=$PATH:$CUDA_HOME/bin
|
||||
```
|
||||
|
||||
**Other errors**
|
||||
|
||||
(Windows) A lot of errors on Windows can be caused by not having the Visual Studio environment loaded.
|
||||
If you run into errors you can't figure out, please try re-activating the Visual Studio environment (as outlined at the top of the Windows instructions on this page) and try again.
|
||||
This activation only lasts within your current terminal session and does not extend to other terminals, but this should only be needed on first run and on updates.
|
||||
@@ -0,0 +1,95 @@
|
||||
Using the viewer
|
||||
================
|
||||
|
||||
The nerfstudio web-based viewer makes it easy to view training in real-time, and to create content videos from your trained models 🌟!
|
||||
|
||||
Connecting to the viewer
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The nerfstudio viewer is launched automatically with each :code:`ns-train` training run! It can also be run separately with :code:`ns-viewer`.
|
||||
To access a viewer, we need to enter the server's address and port in a web browser.
|
||||
|
||||
Accessing on a local machine
|
||||
""""""""""""""""""""""""""""
|
||||
|
||||
You should be able to click the link obtained while running a script to open the viewer in your browser. This should typically look something like :code:`http://localhost:7007`.
|
||||
|
||||
Accessing over an SSH connection
|
||||
""""""""""""""""""""""""""""""""
|
||||
|
||||
If you are training on a remote machine, the viewer will still let you view your trainings. You will need to forward traffic from your viewing host to the listening port on the host running ns-train (7007 by default). You can achieve this by securely tunneling traffic on the specified port through an ssh session. In a terminal window on the viewing host, issue the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ssh -L 7007:127.0.0.1:7007 <username>@<training-host-ip>
|
||||
|
||||
|
||||
.. admonition:: Note
|
||||
|
||||
You can now simply open the link (same one shown in image above) in your browser on your local machine and it should connect!. So long as you don't close this terminal window with this specific active ssh connection, the port will remain open for all your other ssh sessions.
|
||||
|
||||
For example, if you do this in a new terminal window, any existing ssh sessions (terminal windows, VSCode remote connection, etc) will still be able to access the port, but if you close this terminal window, the port will be closed.
|
||||
|
||||
|
||||
.. warning::
|
||||
If the port is being used, you will need to switch the port using the `--viewer.websocket-port` flag tied to the model subcommand.
|
||||
|
||||
|
||||
Accessing via a share link
|
||||
""""""""""""""""""""""""""
|
||||
|
||||
To connect to remote machines without requiring an SSH connection, we also support creating share URLs.
|
||||
This can be generated from the "Share:" icon in the GUI, or from the CLI by specifying the :code:`--viewer.make-share-url True` argument.
|
||||
This will create a publically accessible link that you can share with others to view your training progress.
|
||||
This is useful for sharing training progress with collaborators, or for viewing training progress on a remote machine without requiring an SSH connection.
|
||||
However, it will introduce some latency in the viewer.
|
||||
|
||||
|
||||
.. seealso::
|
||||
|
||||
For a more in-depth developer overview on how to hack with the viewer, see our `developer guide </docs/_build/html/developer_guides/viewer/index.html>`_
|
||||
|
||||
|
||||
Legacy viewer tutorial video
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In the tutorial video below, we walk you through how you can turn a simple capture into a 3D video 📸 🎬
|
||||
|
||||
This video is for the legacy viewer, which was the default in :code:`nerfstudio<1.0`. It has some visual differences from the current viewer. However, the core functionality is the same.
|
||||
The legacy viewer can still be enabled with the :code:`--vis viewer_legacy` option, for example, :code:`ns-train nerfacto --vis viewer_legacy`.
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<iframe width="560" height="315" src="https://www.youtube.com/embed/nSFsugarWzk" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
|
||||
|
||||
For specific sections, click the links below to navigate to the associated portion of the video.
|
||||
|
||||
Getting started
|
||||
"""""""""""""""
|
||||
|
||||
* `Hello from nerfstudio <https://youtu.be/nSFsugarWzk?t=0>`_
|
||||
* `Preprocess your video <https://youtu.be/nSFsugarWzk?t=13>`_
|
||||
* `Launching training and viewer <https://youtu.be/nSFsugarWzk?t=27>`_
|
||||
|
||||
Viewer basics
|
||||
"""""""""""""""
|
||||
|
||||
* `Viewer scene introduction <https://youtu.be/nSFsugarWzk?t=63>`_
|
||||
* `Moving around in viewer <https://youtu.be/nSFsugarWzk?t=80>`_
|
||||
* `Overview of Controls Panel - train speed/output options <https://youtu.be/nSFsugarWzk?t=98>`_
|
||||
* `Overview of Scene Panel - toggle visibility <https://youtu.be/nSFsugarWzk?t=115>`_
|
||||
|
||||
Creating camera trajectories
|
||||
""""""""""""""""""""""""""""
|
||||
|
||||
* `Creating a custom camera path <https://youtu.be/nSFsugarWzk?t=136>`_
|
||||
* `Camera spline options - cycle, speed, smoothness <https://youtu.be/nSFsugarWzk?t=158>`_
|
||||
* `Camera options - move, add, view <https://youtu.be/nSFsugarWzk?t=177>`_
|
||||
|
||||
Rendering a video
|
||||
"""""""""""""""""
|
||||
|
||||
* `Preview camera trajectory <https://youtu.be/nSFsugarWzk?t=206>`_
|
||||
* `How to render final video <https://youtu.be/nSFsugarWzk?t=227>`_
|
||||
|
||||
|
|
||||
Reference in new issue
Block a user