mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-14 11:54:01 +08:00
merging main
This commit is contained in:
terrance committed
commit
80cdb73406
221 files changed
+50645
-2513
No files matched your search
@@ -5,7 +5,7 @@ Training model on existing datasets is only so fun. If you would like to train o
|
||||
To process your own data run:
|
||||
|
||||
```bash
|
||||
ns-process-data {video,images,polycam,insta360,record3d} --data {DATA_PATH} --output-dir {PROCESSED_DATA_DIR}
|
||||
ns-process-data {video,images,polycam,record3d} --data {DATA_PATH} --output-dir {PROCESSED_DATA_DIR}
|
||||
```
|
||||
|
||||
A full set of arguments can be found {doc}`here</reference/cli/ns_process_data>`.
|
||||
@@ -20,6 +20,7 @@ We Currently support the following custom data types:
|
||||
| 📱 [KIRI Engine](kiri) | IOS or Android | [KIRI Engine App](https://www.kiriengine.com/) | 🐇 |
|
||||
| 📱 [Record3D](record3d) | IOS with LiDAR | [Record3D app](https://record3d.app/) | 🐇 |
|
||||
| 🖥 [Metashape](metashape) | Any | [Metashape](https://www.agisoft.com/) | 🐇 |
|
||||
| 🖥 [RealityCapture](realitycapture) | Any | [RealityCapture](https://www.capturingreality.com/realitycapture) | 🐇 |
|
||||
|
||||
(images_and_video)=
|
||||
|
||||
@@ -33,12 +34,6 @@ To assist running on custom data we have a script that will process a video or f
|
||||
- COLMAP can be finicky. Try your best to capture overlapping, non-blurry images.
|
||||
:::
|
||||
|
||||
:::{admonition} Note
|
||||
:class: warning
|
||||
|
||||
- If COLMAP opens in a new window, simply close it to resume processing.
|
||||
:::
|
||||
|
||||
### Processing Data
|
||||
|
||||
```bash
|
||||
@@ -264,6 +259,8 @@ ns-train nerfacto --data {output directory}
|
||||
|
||||
## Metashape
|
||||
|
||||
All images must use the same sensor type (but multiple sensors are supported).
|
||||
|
||||
1. Align your images using Metashape. `File -> Workflow -> Align Photos...`
|
||||
|
||||
```{image} https://user-images.githubusercontent.com/3310961/203389662-12760210-2b52-49d4-ab21-4f23bfa4a2b3.png
|
||||
@@ -292,6 +289,29 @@ ns-process-data metashape --data {data directory} --xml {xml file} --output-dir
|
||||
ns-train nerfacto --data {output directory}
|
||||
```
|
||||
|
||||
<<<<<<< HEAD
|
||||
=======
|
||||
(realitycapture)=
|
||||
|
||||
## RealityCapture
|
||||
|
||||
1. Align your images using RealityCapture. `ALIGNMENT -> Align Images`
|
||||
|
||||
2. Export the camera alignment as a `csv` file. Choose `Internal/External camera parameters`
|
||||
|
||||
3. Convert the data to the nerfstudio format.
|
||||
|
||||
```bash
|
||||
ns-process-data realitycapture --data {data directory} --csv {csv file} --output-dir {output directory}
|
||||
```
|
||||
|
||||
4. Train with nerfstudio!
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --data {output directory}
|
||||
```
|
||||
|
||||
>>>>>>> b1ddb9e695b789fae47e0f760a5c7f59f25af0bb
|
||||
(360_data)=
|
||||
|
||||
## 360 Data (Equirectangular)
|
||||
@@ -303,10 +323,17 @@ Equirectangular data is data that has been taken by a 360 camera such as Insta36
|
||||
For a set of equirectangular images, process the data using the following command:
|
||||
|
||||
```bash
|
||||
<<<<<<< HEAD
|
||||
ns-process-data images --camera-type equirectangular --images-per-equirect {8, or 14} --data {data directory} --output-dir {output directory}
|
||||
```
|
||||
|
||||
The images-per-equirect argument is the number of images that will be sampled from each equirectangular image. We have found that 8 images per equirectangular image is sufficient for most use cases so it defaults to that. However, if you find that there isn't enough detail in the nerf or that colmap is having trouble aligning the images, you can try increasing the number of images per equirectangular image to 14.
|
||||
=======
|
||||
ns-process-data images --camera-type equirectangular --images-per-equirect {8, or 14} --crop-factor {top bottom left right} --data {data directory} --output-dir {output directory}
|
||||
```
|
||||
|
||||
The images-per-equirect argument is the number of images that will be sampled from each equirectangular image. We have found that 8 images per equirectangular image is sufficient for most use cases so it defaults to that. However, if you find that there isn't enough detail in the nerf or that colmap is having trouble aligning the images, you can try increasing the number of images per equirectangular image to 14. See the video section below for details on cropping.
|
||||
>>>>>>> b1ddb9e695b789fae47e0f760a5c7f59f25af0bb
|
||||
|
||||
### Videos
|
||||
|
||||
@@ -326,4 +353,10 @@ The `crop-factor` argument is optional but often very helpful. This is because e
|
||||
|
||||
<img src="imgs/equirect_crop.jpg">
|
||||
|
||||
<<<<<<< HEAD
|
||||
This obscene mesh of human is obviously not helpful in training a nerf so we can remove it by cropping the bottom 20% of the image. This can be done by using the `--crop-factor 0 0.2 0 0` argument.
|
||||
=======
|
||||
The pixels representing the distorted hand and head are obviously not useful in training a nerf so we can remove it by cropping the bottom 20% of the image. This can be done by using the `--crop-factor 0 0.2 0 0` argument.
|
||||
|
||||
If cropping only needs to be done from the bottom, you can use the `--crop-bottom [num]` argument which would be the same as doing `--crop-factor 0.0 [num] 0.0 0.0`
|
||||
>>>>>>> b1ddb9e695b789fae47e0f760a5c7f59f25af0bb
|
||||
@@ -19,7 +19,6 @@ Poisson surface reconstruction gives the highest quality meshes. See the steps b
|
||||
> **Note:**
|
||||
> This will only work with a Model that computes or predicts normals, e.g., nerfacto.
|
||||
|
||||
|
||||
1. Train nerfacto with network settings that predict normals.
|
||||
|
||||
```bash
|
||||
@@ -51,7 +50,7 @@ ns-export --help
|
||||
Say you want to simplify and/or smooth a mesh offline, and then you want to texture it with NeRF. You can do that with the following command. It will work for any mesh filetypes that [PyMeshLab](https://pymeshlab.readthedocs.io/en/latest/) can support, for example a `.ply`.
|
||||
|
||||
```python
|
||||
python scripts/texture.py --load-config CONFIG.yml --input-mesh-filename FILENAME --output-dir OUTPUT_DIR
|
||||
python nerfstudio/scripts/texture.py --load-config CONFIG.yml --input-mesh-filename FILENAME --output-dir OUTPUT_DIR
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
@@ -38,8 +38,6 @@ It is possible to load a pretrained model by running
|
||||
ns-train nerfacto --data data/nerfstudio/poster --load-dir {outputs/.../nerfstudio_models}
|
||||
```
|
||||
|
||||
This will automatically start training. If you do not want it to train, add `--viewer.start-train False` to your training command.
|
||||
|
||||
## Visualize existing run
|
||||
|
||||
Given a pretrained model checkpoint, you can start the viewer by running
|
||||
@@ -98,9 +96,9 @@ ns-train nerfacto <nerfacto optional args> nerfstudio-data --help
|
||||
|
||||
Each script will have some other minor quirks (like the training script dataparser subcommand needing to come after the model subcommand), read up on them [here](../reference/cli/index.md).
|
||||
|
||||
## Tensorboard / WandB
|
||||
## Tensorboard / WandB / Viewer
|
||||
|
||||
We support three different methods to track training progress, using the viewer, [tensorboard](https://www.tensorflow.org/tensorboard), and [Weights and Biases](https://wandb.ai/site). You can specify which visualizer to use by appending `--vis {viewer, tensorboard, wandb}` to the training command. Note that only one may be used at a time. Additionally the viewer only works for methods that are fast (ie. nerfacto, instant-ngp), for slower methods like NeRF, use the other loggers.
|
||||
We support three different methods to track training progress, using the viewer, [tensorboard](https://www.tensorflow.org/tensorboard), and [Weights and Biases](https://wandb.ai/site). You can specify which visualizer to use by appending `--vis {viewer, tensorboard, wandb, viewer+wandb, viewer+tensorboard}` to the training command. Simultaneously utilizing the viewer alongside wandb or tensorboard may cause stuttering issues during evaluation steps. The viewer only works for methods that are fast (ie. nerfacto, instant-ngp), for slower methods like NeRF, use the other loggers.
|
||||
|
||||
## Evaluating Runs
|
||||
|
||||
|
||||
@@ -33,7 +33,7 @@ Install CUDA 11.7. You can find CUDA download links [here](https://developer.nvi
|
||||
|
||||
## Create environment
|
||||
|
||||
Nerfstudio requires `python >= 3.7`. We recommend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html) before proceeding.
|
||||
Nerfstudio requires `python >= 3.8`. We recommend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html) before proceeding.
|
||||
|
||||
```bash
|
||||
conda create --name nerfstudio -y python=3.8
|
||||
@@ -71,7 +71,7 @@ pip uninstall torch torchvision functorch
|
||||
Install pytorch 1.13.1 with CUDA and [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn)
|
||||
|
||||
```bash
|
||||
pip install torch torchvision functorch --extra-index-url https://download.pytorch.org/whl/cu117
|
||||
pip install torch==1.13.1 torchvision functorch --extra-index-url https://download.pytorch.org/whl/cu117
|
||||
```
|
||||
|
||||
:::
|
||||
@@ -79,10 +79,10 @@ pip install torch torchvision functorch --extra-index-url https://download.pytor
|
||||
|
||||
### tinycudann
|
||||
|
||||
After pytorch, install the torch bindings for tinycudann:
|
||||
After pytorch and ninja, install the torch bindings for tinycudann:
|
||||
|
||||
```bash
|
||||
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
|
||||
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
|
||||
```
|
||||
|
||||
## Installing nerfstudio
|
||||
|
||||
Reference in new issue
Block a user