tink/nerfstudio
4
0
Fork 0
mirror of https://github.com/nerfstudio-project/nerfstudio.git synced 2026-11-11 10:34:29 +08:00
Code

Merge branch 'monodepth_regularization' of https://github.com/nerfstudio-project/nerfstudio into monodepth_regularization

Browse Source
This commit is contained in:
AdamRashid96 2023-08-01 23:01:45 +00:00
commit 8a1aaaa74f
9 changed files with 64 additions and 78 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
README.md
View File

@ -92,17 +92,20 @@ Nerfstudio requires `python >= 3.8`. We recommend using conda to manage dependen
```bash
conda create --name nerfstudio -y python=3.8
conda activate nerfstudio
python -m pip install --upgrade pip
pip install --upgrade pip
```
### Dependencies
Install pytorch with CUDA (this repo has been tested with CUDA 11.7 and CUDA 11.8) and [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn)
Install PyTorch with CUDA (this repo has been tested with CUDA 11.7 and CUDA 11.8) and [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn).
`cuda-toolkit` is required for building `tiny-cuda-nn`.
For CUDA 11.7:
```bash
pip install torch==2.0.1+cu117 torchvision==0.15.2+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
conda install -c "nvidia/label/cuda-11.7.1" cuda-toolkit
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
```
@ -110,6 +113,8 @@ For CUDA 11.8:
```bash
pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
```

7
docs/extensions/unreal_engine.md
View File

@ -10,14 +10,9 @@
NeRFStudio models can be used in Unreal Engine if they are converted to an NVOL file. NVOL is a new standard file format to store NeRFs in a fast and efficient way. NVOL files can be obtained from NeRFStudio checkpoints files (.ckpt) using the [Volinga Suite](https://volinga.ai/).
:::{admonition} Note
:class: warning
Volinga Suite is not yet publicly available. If you are interested in converting your models to NVOL, sign up for the beta at [volinga.ai](https://volinga.ai).
:::
## Exporting your model to NVOL
Currently NVOL file only supports Volinga model (which is based on nerfacto). You can train your model using the following command:
Currently NVOL file only supports Volinga model (which is based on nerfacto). To use Volinga model you will need to install [Volinga extension for NeRFStudio](https://github.com/Volinga/volinga-model). You can train your model using the following command:
```bash
ns-train volinga --data /path/to/your/data --vis viewer

15
docs/index.md
View File

@ -200,10 +200,11 @@ If you use this library or find the documentation useful for your research, plea
## Maintainers
| | Nerfstudio Discord | Affiliation |
| -------------------------------------------- | ------------------ | ------------------------------------ |
| [Justin Kerr](https://kerrj.github.io/) | justin.kerr | UC Berkeley |
| [Jonáš Kulhánek](https://jkulhanek.com/) | jkulhanek | Czech Technical University in Prague |
| [Matt Tancik](https://www.matthewtancik.com) | tancik | UC Berkeley |
| [Ethan Weber](https://ethanweber.me/) | ethanweber | UC Berkeley |
| [Brent Yi](https://github.com/brentyi) | brent | UC Berkeley |
| | Nerfstudio Discord | Affiliation |
| ----------------------------------------------- | ------------------ | ------------------------------------ |
| [Justin Kerr](https://kerrj.github.io/) | justin.kerr | UC Berkeley |
| [Jonáš Kulhánek](https://jkulhanek.com/) | jkulhanek | Czech Technical University in Prague |
| [Matt Tancik](https://www.matthewtancik.com) | tancik | Luma AI |
| [Matias Turkulainen](https://maturk.github.io/) | maturk | ETH Zurich |
| [Ethan Weber](https://ethanweber.me/) | ethanweber | UC Berkeley |
| [Brent Yi](https://github.com/brentyi) | brent | UC Berkeley |

41
docs/quickstart/installation.md
View File

@ -33,44 +33,47 @@ python -m pip install --upgrade pip
(pytorch)=
### pytorch
### PyTorch
::::{tab-set}
:::{tab-item} Torch 2.0.1 with CUDA 11.8
- To install 2.0.1 with CUDA 11.8:
Note that if a pytorch version prior to 2.0.1 is installed,
Note that if a PyTorch version prior to 2.0.1 is installed,
the previous version of pytorch, functorch, and tiny-cuda-nn should be uninstalled.
```bash
pip uninstall torch torchvision functorch tinycudann
```
Install pytorch 2.0.1 with CUDA and [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn)
::::{tab-set}
:::{tab-item} Torch 2.0.1 with CUDA 11.8
Install PyTorch 2.0.1 with CUDA 11.8:
```bash
pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
```
To build the necessary CUDA extensions, `cuda-toolkit` is also required. We
recommend installing with conda:
```bash
conda install -c "nvidia/label/cuda-11.7.1" cuda-toolkit
```
:::
:::{tab-item} Torch 2.0.1 with CUDA 11.7
- To install 2.0.1 with CUDA 11.7:
Note that if a pytorch version prior to 2.0.1 is installed,
the previous version of pytorch, functorch, and tiny-cuda-nn should be uninstalled.
```bash
pip uninstall torch torchvision functorch tinycudann
```
Install pytorch 2.0.1 with CUDA and [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn)
Install PyTorch 2.0.1 with CUDA 11.7:
```bash
pip install torch==2.0.1+cu117 torchvision==0.15.2+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
```
To build the necessary CUDA extensions, `cuda-toolkit` is also required. We
recommend installing with conda:
```bash
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
```
:::
::::
@ -258,7 +261,7 @@ While installing tiny-cuda, you run into: `The detected CUDA version mismatches
**Solution**:
Reinstall pytorch with the correct CUDA version.
Reinstall PyTorch with the correct CUDA version.
See [pytorch](pytorch) under Dependencies, above.
<br />

18
nerfstudio/configs/external_methods.py
View File

@ -121,6 +121,24 @@ To enable NeRFPlayer, you must install it first by running:
)
)
# Volinga
external_methods.append(
ExternalMethod(
"""[bold yellow]Volinga[/bold yellow]
For more information visit: https://docs.nerf.studio/en/latest/extensions/unreal_engine.html
To enable Volinga, you must install it first by running:
[grey]pip install git+https://github.com/Volinga/volinga-model[/grey]""",
configurations=[
(
"volinga",
"Real-time rendering model from Volinga. Directly exportable to NVOL format at https://volinga.ai/",
),
],
pip_package="git+https://github.com/Volinga/volinga-model",
)
)
@dataclass
class ExternalMethodTrainerConfig(TrainerConfig):

42
nerfstudio/configs/method_configs.py
View File

@ -68,7 +68,6 @@ method_configs: Dict[str, TrainerConfig] = {}
descriptions = {
"nerfacto": "Recommended real-time model tuned for real captures. This model will be continually updated.",
"depth-nerfacto": "Nerfacto with depth supervision.",
"volinga": "Real-time rendering model from Volinga. Directly exportable to NVOL format at https://volinga.ai/",
"instant-ngp": "Implementation of Instant-NGP. Recommended real-time model for unbounded scenes.",
"instant-ngp-bounded": "Implementation of Instant-NGP. Recommended for bounded real and synthetic scenes",
"mipnerf": "High quality model for bounded scenes. (slow)",
@ -235,47 +234,6 @@ method_configs["depth-nerfacto"] = TrainerConfig(
vis="viewer",
)
method_configs["volinga"] = TrainerConfig(
method_name="volinga",
steps_per_eval_batch=500,
steps_per_save=2000,
max_num_iterations=30000,
mixed_precision=True,
pipeline=VanillaPipelineConfig(
datamanager=VanillaDataManagerConfig(
dataparser=NerfstudioDataParserConfig(),
train_num_rays_per_batch=4096,
eval_num_rays_per_batch=4096,
camera_optimizer=CameraOptimizerConfig(
mode="SO3xR3", optimizer=AdamOptimizerConfig(lr=6e-4, eps=1e-8, weight_decay=1e-2)
),
),
model=NerfactoModelConfig(
eval_num_rays_per_chunk=1 << 15,
hidden_dim=32,
hidden_dim_color=32,
hidden_dim_transient=32,
num_nerf_samples_per_ray=24,
proposal_net_args_list=[
{"hidden_dim": 16, "log2_hashmap_size": 17, "num_levels": 5, "max_res": 128, "use_linear": True},
{"hidden_dim": 16, "log2_hashmap_size": 17, "num_levels": 5, "max_res": 256, "use_linear": True},
],
),
),
optimizers={
"proposal_networks": {
"optimizer": AdamOptimizerConfig(lr=1e-2, eps=1e-15),
"scheduler": None,
},
"fields": {
"optimizer": AdamOptimizerConfig(lr=1e-2, eps=1e-15),
"scheduler": None,
},
},
viewer=ViewerConfig(num_rays_per_chunk=1 << 15),
vis="viewer",
)
method_configs["instant-ngp"] = TrainerConfig(
method_name="instant-ngp",
steps_per_eval_batch=500,

6
nerfstudio/data/datamanagers/base_datamanager.py
View File

@ -114,8 +114,10 @@ class DataManagerConfig(InstantiateConfig):
"""Source of data, may not be used by all models."""
camera_optimizer: Optional[CameraOptimizerConfig] = None
"""Specifies the camera pose optimizer used during training. Helpful if poses are noisy."""
masks_on_gpu: Optional[bool] = None
masks_on_gpu: bool = False
"""Process masks on GPU for speed at the expense of memory, if True."""
images_on_gpu: bool = False
"""Process images on GPU for speed at the expense of memory, if True."""
class DataManager(nn.Module):
@ -402,6 +404,8 @@ class VanillaDataManager(DataManager, Generic[TDataset]):
self.exclude_batch_keys_from_device = self.train_dataset.exclude_batch_keys_from_device
if self.config.masks_on_gpu is True:
self.exclude_batch_keys_from_device.remove("mask")
if self.config.images_on_gpu is True:
self.exclude_batch_keys_from_device.remove("image")
if self.train_dataparser_outputs is not None:
cameras = self.train_dataparser_outputs.cameras

3
nerfstudio/data/dataparsers/sdfstudio_dataparser.py
View File

@ -42,6 +42,8 @@ class SDFStudioDataParserConfig(DataParserConfig):
"""Directory specifying location of data."""
include_mono_prior: bool = False
"""whether or not to load monocular depth and normal """
depth_unit_scale_factor: float = 1e-3
"""Scales the depth values to meters. Default value is 0.001 for a millimeter to meter conversion."""
include_foreground_mask: bool = False
"""whether or not to load foreground mask"""
downscale_factor: int = 1
@ -153,6 +155,7 @@ class SDFStudio(DataParser):
# required for normal maps, these are in colmap format so they require c2w before conversion
"camera_to_worlds": c2w_colmap if len(c2w_colmap) > 0 else None,
"include_mono_prior": self.config.include_mono_prior,
"depth_unit_scale_factor": self.config.depth_unit_scale_factor,
},
)
return dataparser_outputs

1
tests/test_train.py
View File

@ -21,7 +21,6 @@ BLACKLIST = [
"instant-ngp",
"instant-ngp-bounded",
"nerfacto-big",
"volinga",
"phototourism",
"depth-nerfacto",
"neus",