mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-11 10:34:29 +08:00
parent
6a7c8820b8
commit
a9eea19880
@ -73,3 +73,7 @@ After registering your method you should be able to run the method with,
|
||||
```
|
||||
ns-train my-method --data DATA_DIR
|
||||
```
|
||||
|
||||
## Adding to the _nerf.studio_ documentation
|
||||
|
||||
We invite researchers to contribute their own methods to our online documentation. You can find more information on how to do this {ref}`here<own_method_docs>`.
|
||||
|
||||
@ -36,13 +36,13 @@ Details for running Instruct-NeRF2NeRF (built with Nerfstudio!) can be found [he
|
||||
ns-train in2n --help
|
||||
```
|
||||
|
||||
Three varients of Instruct-NeRF2NeRF are provided:
|
||||
Three variants of Instruct-NeRF2NeRF are provided:
|
||||
|
||||
| Method | Description | Memory | Quality |
|
||||
| ----------- | ---------------------------- | ------ | ------- |
|
||||
| `in2n` | Full model, used in paper | ~15GB | Best |
|
||||
| `in2n-small` | Half precision model | ~12GB | Good |
|
||||
| `in2n-tiny` | Half prevision with no LPIPS | ~10GB | Ok |
|
||||
| Method | Description | Memory | Quality |
|
||||
| ------------ | ---------------------------- | ------ | ------- |
|
||||
| `in2n` | Full model, used in paper | ~15GB | Best |
|
||||
| `in2n-small` | Half precision model | ~12GB | Good |
|
||||
| `in2n-tiny` | Half prevision with no LPIPS | ~10GB | Ok |
|
||||
|
||||
## Method
|
||||
|
||||
|
||||
@ -20,7 +20,7 @@ To list the available methods run:
|
||||
ns-train --help
|
||||
```
|
||||
|
||||
## Methods
|
||||
## Implemented Methods
|
||||
|
||||
The following methods are supported in nerfstudio:
|
||||
|
||||
@ -33,3 +33,24 @@ The following methods are supported in nerfstudio:
|
||||
Instruct-NeRF2NeRF<in2n.md>
|
||||
Semantic NeRF-W<semantic_nerfw.md>
|
||||
```
|
||||
|
||||
(own_method_docs)=
|
||||
|
||||
## Adding Your Own Method
|
||||
|
||||
If you're a researcher looking to develop new NeRF-related methods, we hope that you find nerfstudio to be a useful tool. We've provided documentation about integrating with the nerfstudio codebase, which you can find [here](../../developer_guides/new_methods.md).
|
||||
|
||||
We also welcome additions to the list of methods above. To do this, simply create a pull request that adds a markdown file describing the model to the docs/nerfology/methods folder, and update the list in this file. For reference on the layout, you can check out the [Instruct-NeRF2NeRF](in2n) page. Please try to include the following information:
|
||||
|
||||
- Installation instructions
|
||||
- Instructions for running the method
|
||||
- Requirements (dataset, GPU, ect)
|
||||
- Method description (the more detailed the better, treat it like a blog post)
|
||||
|
||||
You are welcome to include assets (such as images or video) in the description, but please host them elsewhere.
|
||||
|
||||
:::{admonition} Note
|
||||
:class: note
|
||||
|
||||
Please ensure that the documentation is clear and easy to understand for other users who may want to try out your method.
|
||||
:::
|
||||
|
||||
@ -7,13 +7,21 @@
|
||||
```bash
|
||||
ns-train nerfacto --help
|
||||
```
|
||||
|
||||
|
||||
We provide a few additional variants:
|
||||
|
||||
| Method | Description | Memory | Speed |
|
||||
| ---------------- | --------------------- | ------ | ------ |
|
||||
| `nerfacto` | Default Model | ~6GB | Fast |
|
||||
| `nerfacto-big` | Larger higher quality | ~12GB | Slower |
|
||||
| `depth-nerfacto` | Supervise on depth | ~6GB | Fast |
|
||||
|
||||
## Method
|
||||
|
||||
|
||||
### Overview
|
||||
|
||||
|
||||
We created the nerfacto model to act as our default for real data captures of static scenes. The model is not existing published work, but rather a combination of many published methods that we have found work well for real data. This guide discusses the details of the model, understanding the [NeRF model](./nerf.md) is a prerequisite.
|
||||
|
||||
|
||||
```{admonition} TLDR
|
||||
We combine the following techniques in this model:
|
||||
* Camera pose refinement
|
||||
@ -22,55 +30,55 @@ We combine the following techniques in this model:
|
||||
* Scene contraction
|
||||
* Hash encoding
|
||||
```
|
||||
|
||||
|
||||
```{warning}
|
||||
🏗️ This guide is under construction 🏗️
|
||||
```
|
||||
|
||||
|
||||
## Pipeline
|
||||
|
||||
|
||||
```{image} imgs/nerfacto/models_nerfacto_pipeline-light.png
|
||||
:align: center
|
||||
:class: only-light
|
||||
```
|
||||
|
||||
|
||||
```{image} imgs/nerfacto/models_nerfacto_pipeline-dark.png
|
||||
:align: center
|
||||
:class: only-dark
|
||||
```
|
||||
|
||||
|
||||
Here is an overview pipeline for nerfacto, we will walk through each component in this guide.
|
||||
|
||||
|
||||
### Pose Refinement
|
||||
|
||||
|
||||
It is not uncommon to have errors in the predicted camera poses. This is even more of a factor when using poses acquired from devices such as phones (ie. if you use the Record3D IOS app to capture data). Misaligned poses result in both cloudy artifacts in the scene and a reduction of sharpness and details. The NeRF framework allows us to backpropagate loss gradients to the input pose calculations. We can use this information to optimize and refine the poses.
|
||||
|
||||
|
||||
### Piecewise Sampler
|
||||
|
||||
|
||||
We use a Piecewise sampler to produce the initial set of samples of the scene. This sampler allocates half of the samples uniformly up to a distance of 1 from the camera. The remaining samples are distributed such that the step size increases with each sample. The step size is chosen such that the [frustums](../model_components/visualize_samples.ipynb) are scaled versions of themselves. By increasing the step sizes, we are able to sample distant objects will still having a dense set of samples for nearby objects.
|
||||
|
||||
|
||||
### Proposal Sampler
|
||||
|
||||
|
||||
The proposal sampler consolidates the sample locations to the regions of the scene that contribute most to the final render (typically the first surface intersection). This greatly improves reconstruction quality. The proposal network sampler requires a density function for the scene. The density function can be implemented in a variety of ways, we find that using a small fused-mlp with a hash encoding has sufficient accuracy and is fast. The proposal network sampler can be chained together with multiple density functions to further consolidate the sampling. We have found that two density functions are better than one. Larger than 2 leads to diminishing returns.
|
||||
|
||||
|
||||
#### Density Field
|
||||
|
||||
|
||||
The density field only needs to represent a coarse density representation of the scene to guide sampling. Combining a hash encoding with a small fused MLP (from [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn)) provides a fast way to query the scene. We can make it more efficient by decreasing the encoding dictionary size and number of feature levels. These simplifications have little impact on the reconstruction quality because the density function does not need to learn high frequency details during the initial passes.
|
||||
|
||||
|
||||
### Nerfacto Field
|
||||
|
||||
|
||||
```{image} imgs/nerfacto/models_nerfacto_field-light.png
|
||||
:align: center
|
||||
:class: only-light
|
||||
:width: 600
|
||||
```
|
||||
|
||||
|
||||
```{image} imgs/nerfacto/models_nerfacto_field-dark.png
|
||||
:align: center
|
||||
:class: only-dark
|
||||
:width: 600
|
||||
```
|
||||
|
||||
|
||||
```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/models/nerfacto.py
|
||||
:color: primary
|
||||
:outline:
|
||||
|
||||
Loading…
Reference in New Issue
Block a user