This commit is contained in:
Justin committed 2023-04-16 00:53:23 +00:00
commit 4b040a175c
11 files changed
+156 -90

No files matched your search

+4
View File
@@ -73,3 +73,7 @@ After registering your method you should be able to run the method with,
```
ns-train my-method --data DATA_DIR
```
## Adding to the _nerf.studio_ documentation
We invite researchers to contribute their own methods to our online documentation. You can find more information on how to do this {ref}`here<own_method_docs>`.
+1 -1
View File
@@ -86,7 +86,7 @@ from nerfstudio.viewer.server.viewer_elements import *
class MyModel(Model):
def __init__(self):
self.a = ViewerButton(name="My Button", call_fn=self.handle_btn)
self.a = ViewerButton(name="My Button", cb_hook=self.handle_btn)
self.b = ViewerNumber(name="Number", default_value=1.0)
self.c = ViewerCheckbox(name="Checkbox", default_value=False)
self.d = ViewerDropdown(name="Dropdown", default_value="A", options=["A", "B"])
+1
View File
@@ -34,6 +34,7 @@ We intend to integrate many of the SDFStudio improvments back into the nerstudio
Supported methods:
- NeuS
- NeuS-facto
## Citation
+6 -6
View File
@@ -36,13 +36,13 @@ Details for running Instruct-NeRF2NeRF (built with Nerfstudio!) can be found [he
ns-train in2n --help
```
Three varients of Instruct-NeRF2NeRF are provided:
Three variants of Instruct-NeRF2NeRF are provided:
| Method | Description | Memory | Quality |
| ----------- | ---------------------------- | ------ | ------- |
| `in2n` | Full model, used in paper | ~15GB | Best |
| `in2n-small` | Half precision model | ~12GB | Good |
| `in2n-tiny` | Half prevision with no LPIPS | ~10GB | Ok |
| Method | Description | Memory | Quality |
| ------------ | ---------------------------- | ------ | ------- |
| `in2n` | Full model, used in paper | ~15GB | Best |
| `in2n-small` | Half precision model | ~12GB | Good |
| `in2n-tiny` | Half prevision with no LPIPS | ~10GB | Ok |
## Method
+22 -1
View File
@@ -20,7 +20,7 @@ To list the available methods run:
ns-train --help
```
## Methods
## Implemented Methods
The following methods are supported in nerfstudio:
@@ -33,3 +33,24 @@ The following methods are supported in nerfstudio:
Instruct-NeRF2NeRF<in2n.md>
Semantic NeRF-W<semantic_nerfw.md>
```
(own_method_docs)=
## Adding Your Own Method
If you're a researcher looking to develop new NeRF-related methods, we hope that you find nerfstudio to be a useful tool. We've provided documentation about integrating with the nerfstudio codebase, which you can find [here](../../developer_guides/new_methods.md).
We also welcome additions to the list of methods above. To do this, simply create a pull request that adds a markdown file describing the model to the docs/nerfology/methods folder, and update the list in this file. For reference on the layout, you can check out the [Instruct-NeRF2NeRF](in2n) page. Please try to include the following information:
- Installation instructions
- Instructions for running the method
- Requirements (dataset, GPU, ect)
- Method description (the more detailed the better, treat it like a blog post)
You are welcome to include assets (such as images or video) in the description, but please host them elsewhere.
:::{admonition} Note
:class: note
Please ensure that the documentation is clear and easy to understand for other users who may want to try out your method.
:::
+29 -21
View File
@@ -7,13 +7,21 @@
```bash
ns-train nerfacto --help
```
We provide a few additional variants:
| Method | Description | Memory | Speed |
| ---------------- | --------------------- | ------ | ------ |
| `nerfacto` | Default Model | ~6GB | Fast |
| `nerfacto-big` | Larger higher quality | ~12GB | Slower |
| `depth-nerfacto` | Supervise on depth | ~6GB | Fast |
## Method
### Overview
We created the nerfacto model to act as our default for real data captures of static scenes. The model is not existing published work, but rather a combination of many published methods that we have found work well for real data. This guide discusses the details of the model, understanding the [NeRF model](./nerf.md) is a prerequisite.
```{admonition} TLDR
We combine the following techniques in this model:
* Camera pose refinement
@@ -22,55 +30,55 @@ We combine the following techniques in this model:
* Scene contraction
* Hash encoding
```
```{warning}
🏗️ This guide is under construction 🏗️
```
## Pipeline
```{image} imgs/nerfacto/models_nerfacto_pipeline-light.png
:align: center
:class: only-light
```
```{image} imgs/nerfacto/models_nerfacto_pipeline-dark.png
:align: center
:class: only-dark
```
Here is an overview pipeline for nerfacto, we will walk through each component in this guide.
### Pose Refinement
It is not uncommon to have errors in the predicted camera poses. This is even more of a factor when using poses acquired from devices such as phones (ie. if you use the Record3D IOS app to capture data). Misaligned poses result in both cloudy artifacts in the scene and a reduction of sharpness and details. The NeRF framework allows us to backpropagate loss gradients to the input pose calculations. We can use this information to optimize and refine the poses.
### Piecewise Sampler
We use a Piecewise sampler to produce the initial set of samples of the scene. This sampler allocates half of the samples uniformly up to a distance of 1 from the camera. The remaining samples are distributed such that the step size increases with each sample. The step size is chosen such that the [frustums](../model_components/visualize_samples.ipynb) are scaled versions of themselves. By increasing the step sizes, we are able to sample distant objects will still having a dense set of samples for nearby objects.
### Proposal Sampler
The proposal sampler consolidates the sample locations to the regions of the scene that contribute most to the final render (typically the first surface intersection). This greatly improves reconstruction quality. The proposal network sampler requires a density function for the scene. The density function can be implemented in a variety of ways, we find that using a small fused-mlp with a hash encoding has sufficient accuracy and is fast. The proposal network sampler can be chained together with multiple density functions to further consolidate the sampling. We have found that two density functions are better than one. Larger than 2 leads to diminishing returns.
#### Density Field
The density field only needs to represent a coarse density representation of the scene to guide sampling. Combining a hash encoding with a small fused MLP (from [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn)) provides a fast way to query the scene. We can make it more efficient by decreasing the encoding dictionary size and number of feature levels. These simplifications have little impact on the reconstruction quality because the density function does not need to learn high frequency details during the initial passes.
### Nerfacto Field
```{image} imgs/nerfacto/models_nerfacto_field-light.png
:align: center
:class: only-light
:width: 600
```
```{image} imgs/nerfacto/models_nerfacto_field-dark.png
:align: center
:class: only-dark
:width: 600
```
```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/models/nerfacto.py
:color: primary
:outline:
+18 -6
View File
@@ -62,11 +62,7 @@ CAMERA_MODEL_TO_TYPE = {
class Cameras(TensorDataclass):
"""Dataparser outputs for the image dataset and the ray generator.
Note: currently only supports cameras with the same principal points and types. The reason we type
the focal lengths, principal points, and image sizes as tensors is to allow for batched cameras
down the line in cases where your batches of camera data don't come from the same cameras.
If a single value is provided, it is broadcasted to all cameras.
If a single value is provided, it is broadcasted to all cameras.
Args:
camera_to_worlds: Camera to world matrices. Tensor of per-image c2w matrices, in [R | t] format
@@ -79,6 +75,8 @@ class Cameras(TensorDataclass):
distortion_params: OpenCV 6 radial distortion coefficients
camera_type: Type of camera model. This will be an int corresponding to the CameraType enum.
times: Timestamps for each camera
metadata: Additional metadata or data needed for interpolation, will mimic shape of the cameras
and will be broadcasted to the rays generated from any derivative RaySamples we create with this
"""
camera_to_worlds: TensorType["num_cameras":..., 3, 4]
@@ -91,6 +89,7 @@ class Cameras(TensorDataclass):
distortion_params: Optional[TensorType["num_cameras":..., 6]]
camera_type: TensorType["num_cameras":..., 1]
times: Optional[TensorType["num_cameras", 1]]
metadata: Optional[Dict]
def __init__(
self,
@@ -111,6 +110,7 @@ class Cameras(TensorDataclass):
]
] = CameraType.PERSPECTIVE,
times: Optional[TensorType["num_cameras"]] = None,
metadata: Optional[Dict] = None,
) -> None:
"""Initializes the Cameras object.
@@ -146,6 +146,8 @@ class Cameras(TensorDataclass):
self.camera_type = self._init_get_camera_type(camera_type)
self.times = self._init_get_times(times)
self.metadata = metadata
self.__post_init__() # This will do the dataclass post_init and broadcast all the tensors
self._use_nerfacc = strtobool(os.environ.get("INTERSECT_WITH_NERFACC", "TRUE"))
@@ -716,13 +718,23 @@ class Cameras(TensorDataclass):
times = self.times[camera_indices, 0] if self.times is not None else None
metadata = (
self._apply_fn_to_dict(self.metadata, lambda x: x[true_indices]) if self.metadata is not None else None
)
if metadata is not None:
metadata["directions_norm"] = directions_norm[0].detach()
else:
metadata = {"directions_norm": directions_norm[0].detach()}
times = self.times[camera_indices, 0] if self.times is not None else None
return RayBundle(
origins=origins,
directions=directions,
pixel_area=pixel_area,
camera_indices=camera_indices,
times=times,
metadata={"directions_norm": directions_norm[0].detach()},
metadata=metadata,
)
def to_json(
+68 -51
View File
@@ -17,14 +17,14 @@
from __future__ import annotations
from abc import abstractmethod
from typing import Any, Callable, List, Optional, Tuple, TypeVar
from typing import Any, Callable, Generic, List, Optional, Tuple, TypeVar
from nerfstudio.viewer.viser import GuiHandle, ViserServer
from nerfstudio.viewer.viser import GuiHandle, GuiSelectHandle, ViserServer
IntOrFloat = TypeVar("IntOrFloat", int, float)
TValue = TypeVar("TValue")
class ViewerElement:
class ViewerElement(Generic[TValue]):
"""Base class for all viewer elements
Args:
@@ -32,9 +32,14 @@ class ViewerElement:
disabled: If the element is disabled
"""
def __init__(self, name: str, disabled: bool = False, cb_hook: Callable = lambda element: None):
def __init__(
self,
name: str,
disabled: bool = False,
cb_hook: Callable = lambda element: None,
) -> None:
self.name = name
self.gui_handle: Optional[GuiHandle] = None
self.gui_handle: Optional[GuiHandle[TValue]] = None
self.disabled = disabled
self.cb_hook = cb_hook
@@ -48,18 +53,20 @@ class ViewerElement:
"""
...
def remove(self):
def remove(self) -> None:
"""Removes the gui element from the viewer"""
if self.gui_handle is not None:
self.gui_handle.remove()
self.gui_handle = None
def set_hidden(self, hidden: bool):
def set_hidden(self, hidden: bool) -> None:
"""Sets the hidden state of the gui element"""
assert self.gui_handle is not None
self.gui_handle.set_hidden(hidden)
def set_disabled(self, disabled: bool):
def set_disabled(self, disabled: bool) -> None:
"""Sets the disabled state of the gui element"""
assert self.gui_handle is not None
self.gui_handle.set_disabled(disabled)
@abstractmethod
@@ -68,17 +75,17 @@ class ViewerElement:
...
class ViewerButton(ViewerElement):
class ViewerButton(ViewerElement[bool]):
"""A button in the viewer
Args:
name: The name of the button
call_fn: The function to call when the button is pressed
cb_hook: The function to call when the button is pressed
disabled: If the button is disabled
"""
def __init__(self, name: str, call_fn: Callable, disabled: bool = False):
super().__init__(name, disabled=disabled, cb_hook=call_fn)
def __init__(self, name: str, cb_hook: Callable[[ViewerButton], Any], disabled: bool = False):
super().__init__(name, disabled=disabled, cb_hook=cb_hook)
def _create_gui_handle(self, viser_server: ViserServer) -> None:
self.gui_handle = viser_server.add_gui_button(self.name)
@@ -87,14 +94,11 @@ class ViewerButton(ViewerElement):
def install(self, viser_server: ViserServer) -> None:
self._create_gui_handle(viser_server)
def call_fn(_: GuiHandle): # pylint: disable=unused-argument
self.cb_hook(self)
assert self.gui_handle is not None
self.gui_handle.on_update(call_fn)
self.gui_handle.on_update(lambda _: self.cb_hook(self))
class ViewerParameter(ViewerElement):
class ViewerParameter(ViewerElement[TValue], Generic[TValue]):
"""A viewer element with state
Args:
@@ -104,9 +108,15 @@ class ViewerParameter(ViewerElement):
cb_hook: Callback to call on update
"""
def __init__(self, name: str, default_value: Any, disabled: bool = False, cb_hook: Callable = lambda element: None):
def __init__(
self,
name: str,
default_value: TValue,
disabled: bool = False,
cb_hook: Callable = lambda element: None,
) -> None:
super().__init__(name, disabled=disabled, cb_hook=cb_hook)
self.def_value = default_value
self.default_value = default_value
def install(self, viser_server: ViserServer) -> None:
"""
@@ -117,32 +127,32 @@ class ViewerParameter(ViewerElement):
"""
self._create_gui_handle(viser_server)
def update_fn(_: GuiHandle):
self.cb_hook(self)
assert self.gui_handle is not None
self.gui_handle.on_update(update_fn)
self.gui_handle.on_update(lambda _: self.cb_hook(self))
@abstractmethod
def _create_gui_handle(self, viser_server: ViserServer) -> None:
...
@property
def value(self) -> Any:
def value(self) -> TValue:
"""Returns the current value of the viewer element"""
if self.gui_handle is None:
return self.def_value
return self.default_value
return self.gui_handle.get_value()
@value.setter
def value(self, value: Any):
def value(self, value: TValue) -> None:
if self.gui_handle is not None:
self.gui_handle.set_value(value)
else:
self.def_value = value
self.default_value = value
class ViewerSlider(ViewerParameter):
IntOrFloat = TypeVar("IntOrFloat", int, float)
class ViewerSlider(ViewerParameter[IntOrFloat], Generic[IntOrFloat]):
"""A slider in the viewer
Args:
@@ -163,7 +173,7 @@ class ViewerSlider(ViewerParameter):
max_value: IntOrFloat,
step: IntOrFloat = 0.1,
disabled: bool = False,
cb_hook: Callable = lambda element: None,
cb_hook: Callable[[ViewerSlider], Any] = lambda element: None,
):
assert isinstance(default_value, (float, int))
super().__init__(name, default_value, disabled=disabled, cb_hook=cb_hook)
@@ -173,11 +183,11 @@ class ViewerSlider(ViewerParameter):
def _create_gui_handle(self, viser_server: ViserServer) -> None:
assert self.gui_handle is None, "gui_handle should be initialized once"
self.gui_handle = viser_server.add_gui_slider(self.name, self.min, self.max, self.step, self.def_value)
self.gui_handle = viser_server.add_gui_slider(self.name, self.min, self.max, self.step, self.default_value)
self.gui_handle.set_disabled(self.disabled)
class ViewerText(ViewerParameter):
class ViewerText(ViewerParameter[str]):
"""A text field in the viewer
Args:
@@ -192,18 +202,18 @@ class ViewerText(ViewerParameter):
name: str,
default_value: str,
disabled: bool = False,
cb_hook: Callable = lambda element: None,
cb_hook: Callable[[ViewerText], Any] = lambda element: None,
):
assert isinstance(default_value, str)
super().__init__(name, default_value, disabled=disabled, cb_hook=cb_hook)
def _create_gui_handle(self, viser_server: ViserServer) -> None:
assert self.gui_handle is None, "gui_handle should be initialized once"
self.gui_handle = viser_server.add_gui_text(self.name, self.def_value)
self.gui_handle = viser_server.add_gui_text(self.name, self.default_value)
self.gui_handle.set_disabled(self.disabled)
class ViewerNumber(ViewerParameter):
class ViewerNumber(ViewerParameter[IntOrFloat], Generic[IntOrFloat]):
"""A number field in the viewer
Args:
@@ -213,22 +223,24 @@ class ViewerNumber(ViewerParameter):
cb_hook: Callback to call on update
"""
default_value: IntOrFloat
def __init__(
self,
name: str,
default_value: IntOrFloat,
disabled: bool = False,
cb_hook: Callable = lambda element: None,
cb_hook: Callable[[ViewerNumber], Any] = lambda element: None,
):
assert isinstance(default_value, (float, int))
super().__init__(name, default_value, disabled=disabled, cb_hook=cb_hook)
def _create_gui_handle(self, viser_server: ViserServer) -> None:
assert self.gui_handle is None, "gui_handle should be initialized once"
self.gui_handle = viser_server.add_gui_number(self.name, self.def_value)
self.gui_handle = viser_server.add_gui_number(self.name, self.default_value)
class ViewerCheckbox(ViewerParameter):
class ViewerCheckbox(ViewerParameter[bool]):
"""A checkbox in the viewer
Args:
@@ -243,18 +255,18 @@ class ViewerCheckbox(ViewerParameter):
name: str,
default_value: bool,
disabled: bool = False,
cb_hook: Callable = lambda element: None,
cb_hook: Callable[[ViewerCheckbox], Any] = lambda element: None,
):
assert isinstance(default_value, bool)
super().__init__(name, default_value, disabled=disabled, cb_hook=cb_hook)
def _create_gui_handle(self, viser_server: ViserServer) -> None:
assert self.gui_handle is None, "gui_handle should be initialized once"
self.gui_handle = viser_server.add_gui_checkbox(self.name, self.def_value)
self.gui_handle = viser_server.add_gui_checkbox(self.name, self.default_value)
self.gui_handle.set_disabled(self.disabled)
class ViewerDropdown(ViewerParameter):
class ViewerDropdown(ViewerParameter[str]):
"""A dropdown in the viewer
Args:
@@ -265,13 +277,15 @@ class ViewerDropdown(ViewerParameter):
cb_hook: Callback to call on update
"""
gui_handle: Optional[GuiSelectHandle[str]]
def __init__(
self,
name: str,
default_value: str,
options: List[str],
disabled: bool = False,
cb_hook: Callable = lambda element: None,
cb_hook: Callable[[ViewerDropdown], Any] = lambda element: None,
):
assert default_value in options
super().__init__(name, default_value, disabled=disabled, cb_hook=cb_hook)
@@ -279,19 +293,22 @@ class ViewerDropdown(ViewerParameter):
def _create_gui_handle(self, viser_server: ViserServer) -> None:
assert self.gui_handle is None, "gui_handle should be initialized once"
self.gui_handle = viser_server.add_gui_select(self.name, self.options, self.def_value)
self.gui_handle = viser_server.add_gui_select(self.name, self.options, self.default_value)
self.gui_handle.set_disabled(self.disabled)
def set_options(self, new_options: List[str]) -> None:
"""
Sets the options of the dropdown,
new_options: The new options. if the current option isn't in the new options, the first option is selected
Args:
new_options: The new options. If the current option isn't in the new options, the first option is selected.
"""
self.options = new_options
self.gui_handle.set_options(new_options)
if self.gui_handle is not None:
self.gui_handle.set_options(new_options)
class ViewerRGB(ViewerParameter):
class ViewerRGB(ViewerParameter[Tuple[int, int, int]]):
"""
An RGB color picker for the viewer
"""
@@ -301,17 +318,17 @@ class ViewerRGB(ViewerParameter):
name,
default_value: Tuple[int, int, int],
disabled=False,
cb_hook: Callable = lambda element: None,
cb_hook: Callable[[ViewerRGB], Any] = lambda element: None,
):
assert len(default_value) == 3
super().__init__(name, default_value, disabled=disabled, cb_hook=cb_hook)
def _create_gui_handle(self, viser_server: ViserServer) -> None:
self.gui_handle = viser_server.add_gui_rgb(self.name, self.def_value)
self.gui_handle = viser_server.add_gui_rgb(self.name, self.default_value)
self.gui_handle.set_disabled(self.disabled)
class ViewerVec3(ViewerParameter):
class ViewerVec3(ViewerParameter[Tuple[float, float, float]]):
"""
3 number boxes in a row to input a vector
"""
@@ -322,12 +339,12 @@ class ViewerVec3(ViewerParameter):
default_value: Tuple[float, float, float],
step=0.1,
disabled=False,
cb_hook: Callable = lambda element: None,
cb_hook: Callable[[ViewerVec3], Any] = lambda element: None,
):
assert len(default_value) == 3
super().__init__(name, default_value, disabled=disabled, cb_hook=cb_hook)
self.step = step
def _create_gui_handle(self, viser_server: ViserServer) -> None:
self.gui_handle = viser_server.add_gui_vector3(self.name, self.def_value, self.step)
self.gui_handle = viser_server.add_gui_vector3(self.name, self.default_value, self.step)
self.gui_handle.set_disabled(self.disabled)
+1
View File
@@ -16,5 +16,6 @@
# pylint: disable=useless-import-alias
from .message_api import GuiHandle as GuiHandle
from .message_api import GuiSelectHandle as GuiSelectHandle
from .messages import NerfstudioMessage as NerfstudioMessage
from .server import ViserServer as ViserServer
+5 -3
View File
@@ -180,15 +180,17 @@ class GuiSelectHandle(GuiHandle[TString], Generic[TString]):
inferred where possible when handles are instantiated; for the most flexibility,
we can declare handles as `GuiHandle[str]`.
"""
overwrite_value = False
# Make sure initial value is in options.
self._impl.leva_conf["options"] = options
if self._impl.leva_conf["value"] not in options:
self._impl.leva_conf["value"] = options[0]
overwrite_value = True
# Update options.
self._impl.api._queue(
GuiSetLevaConfMessage(self._impl.name, self._impl.leva_conf),
)
if overwrite_value:
# Make sure current value is in options.
if self.get_value() not in options:
self.set_value(options[0])
+1 -1
View File
@@ -51,7 +51,7 @@ dependencies = [
"torchmetrics[image]>=0.9.3",
"torchtyping>=0.1.4",
"torchvision>=0.13.0",
"viser==0.0.2",
"viser>=0.0.5",
"nuscenes-devkit>=1.1.1",
"wandb>=0.13.3",
"xatlas",