merging main

This commit is contained in:
terrance committed 2023-05-10 21:26:04 +00:00
commit 80cdb73406
221 files changed
+50645 -2513

No files matched your search

-30
View File
@@ -118,33 +118,3 @@ Often times, you just want to play with the parameters of an existing model with
# NOTE: the dataparser and associated configurations go at the end of the command
ns-train {METHOD_NAME} --vis viewer {DATA_PARSER} --scale-factor 0.5
```
### Extending Nerfstudio with custom methods
In order to extend the Nerfstudio and register your own methods, you can package your code as a python package
and register it with Nerfstudio as a `nerfstudio.method_configs` entrypoint in the `pyproject.toml` file.
The Nerfstudio will automatically look for all registered methods and will register them to be used
by method such as `ns-train`.
Here is an example:
```python
"""my_package/my_config.py"""
from nerfstudio.engine.trainer import TrainerConfig
from nerfstudio.plugins.types import MethodSpecification
MyMethod = MethodSpecification(
config=TrainerConfig(
method_name="my-method",
...
),
description="Custom description"
)
"""pyproject.toml"""
[project]
name = "my_package"
...
[project.entry-points.'nerfstudio.method_configs']
my-method = 'my_package.my_config:MyMethod'
```
@@ -9,7 +9,7 @@ To launch training jobs automatically on each of these items, you can call:
```bash
./scripts/benchmarking/launch_train_blender.sh -m {METHOD_NAME} [-s] [-v {VIS}] [{GPU_LIST}]
./nerfstudio/scripts/benchmarking/launch_train_blender.sh -m {METHOD_NAME} [-s] [-v {VIS}] [{GPU_LIST}]
```
Simply replace the arguments in brackets with the correct arguments.
@@ -18,31 +18,31 @@ Simply replace the arguments in brackets with the correct arguments.
- `-s`: Launch a single job per GPU.
- `-v {VIS}`: Use another visualization than wandb, which is the default. Only other option is tensorboard.
- `{GPU_LIST}`: (optional) Specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs.
:::{admonition} Tip
:class: info
To view all the arguments and annotations, you can run `./scripts/benchmarking/launch_train_blender.sh --help`
:::
To view all the arguments and annotations, you can run `./nerfstudio/scripts/benchmarking/launch_train_blender.sh --help`
:::
A full example would be:
- Specifying gpus
```bash
./scripts/benchmarking/launch_train_blender.sh -m nerfacto 0 1 2 3
```
```bash
./nerfstudio/scripts/benchmarking/launch_train_blender.sh -m nerfacto 0 1 2 3
```
- Automatically find available gpus
```bash
./scripts/benchmarking/launch_train_blender.sh -m nerfacto
```
```bash
./nerfstudio/scripts/benchmarking/launch_train_blender.sh -m nerfacto
```
The script will automatically launch training on all of the items and save the checkpoints in an output directory with the experiment name and current timestamp.
## Evaluating trained Blender models
Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/launch_eval_blender.sh`.
Once you have launched training, and training converges, you can test your method with `nerfstudio/scripts/benchmarking/launch_eval_blender.sh`.
Say we ran a benchmark on 08-10-2022 for `instant-ngp`. By default, the train script will save the benchmarks in the following format:
@@ -65,7 +65,7 @@ If we wanted to run the benchmark on all the blender data for the above example,
```bash
./scripts/benchmarking/launch_eval_blender.sh -m instant-ngp -o outputs/ -t 2022-08-10_172517 [{GPU_LIST}]
./nerfstudio/scripts/benchmarking/launch_eval_blender.sh -m instant-ngp -o outputs/ -t 2022-08-10_172517 [{GPU_LIST}]
```
The flags used in the benchmarking script are defined as follows:
@@ -90,4 +90,4 @@ outputs
:class: warning
Since we are running multiple backgrounded processes concurrently with this script, please note the terminal logs may be messy.
:::
:::
@@ -23,7 +23,24 @@ class RayGenerator(nn.Module):
...
```
At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag.
Alternatively, you can also time parts of the code:
```python
...
def forward(self, ray_indices: TensorType["num_rays", 3]) -> RayBundle:
# implementation here
with profiler.time_function("code1"):
# some code here
...
with profiler.time_function("code2"):
# some code here
...
...
```
At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions or code blocks that have the profiler tag.
:::{admonition} Tip
:class: info
@@ -32,6 +49,14 @@ Use this profiler if there are *specific/individual functions* you want to measu
:::
#### Profiling with PyTorch profiler
If you want to profile the training or evaluation code and track the memory and CUDA kernel launches, consider using [PyTorch profiler](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html).
It will run the profiler for some selected step numbers, once with `CUDA_LAUNCH_BLOCKING=1`, once with `CUDA_LAUNCH_BLOCKING=0`.
The PyTorch profiler can be enabled with `--logging.profiler=pytorch` flag.
The outputs of the profiler are trace files stored in `{PATH_TO_MODEL_OUTPUT}/profiler_traces`, and can be loaded in Google Chrome by typing `chrome://tracing`.
#### Profiling with PySpy
If you want to profile the entire codebase, consider using [PySpy](https://github.com/benfred/py-spy).
+89
View File
@@ -0,0 +1,89 @@
# Adding a New Method
Nerfstudio aims to offer researchers a codebase that they can utilize to extend and develop novel methods. Our vision is for users to establish a distinct repository that imports nerfstudio and overrides pipeline components to cater to specific functionality requirements of the new approach. If any of the new features require modifications to the core nerfstudio repository and can be generally useful, we encourage you to submit a PR to enable others to benefit from it.
Examples are often the best way to learn, take a look at the [LERF](https://github.com/kerrj/lerf) repository for good example of how to use nerfstudio in your projects.
## File Structure
We recommend the following file structure:
```
├── my_method
│ ├── __init__.py
│ ├── my_config.py
│ ├── custom_pipeline.py [optional]
│ ├── custom_model.py [optional]
│ ├── custom_field.py [optional]
│ ├── custom_datamanger.py [optional]
│ ├── ...
├── pyproject.toml
```
## Registering custom model with nerfstudio
In order to extend the Nerfstudio and register your own methods, you can package your code as a python package
and register it with Nerfstudio as a `nerfstudio.method_configs` entrypoint in the `pyproject.toml` file.
Nerfstudio will automatically look for all registered methods and will register them to be used
by methods such as `ns-train`.
First create a config file:
```python
"""my_method/my_config.py"""
from nerfstudio.engine.trainer import TrainerConfig
from nerfstudio.plugins.types import MethodSpecification
MyMethod = MethodSpecification(
config=TrainerConfig(
method_name="my-method",
pipeline=...
...
),
description="Custom description"
)
```
Then create a `pyproject.toml` file. This is where the entrypoint to your method is set and also where you can specify additional dependencies required by your codebase.
```python
"""pyproject.toml"""
[project]
name = "my_method"
dependencies = [
"nerfstudio" # you may want to consider pinning the version, ie "nerfstudio==0.1.19"
]
[tool.setuptools.packages.find]
include = ["my_method*"]
[project.entry-points.'nerfstudio.method_configs']
my-method = 'my_method.my_config:MyMethod'
```
finally run the following to register the method,
```
pip install -e .
```
When developing a new method you don't always want to install your code as a package.
Instead, you may use the `NERFSTUDIO_METHOD_CONFIGS` environment variable to temporarily register your custom method.
```
export NERFSTUDIO_METHOD_CONFIGS="my-method=my_method.my_config:MyMethod"
```
## Running custom method
After registering your method you should be able to run the method with,
```
ns-train my-method --data DATA_DIR
```
## Adding to the _nerf.studio_ documentation
We invite researchers to contribute their own methods to our online documentation. You can find more information on how to do this {ref}`here<own_method_docs>`.
+16 -44
View File
@@ -30,8 +30,16 @@ class DataparserOutputs:
"""Color of dataset background."""
scene_box: SceneBox = SceneBox()
"""Scene box of dataset. Used to bound the scene or provide the scene scale depending on model."""
semantics: Optional[Semantics] = None
"""Semantics information."""
mask_filenames: Optional[List[Path]] = None
"""Filenames for any masks that are required"""
metadata: Dict[str, Any] = to_immutable_dict({})
"""Dictionary of any metadata that be required for the given experiment.
Will be processed by the InputDataset to create any additional tensors that may be required.
"""
dataparser_transform: TensorType[3, 4] = torch.eye(4)[:3, :]
"""Transform applied by the dataparser."""
dataparser_scale: float = 1.0
"""Scale applied by the dataparser."""
@dataclass
class DataParser:
@@ -115,48 +123,12 @@ You can also pull out information from the DataParserOutputs for other DataManga
ray_generator = RayGenerator(dataparser_outputs.cameras)
```
## Our Implementations
## Included DataParsers
Below we enumerate the various dataparsers that we have implemented in our codebase. Feel free to use ours or add your own. Also any contributions are welcome and appreciated!
```{toctree}
---
maxdepth: 2
---
###### Nerfstudio
This is our custom dataparser. We have a script to convert images or videos with COLMAP to this format.
```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/nerfstudio_dataparser.py
:color: primary
:outline:
See the code!
```
###### Blender
We support the synthetic Blender dataset from the original NeRF paper.
```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/blender_dataparser.py
:color: primary
:outline:
See the code!
```
###### Instant NGP
This supports the Instant NGP dataset.
```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/instant_ngp_dataparser.py
:color: primary
:outline:
See the code!
```
###### Record3D
This dataparser can use recorded data from a >= iPhone 12 Pro using the [Record3D app](https://record3d.app/) . Record a video and export with the `EXR + JPG sequence` format. Unzip export and `rgb` folder before training.
For more information on capturing with Record3D, see the [Custom Dataset Docs](/quickstart/custom_dataset.md).
```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/record3d_dataparser.py
:color: primary
:outline:
See the code!
../../reference/api/data/dataparsers
```
+105
View File
@@ -0,0 +1,105 @@
# Custom GUI
We provide support for custom viewer GUI elements that can be defined in any `nn.Module`. Although we don't have any specific use cases in mind, here are some examples of what can be achieved with this feature:
- Using text input to modify the rendering
- Logging numerical values to the viewer
- Using checkboxes to turn off and on losses
- Using a dropdown to switch between appearances
## Adding an Element
To define a custom element, create an instance of one of the provided classes in `nerfstudio.viewer.server.viewer_elements`, and assign it as a class variable in your `nn.Module`.
```python
from nerfstudio.viewer.server.viewer_elements import ViewerNumber
class MyClass(nn.Module):#must inherit from nn.Module
def __init__(self):
# Must be a class variable
self.custom_value = ViewerNumber(name="My Value", default_value=1.0)
```
**Element Hierarchy**
The viewer recursively searches all `nn.Module` children of the base `Pipeline` object, and arranges parameters into folders based on their variable names.
For example, a `ViewerElement` defined in `pipeline.model.field` will be in the "Custom/model/field" folder in the GUI.
**Reading the value**
To read the value of a custom element, simply access its `value` attribute. In this case it will be `1.0` unless modified by the user in the viewer.
```python
current_value = self.custom_value.value
```
**Callbacks**
You can register a callback that will be called whenever a new value for your GUI element is available. For example, one can use a callback to update config parameters when elements are changed:
```python
def on_change_callback(handle: ViewerCheckbox) -> None:
self.config.example_parameter = handle.value
self.custom_checkbox = ViewerCheckbox(
name="Checkbox",
default_value=False,
cb_hook=on_change_callback,
)
```
**Thread safety**
Note that `ViewerElement` values can change asynchronously to model execution. So, it's best practice to store the value of a viewer element once at the beginning
of a forward pass and refer to the static variable afterwards.
```python
class MyModel(Model):
def __init__(self):
self.slider = ViewerSlider(name="Slider", default_value=0.5, min_value=0.0, max_value=1.0)
def get_outputs(self,ray)
slider_val = self.slider.value
#self.slider.value could change after this, unsafe to use
```
**Writing to the element**
You can write to a viewer element in Python, which provides a convenient way to track values in your code without the need for wandb/tensorboard or relying on `print` statements.
```python
self.custom_value.value = x
```
:::{admonition} Warning
:class: warning
Updating module state while training can have unexpected side effects. It is up to the user to ensure that GUI actions are safe. Conditioning on `self.training` can help determine whether effects are applied during forward passes for training or rendering.
:::
## Example Elements
```{image} imgs/custom_controls.png
:align: center
:width: 400
```
This was created with the following elements:
```python
from nerfstudio.viewer.server.viewer_elements import *
class MyModel(Model):
def __init__(self):
self.a = ViewerButton(name="My Button", cb_hook=self.handle_btn)
self.b = ViewerNumber(name="Number", default_value=1.0)
self.c = ViewerCheckbox(name="Checkbox", default_value=False)
self.d = ViewerDropdown(name="Dropdown", default_value="A", options=["A", "B"])
self.e = ViewerSlider(name="Slider", default_value=0.5, min_value=0.0, max_value=1.0)
self.f = ViewerText(name="Text", default_value="Hello World")
self.g = ViewerVec3(name="3D Vector", default_value=(0.1, 0.7, 0.1))
self.rgb_renderer = RGBRenderer()
...
class RGBRenderer(nn.Module):
def __init__(self):
#lives in "Custom/model/rgb_renderer" GUI folder
self.a = ViewerRGB(name="F", default_value=(0.1, 0.7, 0.1))
...
```
For more information on the available classes and their arguments, refer to the [API documentation](../../reference/api/viewer.rst)
Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 384 KiB

+22
View File
@@ -0,0 +1,22 @@
# Viewer
> We have a real-time web viewer that requires no installation. It's available at [https://viewer.nerf.studio/](https://viewer.nerf.studio/), where you can connect to your training job.
The viewer is built on [Viser](https://github.com/brentyi/viser/tree/main/viser) using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine.
```{toctree}
:titlesonly:
custom_gui
local_viewer
```
## Acknowledgements and references
We thank the authors and contributors to the following repos, which we've started, used, and modified for our use-cases.
- [Viser](https://github.com/brentyi/viser/tree/main/viser) - made by [Brent Yi](https://github.com/brentyi)
- [meshcat-python](https://github.com/rdeits/meshcat-python) - made by [Robin Deits](https://github.com/rdeits)
- [meshcat](https://github.com/rdeits/meshcat) - made by [Robin Deits](https://github.com/rdeits)
- [ThreeJS](https://threejs.org/)
- [ReactJS](https://reactjs.org/)
@@ -0,0 +1,62 @@
# Local Server
If you are unable to connect to `https://viewer.nerf.studio`, want to use Safari, or want develop the viewer codebase, you can launch your own local viewer.
## Installing Dependencies
```shell
cd nerfstudio/viewer/app
```
Install npm (to install yarn) and yarn
```shell
sudo apt-get install npm
npm install --global yarn
```
Install nvm and set the node version
Install nvm with [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
```shell
nvm install 17.8.0
```
Now running `node --version` in the shell should print "v17.8.0".
Install package.json dependencies and start the client viewer app:
```shell
yarn install
```
## Launch the web client
From the `nerfstudio/viewer/app` folder, run:
```shell
yarn start
```
The local webserver runs on port 4000 by default,
so when `ns-train` is running, you can connect to the viewer locally at
[http://localhost:4000/?websocket_url=ws://localhost:7007](http://localhost:4000/?websocket_url=ws://localhost:7007)
## FAQ
### Engine node incompatible
While running `yarn install`, you run into: `The engine "node" is incompatible with this module.`
**Solution**:
Install nvm with instructions at [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
```shell
nvm install 17.8.0
```
If you cannot install nvm, try ignoring the engines
```shell
yarn install --ignore-engines
```