mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-16 21:03:46 +08:00
merging main
This commit is contained in:
commit
80cdb73406
221 files changed
+50645
-2513
No files matched your search
@@ -118,33 +118,3 @@ Often times, you just want to play with the parameters of an existing model with
|
||||
# NOTE: the dataparser and associated configurations go at the end of the command
|
||||
ns-train {METHOD_NAME} --vis viewer {DATA_PARSER} --scale-factor 0.5
|
||||
```
|
||||
|
||||
### Extending Nerfstudio with custom methods
|
||||
In order to extend the Nerfstudio and register your own methods, you can package your code as a python package
|
||||
and register it with Nerfstudio as a `nerfstudio.method_configs` entrypoint in the `pyproject.toml` file.
|
||||
The Nerfstudio will automatically look for all registered methods and will register them to be used
|
||||
by method such as `ns-train`.
|
||||
|
||||
Here is an example:
|
||||
```python
|
||||
"""my_package/my_config.py"""
|
||||
|
||||
from nerfstudio.engine.trainer import TrainerConfig
|
||||
from nerfstudio.plugins.types import MethodSpecification
|
||||
|
||||
MyMethod = MethodSpecification(
|
||||
config=TrainerConfig(
|
||||
method_name="my-method",
|
||||
...
|
||||
),
|
||||
description="Custom description"
|
||||
)
|
||||
|
||||
"""pyproject.toml"""
|
||||
[project]
|
||||
name = "my_package"
|
||||
...
|
||||
|
||||
[project.entry-points.'nerfstudio.method_configs']
|
||||
my-method = 'my_package.my_config:MyMethod'
|
||||
```
|
||||
@@ -9,7 +9,7 @@ To launch training jobs automatically on each of these items, you can call:
|
||||
|
||||
```bash
|
||||
|
||||
./scripts/benchmarking/launch_train_blender.sh -m {METHOD_NAME} [-s] [-v {VIS}] [{GPU_LIST}]
|
||||
./nerfstudio/scripts/benchmarking/launch_train_blender.sh -m {METHOD_NAME} [-s] [-v {VIS}] [{GPU_LIST}]
|
||||
```
|
||||
|
||||
Simply replace the arguments in brackets with the correct arguments.
|
||||
@@ -18,31 +18,31 @@ Simply replace the arguments in brackets with the correct arguments.
|
||||
- `-s`: Launch a single job per GPU.
|
||||
- `-v {VIS}`: Use another visualization than wandb, which is the default. Only other option is tensorboard.
|
||||
- `{GPU_LIST}`: (optional) Specify the list of gpus you want to use on your machine space separated. for instance, if you want to use GPU's 0-3, you will need to pass in `0 1 2 3`. If left empty, the script will automatically find available GPU's and distribute training jobs on the available GPUs.
|
||||
|
||||
|
||||
:::{admonition} Tip
|
||||
:class: info
|
||||
|
||||
To view all the arguments and annotations, you can run `./scripts/benchmarking/launch_train_blender.sh --help`
|
||||
:::
|
||||
|
||||
To view all the arguments and annotations, you can run `./nerfstudio/scripts/benchmarking/launch_train_blender.sh --help`
|
||||
:::
|
||||
|
||||
A full example would be:
|
||||
|
||||
- Specifying gpus
|
||||
```bash
|
||||
./scripts/benchmarking/launch_train_blender.sh -m nerfacto 0 1 2 3
|
||||
```
|
||||
|
||||
```bash
|
||||
./nerfstudio/scripts/benchmarking/launch_train_blender.sh -m nerfacto 0 1 2 3
|
||||
```
|
||||
|
||||
- Automatically find available gpus
|
||||
```bash
|
||||
./scripts/benchmarking/launch_train_blender.sh -m nerfacto
|
||||
```
|
||||
```bash
|
||||
./nerfstudio/scripts/benchmarking/launch_train_blender.sh -m nerfacto
|
||||
```
|
||||
|
||||
The script will automatically launch training on all of the items and save the checkpoints in an output directory with the experiment name and current timestamp.
|
||||
|
||||
## Evaluating trained Blender models
|
||||
|
||||
Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/launch_eval_blender.sh`.
|
||||
Once you have launched training, and training converges, you can test your method with `nerfstudio/scripts/benchmarking/launch_eval_blender.sh`.
|
||||
|
||||
Say we ran a benchmark on 08-10-2022 for `instant-ngp`. By default, the train script will save the benchmarks in the following format:
|
||||
|
||||
@@ -65,7 +65,7 @@ If we wanted to run the benchmark on all the blender data for the above example,
|
||||
|
||||
```bash
|
||||
|
||||
./scripts/benchmarking/launch_eval_blender.sh -m instant-ngp -o outputs/ -t 2022-08-10_172517 [{GPU_LIST}]
|
||||
./nerfstudio/scripts/benchmarking/launch_eval_blender.sh -m instant-ngp -o outputs/ -t 2022-08-10_172517 [{GPU_LIST}]
|
||||
```
|
||||
|
||||
The flags used in the benchmarking script are defined as follows:
|
||||
@@ -90,4 +90,4 @@ outputs
|
||||
:class: warning
|
||||
|
||||
Since we are running multiple backgrounded processes concurrently with this script, please note the terminal logs may be messy.
|
||||
:::
|
||||
:::
|
||||
@@ -23,7 +23,24 @@ class RayGenerator(nn.Module):
|
||||
...
|
||||
```
|
||||
|
||||
At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag.
|
||||
Alternatively, you can also time parts of the code:
|
||||
```python
|
||||
...
|
||||
|
||||
def forward(self, ray_indices: TensorType["num_rays", 3]) -> RayBundle:
|
||||
# implementation here
|
||||
with profiler.time_function("code1"):
|
||||
# some code here
|
||||
...
|
||||
|
||||
with profiler.time_function("code2"):
|
||||
# some code here
|
||||
...
|
||||
...
|
||||
```
|
||||
|
||||
|
||||
At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions or code blocks that have the profiler tag.
|
||||
|
||||
:::{admonition} Tip
|
||||
:class: info
|
||||
@@ -32,6 +49,14 @@ Use this profiler if there are *specific/individual functions* you want to measu
|
||||
:::
|
||||
|
||||
|
||||
#### Profiling with PyTorch profiler
|
||||
|
||||
If you want to profile the training or evaluation code and track the memory and CUDA kernel launches, consider using [PyTorch profiler](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html).
|
||||
It will run the profiler for some selected step numbers, once with `CUDA_LAUNCH_BLOCKING=1`, once with `CUDA_LAUNCH_BLOCKING=0`.
|
||||
The PyTorch profiler can be enabled with `--logging.profiler=pytorch` flag.
|
||||
The outputs of the profiler are trace files stored in `{PATH_TO_MODEL_OUTPUT}/profiler_traces`, and can be loaded in Google Chrome by typing `chrome://tracing`.
|
||||
|
||||
|
||||
#### Profiling with PySpy
|
||||
|
||||
If you want to profile the entire codebase, consider using [PySpy](https://github.com/benfred/py-spy).
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# Adding a New Method
|
||||
|
||||
Nerfstudio aims to offer researchers a codebase that they can utilize to extend and develop novel methods. Our vision is for users to establish a distinct repository that imports nerfstudio and overrides pipeline components to cater to specific functionality requirements of the new approach. If any of the new features require modifications to the core nerfstudio repository and can be generally useful, we encourage you to submit a PR to enable others to benefit from it.
|
||||
|
||||
Examples are often the best way to learn, take a look at the [LERF](https://github.com/kerrj/lerf) repository for good example of how to use nerfstudio in your projects.
|
||||
|
||||
## File Structure
|
||||
|
||||
We recommend the following file structure:
|
||||
|
||||
```
|
||||
├── my_method
|
||||
│ ├── __init__.py
|
||||
│ ├── my_config.py
|
||||
│ ├── custom_pipeline.py [optional]
|
||||
│ ├── custom_model.py [optional]
|
||||
│ ├── custom_field.py [optional]
|
||||
│ ├── custom_datamanger.py [optional]
|
||||
│ ├── ...
|
||||
├── pyproject.toml
|
||||
```
|
||||
|
||||
## Registering custom model with nerfstudio
|
||||
|
||||
In order to extend the Nerfstudio and register your own methods, you can package your code as a python package
|
||||
and register it with Nerfstudio as a `nerfstudio.method_configs` entrypoint in the `pyproject.toml` file.
|
||||
Nerfstudio will automatically look for all registered methods and will register them to be used
|
||||
by methods such as `ns-train`.
|
||||
|
||||
First create a config file:
|
||||
|
||||
```python
|
||||
"""my_method/my_config.py"""
|
||||
|
||||
from nerfstudio.engine.trainer import TrainerConfig
|
||||
from nerfstudio.plugins.types import MethodSpecification
|
||||
|
||||
MyMethod = MethodSpecification(
|
||||
config=TrainerConfig(
|
||||
method_name="my-method",
|
||||
pipeline=...
|
||||
...
|
||||
),
|
||||
description="Custom description"
|
||||
)
|
||||
```
|
||||
|
||||
Then create a `pyproject.toml` file. This is where the entrypoint to your method is set and also where you can specify additional dependencies required by your codebase.
|
||||
|
||||
```python
|
||||
"""pyproject.toml"""
|
||||
|
||||
[project]
|
||||
name = "my_method"
|
||||
|
||||
dependencies = [
|
||||
"nerfstudio" # you may want to consider pinning the version, ie "nerfstudio==0.1.19"
|
||||
]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
include = ["my_method*"]
|
||||
|
||||
[project.entry-points.'nerfstudio.method_configs']
|
||||
my-method = 'my_method.my_config:MyMethod'
|
||||
```
|
||||
|
||||
finally run the following to register the method,
|
||||
|
||||
```
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
When developing a new method you don't always want to install your code as a package.
|
||||
Instead, you may use the `NERFSTUDIO_METHOD_CONFIGS` environment variable to temporarily register your custom method.
|
||||
```
|
||||
export NERFSTUDIO_METHOD_CONFIGS="my-method=my_method.my_config:MyMethod"
|
||||
```
|
||||
|
||||
## Running custom method
|
||||
|
||||
After registering your method you should be able to run the method with,
|
||||
|
||||
```
|
||||
ns-train my-method --data DATA_DIR
|
||||
```
|
||||
|
||||
## Adding to the _nerf.studio_ documentation
|
||||
|
||||
We invite researchers to contribute their own methods to our online documentation. You can find more information on how to do this {ref}`here<own_method_docs>`.
|
||||
@@ -30,8 +30,16 @@ class DataparserOutputs:
|
||||
"""Color of dataset background."""
|
||||
scene_box: SceneBox = SceneBox()
|
||||
"""Scene box of dataset. Used to bound the scene or provide the scene scale depending on model."""
|
||||
semantics: Optional[Semantics] = None
|
||||
"""Semantics information."""
|
||||
mask_filenames: Optional[List[Path]] = None
|
||||
"""Filenames for any masks that are required"""
|
||||
metadata: Dict[str, Any] = to_immutable_dict({})
|
||||
"""Dictionary of any metadata that be required for the given experiment.
|
||||
Will be processed by the InputDataset to create any additional tensors that may be required.
|
||||
"""
|
||||
dataparser_transform: TensorType[3, 4] = torch.eye(4)[:3, :]
|
||||
"""Transform applied by the dataparser."""
|
||||
dataparser_scale: float = 1.0
|
||||
"""Scale applied by the dataparser."""
|
||||
|
||||
@dataclass
|
||||
class DataParser:
|
||||
@@ -115,48 +123,12 @@ You can also pull out information from the DataParserOutputs for other DataManga
|
||||
ray_generator = RayGenerator(dataparser_outputs.cameras)
|
||||
```
|
||||
|
||||
## Our Implementations
|
||||
## Included DataParsers
|
||||
|
||||
Below we enumerate the various dataparsers that we have implemented in our codebase. Feel free to use ours or add your own. Also any contributions are welcome and appreciated!
|
||||
```{toctree}
|
||||
---
|
||||
maxdepth: 2
|
||||
---
|
||||
|
||||
###### Nerfstudio
|
||||
|
||||
This is our custom dataparser. We have a script to convert images or videos with COLMAP to this format.
|
||||
|
||||
```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/nerfstudio_dataparser.py
|
||||
:color: primary
|
||||
:outline:
|
||||
See the code!
|
||||
```
|
||||
|
||||
###### Blender
|
||||
|
||||
We support the synthetic Blender dataset from the original NeRF paper.
|
||||
|
||||
```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/blender_dataparser.py
|
||||
:color: primary
|
||||
:outline:
|
||||
See the code!
|
||||
```
|
||||
|
||||
###### Instant NGP
|
||||
|
||||
This supports the Instant NGP dataset.
|
||||
|
||||
```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/instant_ngp_dataparser.py
|
||||
:color: primary
|
||||
:outline:
|
||||
See the code!
|
||||
```
|
||||
|
||||
###### Record3D
|
||||
|
||||
This dataparser can use recorded data from a >= iPhone 12 Pro using the [Record3D app](https://record3d.app/) . Record a video and export with the `EXR + JPG sequence` format. Unzip export and `rgb` folder before training.
|
||||
|
||||
For more information on capturing with Record3D, see the [Custom Dataset Docs](/quickstart/custom_dataset.md).
|
||||
|
||||
```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/record3d_dataparser.py
|
||||
:color: primary
|
||||
:outline:
|
||||
See the code!
|
||||
../../reference/api/data/dataparsers
|
||||
```
|
||||
@@ -0,0 +1,105 @@
|
||||
# Custom GUI
|
||||
|
||||
We provide support for custom viewer GUI elements that can be defined in any `nn.Module`. Although we don't have any specific use cases in mind, here are some examples of what can be achieved with this feature:
|
||||
|
||||
- Using text input to modify the rendering
|
||||
- Logging numerical values to the viewer
|
||||
- Using checkboxes to turn off and on losses
|
||||
- Using a dropdown to switch between appearances
|
||||
|
||||
## Adding an Element
|
||||
|
||||
To define a custom element, create an instance of one of the provided classes in `nerfstudio.viewer.server.viewer_elements`, and assign it as a class variable in your `nn.Module`.
|
||||
|
||||
```python
|
||||
from nerfstudio.viewer.server.viewer_elements import ViewerNumber
|
||||
|
||||
class MyClass(nn.Module):#must inherit from nn.Module
|
||||
def __init__(self):
|
||||
# Must be a class variable
|
||||
self.custom_value = ViewerNumber(name="My Value", default_value=1.0)
|
||||
```
|
||||
**Element Hierarchy**
|
||||
The viewer recursively searches all `nn.Module` children of the base `Pipeline` object, and arranges parameters into folders based on their variable names.
|
||||
For example, a `ViewerElement` defined in `pipeline.model.field` will be in the "Custom/model/field" folder in the GUI.
|
||||
|
||||
**Reading the value**
|
||||
To read the value of a custom element, simply access its `value` attribute. In this case it will be `1.0` unless modified by the user in the viewer.
|
||||
|
||||
```python
|
||||
current_value = self.custom_value.value
|
||||
```
|
||||
|
||||
**Callbacks**
|
||||
You can register a callback that will be called whenever a new value for your GUI element is available. For example, one can use a callback to update config parameters when elements are changed:
|
||||
```python
|
||||
def on_change_callback(handle: ViewerCheckbox) -> None:
|
||||
self.config.example_parameter = handle.value
|
||||
|
||||
self.custom_checkbox = ViewerCheckbox(
|
||||
name="Checkbox",
|
||||
default_value=False,
|
||||
cb_hook=on_change_callback,
|
||||
)
|
||||
```
|
||||
|
||||
**Thread safety**
|
||||
Note that `ViewerElement` values can change asynchronously to model execution. So, it's best practice to store the value of a viewer element once at the beginning
|
||||
of a forward pass and refer to the static variable afterwards.
|
||||
```python
|
||||
class MyModel(Model):
|
||||
def __init__(self):
|
||||
self.slider = ViewerSlider(name="Slider", default_value=0.5, min_value=0.0, max_value=1.0)
|
||||
|
||||
def get_outputs(self,ray)
|
||||
slider_val = self.slider.value
|
||||
#self.slider.value could change after this, unsafe to use
|
||||
|
||||
```
|
||||
|
||||
|
||||
**Writing to the element**
|
||||
You can write to a viewer element in Python, which provides a convenient way to track values in your code without the need for wandb/tensorboard or relying on `print` statements.
|
||||
|
||||
```python
|
||||
self.custom_value.value = x
|
||||
```
|
||||
|
||||
:::{admonition} Warning
|
||||
:class: warning
|
||||
|
||||
Updating module state while training can have unexpected side effects. It is up to the user to ensure that GUI actions are safe. Conditioning on `self.training` can help determine whether effects are applied during forward passes for training or rendering.
|
||||
:::
|
||||
|
||||
## Example Elements
|
||||
|
||||
```{image} imgs/custom_controls.png
|
||||
:align: center
|
||||
:width: 400
|
||||
```
|
||||
|
||||
This was created with the following elements:
|
||||
|
||||
```python
|
||||
from nerfstudio.viewer.server.viewer_elements import *
|
||||
|
||||
class MyModel(Model):
|
||||
def __init__(self):
|
||||
self.a = ViewerButton(name="My Button", cb_hook=self.handle_btn)
|
||||
self.b = ViewerNumber(name="Number", default_value=1.0)
|
||||
self.c = ViewerCheckbox(name="Checkbox", default_value=False)
|
||||
self.d = ViewerDropdown(name="Dropdown", default_value="A", options=["A", "B"])
|
||||
self.e = ViewerSlider(name="Slider", default_value=0.5, min_value=0.0, max_value=1.0)
|
||||
self.f = ViewerText(name="Text", default_value="Hello World")
|
||||
self.g = ViewerVec3(name="3D Vector", default_value=(0.1, 0.7, 0.1))
|
||||
|
||||
self.rgb_renderer = RGBRenderer()
|
||||
...
|
||||
class RGBRenderer(nn.Module):
|
||||
def __init__(self):
|
||||
#lives in "Custom/model/rgb_renderer" GUI folder
|
||||
self.a = ViewerRGB(name="F", default_value=(0.1, 0.7, 0.1))
|
||||
...
|
||||
```
|
||||
|
||||
For more information on the available classes and their arguments, refer to the [API documentation](../../reference/api/viewer.rst)
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 384 KiB |
@@ -0,0 +1,22 @@
|
||||
# Viewer
|
||||
|
||||
> We have a real-time web viewer that requires no installation. It's available at [https://viewer.nerf.studio/](https://viewer.nerf.studio/), where you can connect to your training job.
|
||||
|
||||
The viewer is built on [Viser](https://github.com/brentyi/viser/tree/main/viser) using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine.
|
||||
|
||||
```{toctree}
|
||||
:titlesonly:
|
||||
|
||||
custom_gui
|
||||
local_viewer
|
||||
```
|
||||
|
||||
## Acknowledgements and references
|
||||
|
||||
We thank the authors and contributors to the following repos, which we've started, used, and modified for our use-cases.
|
||||
|
||||
- [Viser](https://github.com/brentyi/viser/tree/main/viser) - made by [Brent Yi](https://github.com/brentyi)
|
||||
- [meshcat-python](https://github.com/rdeits/meshcat-python) - made by [Robin Deits](https://github.com/rdeits)
|
||||
- [meshcat](https://github.com/rdeits/meshcat) - made by [Robin Deits](https://github.com/rdeits)
|
||||
- [ThreeJS](https://threejs.org/)
|
||||
- [ReactJS](https://reactjs.org/)
|
||||
@@ -0,0 +1,62 @@
|
||||
# Local Server
|
||||
|
||||
If you are unable to connect to `https://viewer.nerf.studio`, want to use Safari, or want develop the viewer codebase, you can launch your own local viewer.
|
||||
|
||||
## Installing Dependencies
|
||||
|
||||
```shell
|
||||
cd nerfstudio/viewer/app
|
||||
```
|
||||
|
||||
Install npm (to install yarn) and yarn
|
||||
|
||||
```shell
|
||||
sudo apt-get install npm
|
||||
npm install --global yarn
|
||||
```
|
||||
|
||||
Install nvm and set the node version
|
||||
Install nvm with [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
|
||||
|
||||
```shell
|
||||
nvm install 17.8.0
|
||||
```
|
||||
|
||||
Now running `node --version` in the shell should print "v17.8.0".
|
||||
Install package.json dependencies and start the client viewer app:
|
||||
|
||||
```shell
|
||||
yarn install
|
||||
```
|
||||
|
||||
## Launch the web client
|
||||
|
||||
From the `nerfstudio/viewer/app` folder, run:
|
||||
|
||||
```shell
|
||||
yarn start
|
||||
```
|
||||
|
||||
The local webserver runs on port 4000 by default,
|
||||
so when `ns-train` is running, you can connect to the viewer locally at
|
||||
[http://localhost:4000/?websocket_url=ws://localhost:7007](http://localhost:4000/?websocket_url=ws://localhost:7007)
|
||||
|
||||
## FAQ
|
||||
|
||||
### Engine node incompatible
|
||||
|
||||
While running `yarn install`, you run into: `The engine "node" is incompatible with this module.`
|
||||
|
||||
**Solution**:
|
||||
|
||||
Install nvm with instructions at [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
|
||||
|
||||
```shell
|
||||
nvm install 17.8.0
|
||||
```
|
||||
|
||||
If you cannot install nvm, try ignoring the engines
|
||||
|
||||
```shell
|
||||
yarn install --ignore-engines
|
||||
```
|
||||
Reference in new issue
Block a user