mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-20 14:45:38 +08:00
Viewer backend refactor (#1702)
* Copy in `viser` communication infrastructure * Update background image * Add build directory to fix viser HTTP server May want to remove these later * Copy in `viser` communication infrastructure * Update background image * Pass dataset vis using viser * initial impl of ViewerParameter * idk what i did but brent and matt know * Suppress eslint typescript warnings * implement ViewerElement base class, then ViewerButton, ViewerSlider, etc * client -> server message setup * multi folder stuff * Stream controls over viser * import * add types to viewer_param.py * Move another message to viser * Update folder color * Use non throttled messages * multiple folders works * Cleanup, more porting * starting impl of control panel in viser * More porting * scenebox to viser * Cleanup * Fix render updates * More cleanup, docs, typing * start control_panel.py * set_hidden logic but not merged in * control_panel replicates behavior of existing control panel (visually), except for setting the list of output options. not yet integrated with viewer_utils.py for logic flow. also implemented set_disabled and set_hidden in javscript/ViewerElement side * Remove some original leva components * add some train util logic, attempt set_value, doesn't work on javascript side fully yet (value updates but not slider visually * bye zmq * missing call to decoder * 'gui_set' works properly, control panel now replicates behavior of original * remove print * Fix status viewer message * More cleanup * support set_options in dropdown * hiding buttons works correctly * fix set_options, switch from cur_value to get_value() * Fix slowdown * output options * colormap logic converted * add handle to cb_hook * fix some lambda bugs * cleanup * Save checkpoint logic readded * Cleanup and minor fixes * snappier ui * Refactoring * Fix state update bugs * Bug fixes * Write less to redux * added comments to reducer code * Render state machine (#1727) * state machine works, but train step throttling not implemented * more step logic, cleanup * remove random try catcH * mutex logic for render * Update lock logic * types and docs * added some state machine logic to help rerenders * merge --------- Co-authored-by: Justin <justin.g.kerr@gmail.com> * Refactor and cleanup * More cleanup and docs * docstring to gui_utils * docstrings, rename viewer_param.py to viewer_elements.py * Fix viewer, remove bridge stuff * Render on connect * Fix lint errors * More eslint fixes * render_freq logic * better rerender logic for crop * More cleanup * Purge * Cleanup * Lint fixes * Fix cache * better parsing logic * Fix infinite test * More cleanup * typo * Update webdocs * Update docs * rerender on custom hooks * add more docs * Use shared utilities in `viser.infra` * rework callback to use elements instead of handles * Fix ESlint errors * change naming of handle to element, add setter for value * cleanup --------- Co-authored-by: Matt Tancik <mattt4001@gmail.com> Co-authored-by: Justin <justin.g.kerr@gmail.com> Co-authored-by: Ethan Weber <eweb0124@gmail.com> Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
This commit is contained in:
68 files changed
+3700
-2584
No files matched your search
@@ -0,0 +1,106 @@
|
||||
# Custom GUI
|
||||
|
||||
We provide support for custom viewer GUI elements that can be defined in any `nn.Module`. Although we don't have any specific use cases in mind, here are some examples of what can be achieved with this feature:
|
||||
|
||||
- Using text input to modify the rendering
|
||||
- Logging numerical values to the viewer
|
||||
- Using checkboxes to turn off and on losses
|
||||
- Using a dropdown to switch between appearances
|
||||
|
||||
## Adding an Element
|
||||
|
||||
To define a custom element, create an instance of one of the provided classes in `nerfstudio.viewer.server.viewer_elements`, and assign it as a class variable in your `nn.Module`.
|
||||
|
||||
```python
|
||||
from nerfstudio.viewer.server.viewer_elements import ViewerNumber
|
||||
|
||||
class MyClass(nn.Module):#must inherit from nn.Module
|
||||
def __init__(self):
|
||||
# Must be a class variable
|
||||
self.custom_value = ViewerNumber(name="My Value", default_value=1.0)
|
||||
```
|
||||
**Element Hierarchy**
|
||||
The viewer recursively searches all `nn.Module` children of the base `Pipeline` object, and arranges parameters into folders based on their variable names.
|
||||
For example, a `ViewerElement` defined in `pipeline.model.field` will be in the "Custom/model/field" folder in the GUI.
|
||||
|
||||
**Reading the value**
|
||||
To read the value of a custom element, simply access its `value` attribute. In this case it will be `1.0` unless modified by the user in the viewer.
|
||||
|
||||
```python
|
||||
current_value = self.custom_value.value
|
||||
```
|
||||
|
||||
**Callbacks**
|
||||
You can register a callback that will be called whenever a new value for your GUI element is available. For example, one can use a callback to update config parameters when elements are changed:
|
||||
```python
|
||||
def on_change_callback(handle: GuiHandle):
|
||||
self.config.example_parameter = handle.get_value()
|
||||
|
||||
self.custom_checkbox = ViewerCheckbox(
|
||||
name="Checkbox",
|
||||
default_value = False,
|
||||
cb_hook = on_change_callback
|
||||
)
|
||||
```
|
||||
The GuiHandle value is a viser object which you can use to get the new value.
|
||||
|
||||
**Thread safety**
|
||||
Note that `ViewerElement` values can change asynchronously to model execution. So, it's best practice to store the value of a viewer element once at the beginning
|
||||
of a forward pass and refer to the static variable afterwards.
|
||||
```python
|
||||
class MyModel(Model):
|
||||
def __init__(self):
|
||||
self.slider = ViewerSlider(name="Slider", default_value=0.5, min_value=0.0, max_value=1.0)
|
||||
|
||||
def get_outputs(self,ray)
|
||||
slider_val = self.slider.value
|
||||
#self.slider.value could change after this, unsafe to use
|
||||
|
||||
```
|
||||
|
||||
|
||||
**Writing to the element**
|
||||
You can write to a viewer element in Python, which provides a convenient way to track values in your code without the need for wandb/tensorboard or relying on `print` statements.
|
||||
|
||||
```python
|
||||
self.custom_value = x
|
||||
```
|
||||
|
||||
:::{admonition} Warning
|
||||
:class: warning
|
||||
|
||||
Note that updating module state while training can have unexpected side effects. It is up to the user to ensure that GUI actions are safe. In most cases, we recommend wrapping GUI related logic in `if self.training:` to ensure that the effects are only applied during inference.
|
||||
:::
|
||||
|
||||
## Example Elements
|
||||
|
||||
```{image} imgs/custom_controls.png
|
||||
:align: center
|
||||
:width: 400
|
||||
```
|
||||
|
||||
This was created with the following elements:
|
||||
|
||||
```python
|
||||
from nerfstudio.viewer.server.viewer_elements import *
|
||||
|
||||
class MyModel(Model):
|
||||
def __init__(self):
|
||||
self.a = ViewerButton(name="My Button", call_fn=self.handle_btn)
|
||||
self.b = ViewerNumber(name="Number", default_value=1.0)
|
||||
self.c = ViewerCheckbox(name="Checkbox", default_value=False)
|
||||
self.d = ViewerDropdown(name="Dropdown", default_value="A", options=["A", "B"])
|
||||
self.e = ViewerSlider(name="Slider", default_value=0.5, min_value=0.0, max_value=1.0)
|
||||
self.f = ViewerText(name="Text", default_value="Hello World")
|
||||
self.g = ViewerVec3(name="3D Vector", default_value=(0.1, 0.7, 0.1))
|
||||
|
||||
self.rgb_renderer = RGBRenderer()
|
||||
...
|
||||
class RGBRenderer(nn.Module):
|
||||
def __init__(self):
|
||||
#lives in "Custom/model/rgb_renderer" GUI folder
|
||||
self.a = ViewerRGB(name="F", default_value=(0.1, 0.7, 0.1))
|
||||
...
|
||||
```
|
||||
|
||||
For more information on the available classes and their arguments, refer to the [API documentation](../../reference/api/viewer.rst)
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 582 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 384 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 593 KiB |
@@ -0,0 +1,22 @@
|
||||
# Viewer
|
||||
|
||||
> We have a real-time web viewer that requires no installation. It's available at [https://viewer.nerf.studio/](https://viewer.nerf.studio/), where you can connect to your training job.
|
||||
|
||||
The viewer is built on [Viser](https://github.com/brentyi/viser/tree/main/viser) using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine.
|
||||
|
||||
```{toctree}
|
||||
:titlesonly:
|
||||
|
||||
custom_gui
|
||||
local_viewer
|
||||
```
|
||||
|
||||
## Acknowledgements and references
|
||||
|
||||
We thank the authors and contributors to the following repos, which we've started, used, and modified for our use-cases.
|
||||
|
||||
- [Viser](https://github.com/brentyi/viser/tree/main/viser) - made by [Brent Yi](https://github.com/brentyi)
|
||||
- [meshcat-python](https://github.com/rdeits/meshcat-python) - made by [Robin Deits](https://github.com/rdeits)
|
||||
- [meshcat](https://github.com/rdeits/meshcat) - made by [Robin Deits](https://github.com/rdeits)
|
||||
- [ThreeJS](https://threejs.org/)
|
||||
- [ReactJS](https://reactjs.org/)
|
||||
@@ -0,0 +1,62 @@
|
||||
# Local Server
|
||||
|
||||
If you are unable to connect to `https://viewer.nerf.studio`, want to use Safari, or want develop the viewer codebase, you can launch your own local viewer.
|
||||
|
||||
## Installing Dependencies
|
||||
|
||||
```shell
|
||||
cd nerfstudio/viewer/app
|
||||
```
|
||||
|
||||
Install npm (to install yarn) and yarn
|
||||
|
||||
```shell
|
||||
sudo apt-get install npm
|
||||
npm install --global yarn
|
||||
```
|
||||
|
||||
Install nvm and set the node version
|
||||
Install nvm with [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
|
||||
|
||||
```shell
|
||||
nvm install 17.8.0
|
||||
```
|
||||
|
||||
Now running `node --version` in the shell should print "v17.8.0".
|
||||
Install package.json dependencies and start the client viewer app:
|
||||
|
||||
```shell
|
||||
yarn install
|
||||
```
|
||||
|
||||
## Launch the web client
|
||||
|
||||
From the `nerfstudio/viewer/app` folder, run:
|
||||
|
||||
```shell
|
||||
yarn start
|
||||
```
|
||||
|
||||
The local webserver runs on port 4000 by default,
|
||||
so when `ns-train` is running, you can connect to the viewer locally at
|
||||
[http://localhost:4000/?websocket_url=ws://localhost:7007](http://localhost:4000/?websocket_url=ws://localhost:7007)
|
||||
|
||||
## FAQ
|
||||
|
||||
### Engine node incompatible
|
||||
|
||||
While running `yarn install`, you run into: `The engine "node" is incompatible with this module.`
|
||||
|
||||
**Solution**:
|
||||
|
||||
Install nvm with instructions at [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
|
||||
|
||||
```shell
|
||||
nvm install 17.8.0
|
||||
```
|
||||
|
||||
If you cannot install nvm, try ignoring the engines
|
||||
|
||||
```shell
|
||||
yarn install --ignore-engines
|
||||
```
|
||||
@@ -1,98 +0,0 @@
|
||||
# Viewer overview
|
||||
|
||||
> We have a real-time web viewer that requires no installation. It's available at [https://viewer.nerf.studio/](https://viewer.nerf.studio/), where you can connect to your training job. If you want to understand how the viewer works and contribute to it's features, this section is for you!
|
||||
|
||||
The viewer is built using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine. The following figure helps to illustrate how our viewer framework works:
|
||||
|
||||
```{image} imgs/viewer_light.png
|
||||
:align: center
|
||||
:class: only-light
|
||||
```
|
||||
|
||||
```{image} imgs/viewer_dark.png
|
||||
:align: center
|
||||
:class: only-dark
|
||||
```
|
||||
|
||||
## System design
|
||||
|
||||
#### Connection between nerfstudio and the Bridge Server
|
||||
|
||||
In the center, we have the Bridge Server, which facilitates the connection between nerfstudio code and the Client App. This server runs on the same machine that you are using nerfstudio. It has a TCP Request/Reply (REQ/REP) connection that nerfstudio can connect to with the Viewer object (left). We use [ZeroMQ](https://zeromq.org/), an open-sourced messaging library, to implement this lightweight TCP connection. The Viewer class can send commands to the Bridge Server and receive replies. The Bridge Server will either dispatch commands to the Client App via a websocket or it will return information stored in the Bridge Server state.
|
||||
|
||||
#### Connection between the Bridge Server and the Client App
|
||||
|
||||
The connection between the Bridge Server and the Client App works with WebSockets.
|
||||
|
||||
- **WebSocket connection** - The WebSocket is used by the Bridge Server to dispatch commands coming from the nerfstudio TCP connection. Commands can be used for drawing primitives, for setting the transform of objects, for the setting various properties, and more.
|
||||
|
||||
## Installing and running locally
|
||||
|
||||
#### Running the Client App
|
||||
|
||||
```shell
|
||||
cd nerfstudio/viewer/app
|
||||
```
|
||||
|
||||
Install npm (to install yarn) and yarn
|
||||
|
||||
```shell
|
||||
sudo apt-get install npm
|
||||
npm install --global yarn
|
||||
```
|
||||
|
||||
Install nvm and set the node version
|
||||
Install nvm with [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
|
||||
|
||||
```shell
|
||||
nvm install 17.8.0
|
||||
```
|
||||
|
||||
Now running `node --version` in the shell should print "v17.8.0".
|
||||
Install package.json dependencies and start the client viewer app
|
||||
|
||||
```shell
|
||||
yarn install
|
||||
yarn start
|
||||
```
|
||||
|
||||
The local webserver runs on port 4000 by default,
|
||||
so when `ns-train` is running, you can connect to the viewer locally at
|
||||
[http://localhost:4000/?websocket_url=ws://localhost:7007](http://localhost:4000/?websocket_url=ws://localhost:7007)
|
||||
|
||||
#### Running the Bridge Server Manually
|
||||
|
||||
The viewer bridge server runs on the same machine that you use for training. The training code will connect to this server with a lightweight TCP connection using ZMQ. The training job will launch the viewer bridge server is you specify `--viewer.launch-bridge-server` in the terminal. Otherwise, you can launch the bridge server manually with the following script.
|
||||
|
||||
```python
|
||||
ns-bridge-server --help
|
||||
```
|
||||
|
||||
## Acknowledgements and references
|
||||
|
||||
We thank the authors and contributors to the following repos, which we've started, used, and modified for our use-cases.
|
||||
|
||||
- [meshcat-python](https://github.com/rdeits/meshcat-python) - made by [Robin Deits](https://github.com/rdeits)
|
||||
- [meshcat](https://github.com/rdeits/meshcat) - made by [Robin Deits](https://github.com/rdeits)
|
||||
- [ThreeJS](https://threejs.org/)
|
||||
- [ReactJS](https://reactjs.org/)
|
||||
|
||||
## FAQ
|
||||
|
||||
#### Engine node incompatible
|
||||
|
||||
While running `yarn install`, you run into: `The engine "node" is incompatible with this module.`
|
||||
|
||||
**Solution**:
|
||||
|
||||
Install nvm with instructions at [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
|
||||
|
||||
```shell
|
||||
nvm install 17.8.0
|
||||
```
|
||||
|
||||
If you cannot install nvm, try ignoring the engines
|
||||
|
||||
```
|
||||
yarn install --ignore-engines
|
||||
```
|
||||
Reference in new issue
Block a user