* Copy in `viser` communication infrastructure * Update background image * Add build directory to fix viser HTTP server May want to remove these later * Copy in `viser` communication infrastructure * Update background image * Pass dataset vis using viser * initial impl of ViewerParameter * idk what i did but brent and matt know * Suppress eslint typescript warnings * implement ViewerElement base class, then ViewerButton, ViewerSlider, etc * client -> server message setup * multi folder stuff * Stream controls over viser * import * add types to viewer_param.py * Move another message to viser * Update folder color * Use non throttled messages * multiple folders works * Cleanup, more porting * starting impl of control panel in viser * More porting * scenebox to viser * Cleanup * Fix render updates * More cleanup, docs, typing * start control_panel.py * set_hidden logic but not merged in * control_panel replicates behavior of existing control panel (visually), except for setting the list of output options. not yet integrated with viewer_utils.py for logic flow. also implemented set_disabled and set_hidden in javscript/ViewerElement side * Remove some original leva components * add some train util logic, attempt set_value, doesn't work on javascript side fully yet (value updates but not slider visually * bye zmq * missing call to decoder * 'gui_set' works properly, control panel now replicates behavior of original * remove print * Fix status viewer message * More cleanup * support set_options in dropdown * hiding buttons works correctly * fix set_options, switch from cur_value to get_value() * Fix slowdown * output options * colormap logic converted * add handle to cb_hook * fix some lambda bugs * cleanup * Save checkpoint logic readded * Cleanup and minor fixes * snappier ui * Refactoring * Fix state update bugs * Bug fixes * Write less to redux * added comments to reducer code * Render state machine (#1727) * state machine works, but train step throttling not implemented * more step logic, cleanup * remove random try catcH * mutex logic for render * Update lock logic * types and docs * added some state machine logic to help rerenders * merge --------- Co-authored-by: Justin <justin.g.kerr@gmail.com> * Refactor and cleanup * More cleanup and docs * docstring to gui_utils * docstrings, rename viewer_param.py to viewer_elements.py * Fix viewer, remove bridge stuff * Render on connect * Fix lint errors * More eslint fixes * render_freq logic * better rerender logic for crop * More cleanup * Purge * Cleanup * Lint fixes * Fix cache * better parsing logic * Fix infinite test * More cleanup * typo * Update webdocs * Update docs * rerender on custom hooks * add more docs * Use shared utilities in `viser.infra` * rework callback to use elements instead of handles * Fix ESlint errors * change naming of handle to element, add setter for value * cleanup --------- Co-authored-by: Matt Tancik <mattt4001@gmail.com> Co-authored-by: Justin <justin.g.kerr@gmail.com> Co-authored-by: Ethan Weber <eweb0124@gmail.com> Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
3.8 KiB
Viewer overview
We have a real-time web viewer that requires no installation. It's available at https://viewer.nerf.studio/, where you can connect to your training job. If you want to understand how the viewer works and contribute to it's features, this section is for you!
The viewer is built using ThreeJS and packaged into a ReactJS application. This client viewer application will connect via a websocket to a server running on your machine. The following figure helps to illustrate how our viewer framework works:
:align: center
:class: only-light
:align: center
:class: only-dark
System design
Connection between nerfstudio and the Bridge Server
In the center, we have the Bridge Server, which facilitates the connection between nerfstudio code and the Client App. This server runs on the same machine that you are using nerfstudio. It has a TCP Request/Reply (REQ/REP) connection that nerfstudio can connect to with the Viewer object (left). We use ZeroMQ, an open-sourced messaging library, to implement this lightweight TCP connection. The Viewer class can send commands to the Bridge Server and receive replies. The Bridge Server will either dispatch commands to the Client App via a websocket or it will return information stored in the Bridge Server state.
Connection between the Bridge Server and the Client App
The connection between the Bridge Server and the Client App works with WebSockets.
- WebSocket connection - The WebSocket is used by the Bridge Server to dispatch commands coming from the nerfstudio TCP connection. Commands can be used for drawing primitives, for setting the transform of objects, for the setting various properties, and more.
Installing and running locally
Running the Client App
cd nerfstudio/viewer/app
Install npm (to install yarn) and yarn
sudo apt-get install npm
npm install --global yarn
Install nvm and set the node version Install nvm with instructions.
nvm install 17.8.0
Now running node --version in the shell should print "v17.8.0".
Install package.json dependencies and start the client viewer app
yarn install
yarn start
The local webserver runs on port 4000 by default,
so when ns-train is running, you can connect to the viewer locally at
http://localhost:4000/?websocket_url=ws://localhost:7007
Running the Bridge Server Manually
The viewer bridge server runs on the same machine that you use for training. The training code will connect to this server with a lightweight TCP connection using ZMQ. The training job will launch the viewer bridge server is you specify --viewer.launch-bridge-server in the terminal. Otherwise, you can launch the bridge server manually with the following script.
ns-bridge-server --help
Acknowledgements and references
We thank the authors and contributors to the following repos, which we've started, used, and modified for our use-cases.
- meshcat-python - made by Robin Deits
- meshcat - made by Robin Deits
- ThreeJS
- ReactJS
FAQ
Engine node incompatible
While running yarn install, you run into: The engine "node" is incompatible with this module.
Solution:
Install nvm with instructions at instructions.
nvm install 17.8.0
If you cannot install nvm, try ignoring the engines
yarn install --ignore-engines