mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-11 02:24:38 +08:00
63af268495
* Remove webrtc * cleanup * lint * cleanup * Lower quailty * Update viewer version * Update docs * remove cryptography req
99 lines
3.8 KiB
Markdown
99 lines
3.8 KiB
Markdown
# Viewer overview
|
|
|
|
> We have a real-time web viewer that requires no installation. It's available at [https://viewer.nerf.studio/](https://viewer.nerf.studio/), where you can connect to your training job. If you want to understand how the viewer works and contribute to it's features, this section is for you!
|
|
|
|
The viewer is built using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine. The following figure helps to illustrate how our viewer framework works:
|
|
|
|
```{image} imgs/viewer_light.png
|
|
:align: center
|
|
:class: only-light
|
|
```
|
|
|
|
```{image} imgs/viewer_dark.png
|
|
:align: center
|
|
:class: only-dark
|
|
```
|
|
|
|
## System design
|
|
|
|
#### Connection between nerfstudio and the Bridge Server
|
|
|
|
In the center, we have the Bridge Server, which facilitates the connection between nerfstudio code and the Client App. This server runs on the same machine that you are using nerfstudio. It has a TCP Request/Reply (REQ/REP) connection that nerfstudio can connect to with the Viewer object (left). We use [ZeroMQ](https://zeromq.org/), an open-sourced messaging library, to implement this lightweight TCP connection. The Viewer class can send commands to the Bridge Server and receive replies. The Bridge Server will either dispatch commands to the Client App via a websocket or it will return information stored in the Bridge Server state.
|
|
|
|
#### Connection between the Bridge Server and the Client App
|
|
|
|
The connection between the Bridge Server and the Client App works with WebSockets.
|
|
|
|
- **WebSocket connection** - The WebSocket is used by the Bridge Server to dispatch commands coming from the nerfstudio TCP connection. Commands can be used for drawing primitives, for setting the transform of objects, for the setting various properties, and more.
|
|
|
|
## Installing and running locally
|
|
|
|
#### Running the Client App
|
|
|
|
```shell
|
|
cd nerfstudio/viewer/app
|
|
```
|
|
|
|
Install npm (to install yarn) and yarn
|
|
|
|
```shell
|
|
sudo apt-get install npm
|
|
npm install --global yarn
|
|
```
|
|
|
|
Install nvm and set the node version
|
|
Install nvm with [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
|
|
|
|
```shell
|
|
nvm install 17.8.0
|
|
```
|
|
|
|
Now running `node --version` in the shell should print "v17.8.0".
|
|
Install package.json dependencies and start the client viewer app
|
|
|
|
```shell
|
|
yarn install
|
|
yarn start
|
|
```
|
|
|
|
The local webserver runs on port 4000 by default,
|
|
so when `ns-train` is running, you can connect to the viewer locally at
|
|
[http://localhost:4000/?websocket_url=ws://localhost:7007](http://localhost:4000/?websocket_url=ws://localhost:7007)
|
|
|
|
#### Running the Bridge Server Manually
|
|
|
|
The viewer bridge server runs on the same machine that you use for training. The training code will connect to this server with a lightweight TCP connection using ZMQ. The training job will launch the viewer bridge server is you specify `--viewer.launch-bridge-server` in the terminal. Otherwise, you can launch the bridge server manually with the following script.
|
|
|
|
```python
|
|
ns-bridge-server --help
|
|
```
|
|
|
|
## Acknowledgements and references
|
|
|
|
We thank the authors and contributors to the following repos, which we've started, used, and modified for our use-cases.
|
|
|
|
- [meshcat-python](https://github.com/rdeits/meshcat-python) - made by [Robin Deits](https://github.com/rdeits)
|
|
- [meshcat](https://github.com/rdeits/meshcat) - made by [Robin Deits](https://github.com/rdeits)
|
|
- [ThreeJS](https://threejs.org/)
|
|
- [ReactJS](https://reactjs.org/)
|
|
|
|
## FAQ
|
|
|
|
#### Engine node incompatible
|
|
|
|
While running `yarn install`, you run into: `The engine "node" is incompatible with this module.`
|
|
|
|
**Solution**:
|
|
|
|
Install nvm with instructions at [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/).
|
|
|
|
```shell
|
|
nvm install 17.8.0
|
|
```
|
|
|
|
If you cannot install nvm, try ignoring the engines
|
|
|
|
```
|
|
yarn install --ignore-engines
|
|
```
|