diff --git a/docs/developer_guides/viewer/faq.md b/docs/developer_guides/viewer/faq.md deleted file mode 100644 index d140a4221..000000000 --- a/docs/developer_guides/viewer/faq.md +++ /dev/null @@ -1,21 +0,0 @@ -# FAQ - -* [Engine node incompatible](engine-error) - -(engine-error)= -#### Engine node incompatible - -While running `yarn install`, you run into: `The engine "node" is incompatible with this module.` - -**Solution**: - -Install nvm with instructions at [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/). - -```shell -nvm install 17.8.0 -``` - -If you cannot install nvm, try ignoring the engines -``` -yarn install --ignore-engines -``` \ No newline at end of file diff --git a/docs/developer_guides/viewer/index.rst b/docs/developer_guides/viewer/index.rst deleted file mode 100644 index a67f84cf5..000000000 --- a/docs/developer_guides/viewer/index.rst +++ /dev/null @@ -1,12 +0,0 @@ -Real-time web viewer -==================== - -# TODO: high level summary - -.. toctree:: - :maxdepth: 1 - - viewer_quickstart.md - for_developers.md - viewer_notebook.ipynb - faq.md \ No newline at end of file diff --git a/docs/developer_guides/viewer/viewer_notebook.ipynb b/docs/developer_guides/viewer/viewer_notebook.ipynb deleted file mode 100644 index d791ac338..000000000 --- a/docs/developer_guides/viewer/viewer_notebook.ipynb +++ /dev/null @@ -1,521 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "3c00934d-0f14-4c1a-a563-94d627b377e5", - "metadata": {}, - "source": [ - "# Programming the viewer\n", - "\n", - "The real-time web viewer enables communication between Python and the ReactJS Client Viewer Application. Here we provide a walkthrough for how to use the viewer to draw information as well as read information from the GUI.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 72, - "id": "b15dfdc1-5054-40bc-9631-4c7426eecfd9", - "metadata": { - "tags": [ - "remove-cell" - ] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "The autoreload extension is already loaded. To reload it, use:\n", - " %reload_ext autoreload\n" - ] - } - ], - "source": [ - "# HIDDEN\n", - "%load_ext autoreload\n", - "%autoreload 2\n", - "import pprint\n", - "import torch\n", - "import numpy as np\n", - "import torch\n", - "import mediapy as media\n", - "from nerfstudio.data.scene_box import SceneBox\n", - "from nerfstudio.cameras.cameras import PinholeCamera\n", - "import time" - ] - }, - { - "cell_type": "markdown", - "id": "a8709c84", - "metadata": {}, - "source": [ - "## Open the Nerfviewer client\n", - "\n", - "Open the Nerfviewer client by visiting http://viewer.nerf.studio/ or running the code locally. *We refer to this document for running the code locally.* You should see a GUI like the following:\n", - "\n", - "```{image} imgs/nerfviewer.png\n", - ":align: center\n", - ":class: only-light\n", - ":width: 600\n", - "```" - ] - }, - { - "cell_type": "markdown", - "id": "052af8ce", - "metadata": {}, - "source": [ - "## Start the Bridge Server\n", - "\n", - "Run the following code in the repository home directory\n", - "\n", - "```\n", - "python scripts/run_viewer_bridge_server.py\n", - "```\n", - "\n", - "This will start the bridge server that both the Python code and Client Viewer App will connect to.\n", - "\n", - "Verify that the Client Viewer App is connected." - ] - }, - { - "cell_type": "code", - "execution_count": 73, - "id": "189b2282", - "metadata": {}, - "outputs": [], - "source": [ - "# from IPython.display import HTML\n", - "\n", - "# url=\"http://localhost:4000/?localhost:18051\"\n", - "# height = 500\n", - "\n", - "# HTML(\"\"\"\n", - "#
\n", - "# \n", - "#
\n", - "# \"\"\".format(url=url, height=height))" - ] - }, - { - "cell_type": "markdown", - "id": "46c89306", - "metadata": {}, - "source": [ - "## Communicate with the Client Viewer App from Python\n", - "\n", - "Run the following code in the repository home directory\n", - "\n", - "```\n", - "python scripts/run_viewer_bridge_server.py\n", - "```\n", - "\n", - "This will start the bridge server that both the Python code and Client Viewer App will connect to.\n", - "\n", - "Verify that the Client Viewer App is connected." - ] - }, - { - "cell_type": "code", - "execution_count": 74, - "id": "8919ad68", - "metadata": {}, - "outputs": [], - "source": [ - "# connect to the viewer from Python\n", - "from nerfstudio.viewer.server.visualizer import Viewer\n", - "\n", - "vis = Viewer(zmq_url=\"tcp://127.0.0.1:6000\")" - ] - }, - { - "cell_type": "code", - "execution_count": 75, - "id": "605e0131-bfbe-4309-b338-c49a46923d8f", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "/sceneState/sceneBounds\n" - ] - } - ], - "source": [ - "# draw scene box in the viewer\n", - "s = 1\n", - "scene_box = SceneBox(aabb=torch.tensor([[-s, -s, -s], [s, s, s]]))\n", - "scene_box_json = scene_box.to_json()\n", - "\n", - "# draw scene box in the viewer\n", - "vis[\"sceneState/sceneBox\"].write(scene_box_json)\n", - "\n", - "# draw scene box in plotly\n", - "import plotly.graph_objects as go\n", - "\n", - "# TODO:" - ] - }, - { - "cell_type": "code", - "execution_count": 76, - "id": "fa0acb3c", - "metadata": { - "tags": [ - "remove-cell" - ] - }, - "outputs": [], - "source": [ - "# HIDDEN\n", - "def get_random_rotation_matrix():\n", - " roll = torch.randn(1)\n", - " yaw = torch.randn(1)\n", - " pitch = torch.randn(1)\n", - "\n", - " tensor_0 = torch.zeros(1)\n", - " tensor_1 = torch.ones(1)\n", - "\n", - " RX = torch.stack(\n", - " [\n", - " torch.stack([tensor_1, tensor_0, tensor_0]),\n", - " torch.stack([tensor_0, torch.cos(roll), -torch.sin(roll)]),\n", - " torch.stack([tensor_0, torch.sin(roll), torch.cos(roll)]),\n", - " ]\n", - " ).reshape(3, 3)\n", - "\n", - " RY = torch.stack(\n", - " [\n", - " torch.stack([torch.cos(pitch), tensor_0, torch.sin(pitch)]),\n", - " torch.stack([tensor_0, tensor_1, tensor_0]),\n", - " torch.stack([-torch.sin(pitch), tensor_0, torch.cos(pitch)]),\n", - " ]\n", - " ).reshape(3, 3)\n", - "\n", - " RZ = torch.stack(\n", - " [\n", - " torch.stack([torch.cos(yaw), -torch.sin(yaw), tensor_0]),\n", - " torch.stack([torch.sin(yaw), torch.cos(yaw), tensor_0]),\n", - " torch.stack([tensor_0, tensor_0, tensor_1]),\n", - " ]\n", - " ).reshape(3, 3)\n", - "\n", - " R = torch.mm(RZ, RY)\n", - " R = torch.mm(R, RX)\n", - " return R" - ] - }, - { - "cell_type": "code", - "execution_count": 77, - "id": "88b145d3", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "/sceneState/cameras\n" - ] - }, - { - "data": { - "text/html": [ - "
" - ], - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "/sceneState/cameras/000000\n", - "/sceneState/cameras/000001\n" - ] - }, - { - "data": { - "text/plain": [ - "b'ok'" - ] - }, - "execution_count": 77, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# delete any current cameras\n", - "vis[f\"sceneState/cameras\"].delete()\n", - "\n", - "# draw a camera with with an image\n", - "image = torch.rand(256, 256, 3)\n", - "media.show_image(image)\n", - "camera = PinholeCamera(cx=100, cy=100, fx=50, fy=50)\n", - "camera_json = camera.to_json(image=image)\n", - "vis[f\"sceneState/cameras/000000\"].write(camera_json)\n", - "\n", - "# draw a camera with rotation and translation (with a black image)\n", - "image = torch.zeros(256, 256, 3)\n", - "c2w = torch.eye(4)\n", - "c2w[:3, :3] = get_random_rotation_matrix()\n", - "random_translation = torch.rand(1) * 5\n", - "c2w[:3, 3] = random_translation\n", - "camera = PinholeCamera(cx=100, cy=100, fx=50, fy=50, camera_to_world=c2w)\n", - "camera_json = camera.to_json(image=image)\n", - "vis[f\"sceneState/cameras/000001\"].write(camera_json)" - ] - }, - { - "cell_type": "code", - "execution_count": 78, - "id": "18991f81", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "/sceneState/cameras/000000\n" - ] - }, - { - "data": { - "text/plain": [ - "b'ok'" - ] - }, - "execution_count": 78, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# delete a camera from the Nerfviewer\n", - "vis[\"sceneState/cameras/000000\"].delete()" - ] - }, - { - "cell_type": "markdown", - "id": "6b798e26", - "metadata": {}, - "source": [ - "## Read from the GUI sliders" - ] - }, - { - "cell_type": "code", - "execution_count": 79, - "id": "31cd0de1", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "80\n" - ] - } - ], - "source": [ - "# read the field of view\n", - "camera = vis[f\"renderingState/camera\"].read()\n", - "fov = camera[\"object\"][\"fov\"]\n", - "print(fov)" - ] - }, - { - "cell_type": "code", - "execution_count": 86, - "id": "feb83aaa", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "/renderingState/field_of_view\n" - ] - }, - { - "data": { - "text/plain": [ - "b'ok'" - ] - }, - "execution_count": 86, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "vis[f\"renderingState/field_of_view\"].write(40)" - ] - }, - { - "cell_type": "code", - "execution_count": 81, - "id": "1abb93e5", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "None\n" - ] - } - ], - "source": [ - "fov = vis[f\"renderingState/camera/object/fov\"].read()\n", - "print(fov)" - ] - }, - { - "cell_type": "code", - "execution_count": 82, - "id": "a0304523", - "metadata": {}, - "outputs": [], - "source": [ - "# set the field of view\n", - "camera = vis[f\"renderingState/camera\"].read()\n", - "fov = camera[\"object\"][\"fov\"]" - ] - }, - { - "cell_type": "code", - "execution_count": 83, - "id": "b3ba0ee7", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "/renderingState/minResolution\n" - ] - }, - { - "data": { - "text/plain": [ - "b'ok'" - ] - }, - "execution_count": 83, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "vis[f\"renderingState/minResolution\"].write(20)" - ] - }, - { - "cell_type": "code", - "execution_count": 84, - "id": "bba59227", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "20" - ] - }, - "execution_count": 84, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# read from a slider\n", - "vis[f\"renderingState/minResolution\"].read()" - ] - }, - { - "cell_type": "code", - "execution_count": 87, - "id": "fe922185", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'depth'" - ] - }, - "execution_count": 87, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# read from output choice\n", - "vis[\"renderingState/output_choice\"].read()" - ] - }, - { - "cell_type": "code", - "execution_count": 85, - "id": "75c37760", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "['rgb', 'accumulation', 'depth', 'alive_ray_mask']\n", - "None\n" - ] - } - ], - "source": [ - "# read from a dropdown\n", - "output_options = vis[f\"renderingState/output_options\"].read()\n", - "print(output_options)\n", - "output_choice = vis[f\"renderingState/output_choice\"].read()\n", - "print(output_choice)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "98aaa12c", - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.8.12 ('nerfactory')", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.12" - }, - "vscode": { - "interpreter": { - "hash": "34c28001ff35fb390494047002768a8182dcf55b1b11415165e62ea61557ab83" - } - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/docs/developer_guides/viewer/for_developers.md b/docs/developer_guides/viewer/viewer_overview.md similarity index 58% rename from docs/developer_guides/viewer/for_developers.md rename to docs/developer_guides/viewer/viewer_overview.md index d0f8cb28a..5d45fa5f5 100644 --- a/docs/developer_guides/viewer/for_developers.md +++ b/docs/developer_guides/viewer/viewer_overview.md @@ -1,6 +1,12 @@ -# For developers +# Viewer Overview -The tutorial below shows you how to host the viewer yourself and how you can use it to visualize training. +> We have a real-time web viewer that requires no installation. It's available at [https://viewer.nerf.studio/](https://viewer.nerf.studio/), where you can connect to your training job. If you want to understand how the viewer works and contribute to it's features, this section is for you! + +The viewer is built using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine. The following figure helps to illustrate how our viewer framework works: + +![visualize_dataset](imgs/viewer_figure.png) + +## System design #### Connection between nerfstudio and the Bridge Server @@ -14,27 +20,18 @@ The connection between the Bridge Server and the Client App works with WebSocket - **WebRTC connection** - We use WebRTC to stream images being rendered from nerfstudio. The websocket connection if first used to establish the WebRTC connection. Then, the Client App constantly publishes camera pose information to the Bridge Server and stores the camera information (intrinsics and extrinsics). This information is then queried from the nerfstudio code, used to render an image with some Graph, and then the image is send over the TCP connection and dispatched via WebRTC to render the stream of images. -## Getting started +## Installing and running locally #### Running the Bridge Server -The viewer server runs on the same machine that you use for training. The training code will connect to the server with a lightweight TCP connection. +The viewer bridge server runs on the same machine that you use for training. The training code will connect to this server with a lightweight TCP connection using ZMQ. The training job will launch the viewer bridge server is you specify `--viewer.launch-bridge-server` in the terminal. Otherwise, you can launch the bridge server manually with the following script. -``` -cd nerfstudio - -# run the server on your machine -# this will run in the background -python scripts/run_viewer_bridge_server.py - -It should print out something of the form: -"ZMQWebSocketBridge using zmq_url=tcp://127.0.0.1:6000 and websocket_port=8051" +```python +python scripts/viewer/run_bridge_server.py --help ``` #### Running the Client App -> We will host the viewer online in the future, but for now we have to run it locally. - ```shell cd nerfstudio/viewer/app ``` @@ -61,43 +58,32 @@ yarn install yarn start ``` -Forward port and open in your local browser. The URL takes the following form, `http://localhost:?localhost:`. Notice the "?" character. - -- http://localhost:4000?localhost:8051 - -Possibly edit the port in `app/.env.development` file with the following to adjust the PORT, for example. - -``` -BROWSER=none -FAST_REFRESH=false -HOST=0.0.0.0 -PORT=4000 -ESLINT_NO_DEV_ERRORS=true -``` - -#### Running the nerfstudio Code - -You can now simply run a training job and visualize progress by enabling the viewer during training: - -```bash -python scripts/train.py --config-name=graph_instant_ngp.yaml viewer.enable=true -``` - -- **Notebook demo** - See [Programming the viewer](viewer_notebook.ipynb) for an overview for how to interact with the viewer with the Viewer object from nerfstudio. - ## Acknowledgements and references -We thank [Robin Deits](https://github.com/rdeits) and other contributors to the following repos, which we've started with and modified and extended for our use. - -- [meshcat-python](https://github.com/rdeits/meshcat-python) -- [meshcat](https://github.com/rdeits/meshcat) - -Here are other resources that we've used and/or have found helpful while building the viewer code. +We thank the authors and contributors to the following repos, which we've started, used, and modified for our use-cases. +- [meshcat-python](https://github.com/rdeits/meshcat-python) - made by [Robin Deits](https://github.com/rdeits) +- [meshcat](https://github.com/rdeits/meshcat) - made by [Robin Deits](https://github.com/rdeits) - [ThreeJS](https://threejs.org/) - [ReactJS](https://reactjs.org/) - [WebRTC](https://webrtc.org/) - WebRTC is a framework for real-time communication that allows two peers to send video, audio, or general data to and from each other with low latency. We've adopted WebRTC to stream rendered images to our viewer. +## FAQ + +#### Engine node incompatible + +While running `yarn install`, you run into: `The engine "node" is incompatible with this module.` + +**Solution**: + +Install nvm with instructions at [instructions](https://heynode.com/tutorial/install-nodejs-locally-nvm/). + +```shell +nvm install 17.8.0 ``` +If you cannot install nvm, try ignoring the engines + +``` +yarn install --ignore-engines ``` diff --git a/docs/developer_guides/viewer/viewer_quickstart.md b/docs/developer_guides/viewer/viewer_quickstart.md deleted file mode 100644 index 191a6bd01..000000000 --- a/docs/developer_guides/viewer/viewer_quickstart.md +++ /dev/null @@ -1,22 +0,0 @@ -# Quickstart - -We provide a real-time, web-based viewer that can be used during training or to view a trained model. The purpose of the viewer is to be able to freely move the camera and see the reconstruction from novel viewpoints. It can also be used to draw various primitives (e.g., bounding boxes, camera frustums, etc.). - -The viewer is built using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine. The following figure helps to illustrate how our viewer framework works: - -![visualize_dataset](imgs/viewer_figure.png) - -## Run hosted web viewer - -1. Open the Client App Viewer at ... [https://viewer.nerf.studio/latest/?localhost:7007](https://viewer.nerf.studio/latest/?localhost:8051) but make sure to change `localhost:7007` accordingly. - -2. Run training script with viewer enabled. Note that you should only use the viewer with a fast pipeline, such as instant ngp. - ```shell - python scripts/train.py --config-name=graph_instant_ngp.yaml viewer.enable=True - ``` - -This pulls up a hosted web-viewer which is the fastest way to get started with visualizing your nerfstudio training progress! - -![viewer](imgs/bulldozer.png) - -If you want to host the viewer locally/yourself, please proceed to "For devleopers" section. diff --git a/docs/index.md b/docs/index.md index a14958889..9feaf4d94 100644 --- a/docs/index.md +++ b/docs/index.md @@ -57,7 +57,7 @@ nerfology/model_components/index :caption: Developer Guides developer_guides/pipelines/index -developer_guides/viewer/index +developer_guides/viewer/viewer_overview developer_guides/config developer_guides/debugging_tools/index ``` diff --git a/docs/quickstart/first_nerf.md b/docs/quickstart/first_nerf.md index a739de26d..f8e096862 100644 --- a/docs/quickstart/first_nerf.md +++ b/docs/quickstart/first_nerf.md @@ -24,8 +24,8 @@ The resulting script should download and unpack the dataset as follows: | ├─ fern/ | ├─ lego/ ... - |- / - |- + |- nerfstudio/ + |- poster ... ``` @@ -48,9 +48,9 @@ Run a nerfacto model. ns-train nerfacto ``` -Run with nerfstudio data with our web-based viewer enabled. +Run a nerfacto model with different data and port. ``` -ns-train nerfacto --vis viewer --viewer.zmq-port 8001 --viewer.websocket-port 8002 nerfstudio-data --pipeline.datamanager.dataparser.data-directory data/nerfstudio/poster --pipeline.datamanager.dataparser.downscale-factor 4 +ns-train nerfacto --vis viewer --data data/nerfstudio/poster --viewer.websocket-port 7007 ``` :::{admonition} Warning diff --git a/scripts/viewer/run_bridge_server.py b/scripts/viewer/run_bridge_server.py new file mode 100644 index 000000000..7aa27a348 --- /dev/null +++ b/scripts/viewer/run_bridge_server.py @@ -0,0 +1,7 @@ +""" +Runs the viewer bridge server. +""" +from nerfstudio.viewer.server.server import main + +if __name__ == "__main__": + main()