{ "cells": [ { "cell_type": "markdown", "id": "3c00934d-0f14-4c1a-a563-94d627b377e5", "metadata": {}, "source": [ "# Programming the viewer\n", "\n", "The real-time web viewer enables communication between Python and the ReactJS Client Viewer Application. Here we provide a walkthrough for how to use the viewer to draw information as well as read information from the GUI.\n" ] }, { "cell_type": "code", "execution_count": 72, "id": "b15dfdc1-5054-40bc-9631-4c7426eecfd9", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The autoreload extension is already loaded. To reload it, use:\n", " %reload_ext autoreload\n" ] } ], "source": [ "# HIDDEN\n", "%load_ext autoreload\n", "%autoreload 2\n", "import pprint\n", "import torch\n", "import numpy as np\n", "import torch\n", "import mediapy as media\n", "from nerfactory.data.structs import SceneBounds\n", "from nerfactory.cameras.cameras import PinholeCamera\n", "import time" ] }, { "cell_type": "markdown", "id": "a8709c84", "metadata": {}, "source": [ "## Open the Nerfviewer client\n", "\n", "Open the Nerfviewer client by visiting http://nerfviewer.plenoptix.ai/ or running the code locally. *We refer to this document for running the code locally.* You should see a GUI like the following:\n", "\n", "```{image} imgs/nerfviewer.png\n", ":align: center\n", ":class: only-light\n", ":width: 600\n", "```" ] }, { "cell_type": "markdown", "id": "052af8ce", "metadata": {}, "source": [ "## Start the Bridge Server\n", "\n", "Run the following code in the repository home directory\n", "\n", "```\n", "python scripts/run_viewer_bridge_server.py\n", "```\n", "\n", "This will start the bridge server that both the Python code and Client Viewer App will connect to.\n", "\n", "Verify that the Client Viewer App is connected." ] }, { "cell_type": "code", "execution_count": 73, "id": "189b2282", "metadata": {}, "outputs": [], "source": [ "# from IPython.display import HTML\n", "\n", "# url=\"http://localhost:4000/?localhost:18051\"\n", "# height = 500\n", "\n", "# HTML(\"\"\"\n", "#
\n", "# \n", "#
\n", "# \"\"\".format(url=url, height=height))" ] }, { "cell_type": "markdown", "id": "46c89306", "metadata": {}, "source": [ "## Communicate with the Client Viewer App from Python\n", "\n", "Run the following code in the repository home directory\n", "\n", "```\n", "python scripts/run_viewer_bridge_server.py\n", "```\n", "\n", "This will start the bridge server that both the Python code and Client Viewer App will connect to.\n", "\n", "Verify that the Client Viewer App is connected." ] }, { "cell_type": "code", "execution_count": 74, "id": "8919ad68", "metadata": {}, "outputs": [], "source": [ "# connect to the viewer from Python\n", "from nerfactory.viewer.server.visualizer import Viewer\n", "\n", "vis = Viewer(zmq_url=\"tcp://127.0.0.1:6000\")" ] }, { "cell_type": "code", "execution_count": 75, "id": "605e0131-bfbe-4309-b338-c49a46923d8f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/sceneState/sceneBounds\n" ] } ], "source": [ "# draw scene bounds in the viewer\n", "s = 1\n", "scene_bounds = SceneBounds(aabb=torch.tensor([[-s, -s, -s], [s, s, s]]))\n", "scene_bounds_json = scene_bounds.to_json()\n", "\n", "# draw scene bounds in the viewer\n", "vis[\"sceneState/sceneBounds\"].write(scene_bounds_json)\n", "\n", "# draw scene bounds in plotly\n", "import plotly.graph_objects as go\n", "\n", "# TODO:" ] }, { "cell_type": "code", "execution_count": 76, "id": "fa0acb3c", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "# HIDDEN\n", "def get_random_rotation_matrix():\n", " roll = torch.randn(1)\n", " yaw = torch.randn(1)\n", " pitch = torch.randn(1)\n", "\n", " tensor_0 = torch.zeros(1)\n", " tensor_1 = torch.ones(1)\n", "\n", " RX = torch.stack(\n", " [\n", " torch.stack([tensor_1, tensor_0, tensor_0]),\n", " torch.stack([tensor_0, torch.cos(roll), -torch.sin(roll)]),\n", " torch.stack([tensor_0, torch.sin(roll), torch.cos(roll)]),\n", " ]\n", " ).reshape(3, 3)\n", "\n", " RY = torch.stack(\n", " [\n", " torch.stack([torch.cos(pitch), tensor_0, torch.sin(pitch)]),\n", " torch.stack([tensor_0, tensor_1, tensor_0]),\n", " torch.stack([-torch.sin(pitch), tensor_0, torch.cos(pitch)]),\n", " ]\n", " ).reshape(3, 3)\n", "\n", " RZ = torch.stack(\n", " [\n", " torch.stack([torch.cos(yaw), -torch.sin(yaw), tensor_0]),\n", " torch.stack([torch.sin(yaw), torch.cos(yaw), tensor_0]),\n", " torch.stack([tensor_0, tensor_0, tensor_1]),\n", " ]\n", " ).reshape(3, 3)\n", "\n", " R = torch.mm(RZ, RY)\n", " R = torch.mm(R, RX)\n", " return R" ] }, { "cell_type": "code", "execution_count": 77, "id": "88b145d3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/sceneState/cameras\n" ] }, { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "/sceneState/cameras/000000\n", "/sceneState/cameras/000001\n" ] }, { "data": { "text/plain": [ "b'ok'" ] }, "execution_count": 77, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# delete any current cameras\n", "vis[f\"sceneState/cameras\"].delete()\n", "\n", "# draw a camera with with an image\n", "image = torch.rand(256, 256, 3)\n", "media.show_image(image)\n", "camera = PinholeCamera(cx=100, cy=100, fx=50, fy=50)\n", "camera_json = camera.to_json(image=image)\n", "vis[f\"sceneState/cameras/000000\"].write(camera_json)\n", "\n", "# draw a camera with rotation and translation (with a black image)\n", "image = torch.zeros(256, 256, 3)\n", "c2w = torch.eye(4)\n", "c2w[:3, :3] = get_random_rotation_matrix()\n", "random_translation = torch.rand(1) * 5\n", "c2w[:3, 3] = random_translation\n", "camera = PinholeCamera(cx=100, cy=100, fx=50, fy=50, camera_to_world=c2w)\n", "camera_json = camera.to_json(image=image)\n", "vis[f\"sceneState/cameras/000001\"].write(camera_json)" ] }, { "cell_type": "code", "execution_count": 78, "id": "18991f81", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/sceneState/cameras/000000\n" ] }, { "data": { "text/plain": [ "b'ok'" ] }, "execution_count": 78, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# delete a camera from the Nerfviewer\n", "vis[\"sceneState/cameras/000000\"].delete()" ] }, { "cell_type": "markdown", "id": "6b798e26", "metadata": {}, "source": [ "## Read from the GUI sliders" ] }, { "cell_type": "code", "execution_count": 79, "id": "31cd0de1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "80\n" ] } ], "source": [ "# read the field of view\n", "camera = vis[f\"renderingState/camera\"].read()\n", "fov = camera[\"object\"][\"fov\"]\n", "print(fov)" ] }, { "cell_type": "code", "execution_count": 86, "id": "feb83aaa", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/renderingState/field_of_view\n" ] }, { "data": { "text/plain": [ "b'ok'" ] }, "execution_count": 86, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vis[f\"renderingState/field_of_view\"].write(40)" ] }, { "cell_type": "code", "execution_count": 81, "id": "1abb93e5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "None\n" ] } ], "source": [ "fov = vis[f\"renderingState/camera/object/fov\"].read()\n", "print(fov)" ] }, { "cell_type": "code", "execution_count": 82, "id": "a0304523", "metadata": {}, "outputs": [], "source": [ "# set the field of view\n", "camera = vis[f\"renderingState/camera\"].read()\n", "fov = camera[\"object\"][\"fov\"]" ] }, { "cell_type": "code", "execution_count": 83, "id": "b3ba0ee7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/renderingState/minResolution\n" ] }, { "data": { "text/plain": [ "b'ok'" ] }, "execution_count": 83, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vis[f\"renderingState/minResolution\"].write(20)" ] }, { "cell_type": "code", "execution_count": 84, "id": "bba59227", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "20" ] }, "execution_count": 84, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# read from a slider\n", "vis[f\"renderingState/minResolution\"].read()" ] }, { "cell_type": "code", "execution_count": 87, "id": "fe922185", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'depth'" ] }, "execution_count": 87, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# read from output choice\n", "vis[\"renderingState/output_choice\"].read()" ] }, { "cell_type": "code", "execution_count": 85, "id": "75c37760", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['rgb', 'accumulation', 'depth', 'alive_ray_mask']\n", "None\n" ] } ], "source": [ "# read from a dropdown\n", "output_options = vis[f\"renderingState/output_options\"].read()\n", "print(output_options)\n", "output_choice = vis[f\"renderingState/output_choice\"].read()\n", "print(output_choice)" ] }, { "cell_type": "code", "execution_count": null, "id": "98aaa12c", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3.8.13 ('nerfactory')", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.13" }, "vscode": { "interpreter": { "hash": "100252972d4ee5286ab455ef1c778abde05b4affca9d1be78633d3797abf7e8c" } } }, "nbformat": 4, "nbformat_minor": 5 }