nerfstudio/docs/tutorials/viewer/viewer_notebook.ipynb

Programming the viewer

The real-time web viewer enables communication between Python and the ReactJS Client Viewer Application. Here we provide a walkthrough for how to use the viewer to draw information as well as read information from the GUI.

In [72]:

# HIDDEN
%load_ext autoreload
%autoreload 2
import pprint
import torch
import numpy as np
import torch
import mediapy as media
from nerfstudio.data.scene_box import SceneBox
from nerfstudio.cameras.cameras import PinholeCamera
import time

The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

Open the Nerfviewer client

Open the Nerfviewer client by visiting http://nerfviewer.plenoptix.ai/ or running the code locally. We refer to this document for running the code locally. You should see a GUI like the following:

:align: center
:class: only-light
:width: 600

Start the Bridge Server

Run the following code in the repository home directory

python scripts/run_viewer_bridge_server.py

This will start the bridge server that both the Python code and Client Viewer App will connect to.

Verify that the Client Viewer App is connected.

In [73]:

# from IPython.display import HTML

# url="http://localhost:4000/?localhost:18051"
# height = 500

# HTML("""
# <div style="height: {height}px; width: 100%; overflow-x: auto; overflow-y: hidden; resize: both">
# <iframe src="{url}" style="width: 100%; height: 100%; border: none"></iframe>
# </div>
# """.format(url=url, height=height))

Communicate with the Client Viewer App from Python

Run the following code in the repository home directory

python scripts/run_viewer_bridge_server.py

This will start the bridge server that both the Python code and Client Viewer App will connect to.

Verify that the Client Viewer App is connected.

In [74]:

# connect to the viewer from Python
from nerfstudio.viewer.server.visualizer import Viewer

vis = Viewer(zmq_url="tcp://127.0.0.1:6000")

In [75]:

# draw scene box in the viewer
s = 1
scene_box = SceneBox(aabb=torch.tensor([[-s, -s, -s], [s, s, s]]))
scene_box_json = scene_box.to_json()

# draw scene box in the viewer
vis["sceneState/sceneBox"].write(scene_box_json)

# draw scene box in plotly
import plotly.graph_objects as go

# TODO:

/sceneState/sceneBounds

In [76]:

# HIDDEN
def get_random_rotation_matrix():
    roll = torch.randn(1)
    yaw = torch.randn(1)
    pitch = torch.randn(1)

    tensor_0 = torch.zeros(1)
    tensor_1 = torch.ones(1)

    RX = torch.stack(
        [
            torch.stack([tensor_1, tensor_0, tensor_0]),
            torch.stack([tensor_0, torch.cos(roll), -torch.sin(roll)]),
            torch.stack([tensor_0, torch.sin(roll), torch.cos(roll)]),
        ]
    ).reshape(3, 3)

    RY = torch.stack(
        [
            torch.stack([torch.cos(pitch), tensor_0, torch.sin(pitch)]),
            torch.stack([tensor_0, tensor_1, tensor_0]),
            torch.stack([-torch.sin(pitch), tensor_0, torch.cos(pitch)]),
        ]
    ).reshape(3, 3)

    RZ = torch.stack(
        [
            torch.stack([torch.cos(yaw), -torch.sin(yaw), tensor_0]),
            torch.stack([torch.sin(yaw), torch.cos(yaw), tensor_0]),
            torch.stack([tensor_0, tensor_0, tensor_1]),
        ]
    ).reshape(3, 3)

    R = torch.mm(RZ, RY)
    R = torch.mm(R, RX)
    return R

In [77]:

# delete any current cameras
vis[f"sceneState/cameras"].delete()

# draw a camera with with an image
image = torch.rand(256, 256, 3)
media.show_image(image)
camera = PinholeCamera(cx=100, cy=100, fx=50, fy=50)
camera_json = camera.to_json(image=image)
vis[f"sceneState/cameras/000000"].write(camera_json)

# draw a camera with rotation and translation (with a black image)
image = torch.zeros(256, 256, 3)
c2w = torch.eye(4)
c2w[:3, :3] = get_random_rotation_matrix()
random_translation = torch.rand(1) * 5
c2w[:3, 3] = random_translation
camera = PinholeCamera(cx=100, cy=100, fx=50, fy=50, camera_to_world=c2w)
camera_json = camera.to_json(image=image)
vis[f"sceneState/cameras/000001"].write(camera_json)

Out [77]:

/sceneState/cameras

/sceneState/cameras/000000
/sceneState/cameras/000001

b'ok'

In [78]:

# delete a camera from the Nerfviewer
vis["sceneState/cameras/000000"].delete()

Out [78]:

/sceneState/cameras/000000

b'ok'

Read from the GUI sliders

In [79]:

# read the field of view
camera = vis[f"renderingState/camera"].read()
fov = camera["object"]["fov"]
print(fov)

80

In [86]:

vis[f"renderingState/field_of_view"].write(40)

Out [86]:

/renderingState/field_of_view

b'ok'

In [81]:

fov = vis[f"renderingState/camera/object/fov"].read()
print(fov)

None

In [82]:

# set the field of view
camera = vis[f"renderingState/camera"].read()
fov = camera["object"]["fov"]

In [83]:

vis[f"renderingState/minResolution"].write(20)

Out [83]:

/renderingState/minResolution

b'ok'

In [84]:

# read from a slider
vis[f"renderingState/minResolution"].read()

Out [84]:

20

In [87]:

# read from output choice
vis["renderingState/output_choice"].read()

Out [87]:

'depth'

In [85]:

# read from a dropdown
output_options = vis[f"renderingState/output_options"].read()
print(output_options)
output_choice = vis[f"renderingState/output_choice"].read()
print(output_choice)

['rgb', 'accumulation', 'depth', 'alive_ray_mask']
None

In [ ]: