nerfstudio/docs/nerfology/model_components/visualize_cameras.ipynb

Camera Models

Everything starts with the cameras. Given a set of images, we need a way to project their pixels into 3D space. This is accomplished by computing the camera ray for each pixel given information about the type of camera and the location of the camera. This guide describes the camera models implemented in nerfstudio.

Each image should have an associated pose which consists of two properties, intrinsic and extrinsic parameters.

Intrinsics

All of the parameters internal to the camera such as lense or sensor properties.

Extrinsics

All of the parameters external to the camera such as the location and rotation relative to the world frame.

In [1]:

# HIDDEN
import plotly.graph_objects as go
import torch

Perspective Camera Model

The simplest and most used camera model. Most standard cameras can be sufficiently approximated using the pinhole camera model.

Intrinsic	Description
cx	Number of pixels in the x dimension
cy	Number of pixels in the y dimension
fx	Focal length in the x dimension
fy	Focal length in the y dimension

In [2]:

# COLLAPSED
from nerfstudio.cameras.cameras import Cameras, CameraType
from nerfstudio.utils import plotly_utils as vis

cx = 20.0
cy = 10.0
fx = 20.0
fy = 20.0

c2w = torch.eye(4)[None, :3, :]

camera = Cameras(fx=fx, fy=fy, cx=cx, cy=cy, camera_to_worlds=c2w, camera_type=CameraType.PERSPECTIVE)
fig = vis.vis_camera_rays(camera)
fig.show()

Fisheye Camera Model

The spacing of rays on a fisheye camera is more even on the edges of the image (most noticeable in wide angle images). This makes them desirable for NeRF reconstructions.

In [3]:

# COLLAPSED
from nerfstudio.cameras.cameras import Cameras, CameraType
from nerfstudio.utils import plotly_utils as vis

cx = 10.0
cy = 10.0
fx = 10.0
fy = 10.0

c2w = torch.eye(4)[None, :3, :]

camera = Cameras(fx=fx, fy=fy, cx=cx, cy=cy, camera_to_worlds=c2w, camera_type=CameraType.FISHEYE)
fig = vis.vis_camera_rays(camera)
fig.show()

Equirectangular/Spherical Camera Model

Rays are produced at all angles around the camera.

In [4]:

# COLLAPSED
from nerfstudio.cameras.cameras import Cameras, CameraType
from nerfstudio.utils import plotly_utils as vis

cx = 20.0  # = width/2
cy = 10.0  # = height/2
fx = 20.0  # = height = width/2
fy = 20.0  # = height = width/2

c2w = torch.eye(4)[None, :3, :]

camera = Cameras(fx=fx, fy=fy, cx=cx, cy=cy, camera_to_worlds=c2w, camera_type=CameraType.EQUIRECTANGULAR)
fig = vis.vis_camera_rays(camera)
fig.show()

Distortion Parameters

Unless you are using a pinhole camera (you probably aren't), modeling the distortion caused by the lenses may be beneficial. We represent the distortion with 6 parameters, [k_1, k_2, k_3, k_4] for radial distortion and [p_1, p_2] for tangential distortion.

In [5]:

# OUTPUT_ONLY
from plotly.subplots import make_subplots

height = 15
width = 15

distortion_names = ("k1", "k2", "p1", "k3", "k4", "p2")
distortion_min = (0.01, 0.001, 0.01, 0.001, -0.05, -0.05)
distortion_max = (0.05, 0.05, 0.05, 0.05, 0.05, 0.05)
fig = make_subplots(rows=2, cols=3, vertical_spacing=0.1, subplot_titles=distortion_names)

num_steps = 19
all_steps = []
for i in range(len(distortion_names)):
    for step in torch.linspace(distortion_min[i], distortion_max[i], num_steps):
        distortion_params = torch.zeros(6)
        distortion_params[i] = step

        from nerfstudio.cameras import camera_utils

        coords = torch.meshgrid(torch.linspace(-1, 1, height), torch.linspace(-1, 1, width), indexing="ij")
        coords = torch.stack(coords, dim=-1)

        coords = camera_utils.radial_and_tangential_undistort(coords, distortion_params)

        fig.add_trace(
            go.Scatter(
                x=coords[..., 0].flatten(),
                y=coords[..., 1].flatten(),
                mode="markers",
                marker=dict(
                    size=10,
                ),
                visible=False,
            ),
            row=i % 2 + 1,
            col=i // 2 + 1,
        )

    fig.data[num_steps * i + num_steps // 2].visible = True

# Create and add slider
steps = []
for i in range(num_steps):
    step = dict(method="update", args=[{"visible": [False] * len(fig.data)}], label="")
    for d in range(len(distortion_names)):
        step["args"][0]["visible"][num_steps * d + i] = True
    steps.append(step)

sliders = [
    dict(
        active=num_steps // 2,
        pad={"t": 10},
        steps=steps,
        len=0.5,
        x=0.5,
        xanchor="center",
    )
]

webdocs_layout = go.Layout(
    height=600,
    sliders=sliders,
    margin=dict(r=0, b=0, l=0, t=20),
    hovermode=False,
    showlegend=False,
    paper_bgcolor="rgba(0,0,0,0)",
    plot_bgcolor="rgba(0,0,0,0)",
    font=dict(color="#a3a3a3", size=18),
)
fig.for_each_yaxis(lambda x: x.update(range=[-1.3, 1.3], constrain="domain", showgrid=False, visible=False))
fig.for_each_xaxis(lambda x: x.update(scaleanchor="y", scaleratio=1, showgrid=False, visible=False))
fig.update_layout(webdocs_layout)

fig.show()