nerfstudio/docs/model_components/visualize_spatial_distortions.ipynb

Spatial Distortions

If you are trying to reconstruct and object floating in an empty void, you can stop reading. However if you are trying to reconstruct a scene or object from images, you may wish to consider adding a spatial distortion.

When rendering a target view of a scene, the camera will emit a camera ray for each pixel and query the scene at points along this ray. We can choose where to query these point using different samplers. These samplers have some notion of bounds that define where the ray should start and terminate. If you know that everything in your scenes exists within some predefined bounds (ie. a cube that a room fits in) then the sampler will properly sample the entire space. If however the scene is unbounded (ie. an outdoor scene) defining where to stop sampling is challenging. One option to increase the far sampling distance to a large value (ie. 1km). Alternatively we can warp the space into a fixed volume. Below are supported distortions.

(scene_contraction)=

Scene Contraction

Contract unbounded space into a sphere of radius 2. This contraction was proposed in MipNeRF-360. Samples within the unit sphere are not modified, whereas sample outside the unit sphere are contracted to fit within the sphere of radius 2.

We use the following contraction equation:

    f(x) = \begin{cases}
        x & ||x|| \leq 1 \\
        (2 - \frac{1}{||x||})(\frac{x}{||x||}) & ||x|| > 1
    \end{cases}

Below we visualize a ray before and after scene contraction. Visualized are 95% confidence intervals for the multivariate Gaussians for each sample location (this guide explains why the samples are represented by Gaussians). We are also visualizing both a unit sphere and a radius 2 sphere.

(before)=

Before Contraction

In [13]:

# HIDDEN
import torch
import plotly.graph_objects as go

torch.manual_seed(4)

webdocs_layout = go.Layout(
    scene=dict(
        aspectmode="data",
        xaxis=dict(showspikes=False),
        yaxis=dict(showspikes=False),
        zaxis=dict(showspikes=False),
        # xaxis_visible=False,
        # yaxis_visible=False,
        # zaxis_visible=False,
    ),
    scene_camera=dict(up=dict(x=0, y=1, z=0)),
    margin=dict(r=0, b=10, l=0, t=10),
    hovermode=False,
    showlegend=False,
    paper_bgcolor="rgba(0,0,0,0)",
)

In [14]:

# COLLAPSED
import nerfactory
from nerfactory.utils import plotly as vis
import torch
import plotly.graph_objects as go

num_rays = 4
num_samples = 15
pixel_area = 0.12
far = 5

origins = torch.rand((num_rays, 1, 3)) * 1 - 0.5
directions = torch.randn((num_rays, 1, 3))
bins = torch.linspace(0.1, far, num_samples + 1)[None, ..., None]
pixel_area = torch.tensor([pixel_area])[None, None, ...]

directions = directions / directions.norm(dim=-1, keepdim=True)

frustums = nerfactory.cameras.rays.Frustums(
    origins=origins, directions=directions, starts=bins[:, :-1, :], ends=bins[:, 1:, :], pixel_area=pixel_area
)

data = []
for i, frustum in enumerate(frustums):
    data += vis.get_gaussian_ellipsoids_list(frustum.get_gaussian_blob(), color=vis.get_random_color(idx=i))
data.append(vis.get_sphere(radius=1.0, color="#111111", opacity=0.05))
data.append(vis.get_sphere(radius=2.0, color="#111111", opacity=0.05))

fig = go.Figure(data=data, layout=webdocs_layout)
fig.show()

(after)=

After Contraction

In [15]:

# COLLAPSED
from nerfactory.fields.modules.spatial_distortions import SceneContraction

data = []
for i, frustum in enumerate(frustums):
    contracted_gaussian = SceneContraction()(frustum.get_gaussian_blob())
    data += vis.get_gaussian_ellipsoids_list(contracted_gaussian, color=vis.get_random_color(idx=i))

data.append(vis.get_sphere(radius=1.0, color="#111111", opacity=0.05))
data.append(vis.get_sphere(radius=2.0, color="#111111", opacity=0.05))

fig = go.Figure(data=data, layout=webdocs_layout)
fig.show()