diff --git a/docs/developer_guides/pipelines/dataparsers.md b/docs/developer_guides/pipelines/dataparsers.md
index 6b81803c2..3a38664f9 100644
--- a/docs/developer_guides/pipelines/dataparsers.md
+++ b/docs/developer_guides/pipelines/dataparsers.md
@@ -67,10 +67,16 @@ class NerfstudioDataParserConfig(DataParserConfig):
"""How much to downscale images. If not set, images are chosen such that the max dimension is <1600px."""
scene_scale: float = 1.0
"""How much to scale the region of interest by."""
- orientation_method: Literal["pca", "up"] = "up"
+ orientation_method: Literal["pca", "up", "vertical", "none"] = "up"
"""The method to use for orientation."""
+ center_method: Literal["poses", "focus", "none"] = "poses"
+ """The method to use to center the poses."""
+ auto_scale_poses: bool = True
+ """Whether to automatically scale the poses to fit in +/- 1 bounding box."""
train_split_fraction: float = 0.9
"""The fraction of images to use for training. The remaining images are for eval."""
+ depth_unit_scale_factor: float = 1e-3
+ """Scales the depth values to meters. Default value is 0.001 for a millimeter to meter conversion."""
@dataclass
class Nerfstudio(DataParser):
diff --git a/docs/developer_guides/viewer/imgs/viewer_dark.png b/docs/developer_guides/viewer/imgs/viewer_dark.png
new file mode 100644
index 000000000..d2abe9a10
Binary files /dev/null and b/docs/developer_guides/viewer/imgs/viewer_dark.png differ
diff --git a/docs/developer_guides/viewer/imgs/viewer_light.png b/docs/developer_guides/viewer/imgs/viewer_light.png
new file mode 100644
index 000000000..2220e3a8c
Binary files /dev/null and b/docs/developer_guides/viewer/imgs/viewer_light.png differ
diff --git a/docs/developer_guides/viewer/viewer_overview.md b/docs/developer_guides/viewer/viewer_overview.md
index 4e10319b6..55dbab2f2 100644
--- a/docs/developer_guides/viewer/viewer_overview.md
+++ b/docs/developer_guides/viewer/viewer_overview.md
@@ -4,7 +4,15 @@
The viewer is built using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine. The following figure helps to illustrate how our viewer framework works:
-
+```{image} imgs/viewer_light.png
+:align: center
+:class: only-light
+```
+
+```{image} imgs/viewer_dark.png
+:align: center
+:class: only-dark
+```
## System design
@@ -14,12 +22,10 @@ In the center, we have the Bridge Server, which facilitates the connection betwe
#### Connection between the Bridge Server and the Client App
-The connection between the Bridge Server and the Client App works with WebSockets and WebRTC.
+The connection between the Bridge Server and the Client App works with WebSockets.
- **WebSocket connection** - The WebSocket is used by the Bridge Server to dispatch commands coming from the nerfstudio TCP connection. Commands can be used for drawing primitives, for setting the transform of objects, for the setting various properties, and more.
-- **WebRTC connection** - We use WebRTC to stream images being rendered from nerfstudio. The websocket connection if first used to establish the WebRTC connection. Then, the Client App constantly publishes camera pose information to the Bridge Server and stores the camera information (intrinsics and extrinsics). This information is then queried from the nerfstudio code, used to render an image with some Graph, and then the image is send over the TCP connection and dispatched via WebRTC to render the stream of images.
-
## Installing and running locally
#### Running the Client App
@@ -50,7 +56,7 @@ yarn install
yarn start
```
-The local webserver runs on port 4000 by default,
+The local webserver runs on port 4000 by default,
so when `ns-train` is running, you can connect to the viewer locally at
[http://localhost:4000/?websocket_url=ws://localhost:7007](http://localhost:4000/?websocket_url=ws://localhost:7007)
@@ -70,7 +76,6 @@ We thank the authors and contributors to the following repos, which we've starte
- [meshcat](https://github.com/rdeits/meshcat) - made by [Robin Deits](https://github.com/rdeits)
- [ThreeJS](https://threejs.org/)
- [ReactJS](https://reactjs.org/)
-- [WebRTC](https://webrtc.org/) - WebRTC is a framework for real-time communication that allows two peers to send video, audio, or general data to and from each other with low latency. We've adopted WebRTC to stream rendered images to our viewer.
## FAQ
diff --git a/docs/nerfology/methods/nerf.md b/docs/nerfology/methods/nerf.md
index 90e89cd59..4c56f3154 100644
--- a/docs/nerfology/methods/nerf.md
+++ b/docs/nerfology/methods/nerf.md
@@ -18,7 +18,7 @@ ns-train vanilla-nerf
If you have arrived to this site, it is likely that you have at least heard of NeRFs. This page will discuss the original NeRF paper, _"NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis"_ by Mildenhall, Srinivasan, Tancik et al. (2020).
-For most tasks, using the original NeRF model is likely not a good choice and hence we provide implementations of various other NeRF related models. It is however useful to understand how NeRF's work as most follow ups follow a similar structure.
+For most tasks, using the original NeRF model is likely not a good choice and hence we provide implementations of various other NeRF related models. It is however useful to understand how NeRF's work as most follow ups follow a similar structure and it doesn't require CUDA to execute (useful for stepping through the code with a debugger if you don't have a GPU at hand).
The goal is to optimize a volumetric representation of a scene that can be rendered from novel viewpoints. This representation is optimized from a set of images and associated camera poses.
diff --git a/docs/nerfology/model_components/visualize_samplers.ipynb b/docs/nerfology/model_components/visualize_samplers.ipynb
index 1ead09832..a23500d58 100644
--- a/docs/nerfology/model_components/visualize_samplers.ipynb
+++ b/docs/nerfology/model_components/visualize_samplers.ipynb
@@ -1,6 +1,7 @@
{
"cells": [
{
+ "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@@ -46,7 +47,7 @@
":class: only-dark\n",
"```\n",
"\n",
- "### Heirarchical Sampling\n",
+ "### Hierarchical Sampling\n",
"\n",
"It is important to sample the scene where it has content otherwise the reconstruction quality will be reduced. \n",
"\n",
@@ -7413,7 +7414,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3.8.12 ('nerfactory')",
+ "display_name": "Python 3",
"language": "python",
"name": "python3"
},
@@ -7427,11 +7428,11 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.8.12"
+ "version": "3.10.7"
},
"vscode": {
"interpreter": {
- "hash": "34c28001ff35fb390494047002768a8182dcf55b1b11415165e62ea61557ab83"
+ "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
}
}
},
diff --git a/docs/quickstart/custom_dataset.md b/docs/quickstart/custom_dataset.md
index ebc6ce313..009cb287c 100644
--- a/docs/quickstart/custom_dataset.md
+++ b/docs/quickstart/custom_dataset.md
@@ -290,3 +290,35 @@ ns-process-data metashape --data {data directory} --xml {xml file} --output-dir
```bash
ns-train nerfacto --data {output directory}
```
+
+## Equirectangular data
+
+Equirectangular data is data that has been taken by a 360 camera such as Insta360. Both equirectangular image sets and videos can be processed by nerfstudio.
+
+### Images
+For a set of equirectangular images, process the data using the following command:
+
+```bash
+ns-process-data images --camera-type equirectangular --images-per-equirect {8, or 14} --data {data directory} --output-dir {output directory}
+```
+
+The images-per-equirect argument is the number of images that will be sampled from each equirectangular image. We have found that 8 images per equirectangular image is sufficient for most use cases so it defaults to that. However, if you find that there isn't enough detail in the nerf or that colmap is having trouble aligning the images, you can try increasing the number of images per equirectangular image to 14.
+
+### Videos
+For videos we reccomend taking a video with the camera held on top of your head. This will result in any unwanted capturer to just be in the bottom of each frame image and therefore can be cropped out.
+
+For a video, process the data using the following command:
+
+```bash
+ns-process-data video --camera-type equirectangular --images-per-equirect {8, or 14} --num-frames-target {num equirectangular frames to sample from} --crop-factor {top bottom left right} --data {data directory} --output-dir {output directory}
+```
+See the equirectangular images section above for a description of the `--images-per-equirect` argument.
+
+The `num-frames-target` argument is optional but it is recomended to set it to 3*(seconds of video) frames. For example, if you have a 30 second video, you would use `--num-frames-target 90` (3*30=90). This number was chosen from a bit of experimentation and seems to work well for most videos. It is by no means a hard rule and you can experiment with different values.
+
+The `crop-factor` argument is optional but often very helpful. This is because equirectangular videos taken by 360 cameras tend to have a portion of the bottom of the image that is the person who was holding the camera over their head.
+
+
+
+This obscene mesh of human is obviously not helpful in training a nerf so we can remove it by cropping the bottom 20% of the image. This can be done by using the `--crop-factor 0 0.2 0 0` argument.
+
diff --git a/docs/quickstart/imgs/equirect_crop.jpg b/docs/quickstart/imgs/equirect_crop.jpg
new file mode 100644
index 000000000..584dab831
Binary files /dev/null and b/docs/quickstart/imgs/equirect_crop.jpg differ
diff --git a/docs/quickstart/installation.md b/docs/quickstart/installation.md
index 2a2db4453..950aefb2a 100644
--- a/docs/quickstart/installation.md
+++ b/docs/quickstart/installation.md
@@ -140,7 +140,7 @@ To restrict to only CUDA architectures that you have available locally, use the
build arg and look up [the compute capability for your GPU](https://developer.nvidia.com/cuda-gpus).
For example, here's how to build with support for GeForce 30xx series GPUs:
```bash
-docker build --build-arg CUDA_ARCHITECTURES=86 -tag nerfstudio-86 .
+docker build --build-arg CUDA_ARCHITECTURES=86 --tag nerfstudio-86 -f Dockerfile .
```
### Using an interactive container
diff --git a/nerfstudio/cameras/camera_utils.py b/nerfstudio/cameras/camera_utils.py
index b1dd5a267..d2da627b7 100644
--- a/nerfstudio/cameras/camera_utils.py
+++ b/nerfstudio/cameras/camera_utils.py
@@ -229,7 +229,7 @@ def get_interpolated_poses_many(
traj = np.stack(traj, axis=0)
k_interp = np.stack(k_interp, axis=0)
- return torch.tensor(traj), torch.tensor(k_interp)
+ return torch.tensor(traj, dtype=torch.float32), torch.tensor(k_interp, dtype=torch.float32)
def normalize(x: torch.Tensor) -> TensorType[...]:
@@ -429,35 +429,89 @@ def rotation_matrix(a: TensorType[3], b: TensorType[3]) -> TensorType[3, 3]:
return torch.eye(3) + skew_sym_mat + skew_sym_mat @ skew_sym_mat * ((1 - c) / (s**2 + 1e-8))
+def focus_of_attention(poses: TensorType["num_poses":..., 4, 4], initial_focus: TensorType[3]) -> TensorType[3]:
+ """Compute the focus of attention of a set of cameras. Only cameras
+ that have the focus of attention in front of them are considered.
+
+ Args:
+ poses: The poses to orient.
+ initial_focus: The 3D point views to decide which cameras are initially activated.
+
+ Returns:
+ The 3D position of the focus of attention.
+ """
+ # References to the same method in third-party code:
+ # https://github.com/google-research/multinerf/blob/1c8b1c552133cdb2de1c1f3c871b2813f6662265/internal/camera_utils.py#L145
+ # https://github.com/bmild/nerf/blob/18b8aebda6700ed659cb27a0c348b737a5f6ab60/load_llff.py#L197
+ active_directions = -poses[:, :3, 2:3]
+ active_origins = poses[:, :3, 3:4]
+ # initial value for testing if the focus_pt is in front or behind
+ focus_pt = initial_focus
+ # Prune cameras which have the current have the focus_pt behind them.
+ active = torch.sum(active_directions.squeeze(-1) * (focus_pt - active_origins.squeeze(-1)), dim=-1) > 0
+ done = False
+ # We need at least two active cameras, else fallback on the previous solution.
+ # This may be the "poses" solution if no cameras are active on first iteration, e.g.
+ # they are in an outward-looking configuration.
+ while torch.sum(active.int()) > 1 and not done:
+ active_directions = active_directions[active]
+ active_origins = active_origins[active]
+ # https://en.wikipedia.org/wiki/Line–line_intersection#In_more_than_two_dimensions
+ m = torch.eye(3) - active_directions * torch.transpose(active_directions, -2, -1)
+ mt_m = torch.transpose(m, -2, -1) @ m
+ focus_pt = torch.linalg.inv(mt_m.mean(0)) @ (mt_m @ active_origins).mean(0)[:, 0]
+ active = torch.sum(active_directions.squeeze(-1) * (focus_pt - active_origins.squeeze(-1)), dim=-1) > 0
+ if active.all():
+ # the set of active cameras did not change, so we're done.
+ done = True
+ return focus_pt
+
+
def auto_orient_and_center_poses(
- poses: TensorType["num_poses":..., 4, 4], method: Literal["pca", "up", "none"] = "up", center_poses: bool = True
+ poses: TensorType["num_poses":..., 4, 4],
+ method: Literal["pca", "up", "vertical", "none"] = "up",
+ center_method: Literal["poses", "focus", "none"] = "poses",
) -> Tuple[TensorType["num_poses":..., 3, 4], TensorType[4, 4]]:
"""Orients and centers the poses. We provide two methods for orientation: pca and up.
- pca: Orient the poses so that the principal component of the points is aligned with the axes.
- This method works well when all of the cameras are in the same plane.
+ pca: Orient the poses so that the principal directions of the camera centers are aligned
+ with the axes, Z corresponding to the smallest principal component.
+ This method works well when all of the cameras are in the same plane, for example when
+ images are taken using a mobile robot.
up: Orient the poses so that the average up vector is aligned with the z axis.
This method works well when images are not at arbitrary angles.
+ vertical: Orient the poses so that the Z 3D direction projects close to the
+ y axis in images. This method works better if cameras are not all
+ looking in the same 3D direction, which may happen in camera arrays or in LLFF.
+ There are two centering methods:
+ poses: The poses are centered around the origin.
+ focus: The origin is set to the focus of attention of all cameras (the
+ closest point to cameras optical axes). Recommended for inward-looking
+ camera configurations.
Args:
poses: The poses to orient.
method: The method to use for orientation.
- center_poses: If True, the poses are centered around the origin.
+ center_method: The method to use to center the poses.
Returns:
Tuple of the oriented poses and the transform matrix.
"""
- translation = poses[..., :3, 3]
+ origins = poses[..., :3, 3]
- mean_translation = torch.mean(translation, dim=0)
- translation_diff = translation - mean_translation
+ mean_origin = torch.mean(origins, dim=0)
+ translation_diff = origins - mean_origin
- if center_poses:
- translation = mean_translation
+ if center_method == "poses":
+ translation = mean_origin
+ elif center_method == "focus":
+ translation = focus_of_attention(poses, mean_origin)
+ elif center_method == "none":
+ translation = torch.zeros_like(mean_origin)
else:
- translation = torch.zeros_like(mean_translation)
+ raise ValueError(f"Unknown value for center_method: {center_method}")
if method == "pca":
_, eigvec = torch.linalg.eigh(translation_diff.T @ translation_diff)
@@ -471,9 +525,41 @@ def auto_orient_and_center_poses(
if oriented_poses.mean(axis=0)[2, 1] < 0:
oriented_poses[:, 1:3] = -1 * oriented_poses[:, 1:3]
- elif method == "up":
+ elif method in ("up", "vertical"):
up = torch.mean(poses[:, :3, 1], dim=0)
up = up / torch.linalg.norm(up)
+ if method == "vertical":
+ # If cameras are not all parallel (e.g. not in an LLFF configuration),
+ # we can find the 3D direction that most projects vertically in all
+ # cameras by minimizing ||Xu|| s.t. ||u||=1. This total least squares
+ # problem is solved by SVD.
+ x_axis_matrix = poses[:, :3, 0]
+ _, S, Vh = torch.linalg.svd(x_axis_matrix, full_matrices=False)
+ # Singular values are S_i=||Xv_i|| for each right singular vector v_i.
+ # ||S|| = sqrt(n) because lines of X are all unit vectors and the v_i
+ # are an orthonormal basis.
+ # ||Xv_i|| = sqrt(sum(dot(x_axis_j,v_i)^2)), thus S_i/sqrt(n) is the
+ # RMS of cosines between x axes and v_i. If the second smallest singular
+ # value corresponds to an angle error less than 10° (cos(80°)=0.17),
+ # this is probably a degenerate camera configuration (typical values
+ # are around 5° average error for the true vertical). In this case,
+ # rather than taking the vector corresponding to the smallest singular
+ # value, we project the "up" vector on the plane spanned by the two
+ # best singular vectors. We could also just fallback to the "up"
+ # solution.
+ if S[1] > 0.17 * math.sqrt(poses.shape[0]):
+ # regular non-degenerate configuration
+ up_vertical = Vh[2, :]
+ # It may be pointing up or down. Use "up" to disambiguate the sign.
+ up = up_vertical if torch.dot(up_vertical, up) > 0 else -up_vertical
+ else:
+ # Degenerate configuration: project "up" on the plane spanned by
+ # the last two right singular vectors (which are orthogonal to the
+ # first). v_0 is a unit vector, no need to divide by its norm when
+ # projecting.
+ up = up - Vh[0, :] * torch.dot(up, Vh[0, :])
+ # re-normalize
+ up = up / torch.linalg.norm(up)
rotation = rotation_matrix(up, torch.Tensor([0, 0, 1]))
transform = torch.cat([rotation, rotation @ -translation[..., None]], dim=-1)
@@ -483,5 +569,7 @@ def auto_orient_and_center_poses(
transform[:3, 3] = -translation
transform = transform[:3, :]
oriented_poses = transform @ poses
+ else:
+ raise ValueError(f"Unknown value for method: {method}")
return oriented_poses, transform
diff --git a/nerfstudio/configs/base_config.py b/nerfstudio/configs/base_config.py
index a2e695c14..9d95cc269 100644
--- a/nerfstudio/configs/base_config.py
+++ b/nerfstudio/configs/base_config.py
@@ -23,8 +23,6 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Any, List, Optional, Tuple, Type
-from typing_extensions import Literal
-
# model instances
from nerfstudio.utils import writer
@@ -148,10 +146,3 @@ class ViewerConfig(PrintableConfig):
actually used in training/evaluation. If -1, display all."""
quit_on_train_completion: bool = False
"""Whether to kill the training job when it has completed. Note this will stop rendering in the viewer."""
- skip_openrelay: bool = False
- """Avoid using openrelay to communicate with the viewer. Try disabling if you have trouble
- connecting to the viewer"""
- codec: Literal["H264", "VP8"] = "VP8"
- """Video codec that viewer will use."""
- local: bool = False
- """If running local server instance, avoid using relays to communicate with the viewer."""
diff --git a/nerfstudio/data/datamanagers/base_datamanager.py b/nerfstudio/data/datamanagers/base_datamanager.py
index dd0109c1d..7ff2f9aaf 100644
--- a/nerfstudio/data/datamanagers/base_datamanager.py
+++ b/nerfstudio/data/datamanagers/base_datamanager.py
@@ -235,24 +235,39 @@ class DataManager(nn.Module):
"""Sets up the data manager for evaluation"""
@abstractmethod
- def next_train(self, step: int) -> Tuple:
+ def next_train(self, step: int) -> Tuple[RayBundle, Dict]:
"""Returns the next batch of data from the train data manager.
- This will be a tuple of all the information that this data manager outputs.
+ Args:
+ step: the step number of the eval image to retrieve
+ Returns:
+ A tuple of the ray bundle for the image, and a dictionary of additional batch information
+ such as the groudtruth image.
"""
raise NotImplementedError
@abstractmethod
- def next_eval(self, step: int) -> Tuple:
+ def next_eval(self, step: int) -> Tuple[RayBundle, Dict]:
"""Returns the next batch of data from the eval data manager.
- This will be a tuple of all the information that this data manager outputs.
+ Args:
+ step: the step number of the eval image to retrieve
+ Returns:
+ A tuple of the ray bundle for the image, and a dictionary of additional batch information
+ such as the groudtruth image.
"""
raise NotImplementedError
@abstractmethod
- def next_eval_image(self, step: int) -> Tuple:
- """Returns the next eval image."""
+ def next_eval_image(self, step: int) -> Tuple[int, RayBundle, Dict]:
+ """Retreive the next eval image.
+
+ Args:
+ step: the step number of the eval image to retrieve
+ Returns:
+ A tuple of the step number, the ray bundle for the image, and a dictionary of
+ additional batch information such as the groudtruth image.
+ """
raise NotImplementedError
@abstractmethod
@@ -358,12 +373,12 @@ class VanillaDataManager(DataManager): # pylint: disable=abstract-method
self.sampler = None
self.test_mode = test_mode
self.test_split = "test" if test_mode in ["test", "inference"] else "val"
- self.dataparser = self.config.dataparser
+ self.dataparser_config = self.config.dataparser
if self.config.data is not None:
self.config.dataparser.data = Path(self.config.data)
else:
self.config.data = self.config.dataparser.data
- self.dataparser = self.dataparser.setup()
+ self.dataparser = self.dataparser_config.setup()
self.train_dataparser_outputs = self.dataparser.get_dataparser_outputs(split="train")
self.train_dataset = self.create_train_dataset()
@@ -453,7 +468,6 @@ class VanillaDataManager(DataManager): # pylint: disable=abstract-method
)
self.eval_dataloader = RandIndicesEvalDataloader(
input_dataset=self.eval_dataset,
- image_indices=self.config.eval_image_indices,
device=self.device,
num_workers=self.world_size * 4,
)
diff --git a/nerfstudio/data/dataparsers/arkitscenes_dataparser.py b/nerfstudio/data/dataparsers/arkitscenes_dataparser.py
index 736f9941b..a92745bc6 100644
--- a/nerfstudio/data/dataparsers/arkitscenes_dataparser.py
+++ b/nerfstudio/data/dataparsers/arkitscenes_dataparser.py
@@ -21,7 +21,9 @@ from typing import Type
import cv2
import numpy as np
import torch
+from typing_extensions import Literal
+from nerfstudio.cameras import camera_utils
from nerfstudio.cameras.cameras import Cameras, CameraType
from nerfstudio.data.dataparsers.base_dataparser import (
DataParser,
@@ -70,11 +72,13 @@ class ARKitScenesDataParserConfig(DataParserConfig):
"""target class to instantiate"""
data: Path = Path("data/ARKitScenes/3dod/Validation/41069021")
"""Path to ARKitScenes folder with densely extracted scenes."""
+ scale_factor: float = 1.0
+ """How much to scale the camera origins by."""
scene_scale: float = 1.0
"""How much to scale the region of interest by."""
- center_poses: bool = True
- """Whether to center the poses."""
- scale_poses: bool = True
+ center_method: Literal["poses", "focus", "none"] = "poses"
+ """The method to use to center the poses."""
+ auto_scale_poses: bool = True
"""Whether to automatically scale the poses to fit in +/- 1 bounding box."""
train_split_fraction: float = 0.9
"""The fraction of images to use for training. The remaining images are for eval."""
@@ -141,11 +145,19 @@ class ARKitScenes(DataParser):
poses = torch.from_numpy(np.stack(poses).astype(np.float32))
intrinsics = torch.from_numpy(np.stack(intrinsics).astype(np.float32))
- if self.config.center_poses:
- poses[:, :3, 3] -= poses[:, :3, 3].mean(dim=0)
+ poses, transform_matrix = camera_utils.auto_orient_and_center_poses(
+ poses,
+ method="none",
+ center_method=self.config.center_method,
+ )
- if self.config.scale_poses:
- poses[:, :3, 3] /= poses[:, :3, 3].abs().max()
+ # Scale poses
+ scale_factor = 1.0
+ if self.config.auto_scale_poses:
+ scale_factor /= float(torch.max(torch.abs(poses[:, :3, 3])))
+ scale_factor *= self.config.scale_factor
+
+ poses[:, :3, 3] *= scale_factor
# Choose image_filenames and poses based on split, but after auto orient and scaling the poses.
image_filenames = [image_filenames[i] for i in indices]
@@ -177,6 +189,8 @@ class ARKitScenes(DataParser):
image_filenames=image_filenames,
cameras=cameras,
scene_box=scene_box,
+ dataparser_scale=scale_factor,
+ dataparser_transform=transform_matrix,
metadata={
"depth_filenames": depth_filenames if len(depth_filenames) > 0 else None,
"depth_unit_scale_factor": self.config.depth_unit_scale_factor,
@@ -208,4 +222,6 @@ class ARKitScenes(DataParser):
frame_pose = np.array(poses_from_traj[str(my_key)])
frame_pose[0:3, 1:3] *= -1
+ frame_pose = frame_pose[np.array([1, 0, 2, 3]), :]
+ frame_pose[2, :] *= -1
return frame_pose
diff --git a/nerfstudio/data/dataparsers/nerfstudio_dataparser.py b/nerfstudio/data/dataparsers/nerfstudio_dataparser.py
index 035e2da2d..adb09eb48 100644
--- a/nerfstudio/data/dataparsers/nerfstudio_dataparser.py
+++ b/nerfstudio/data/dataparsers/nerfstudio_dataparser.py
@@ -54,10 +54,10 @@ class NerfstudioDataParserConfig(DataParserConfig):
"""How much to downscale images. If not set, images are chosen such that the max dimension is <1600px."""
scene_scale: float = 1.0
"""How much to scale the region of interest by."""
- orientation_method: Literal["pca", "up", "none"] = "up"
+ orientation_method: Literal["pca", "up", "vertical", "none"] = "up"
"""The method to use for orientation."""
- center_poses: bool = True
- """Whether to center the poses."""
+ center_method: Literal["poses", "focus", "none"] = "poses"
+ """The method to use to center the poses."""
auto_scale_poses: bool = True
"""Whether to automatically scale the poses to fit in +/- 1 bounding box."""
train_split_fraction: float = 0.9
@@ -209,7 +209,7 @@ class Nerfstudio(DataParser):
poses, transform_matrix = camera_utils.auto_orient_and_center_poses(
poses,
method=orientation_method,
- center_poses=self.config.center_poses,
+ center_method=self.config.center_method,
)
# Scale poses
diff --git a/nerfstudio/data/dataparsers/phototourism_dataparser.py b/nerfstudio/data/dataparsers/phototourism_dataparser.py
index 5b5287d3d..e5a88e10e 100644
--- a/nerfstudio/data/dataparsers/phototourism_dataparser.py
+++ b/nerfstudio/data/dataparsers/phototourism_dataparser.py
@@ -60,12 +60,12 @@ class PhototourismDataParserConfig(DataParserConfig):
"""The fraction of images to use for training. The remaining images are for eval."""
scene_scale: float = 1.0
"""How much to scale the region of interest by."""
- orientation_method: Literal["pca", "up", "none"] = "up"
+ orientation_method: Literal["pca", "up", "vertical", "none"] = "up"
"""The method to use for orientation."""
+ center_method: Literal["poses", "focus", "none"] = "poses"
+ """The method to use to center the poses."""
auto_scale_poses: bool = True
"""Whether to automatically scale the poses to fit in +/- 1 bounding box."""
- center_poses: bool = True
- """Whether to center the poses."""
@dataclass
@@ -147,7 +147,7 @@ class Phototourism(DataParser):
raise ValueError(f"Unknown dataparser split {split}")
poses, transform_matrix = camera_utils.auto_orient_and_center_poses(
- poses, method=self.config.orientation_method, center_poses=self.config.center_poses
+ poses, method=self.config.orientation_method, center_method=self.config.center_method
)
# Scale poses
diff --git a/nerfstudio/data/dataparsers/scannet_dataparser.py b/nerfstudio/data/dataparsers/scannet_dataparser.py
index 80d0dd688..d5a787799 100644
--- a/nerfstudio/data/dataparsers/scannet_dataparser.py
+++ b/nerfstudio/data/dataparsers/scannet_dataparser.py
@@ -21,7 +21,9 @@ from typing import Type
import cv2
import numpy as np
import torch
+from typing_extensions import Literal
+from nerfstudio.cameras import camera_utils
from nerfstudio.cameras.cameras import Cameras, CameraType
from nerfstudio.data.dataparsers.base_dataparser import (
DataParser,
@@ -47,11 +49,13 @@ class ScanNetDataParserConfig(DataParserConfig):
"""target class to instantiate"""
data: Path = Path("data/scannet/scene0423_02")
"""Path to ScanNet folder with densely extracted scenes."""
+ scale_factor: float = 1.0
+ """How much to scale the camera origins by."""
scene_scale: float = 1.0
"""How much to scale the region of interest by."""
- center_poses: bool = True
- """Whether to center the poses."""
- scale_poses: bool = True
+ center_method: Literal["poses", "focus", "none"] = "poses"
+ """The method to use to center the poses."""
+ auto_scale_poses: bool = True
"""Whether to automatically scale the poses to fit in +/- 1 bounding box."""
train_split_fraction: float = 0.9
"""The fraction of images to use for training. The remaining images are for eval."""
@@ -77,15 +81,14 @@ class ScanNet(DataParser):
first_img = cv2.imread(img_dir_sorted[0].as_posix())
h, w, _ = first_img.shape
- image_filenames = []
- depth_filenames = []
- intrinsics = []
- poses = []
+ image_filenames, depth_filenames, intrinsics, poses = [], [], [], []
K = np.loadtxt(self.config.data / "intrinsic" / "intrinsic_color.txt")
for img, depth, pose in zip(img_dir_sorted, depth_dir_sorted, pose_dir_sorted):
pose = np.loadtxt(pose)
pose[0:3, 1:3] *= -1
+ pose = pose[np.array([1, 0, 2, 3]), :]
+ pose[2, :] *= -1
# We cannot accept files directly, as some of the poses are invalid
if np.isinf(pose).any():
@@ -116,11 +119,19 @@ class ScanNet(DataParser):
poses = torch.from_numpy(np.stack(poses).astype(np.float32))
intrinsics = torch.from_numpy(np.stack(intrinsics).astype(np.float32))
- if self.config.center_poses:
- poses[:, :3, 3] -= poses[:, :3, 3].mean(dim=0)
+ poses, transform_matrix = camera_utils.auto_orient_and_center_poses(
+ poses,
+ method="none",
+ center_method=self.config.center_method,
+ )
- if self.config.scale_poses:
- poses[:, :3, 3] /= poses[:, :3, 3].abs().max()
+ # Scale poses
+ scale_factor = 1.0
+ if self.config.auto_scale_poses:
+ scale_factor /= float(torch.max(torch.abs(poses[:, :3, 3])))
+ scale_factor *= self.config.scale_factor
+
+ poses[:, :3, 3] *= scale_factor
# Choose image_filenames and poses based on split, but after auto orient and scaling the poses.
image_filenames = [image_filenames[i] for i in indices]
@@ -152,6 +163,8 @@ class ScanNet(DataParser):
image_filenames=image_filenames,
cameras=cameras,
scene_box=scene_box,
+ dataparser_scale=scale_factor,
+ dataparser_transform=transform_matrix,
metadata={
"depth_filenames": depth_filenames if len(depth_filenames) > 0 else None,
"depth_unit_scale_factor": self.config.depth_unit_scale_factor,
diff --git a/nerfstudio/data/dataparsers/sdfstudio_dataparser.py b/nerfstudio/data/dataparsers/sdfstudio_dataparser.py
index f25c8f3e7..43781a00b 100644
--- a/nerfstudio/data/dataparsers/sdfstudio_dataparser.py
+++ b/nerfstudio/data/dataparsers/sdfstudio_dataparser.py
@@ -116,7 +116,7 @@ class SDFStudio(DataParser):
camera_to_worlds, transform = camera_utils.auto_orient_and_center_poses(
camera_to_worlds,
method="up",
- center_poses=False,
+ center_method="none",
)
# scene box from meta data
diff --git a/nerfstudio/data/pixel_samplers.py b/nerfstudio/data/pixel_samplers.py
index 8ac8e183d..244498df8 100644
--- a/nerfstudio/data/pixel_samplers.py
+++ b/nerfstudio/data/pixel_samplers.py
@@ -261,6 +261,14 @@ class PatchPixelSampler(PixelSampler): # pylint: disable=too-few-public-methods
num_rays = (num_rays_per_batch // (self.patch_size**2)) * (self.patch_size**2)
super().__init__(num_rays, keep_full_image, **kwargs)
+ def set_num_rays_per_batch(self, num_rays_per_batch: int):
+ """Set the number of rays to sample per batch. Overrided to deal with patch-based sampling.
+
+ Args:
+ num_rays_per_batch: number of rays to sample per batch
+ """
+ self.num_rays_per_batch = (num_rays_per_batch // (self.patch_size**2)) * (self.patch_size**2)
+
# overrides base method
def sample_method( # pylint: disable=no-self-use
self,
diff --git a/nerfstudio/data/utils/dataloaders.py b/nerfstudio/data/utils/dataloaders.py
index c4e155102..23d712d7e 100644
--- a/nerfstudio/data/utils/dataloaders.py
+++ b/nerfstudio/data/utils/dataloaders.py
@@ -116,7 +116,7 @@ class CacheDataloader(DataLoader):
"""Returns a collated batch."""
batch_list = self._get_batch_list()
collated_batch = self.collate_fn(batch_list)
- collated_batch = get_dict_to_torch(collated_batch, device=self.device)
+ collated_batch = get_dict_to_torch(collated_batch, device=self.device, exclude=["image"])
return collated_batch
def __iter__(self):
@@ -183,7 +183,7 @@ class EvalDataloader(DataLoader):
"""
ray_bundle = self.cameras.generate_rays(camera_indices=image_idx, keep_shape=True)
batch = self.input_dataset[image_idx]
- batch = get_dict_to_torch(batch, device=self.device)
+ batch = get_dict_to_torch(batch, device=self.device, exclude=["image"])
return ray_bundle, batch
diff --git a/nerfstudio/engine/trainer.py b/nerfstudio/engine/trainer.py
index a4a09e7b1..97ff9244a 100644
--- a/nerfstudio/engine/trainer.py
+++ b/nerfstudio/engine/trainer.py
@@ -243,7 +243,9 @@ class Trainer:
writer.put_dict(name="Train Loss Dict", scalar_dict=loss_dict, step=step)
writer.put_dict(name="Train Metrics Dict", scalar_dict=metrics_dict, step=step)
- self.eval_iteration(step)
+ # Do not perform evaluation if there are no validation images
+ if self.pipeline.datamanager.eval_dataset:
+ self.eval_iteration(step)
if step_check(step, self.config.steps_per_save):
self.save_checkpoint(step)
diff --git a/nerfstudio/exporter/exporter_utils.py b/nerfstudio/exporter/exporter_utils.py
index 9f4168517..2f45badab 100644
--- a/nerfstudio/exporter/exporter_utils.py
+++ b/nerfstudio/exporter/exporter_utils.py
@@ -178,8 +178,8 @@ def generate_point_cloud(
rgbs = torch.cat(rgbs, dim=0)
pcd = o3d.geometry.PointCloud()
- pcd.points = o3d.utility.Vector3dVector(points.float().cpu().numpy())
- pcd.colors = o3d.utility.Vector3dVector(rgbs.float().cpu().numpy())
+ pcd.points = o3d.utility.Vector3dVector(points.double().cpu().numpy())
+ pcd.colors = o3d.utility.Vector3dVector(rgbs.double().cpu().numpy())
ind = None
if remove_outliers:
@@ -203,7 +203,7 @@ def generate_point_cloud(
if ind is not None:
# mask out normals for points that were removed with remove_outliers
normals = normals[ind]
- pcd.normals = o3d.utility.Vector3dVector(normals.float().cpu().numpy())
+ pcd.normals = o3d.utility.Vector3dVector(normals.double().cpu().numpy())
return pcd
diff --git a/nerfstudio/fields/instant_ngp_field.py b/nerfstudio/fields/instant_ngp_field.py
index 102912ba7..b0f4e97e4 100644
--- a/nerfstudio/fields/instant_ngp_field.py
+++ b/nerfstudio/fields/instant_ngp_field.py
@@ -19,6 +19,7 @@ Instant-NGP field implementations using tiny-cuda-nn, torch, ....
from typing import Dict, Optional, Tuple
+import numpy as np
import torch
from nerfacc import ContractionType, contract
from torch.nn.parameter import Parameter
@@ -63,6 +64,7 @@ class TCNNInstantNGPField(Field):
contraction_type: type of contraction
num_levels: number of levels of the hashmap for the base mlp
log2_hashmap_size: size of the hashmap for the base mlp
+ max_res: maximum resolution of the hashmap for the base mlp
"""
def __init__(
@@ -79,6 +81,7 @@ class TCNNInstantNGPField(Field):
contraction_type: ContractionType = ContractionType.UN_BOUNDED_SPHERE,
num_levels: int = 16,
log2_hashmap_size: int = 19,
+ max_res: int = 2048,
) -> None:
super().__init__()
@@ -93,7 +96,8 @@ class TCNNInstantNGPField(Field):
self.appearance_embedding = Embedding(num_images, appearance_embedding_dim)
# TODO: set this properly based on the aabb
- per_level_scale = 1.4472692012786865
+ base_res: int = 16
+ per_level_scale = np.exp((np.log(max_res) - np.log(base_res)) / (num_levels - 1))
self.direction_encoding = tcnn.Encoding(
n_input_dims=3,
@@ -111,7 +115,7 @@ class TCNNInstantNGPField(Field):
"n_levels": num_levels,
"n_features_per_level": 2,
"log2_hashmap_size": log2_hashmap_size,
- "base_resolution": 16,
+ "base_resolution": base_res,
"per_level_scale": per_level_scale,
},
network_config={
diff --git a/nerfstudio/models/instant_ngp.py b/nerfstudio/models/instant_ngp.py
index ace4b77c6..2bdf85b6a 100644
--- a/nerfstudio/models/instant_ngp.py
+++ b/nerfstudio/models/instant_ngp.py
@@ -65,6 +65,10 @@ class InstantNGPModelConfig(ModelConfig):
"""Number of samples in field evaluation."""
grid_resolution: int = 128
"""Resolution of the grid used for the field."""
+ max_res: int = 2048
+ """Maximum resolution of the hashmap for the base mlp."""
+ log2_hashmap_size: int = 19
+ """Size of the hashmap for the base mlp"""
contraction_type: ContractionType = ContractionType.UN_BOUNDED_SPHERE
"""Contraction type used for spatial deformation of the field."""
cone_angle: float = 0.004
@@ -103,6 +107,8 @@ class NGPModel(Model):
contraction_type=self.config.contraction_type,
use_appearance_embedding=self.config.use_appearance_embedding,
num_images=self.num_train_data,
+ log2_hashmap_size=self.config.log2_hashmap_size,
+ max_res=self.config.max_res,
)
self.scene_aabb = Parameter(self.scene_box.aabb.flatten(), requires_grad=False)
diff --git a/nerfstudio/process_data/equirect_utils.py b/nerfstudio/process_data/equirect_utils.py
index 596af58fb..340dd7884 100644
--- a/nerfstudio/process_data/equirect_utils.py
+++ b/nerfstudio/process_data/equirect_utils.py
@@ -15,13 +15,15 @@
"""Helper utils for processing equirectangular data."""
import os
+import sys
from pathlib import Path
-from typing import Tuple
+from typing import List, Tuple
import cv2
import numpy as np
import torch
from equilib import Equi2Pers
+from rich.console import Console
from rich.progress import (
BarColumn,
Progress,
@@ -32,11 +34,86 @@ from rich.progress import (
from nerfstudio.utils.rich_utils import ItersPerSecColumn
+CONSOLE = Console(width=120)
+
+
+def _crop_bottom(bound_arr: list, fov: int, crop_factor: float) -> List[float]:
+ """Returns a list of vertical bounds with the bottom cropped.
+
+ Args:
+ bound_arr (list): List of vertical bounds in ascending order.
+ fov (int): Field of view of the camera.
+ crop_factor (float): Portion of the image to crop from the bottom.
+
+ Returns:
+ list: A new list of bounds with the bottom cropped.
+ """
+ degrees_chopped = 180 * crop_factor
+ new_bottom_start = 90 - degrees_chopped - fov / 2
+ for i, el in reversed(list(enumerate(bound_arr))):
+ if el > new_bottom_start + fov / 2:
+ bound_arr[i] = None
+ elif el > new_bottom_start:
+ diff = el - new_bottom_start
+ bound_arr[i] = new_bottom_start
+ for j in range(i - 1, -1, -1):
+ bound_arr[j] -= diff / (2 ** (i - j))
+ break
+
+ return bound_arr
+
+
+def _crop_top(bound_arr: list, fov: int, crop_factor: float) -> List[float]:
+ """Returns a list of vertical bounds with the top cropped.
+
+ Args:
+ bound_arr (list): List of vertical bounds in ascending order.
+ fov (int): Field of view of the camera.
+ crop_factor (float): Portion of the image to crop from the top.
+
+ Returns:
+ list: A new list of bounds with the top cropped.
+ """
+ degrees_chopped = 180 * crop_factor
+ new_top_start = -90 + degrees_chopped + fov / 2
+ for i, el in enumerate(bound_arr):
+ if el < new_top_start - fov / 2:
+ bound_arr[i] = None
+ elif el < new_top_start:
+ diff = new_top_start - el
+ bound_arr[i] = new_top_start
+ for j in range(i + 1, len(bound_arr)):
+ bound_arr[j] += diff / (2 ** (j - i))
+ break
+
+ return bound_arr
+
+
+def _crop_bound_arr_vertical(
+ bound_arr: list, fov: int, crop_factor: Tuple[float, float, float, float] = (0.0, 0.0, 0.0, 0.0)
+) -> list:
+ """Returns a list of vertical bounds adjusted for cropping.
+
+ Args:
+ bound_arr (list): Original list of vertical bounds in ascending order.
+ fov (int): Field of view of the camera.
+ crop_factor (Tuple[float, float, float, float]): Crop arr (top, bottom, left, right).
+
+ Returns:
+ list: Cropped bound arr
+ """
+ if crop_factor[1] > 0:
+ bound_arr = _crop_bottom(bound_arr, fov, crop_factor[1])
+ if crop_factor[0] > 0:
+ bound_arr = _crop_top(bound_arr, fov, crop_factor[0])
+ return bound_arr
+
def generate_planar_projections_from_equirectangular(
image_dir: Path,
planar_image_size: Tuple[int, int],
samples_per_im: int,
+ crop_factor: Tuple[float, float, float, float] = (0.0, 0.0, 0.0, 0.0),
) -> Path:
"""Generate planar projections from an equirectangular image.
@@ -44,30 +121,53 @@ def generate_planar_projections_from_equirectangular(
image_dir: The directory containing the equirectangular image.
planar_image_size: The size of the planar projections [width, height].
samples_per_im: The number of samples to take per image.
+ crop_factor: The portion of the image to crop from the (top, bottom, left, and right).
+ Values should be in [0, 1].
returns:
The path to the planar projections directory.
"""
+ for i in crop_factor:
+ if i < 0 or i > 1:
+ CONSOLE.print("[bold red] Invalid crop factor. All values must be in [0,1].")
+ sys.exit(1)
+
device = torch.device("cuda")
fov = 120
yaw_pitch_pairs = []
+ left_bound, right_bound = -180, 180
+ if crop_factor[3] > 0:
+ left_bound = -180 + 360 * crop_factor[3]
+ if crop_factor[2] > 0:
+ right_bound = 180 - 360 * crop_factor[2]
+
if samples_per_im == 8:
fov = 120
- for i in np.arange(-180, 180, 90):
- yaw_pitch_pairs.append((i, 0))
- for i in np.arange(-180, 180, 180):
- yaw_pitch_pairs.append((i, 45))
- for i in np.arange(-180, 180, 180):
- yaw_pitch_pairs.append((i, -45))
+ bound_arr = [-45, 0, 45]
+ bound_arr = _crop_bound_arr_vertical(bound_arr, fov, crop_factor)
+ if bound_arr[1] is not None:
+ for i in np.arange(left_bound, right_bound, 90):
+ yaw_pitch_pairs.append((i, bound_arr[1]))
+ if bound_arr[2] is not None:
+ for i in np.arange(left_bound, right_bound, 180):
+ yaw_pitch_pairs.append((i, bound_arr[2]))
+ if bound_arr[0] is not None:
+ for i in np.arange(left_bound, right_bound, 180):
+ yaw_pitch_pairs.append((i, bound_arr[0]))
elif samples_per_im == 14:
fov = 110
- for i in np.arange(-180, 180, 60):
- yaw_pitch_pairs.append((i, 0))
- for i in np.arange(-180, 180, 90):
- yaw_pitch_pairs.append((i, 45))
- for i in np.arange(-180, 180, 90):
- yaw_pitch_pairs.append((i, -45))
+ bound_arr = [-45, 0, 45]
+ bound_arr = _crop_bound_arr_vertical(bound_arr, fov, crop_factor)
+ if bound_arr[1] is not None:
+ for i in np.arange(left_bound, right_bound, 60):
+ yaw_pitch_pairs.append((i, bound_arr[1]))
+ if bound_arr[2] is not None:
+ for i in np.arange(left_bound, right_bound, 90):
+ yaw_pitch_pairs.append((i, bound_arr[2]))
+ if bound_arr[0] is not None:
+ for i in np.arange(left_bound, right_bound, 90):
+ yaw_pitch_pairs.append((i, bound_arr[0]))
equi2pers = Equi2Pers(height=planar_image_size[1], width=planar_image_size[0], fov_x=fov, mode="bilinear")
frame_dir = image_dir
@@ -82,7 +182,6 @@ def generate_planar_projections_from_equirectangular(
TimeRemainingColumn(elapsed_when_finished=True, compact=True),
)
- curr_im = 0
with progress:
for i in progress.track(os.listdir(frame_dir), description="", total=num_ims):
if i.lower().endswith((".jpg", ".png", ".jpeg")):
@@ -97,7 +196,6 @@ def generate_planar_projections_from_equirectangular(
pers_image = (pers_image.permute(1, 2, 0)).type(torch.uint8).to("cpu").numpy()
cv2.imwrite(f"{output_dir}/{i[:-4]}_{count}.jpg", pers_image)
count += 1
- curr_im += 1
return output_dir
diff --git a/nerfstudio/process_data/process_data_utils.py b/nerfstudio/process_data/process_data_utils.py
index 6b94bbaa9..14ae33662 100644
--- a/nerfstudio/process_data/process_data_utils.py
+++ b/nerfstudio/process_data/process_data_utils.py
@@ -100,7 +100,11 @@ def get_num_frames_in_video(video: Path) -> int:
def convert_video_to_images(
- video_path: Path, image_dir: Path, num_frames_target: int, verbose: bool = False
+ video_path: Path,
+ image_dir: Path,
+ num_frames_target: int,
+ crop_factor: Tuple[float, float, float, float] = (0.0, 0.0, 0.0, 0.0),
+ verbose: bool = False,
) -> Tuple[List[str], int]:
"""Converts a video into a sequence of images.
@@ -108,11 +112,17 @@ def convert_video_to_images(
video_path: Path to the video.
output_dir: Path to the output directory.
num_frames_target: Number of frames to extract.
+ crop_factor: Portion of the image to crop. Should be in [0,1] (top, bottom, left, right)
verbose: If True, logs the output of the command.
Returns:
A tuple containing summary of the conversion and the number of extracted frames.
"""
+ for i in crop_factor:
+ if i < 0 or i > 1:
+ CONSOLE.print("[bold red]Error: Invalid crop factor. All crops must be in [0,1].")
+ sys.exit(1)
+
if video_path.is_dir():
CONSOLE.print(f"[bold red]Error: Video path is a directory, not a path: {video_path}")
sys.exit(1)
@@ -135,16 +145,23 @@ def convert_video_to_images(
out_filename = image_dir / "frame_%05d.png"
ffmpeg_cmd = f'ffmpeg -i "{video_path}"'
- spacing = num_frames // num_frames_target
+ crop_cmd = ""
+ if crop_factor != (0.0, 0.0, 0.0, 0.0):
+ height = 1 - crop_factor[0] - crop_factor[1]
+ width = 1 - crop_factor[2] - crop_factor[3]
+ start_x = crop_factor[2]
+ start_y = crop_factor[0]
+ crop_cmd = f',"crop=w=iw*{width}:h=ih*{height}:x=iw*{start_x}:y=ih*{start_y}"'
+
+ spacing = num_frames // num_frames_target
if spacing > 1:
- ffmpeg_cmd += f" -vf thumbnail={spacing},setpts=N/TB -r 1"
+ ffmpeg_cmd += f" -vf thumbnail={spacing},setpts=N/TB{crop_cmd} -r 1"
else:
CONSOLE.print("[bold red]Can't satisfy requested number of frames. Extracting all frames.")
ffmpeg_cmd += " -pix_fmt bgr8"
ffmpeg_cmd += f" {out_filename}"
-
run_command(ffmpeg_cmd, verbose=verbose)
num_final_frames = len(list(image_dir.glob("*.png")))
@@ -399,25 +416,23 @@ def generate_circle_mask(height: int, width: int, percent_radius) -> Optional[np
return mask
-def generate_crop_mask(
- height: int, width: int, percent_crop: Tuple[float, float, float, float]
-) -> Optional[np.ndarray]:
+def generate_crop_mask(height: int, width: int, crop_factor: Tuple[float, float, float, float]) -> Optional[np.ndarray]:
"""generate a crop mask of the given size.
Args:
height: The height of the mask.
width: The width of the mask.
- percent_crop: The percent of the image to crop in each direction [top, bottom, left, right].
+ crop_factor: The percent of the image to crop in each direction [top, bottom, left, right].
Returns:
The mask or None if no cropping is performed.
"""
- if np.all(np.array(percent_crop) == 0.0):
+ if np.all(np.array(crop_factor) == 0.0):
return None
- if np.any(np.array(percent_crop) < 0.0) or np.any(np.array(percent_crop) > 1.0):
+ if np.any(np.array(crop_factor) < 0.0) or np.any(np.array(crop_factor) > 1.0):
CONSOLE.log("[bold red]Invalid crop percentage, must be between 0 and 1.")
sys.exit(1)
- top, bottom, left, right = percent_crop
+ top, bottom, left, right = crop_factor
mask = np.zeros((height, width), dtype=np.uint8)
top = int(top * height)
bottom = int(bottom * height)
@@ -428,20 +443,20 @@ def generate_crop_mask(
def generate_mask(
- height: int, width: int, percent_crop: Tuple[float, float, float, float], percent_radius: float
+ height: int, width: int, crop_factor: Tuple[float, float, float, float], percent_radius: float
) -> Optional[np.ndarray]:
"""generate a mask of the given size.
Args:
height: The height of the mask.
width: The width of the mask.
- percent_crop: The percent of the image to crop in each direction [top, bottom, left, right].
+ crop_factor: The percent of the image to crop in each direction [top, bottom, left, right].
percent_radius: The radius of the circle as a percentage of the image diagonal size.
Returns:
The mask or None if no mask is needed.
"""
- crop_mask = generate_crop_mask(height, width, percent_crop)
+ crop_mask = generate_crop_mask(height, width, crop_factor)
circle_mask = generate_circle_mask(height, width, percent_radius)
if crop_mask is None:
return circle_mask
@@ -453,7 +468,7 @@ def generate_mask(
def save_mask(
image_dir: Path,
num_downscales: int,
- percent_crop: Tuple[float, float, float, float] = (0, 0, 0, 0),
+ crop_factor: Tuple[float, float, float, float] = (0, 0, 0, 0),
percent_radius: float = 1.0,
) -> Optional[Path]:
"""Save a mask for each image in the image directory.
@@ -461,7 +476,7 @@ def save_mask(
Args:
image_dir: The directory containing the images.
num_downscales: The number of downscaling levels.
- percent_crop: The percent of the image to crop in each direction [top, bottom, left, right].
+ crop_factor: The percent of the image to crop in each direction [top, bottom, left, right].
percent_radius: The radius of the circle as a percentage of the image diagonal size.
Returns:
@@ -470,7 +485,7 @@ def save_mask(
image_path = next(image_dir.glob("frame_*"))
image = cv2.imread(str(image_path))
height, width = image.shape[:2]
- mask = generate_mask(height, width, percent_crop, percent_radius)
+ mask = generate_mask(height, width, crop_factor, percent_radius)
if mask is None:
return None
mask *= 255
diff --git a/nerfstudio/viewer/app/package.json b/nerfstudio/viewer/app/package.json
index 171ec8dd1..3525ebd86 100644
--- a/nerfstudio/viewer/app/package.json
+++ b/nerfstudio/viewer/app/package.json
@@ -1,7 +1,7 @@
{
"name": "viewer",
"homepage": ".",
- "version": "23-02-3-0",
+ "version": "23-03-9-0",
"private": true,
"dependencies": {
"@emotion/react": "^11.10.4",
diff --git a/nerfstudio/viewer/app/src/index.scss b/nerfstudio/viewer/app/src/index.scss
index 3a331e37a..a3af6c244 100644
--- a/nerfstudio/viewer/app/src/index.scss
+++ b/nerfstudio/viewer/app/src/index.scss
@@ -132,8 +132,8 @@ $sidebar-width: 100% !default;
padding: 10px;
}
-.StatusPanel .StatusPanel-button-group {
- margin: 10px;
+.StatusPanel .StatusPanel-button {
+ margin-bottom: 4px;
}
.StatusPanel .StatusPanel-hide-scene-button {
@@ -348,7 +348,7 @@ $sidebar-width: 100% !default;
// Viewer Window
.canvas-container-main,
-.WebRTCVideo,
+.RenderWindow,
.ViewerWindow-render-crop-container {
position: absolute;
top: 0px;
@@ -358,7 +358,7 @@ $sidebar-width: 100% !default;
border: $custom-border;
}
-.WebRTCVideo {
+.RenderWindow {
z-index: 1;
}
@@ -373,12 +373,6 @@ $sidebar-width: 100% !default;
z-index: 3;
}
-.WebRTCVideo-video {
- width: 100%;
- height: 100%;
- object-fit: fill;
-}
-
.ViewerWindow-render-crop {
display: flex;
box-shadow: rgb(0 0 0 / 70%) 0px 0px 0px 1600px;
diff --git a/nerfstudio/viewer/app/src/modules/RenderWindow/RenderWindow.jsx b/nerfstudio/viewer/app/src/modules/RenderWindow/RenderWindow.jsx
new file mode 100644
index 000000000..e2e1c53e7
--- /dev/null
+++ b/nerfstudio/viewer/app/src/modules/RenderWindow/RenderWindow.jsx
@@ -0,0 +1,12 @@
+import * as React from 'react';
+import { useSelector } from 'react-redux';
+
+export default function RenderWindow() {
+ const render_img = useSelector((state) => state.render_img);
+
+ return (
+