+ 
+
A collaboration friendly studio for NeRFs
@@ -70,7 +70,7 @@ Nerfstudio initially launched as an opensource project by Berkeley students in [ We are committed to providing learning resources to help you understand the basics of (if you're just getting started), and keep up-to-date with (if you're a seasoned veteran) all things NeRF. As researchers, we know just how hard it is to get onboarded with this next-gen technology. So we're here to help with tutorials, documentation, and more! -Have feature requests? Want to add your brand-spankin'-new NeRF model? Have a new dataset? **We welcome [contributions](https://docs.nerf.studio/en/latest/reference/contributing.html)!** Please do not hesitate to reach out to the nerfstudio team with any questions via [Discord](https://discord.gg/uMbNqcraFc). +Have feature requests? Want to add your brand-spankin'-new NeRF model? Have a new dataset? **We welcome [contributions](https://docs.nerf.studio/reference/contributing.html)!** Please do not hesitate to reach out to the nerfstudio team with any questions via [Discord](https://discord.gg/uMbNqcraFc). Have feedback? We'd love for you to fill out our [Nerfstudio Feedback Form](https://forms.gle/sqN5phJN7LfQVwnP9) if you want to let us know who you are, why you are interested in Nerfstudio, or provide any feedback! @@ -118,7 +118,7 @@ You must have an NVIDIA video card with CUDA installed on the system. This libra ### Create environment -Nerfstudio requires `python >= 3.8`. We recommend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html) before proceeding. +Nerfstudio requires `python >= 3.8`. We recommend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/miniconda.html) before proceeding. ```bash conda create --name nerfstudio -y python=3.8 @@ -250,6 +250,7 @@ Using an existing dataset is great, but likely you want to use your own data! We | 📱 [Record3D](https://docs.nerf.studio/quickstart/custom_dataset.html#record3d-capture) | IOS with LiDAR | [Record3D app](https://record3d.app/) | 🐇 | | 🖥 [Metashape](https://docs.nerf.studio/quickstart/custom_dataset.html#metashape) | Any | [Metashape](https://www.agisoft.com/) | 🐇 | | 🖥 [RealityCapture](https://docs.nerf.studio/quickstart/custom_dataset.html#realitycapture) | Any | [RealityCapture](https://www.capturingreality.com/realitycapture) | 🐇 | +| 🖥 [ODM](https://docs.nerf.studio/quickstart/custom_dataset.html#ODM) | Any | [ODM](https://github.com/OpenDroneMap/ODM) | 🐇 | | 🛠 [Custom](https://docs.nerf.studio/quickstart/data_conventions.html) | Any | Camera Poses | 🐇 | ## 5. Advanced Options @@ -280,23 +281,23 @@ We support four different methods to track training progress, using the viewer[t And that's it for getting started with the basics of nerfstudio. -If you're interested in learning more on how to create your own pipelines, develop with the viewer, run benchmarks, and more, please check out some of the quicklinks below or visit our [documentation](https://docs.nerf.studio/en/latest/) directly. +If you're interested in learning more on how to create your own pipelines, develop with the viewer, run benchmarks, and more, please check out some of the quicklinks below or visit our [documentation](https://docs.nerf.studio/) directly. | Section | Description | | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | -| [Documentation](https://docs.nerf.studio/en/latest/) | Full API documentation and tutorials | +| [Documentation](https://docs.nerf.studio/) | Full API documentation and tutorials | | [Viewer](https://viewer.nerf.studio/) | Home page for our web viewer | | 🎒 **Educational** | -| [Model Descriptions](https://docs.nerf.studio/en/latest/nerfology/methods/index.html) | Description of all the models supported by nerfstudio and explanations of component parts. | -| [Component Descriptions](https://docs.nerf.studio/en/latest/nerfology/model_components/index.html) | Interactive notebooks that explain notable/commonly used modules in various models. | +| [Model Descriptions](https://docs.nerf.studio/nerfology/methods/index.html) | Description of all the models supported by nerfstudio and explanations of component parts. | +| [Component Descriptions](https://docs.nerf.studio/nerfology/model_components/index.html) | Interactive notebooks that explain notable/commonly used modules in various models. | | 🏃 **Tutorials** | -| [Getting Started](https://docs.nerf.studio/en/latest/quickstart/installation.html) | A more in-depth guide on how to get started with nerfstudio from installation to contributing. | -| [Using the Viewer](https://docs.nerf.studio/en/latest/quickstart/viewer_quickstart.html) | A quick demo video on how to navigate the viewer. | +| [Getting Started](https://docs.nerf.studio/quickstart/installation.html) | A more in-depth guide on how to get started with nerfstudio from installation to contributing. | +| [Using the Viewer](https://docs.nerf.studio/quickstart/viewer_quickstart.html) | A quick demo video on how to navigate the viewer. | | [Using Record3D](https://www.youtube.com/watch?v=XwKq7qDQCQk) | Demo video on how to run nerfstudio without using COLMAP. | | 💻 **For Developers** | -| [Creating pipelines](https://docs.nerf.studio/en/latest/developer_guides/pipelines/index.html) | Learn how to easily build new neural rendering pipelines by using and/or implementing new modules. | -| [Creating datasets](https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html) | Have a new dataset? Learn how to run it with nerfstudio. | -| [Contributing](https://docs.nerf.studio/en/latest/reference/contributing.html) | Walk-through for how you can start contributing now. | +| [Creating pipelines](https://docs.nerf.studio/developer_guides/pipelines/index.html) | Learn how to easily build new neural rendering pipelines by using and/or implementing new modules. | +| [Creating datasets](https://docs.nerf.studio/quickstart/custom_dataset.html) | Have a new dataset? Learn how to run it with nerfstudio. | +| [Contributing](https://docs.nerf.studio/reference/contributing.html) | Walk-through for how you can start contributing now. | | 💖 **Community** | | [Discord](https://discord.gg/uMbNqcraFc) | Join our community to discuss more. We would love to hear from you! | | [Twitter](https://twitter.com/nerfstudioteam) | Follow us on Twitter @nerfstudioteam to see cool updates and announcements | diff --git a/docs/nerfology/methods/instant_ngp.md b/docs/nerfology/methods/instant_ngp.md index c7589e295..5f5f7e07f 100644 --- a/docs/nerfology/methods/instant_ngp.md +++ b/docs/nerfology/methods/instant_ngp.md @@ -34,7 +34,7 @@ Instant-NGP breaks NeRF training into 3 pillars and proposes improvements to eac The core idea behind the improved sampling technique is that sampling over empty space should be skipped and sampling behind high density areas should also be skipped. This is achieved by maintaining a set of multiscale occupancy grids which coarsely mark empty and non-empty space. Occupancy is stored as a single bit, and a sample on a ray is skipped if its occupancy is too low. These occupancy grids are stored independently of the trainable encoding and are updated throughout training based on the updated density predictions. The authors find they can increase sampling speed by 10-100x compared to naive approaches. -Nerfstudio uses [NerfAcc](https://www.nerfacc.com/en/latest/index.html) as the sampling algorithm implementation. The details on NerfAcc's sampling and occupancy grid is discussed [here](https://www.nerfacc.com/en/stable/methodology/sampling.html#occupancy-grid-estimator). +Nerfstudio uses [NerfAcc](https://www.nerfacc.com/index.html) as the sampling algorithm implementation. The details on NerfAcc's sampling and occupancy grid is discussed [here](https://www.nerfacc.com/en/stable/methodology/sampling.html#occupancy-grid-estimator). Another major bottleneck for NeRF's training speed has been querying the neural network. The authors of this work implement the network such that it runs entirely on a single CUDA kernel. The network is also shrunk down to be just 4 layers with 64 neurons in each layer. They show that their fully-fused neural network is 5-10x faster than a Tensorflow implementation. diff --git a/docs/quickstart/custom_dataset.md b/docs/quickstart/custom_dataset.md index 723a97c91..46aaa907d 100644 --- a/docs/quickstart/custom_dataset.md +++ b/docs/quickstart/custom_dataset.md @@ -21,6 +21,7 @@ We Currently support the following custom data types: | 📱 [Record3D](record3d) | IOS with LiDAR | [Record3D app](https://record3d.app/) | 🐇 | | 🖥 [Metashape](metashape) | Any | [Metashape](https://www.agisoft.com/) | 🐇 | | 🖥 [RealityCapture](realitycapture) | Any | [RealityCapture](https://www.capturingreality.com/realitycapture) | 🐇 | +| 🖥 [ODM](odm) | Any | [ODM](https://github.com/OpenDroneMap/ODM) | 🐇 | (images_and_video)= @@ -319,6 +320,34 @@ ns-process-data realitycapture --data {data directory} --csv {csv file} --output ns-train nerfacto --data {output directory} ``` +(odm)= + +## ODM + +All images/videos must be captured with the same camera. + +1. Process a dataset using [ODM](https://github.com/OpenDroneMap/ODM#quickstart) + +```bash +$ ls /path/to/dataset +images +odm_report +odm_orthophoto +... +``` + +2. Convert to nerfstudio format. + +```bash +ns-process-data odm --data /path/to/dataset --output-dir {output directory} +``` + +4. Train! + +```bash +ns-train nerfacto --data {output directory} +``` + (360_data)= ## 360 Data (Equirectangular) @@ -377,7 +406,7 @@ This outputs two 180 deg equirectangular renders horizontally stacked, one for e ### Setup instructions -To render for VR video it is essential to adjust the NeRF to have an approximately true-to-life real world scale (adjustable in the camera path) to ensure that the scene depth and IPD (distance between the eyes) is appropriate for the render to be viewable in VR. You can adjust the scene scale with the [Nerfstudio Blender Add-on](https://docs.nerf.studio/en/latest/extensions/blender_addon.html) by appropriately scaling a point cloud representation of the NeRF. +To render for VR video it is essential to adjust the NeRF to have an approximately true-to-life real world scale (adjustable in the camera path) to ensure that the scene depth and IPD (distance between the eyes) is appropriate for the render to be viewable in VR. You can adjust the scene scale with the [Nerfstudio Blender Add-on](https://docs.nerf.studio/extensions/blender_addon.html) by appropriately scaling a point cloud representation of the NeRF. Results may be unviewable if the scale is not set appropriately. The IPD is set at 64mm by default but only is accurate when the NeRF scene is true to scale. For good quality renders, it is recommended to render at high resolutions (For ODS: 4096x2048 per eye, or 2048x1024 per eye. For VR180: 4096x4096 per eye or 2048x2048 per eye). Render resolutions for a single eye are specified in the camera path. For VR180, resolutions must be in a 1:1 aspect ratio. For ODS, resolutions must be in a 2:1 aspect ratio. The final stacked render output will automatically be constructed (with aspect ratios for VR180 as 2:1 and ODS as 1:1). @@ -388,7 +417,7 @@ If you are rendering an image sequence, it is recommended to render as png inste ::: To render with the VR videos camera: -1. Use the [Nerfstudio Blender Add-on](https://docs.nerf.studio/en/latest/extensions/blender_addon.html) to set the scale of the NeRF scene and create the camera path +1. Use the [Nerfstudio Blender Add-on](https://docs.nerf.studio/extensions/blender_addon.html) to set the scale of the NeRF scene and create the camera path - Export a point cloud representation of the NeRF - Import the point cloud representation in Blender and enable the Nerfstudio Blender Add-on - Create a reference object such as a cube which may be 1x1x1 meter. You could also create a cylinder and scale it to an appropriate height of a viewer. diff --git a/docs/quickstart/data_conventions.md b/docs/quickstart/data_conventions.md index 2e9e76eff..a929ff53f 100644 --- a/docs/quickstart/data_conventions.md +++ b/docs/quickstart/data_conventions.md @@ -31,10 +31,10 @@ If all of the images share the same camera intrinsics, the values can be placed "cy": 1000.0, // principal point y "w": 3008, // image width "h": 2000, // image height - "k1": 0.0312, // first radial distorial parameter, used by [OPENCV, OPENCV_FISHEYE] - "k2": 0.0051, // second radial distorial parameter, used by [OPENCV, OPENCV_FISHEYE] - "k3": 0.0006, // third radial distorial parameter, used by [OPENCV_FISHEYE] - "k4": 0.0001, // fourth radial distorial parameter, used by [OPENCV_FISHEYE] + "k1": 0.0312, // first radial distortion parameter, used by [OPENCV, OPENCV_FISHEYE] + "k2": 0.0051, // second radial distortion parameter, used by [OPENCV, OPENCV_FISHEYE] + "k3": 0.0006, // third radial distortion parameter, used by [OPENCV_FISHEYE] + "k4": 0.0001, // fourth radial distortion parameter, used by [OPENCV_FISHEYE] "p1": -6.47e-5, // first tangential distortion parameter, used by [OPENCV] "p2": -1.37e-7, // second tangential distortion parameter, used by [OPENCV] "frames": // ... per-frame intrinsics and extrinsics parameters diff --git a/docs/reference/contributing.md b/docs/reference/contributing.md index 529d154eb..bf0af33db 100644 --- a/docs/reference/contributing.md +++ b/docs/reference/contributing.md @@ -94,7 +94,7 @@ python nerfstudio/scripts/docs/build_docs.py ### Auto build -As you change or add models/components, the auto-generated [Reference API](https://docs.nerf.studio/en/latest/reference/api/index.html) may change. +As you change or add models/components, the auto-generated [Reference API](https://docs.nerf.studio/reference/api/index.html) may change. If you want the code to build on save you can use [sphinx autobuild](https://github.com/executablebooks/sphinx-autobuild). :::{admonition} Tip diff --git a/nerfstudio/configs/external_methods.py b/nerfstudio/configs/external_methods.py index 6a0c78b09..ce4e7bbc2 100644 --- a/nerfstudio/configs/external_methods.py +++ b/nerfstudio/configs/external_methods.py @@ -45,7 +45,7 @@ external_methods = [] external_methods.append( ExternalMethod( """[bold yellow]Instruct-NeRF2NeRF[/bold yellow] -For more information visit: https://docs.nerf.studio/en/latest/nerfology/methods/in2n.html +For more information visit: https://docs.nerf.studio/nerfology/methods/in2n.html To enable Instruct-NeRF2NeRF, you must install it first by running: [grey]pip install git+https://github.com/ayaanzhaque/instruct-nerf2nerf[/grey]""", @@ -62,7 +62,7 @@ To enable Instruct-NeRF2NeRF, you must install it first by running: external_methods.append( ExternalMethod( """[bold yellow]K-Planes[/bold yellow] -For more information visit https://docs.nerf.studio/en/latest/nerfology/methods/kplanes.html +For more information visit https://docs.nerf.studio/nerfology/methods/kplanes.html To enable K-Planes, you must install it first by running: [grey]pip install kplanes-nerfstudio[/grey]""", @@ -78,7 +78,7 @@ To enable K-Planes, you must install it first by running: external_methods.append( ExternalMethod( """[bold yellow]LERF[/bold yellow] -For more information visit: https://docs.nerf.studio/en/latest/nerfology/methods/lerf.html +For more information visit: https://docs.nerf.studio/nerfology/methods/lerf.html To enable LERF, you must install it first by running: [grey]pip install git+https://github.com/kerrj/lerf[/grey]""", @@ -95,7 +95,7 @@ To enable LERF, you must install it first by running: external_methods.append( ExternalMethod( """[bold yellow]Tetra-NeRF[/bold yellow] -For more information visit: https://docs.nerf.studio/en/latest/nerfology/methods/tetranerf.html +For more information visit: https://docs.nerf.studio/nerfology/methods/tetranerf.html To enable Tetra-NeRF, you must install it first. Please follow the instructions here: https://github.com/jkulhanek/tetra-nerf/blob/master/README.md#installation""", @@ -110,7 +110,7 @@ To enable Tetra-NeRF, you must install it first. Please follow the instructions external_methods.append( ExternalMethod( """[bold yellow]NeRFPlayer[/bold yellow] -For more information visit: https://docs.nerf.studio/en/latest/nerfology/methods/nerfplayer.html +For more information visit: https://docs.nerf.studio/nerfology/methods/nerfplayer.html To enable NeRFPlayer, you must install it first by running: [grey]pip install git+https://github.com/lsongx/nerfplayer-nerfstudio[/grey]""", @@ -125,7 +125,7 @@ To enable NeRFPlayer, you must install it first by running: external_methods.append( ExternalMethod( """[bold yellow]Volinga[/bold yellow] -For more information visit: https://docs.nerf.studio/en/latest/extensions/unreal_engine.html +For more information visit: https://docs.nerf.studio/extensions/unreal_engine.html To enable Volinga, you must install it first by running: [grey]pip install git+https://github.com/Volinga/volinga-model[/grey]""", diff --git a/nerfstudio/data/datamanagers/base_datamanager.py b/nerfstudio/data/datamanagers/base_datamanager.py index 2c0f602ef..821bbda7f 100644 --- a/nerfstudio/data/datamanagers/base_datamanager.py +++ b/nerfstudio/data/datamanagers/base_datamanager.py @@ -555,9 +555,13 @@ class VanillaDataManager(DataManager, Generic[TDataset]): raise ValueError("No more eval images") def get_train_rays_per_batch(self) -> int: + if self.train_pixel_sampler is not None: + return self.train_pixel_sampler.num_rays_per_batch return self.config.train_num_rays_per_batch def get_eval_rays_per_batch(self) -> int: + if self.eval_pixel_sampler is not None: + return self.eval_pixel_sampler.num_rays_per_batch return self.config.eval_num_rays_per_batch def get_datapath(self) -> Path: diff --git a/nerfstudio/data/datamanagers/parallel_datamanager.py b/nerfstudio/data/datamanagers/parallel_datamanager.py index 7a2e5ac55..5c4d5c4a2 100644 --- a/nerfstudio/data/datamanagers/parallel_datamanager.py +++ b/nerfstudio/data/datamanagers/parallel_datamanager.py @@ -303,10 +303,14 @@ class ParallelDataManager(DataManager, Generic[TDataset]): def get_train_rays_per_batch(self) -> int: """Returns the number of rays per batch for training.""" + if self.train_pixel_sampler is not None: + return self.train_pixel_sampler.num_rays_per_batch return self.config.train_num_rays_per_batch def get_eval_rays_per_batch(self) -> int: """Returns the number of rays per batch for evaluation.""" + if self.eval_pixel_sampler is not None: + return self.eval_pixel_sampler.num_rays_per_batch return self.config.eval_num_rays_per_batch def get_datapath(self) -> Path: diff --git a/nerfstudio/exporter/marching_cubes.py b/nerfstudio/exporter/marching_cubes.py index a83307501..ee13422b1 100644 --- a/nerfstudio/exporter/marching_cubes.py +++ b/nerfstudio/exporter/marching_cubes.py @@ -249,7 +249,7 @@ def generate_mesh_with_multires_marching_cubes( ) verts = verts + np.array([x_min, y_min, z_min]) - meshcrop = trimesh.Trimesh(verts, faces, normals) + meshcrop = trimesh.Trimesh(verts, faces, normals) # type: ignore meshes.append(meshcrop) combined_mesh: trimesh.Trimesh = trimesh.util.concatenate(meshes) # type: ignore diff --git a/nerfstudio/process_data/hloc_utils.py b/nerfstudio/process_data/hloc_utils.py index 85049b5bf..a833f51ed 100644 --- a/nerfstudio/process_data/hloc_utils.py +++ b/nerfstudio/process_data/hloc_utils.py @@ -29,7 +29,7 @@ from nerfstudio.utils.rich_utils import CONSOLE try: # TODO(1480) un-hide pycolmap import import pycolmap - from hloc import ( + from hloc import ( # type: ignore extract_features, match_features, pairs_from_exhaustive, @@ -38,6 +38,7 @@ try: ) except ImportError: _HAS_HLOC = False + else: _HAS_HLOC = True @@ -95,24 +96,24 @@ def run_hloc( features = outputs / "features.h5" matches = outputs / "matches.h5" - retrieval_conf = extract_features.confs["netvlad"] - feature_conf = extract_features.confs[feature_type] - matcher_conf = match_features.confs[matcher_type] + retrieval_conf = extract_features.confs["netvlad"] # type: ignore + feature_conf = extract_features.confs[feature_type] # type: ignore + matcher_conf = match_features.confs[matcher_type] # type: ignore references = [p.relative_to(image_dir).as_posix() for p in image_dir.iterdir()] - extract_features.main(feature_conf, image_dir, image_list=references, feature_path=features) + extract_features.main(feature_conf, image_dir, image_list=references, feature_path=features) # type: ignore if matching_method == "exhaustive": - pairs_from_exhaustive.main(sfm_pairs, image_list=references) + pairs_from_exhaustive.main(sfm_pairs, image_list=references) # type: ignore else: - retrieval_path = extract_features.main(retrieval_conf, image_dir, outputs) + retrieval_path = extract_features.main(retrieval_conf, image_dir, outputs) # type: ignore if num_matched >= len(references): num_matched = len(references) - pairs_from_retrieval.main(retrieval_path, sfm_pairs, num_matched=num_matched) - match_features.main(matcher_conf, sfm_pairs, features=features, matches=matches) + pairs_from_retrieval.main(retrieval_path, sfm_pairs, num_matched=num_matched) # type: ignore + match_features.main(matcher_conf, sfm_pairs, features=features, matches=matches) # type: ignore - image_options = pycolmap.ImageReaderOptions(camera_model=camera_model.value) + image_options = pycolmap.ImageReaderOptions(camera_model=camera_model.value) # type: ignore if refine_pixsfm: - sfm = PixSfM( + sfm = PixSfM( # type: ignore conf={ "dense_features": {"use_cache": True}, "KA": {"dense_features": {"use_cache": True}, "max_kps_per_problem": 1000}, @@ -126,20 +127,20 @@ def run_hloc( features, matches, image_list=references, - camera_mode=pycolmap.CameraMode.SINGLE, + camera_mode=pycolmap.CameraMode.SINGLE, # type: ignore image_options=image_options, verbose=verbose, ) print("Refined", refined.summary()) else: - reconstruction.main( + reconstruction.main( # type: ignore sfm_dir, image_dir, sfm_pairs, features, matches, - camera_mode=pycolmap.CameraMode.SINGLE, + camera_mode=pycolmap.CameraMode.SINGLE, # type: ignore image_options=image_options, verbose=verbose, ) diff --git a/nerfstudio/process_data/odm_utils.py b/nerfstudio/process_data/odm_utils.py new file mode 100644 index 000000000..aa9b69541 --- /dev/null +++ b/nerfstudio/process_data/odm_utils.py @@ -0,0 +1,148 @@ +# Copyright 2022 the Regents of the University of California, Nerfstudio Team and contributors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Helper utils for processing ODM data into the nerfstudio format.""" + +import json +from pathlib import Path +from typing import Dict, List +import os +import sys +import math + +import numpy as np + +from nerfstudio.process_data.process_data_utils import CAMERA_MODELS + + +def rodrigues_vec_to_rotation_mat(rodrigues_vec: np.ndarray) -> np.ndarray: + theta = np.linalg.norm(rodrigues_vec) + if theta < sys.float_info.epsilon: + rotation_mat = np.eye(3, dtype=float) + else: + r = rodrigues_vec / theta + ident = np.eye(3, dtype=float) + r_rT = np.array( + [ + [r[0] * r[0], r[0] * r[1], r[0] * r[2]], + [r[1] * r[0], r[1] * r[1], r[1] * r[2]], + [r[2] * r[0], r[2] * r[1], r[2] * r[2]], + ] + ) + r_cross = np.array([[0, -r[2], r[1]], [r[2], 0, -r[0]], [-r[1], r[0], 0]], dtype=float) + rotation_mat = math.cos(theta) * ident + (1 - math.cos(theta)) * r_rT + math.sin(theta) * r_cross + return rotation_mat + + +def cameras2nerfds( + image_filename_map: Dict[str, Path], + cameras_file: Path, + shots_file: Path, + output_dir: Path, + verbose: bool = False, +) -> List[str]: + """Convert ODM cameras into a nerfstudio dataset. + + Args: + image_filename_map: Mapping of original image filenames to their saved locations. + shots_file: Path to ODM's shots.geojson + output_dir: Path to the output directory. + verbose: Whether to print verbose output. + + Returns: + Summary of the conversion. + """ + + with open(cameras_file, "r", encoding="utf-8") as f: + cameras = json.loads(f.read()) + with open(shots_file, "r", encoding="utf-8") as f: + shots = json.loads(f.read()) + + camera_ids = list(cameras.keys()) + if len(camera_ids) > 1: + raise ValueError("Only one camera is supported") + camera_id = camera_ids[0] + camera = cameras[camera_id] + data = {} + if camera["projection_type"] in ["brown", "perspective"]: + data["camera_model"] = CAMERA_MODELS["perspective"].value + elif camera["projection_type"] in ["fisheye", "fisheye_opencv"]: + data["camera_model"] = CAMERA_MODELS["fisheye"].value + elif camera["projection_type"] in ["spherical", "equirectangular"]: + data["camera_model"] = CAMERA_MODELS["equirectangular"].value + else: + raise ValueError("Unsupported ODM camera model: " + data["camera_model"]) + + sensor_dict = {} + s = {"w": int(camera["width"]), "h": int(camera["height"])} + + s["fl_x"] = camera.get("focal_x", camera.get("focal")) * max(s["w"], s["h"]) + s["fl_y"] = camera.get("focal_y", camera.get("focal")) * max(s["w"], s["h"]) + + s["cx"] = camera["c_x"] + (s["w"] - 1.0) / 2.0 + s["cy"] = camera["c_y"] + (s["h"] - 1.0) / 2.0 + + for p in ["k1", "k2", "p1", "p2", "k3"]: + if p in camera: + s[p] = camera[p] + + sensor_dict[camera_id] = s + + shots = shots["features"] + shots_dict = {} + for shot in shots: + props = shot["properties"] + filename = props["filename"] + rotation = rodrigues_vec_to_rotation_mat(np.array(props["rotation"]) * -1) + translation = np.array(props["translation"]) + + m = np.eye(4) + m[:3, :3] = rotation + m[:3, 3] = translation + + name, ext = os.path.splitext(filename) + shots_dict[name] = m + + frames = [] + num_skipped = 0 + + for fname in shots_dict: + transform = shots_dict[fname] + if fname not in image_filename_map: + num_skipped += 1 + continue + + frame = {} + frame["file_path"] = image_filename_map[fname].as_posix() + frame.update(sensor_dict[camera_id]) + + transform = transform[[2, 0, 1, 3], :] + transform[:, 1:3] *= -1 + frame["transform_matrix"] = transform.tolist() + frames.append(frame) + + data["frames"] = frames + + with open(output_dir / "transforms.json", "w", encoding="utf-8") as f: + json.dump(data, f, indent=4) + + summary = [] + if num_skipped == 1: + summary.append(f"{num_skipped} image skipped because it was missing its camera pose.") + if num_skipped > 1: + summary.append(f"{num_skipped} images were skipped because they were missing camera poses.") + + summary.append(f"Final dataset is {len(data['frames'])} frames.") + + return summary diff --git a/nerfstudio/scripts/process_data.py b/nerfstudio/scripts/process_data.py index 8b4151f75..83cdcca47 100644 --- a/nerfstudio/scripts/process_data.py +++ b/nerfstudio/scripts/process_data.py @@ -32,6 +32,7 @@ from nerfstudio.process_data import ( process_data_utils, realitycapture_utils, record3d_utils, + odm_utils, ) from nerfstudio.process_data.colmap_converter_to_nerfstudio_dataset import BaseConverterToNerfstudioDataset from nerfstudio.process_data.images_to_nerfstudio_dataset import ImagesToNerfstudioDataset @@ -387,6 +388,93 @@ class ProcessRealityCapture(BaseConverterToNerfstudioDataset, _NoDefaultProcessR CONSOLE.rule() +@dataclass +class ProcessODM(BaseConverterToNerfstudioDataset): + """Process ODM data into a nerfstudio dataset. + + This script does the following: + + 1. Scales images to a specified size. + 2. Converts ODM poses into the nerfstudio format. + """ + + num_downscales: int = 3 + """Number of times to downscale the images. Downscales by 2 each time. For example a value of 3 + will downscale the images by 2x, 4x, and 8x.""" + max_dataset_size: int = 600 + """Max number of images to train on. If the dataset has more, images will be sampled approximately evenly. If -1, + use all images.""" + + def main(self) -> None: + """Process images into a nerfstudio dataset.""" + + orig_images_dir = self.data / "images" + cameras_file = self.data / "cameras.json" + shots_file = self.data / "odm_report" / "shots.geojson" + + if not shots_file.exists: + raise ValueError(f"shots file {shots_file} doesn't exist") + if not shots_file.exists: + raise ValueError(f"cameras file {cameras_file} doesn't exist") + + if not orig_images_dir.exists: + raise ValueError(f"Images dir {orig_images_dir} doesn't exist") + + if self.eval_data is not None: + raise ValueError("Cannot use eval_data since cameras were already aligned with ODM.") + + self.output_dir.mkdir(parents=True, exist_ok=True) + image_dir = self.output_dir / "images" + image_dir.mkdir(parents=True, exist_ok=True) + + summary_log = [] + + # Copy images to output directory + image_filenames, num_orig_images = process_data_utils.get_image_filenames( + orig_images_dir, self.max_dataset_size + ) + copied_image_paths = process_data_utils.copy_images_list( + image_filenames, + image_dir=image_dir, + verbose=self.verbose, + num_downscales=self.num_downscales, + ) + num_frames = len(copied_image_paths) + + copied_image_paths = [Path("images/" + copied_image_path.name) for copied_image_path in copied_image_paths] + original_names = [image_path.stem for image_path in image_filenames] + image_filename_map = dict(zip(original_names, copied_image_paths)) + + if self.max_dataset_size > 0 and num_frames != num_orig_images: + summary_log.append(f"Started with {num_frames} images out of {num_orig_images} total") + summary_log.append( + "To change the size of the dataset add the argument [yellow]--max_dataset_size[/yellow] to " + f"larger than the current value ({self.max_dataset_size}), or -1 to use all images." + ) + else: + summary_log.append(f"Started with {num_frames} images") + + # Save json + if num_frames == 0: + CONSOLE.print("[bold red]No images found, exiting") + sys.exit(1) + summary_log.extend( + odm_utils.cameras2nerfds( + image_filename_map=image_filename_map, + cameras_file=cameras_file, + shots_file=shots_file, + output_dir=self.output_dir, + verbose=self.verbose, + ) + ) + + CONSOLE.rule("[bold green]:tada: :tada: :tada: All DONE :tada: :tada: :tada:") + + for summary in summary_log: + CONSOLE.print(summary, justify="center") + CONSOLE.rule() + + Commands = Union[ Annotated[ImagesToNerfstudioDataset, tyro.conf.subcommand(name="images")], Annotated[VideoToNerfstudioDataset, tyro.conf.subcommand(name="video")], @@ -394,6 +482,7 @@ Commands = Union[ Annotated[ProcessMetashape, tyro.conf.subcommand(name="metashape")], Annotated[ProcessRealityCapture, tyro.conf.subcommand(name="realitycapture")], Annotated[ProcessRecord3D, tyro.conf.subcommand(name="record3d")], + Annotated[ProcessODM, tyro.conf.subcommand(name="odm")], ] diff --git a/nerfstudio/scripts/render.py b/nerfstudio/scripts/render.py index f20c4c7b2..2b419a15c 100644 --- a/nerfstudio/scripts/render.py +++ b/nerfstudio/scripts/render.py @@ -18,12 +18,12 @@ render.py """ from __future__ import annotations +import gzip import json import os -import struct import shutil +import struct import sys -import gzip from contextlib import ExitStack, contextmanager from dataclasses import dataclass, field from pathlib import Path @@ -54,11 +54,14 @@ from nerfstudio.cameras.camera_paths import ( get_spiral_path, ) from nerfstudio.cameras.cameras import Cameras, CameraType, RayBundle +from nerfstudio.data.datamanagers.base_datamanager import ( + VanillaDataManager, + VanillaDataManagerConfig, +) from nerfstudio.data.datasets.base_dataset import Dataset -from nerfstudio.data.datamanagers.base_datamanager import VanillaDataManager, VanillaDataManagerConfig +from nerfstudio.data.scene_box import OrientedBox from nerfstudio.data.utils.dataloaders import FixedIndicesEvalDataloader from nerfstudio.engine.trainer import TrainerConfig -from nerfstudio.data.scene_box import OrientedBox from nerfstudio.model_components import renderers from nerfstudio.pipelines.base_pipeline import Pipeline from nerfstudio.utils import colormaps, install_checks @@ -322,6 +325,9 @@ def get_crop_from_json(camera_json: Dict[str, Any]) -> Optional[CropData]: center = camera_json["crop"]["crop_center"] scale = camera_json["crop"]["crop_scale"] rot = (0.0, 0.0, 0.0) if "crop_rot" not in camera_json["crop"] else tuple(camera_json["crop"]["crop_rot"]) + assert len(center) == 3 + assert len(scale) == 3 + assert len(rot) == 3 return CropData( background_color=torch.Tensor([bg_color["r"] / 255.0, bg_color["g"] / 255.0, bg_color["b"] / 255.0]), obb=OrientedBox.from_params(center, rot, scale), diff --git a/nerfstudio/utils/plotly_utils.py b/nerfstudio/utils/plotly_utils.py index 00cab4760..1ab7da1fb 100644 --- a/nerfstudio/utils/plotly_utils.py +++ b/nerfstudio/utils/plotly_utils.py @@ -239,6 +239,7 @@ def get_cube( if isinstance(side_length, float): pts *= side_length / 2.0 else: + assert isinstance(side_length, torch.Tensor) pts[0] *= side_length[0].item() / 2.0 pts[1] *= side_length[1].item() / 2.0 pts[2] *= side_length[2].item() / 2.0 diff --git a/nerfstudio/viewer/app/src/modules/Banner/Banner.jsx b/nerfstudio/viewer/app/src/modules/Banner/Banner.jsx index baedf714d..e2156606a 100644 --- a/nerfstudio/viewer/app/src/modules/Banner/Banner.jsx +++ b/nerfstudio/viewer/app/src/modules/Banner/Banner.jsx @@ -62,7 +62,7 @@ export default function Banner() {