diff --git a/README.md b/README.md index 57d77724b..d2e320041 100644 --- a/README.md +++ b/README.md @@ -59,12 +59,14 @@ # About +_It’s as simple as plug and play with nerfstudio!_ + Nerfstudio provides a simple API that allows for a simplified end-to-end process of creating, training, and testing NeRFs. The library supports a **more interpretable implementation of NeRFs by modularizing each component.** With more modular NeRFs, we hope to create a more user-friendly experience in exploring the technology. -Nerfstudio is a contributor-friendly repo with the goal of building a community where users can more easily build upon each other's contributions. -It’s as simple as plug and play with nerfstudio! +This is a contributor-friendly repo with the goal of building a community where users can more easily build upon each other's contributions. +Nerfstudio initially launched as an opensource project by Berkeley students in [KAIR lab](https://people.eecs.berkeley.edu/~kanazawa/index.html#kair) at [Berkeley AI Research (BAIR)](https://bair.berkeley.edu/) in October 2022 as a part of a research project ([paper](https://arxiv.org/abs/2302.04264)). It is currently developed by Berkeley students and community contributors. We are committed to providing learning resources to help you understand the basics of (if you're just getting started), and keep up-to-date with (if you're a seasoned veteran) all things NeRF. As researchers, we know just how hard it is to get onboarded with this next-gen technology. So we're here to help with tutorials, documentation, and more! diff --git a/docs/developer_guides/debugging_tools/profiling.md b/docs/developer_guides/debugging_tools/profiling.md index ed8796cec..aabada5f4 100644 --- a/docs/developer_guides/debugging_tools/profiling.md +++ b/docs/developer_guides/debugging_tools/profiling.md @@ -23,7 +23,24 @@ class RayGenerator(nn.Module): ... ``` -At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions that have the profiler tag. +Alternatively, you can also time parts of the code: +```python +... + +def forward(self, ray_indices: TensorType["num_rays", 3]) -> RayBundle: + # implementation here + with profiler.time_function("code1"): + # some code here + ... + + with profiler.time_function("code2"): + # some code here + ... + ... +``` + + +At termination of training or end of the training run, the profiler will print out the average execution time for all of the functions or code blocks that have the profiler tag. :::{admonition} Tip :class: info @@ -32,6 +49,14 @@ Use this profiler if there are *specific/individual functions* you want to measu ::: +#### Profiling with PyTorch profiler + +If you want to profile the training or evaluation code and track the memory and CUDA kernel launches, consider using [PyTorch profiler](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html). +It will run the profiler for some selected step numbers, once with `CUDA_LAUNCH_BLOCKING=1`, once with `CUDA_LAUNCH_BLOCKING=0`. +The PyTorch profiler can be enabled with `--logging.profiler=pytorch` flag. +The outputs of the profiler are trace files stored in `{PATH_TO_MODEL_OUTPUT}/profiler_traces`, and can be loaded in Google Chrome by typing `chrome://tracing`. + + #### Profiling with PySpy If you want to profile the entire codebase, consider using [PySpy](https://github.com/benfred/py-spy). diff --git a/docs/developer_guides/new_methods.md b/docs/developer_guides/new_methods.md index 661629d15..26376d655 100644 --- a/docs/developer_guides/new_methods.md +++ b/docs/developer_guides/new_methods.md @@ -10,6 +10,7 @@ We recommend the following file structure: ``` ├── my_method +│ ├── __init__.py │ ├── my_config.py │ ├── custom_pipeline.py [optional] │ ├── custom_model.py [optional] @@ -56,6 +57,9 @@ dependencies = [ "nerfstudio" # you may want to consider pinning the version, ie "nerfstudio==0.1.19" ] +[tool.setuptools.packages.find] +include = ["my_method*"] + [project.entry-points.'nerfstudio.method_configs'] my-method = 'my_method.my_config:MyMethod' ``` diff --git a/docs/developer_guides/viewer/custom_gui.md b/docs/developer_guides/viewer/custom_gui.md index 09275dac1..e11381d02 100644 --- a/docs/developer_guides/viewer/custom_gui.md +++ b/docs/developer_guides/viewer/custom_gui.md @@ -33,16 +33,15 @@ current_value = self.custom_value.value **Callbacks** You can register a callback that will be called whenever a new value for your GUI element is available. For example, one can use a callback to update config parameters when elements are changed: ```python -def on_change_callback(handle: GuiHandle): - self.config.example_parameter = handle.get_value() +def on_change_callback(handle: ViewerCheckbox) -> None: + self.config.example_parameter = handle.value self.custom_checkbox = ViewerCheckbox( name="Checkbox", - default_value = False, - cb_hook = on_change_callback + default_value=False, + cb_hook=on_change_callback, ) ``` -The GuiHandle value is a viser object which you can use to get the new value. **Thread safety** Note that `ViewerElement` values can change asynchronously to model execution. So, it's best practice to store the value of a viewer element once at the beginning @@ -63,13 +62,13 @@ class MyModel(Model): You can write to a viewer element in Python, which provides a convenient way to track values in your code without the need for wandb/tensorboard or relying on `print` statements. ```python -self.custom_value = x +self.custom_value.value = x ``` :::{admonition} Warning :class: warning -Note that updating module state while training can have unexpected side effects. It is up to the user to ensure that GUI actions are safe. In most cases, we recommend wrapping GUI related logic in `if self.training:` to ensure that the effects are only applied during inference. +Updating module state while training can have unexpected side effects. It is up to the user to ensure that GUI actions are safe. Conditioning on `self.training` can help determine whether effects are applied during forward passes for training or rendering. ::: ## Example Elements diff --git a/docs/nerfology/methods/index.md b/docs/nerfology/methods/index.md index 68912bb7b..db2677524 100644 --- a/docs/nerfology/methods/index.md +++ b/docs/nerfology/methods/index.md @@ -31,6 +31,7 @@ The following methods are supported in nerfstudio: Nerfacto Instant-NGP Instruct-NeRF2NeRF + LERF Semantic NeRF-W ``` diff --git a/docs/nerfology/methods/lerf.md b/docs/nerfology/methods/lerf.md new file mode 100644 index 000000000..509c88fac --- /dev/null +++ b/docs/nerfology/methods/lerf.md @@ -0,0 +1,91 @@ +# LERF + +

📎 Language Embedded Radiance Fields 🚜

+ +```{button-link} https://www.lerf.io/ +:color: primary +:outline: +Paper Website +``` + +```{button-link} https://github.com/kerrj/lerf +:color: primary +:outline: +Code +``` + + + +**Grounding CLIP vectors volumetrically inside NeRF allows flexible natural language queries in 3D** + +## Installation + +First install nerfstudio dependencies. Then run: + +```bash +pip install git+https://github.com/kerrj/lerf +``` + +## Running LERF + +Details for running LERF (built with Nerfstudio!) can be found [here](https://github.com/kerrj/lerf). Once installed, run: + +```bash +ns-train lerf --help +``` + +Three variants of LERF are provided: + +| Method | Description | Memory | Quality | +| ----------- | ----------------------------------------------- | ------ | ------- | +| `lerf-big` | LERF with OpenCLIP ViT-L/14 | ~22 GB | Best | +| `lerf` | Model with OpenCLIP ViT-B/16, used in paper | ~15 GB | Good | +| `lerf-lite` | LERF with smaller network and less LERF samples | ~8 GB | Ok | + +`lerf-lite` should work on a single NVIDIA 2080. +`lerf-big` is experimental, and needs further tuning. + +## Method + +LERF enables pixel-aligned queries of the distilled 3D CLIP embeddings without relying on region proposals, masks, or fine-tuning, supporting long-tail open-vocabulary queries hierarchically across the volume. + +### Multi-scale supervision + +To supervise language embeddings, we pre-compute an image pyramid of CLIP features for each training view. Then, each sampled ray during optimization is supervised by interpolating the CLIP embedding within this pyramid. + + + +### LERF Optimization + +LERF optimizes a dense, multi-scale language 3D field by volume rendering CLIP embeddings along training rays, supervising these embeddings with multi-scale CLIP features across multi-view training images. + +Inspired by Distilled Feature Fields (DFF), we use DINO features to regularize CLIP features. This leads to qualitative improvements in object boundaries, as CLIP embeddings in 3D can be sensitive to floaters and regions with few views. + +After optimization, LERF can extract 3D relevancy maps for language queries interactively in real-time. + + + +### Visualizing relevancy for text queries + +Set the "Output Render" type to `relevancy_0`, and enter the text query in the "LERF Positives" textbox (see image). The output render will show the 3D relevancy map for the query. View the [project page](https://lerf.io) for more examples and details about the relevancy map normalization. + +
+ +
+ +## Results + +For results, view the [project page](https://lerf.io)! + +Datasets used in the original paper can be found [here](https://drive.google.com/drive/folders/1LUzwEvBCE19PNYcwfmrG-9FLpZLbi4on?usp=sharing). + +```none +@article{lerf2023, + author = {Kerr, Justin and Kim, Chung Min and Goldberg, Ken and Kanazawa, Angjoo and Tancik, Matthew}, + title = {LERF: Language Embedded Radiance Fields}, + journal = {arXiv preprint arXiv:2303.09553}, + year = {2023}, +} +``` diff --git a/docs/quickstart/custom_dataset.md b/docs/quickstart/custom_dataset.md index 65e6c22d4..a83d59218 100644 --- a/docs/quickstart/custom_dataset.md +++ b/docs/quickstart/custom_dataset.md @@ -341,4 +341,6 @@ The `crop-factor` argument is optional but often very helpful. This is because e -This obscene mesh of human is obviously not helpful in training a nerf so we can remove it by cropping the bottom 20% of the image. This can be done by using the `--crop-factor 0 0.2 0 0` argument. +The pixels representing the distorted hand and head are obviously not useful in training a nerf so we can remove it by cropping the bottom 20% of the image. This can be done by using the `--crop-factor 0 0.2 0 0` argument. + +If cropping only needs to be done from the bottom, you can use the `--crop-bottom [num]` argument which would be the same as doing `--crop-factor 0.0 [num] 0.0 0.0` diff --git a/nerfstudio/cameras/cameras.py b/nerfstudio/cameras/cameras.py index bda664642..0dec377c9 100644 --- a/nerfstudio/cameras/cameras.py +++ b/nerfstudio/cameras/cameras.py @@ -633,7 +633,7 @@ class Cameras(TensorDataclass): if distortion_params is not None: mask = (self.camera_type[true_indices] != CameraType.EQUIRECTANGULAR.value).squeeze(-1) # (num_rays) coord_mask = torch.stack([mask, mask, mask], dim=0) - if mask.any(): + if mask.any() and (distortion_params != 0).any(): coord_stack[coord_mask, :] = camera_utils.radial_and_tangential_undistort( coord_stack[coord_mask, :].reshape(3, -1, 2), distortion_params[mask, :], diff --git a/nerfstudio/configs/base_config.py b/nerfstudio/configs/base_config.py index fd5608b4e..5731091ac 100644 --- a/nerfstudio/configs/base_config.py +++ b/nerfstudio/configs/base_config.py @@ -119,9 +119,11 @@ class LoggingConfig(PrintableConfig): e.g. if 20, averages will be computed over past 20 occurrences.""" local_writer: LocalWriterConfig = LocalWriterConfig(enable=True) """if provided, will print stats locally. if None, will disable printing""" - enable_profiler: bool = True - """whether to enable profiling code; prints speed of functions at the end of a program. - profiler logs run times of functions and prints at end of training""" + profiler: Literal["none", "basic", "pytorch"] = "basic" + """how to profile the code; + "basic" - prints speed of all decorated functions at the end of a program. + "pytorch" - same as basic, but it also traces few training steps. + """ # Viewer related configs @@ -135,6 +137,8 @@ class ViewerConfig(PrintableConfig): """The websocket port to connect to. If None, find an available port.""" websocket_port_default: int = 7007 """The default websocket port to connect to if websocket_port is not specified""" + websocket_host: str = "0.0.0.0" + """The host address to bind the websocket server to.""" num_rays_per_chunk: int = 32768 """number of rays per chunk to render with viewer""" max_num_display_images: int = 512 diff --git a/nerfstudio/configs/method_configs.py b/nerfstudio/configs/method_configs.py index b4e697585..ba645c8c3 100644 --- a/nerfstudio/configs/method_configs.py +++ b/nerfstudio/configs/method_configs.py @@ -21,7 +21,6 @@ from __future__ import annotations from typing import Dict import tyro -from nerfacc import ContractionType from nerfstudio.cameras.camera_optimizers import CameraOptimizerConfig from nerfstudio.configs.base_config import ViewerConfig @@ -247,16 +246,20 @@ method_configs["instant-ngp"] = TrainerConfig( max_num_iterations=30000, mixed_precision=True, pipeline=DynamicBatchPipelineConfig( - datamanager=VanillaDataManagerConfig(dataparser=NerfstudioDataParserConfig(), train_num_rays_per_batch=8192), + datamanager=VanillaDataManagerConfig( + dataparser=NerfstudioDataParserConfig(), + train_num_rays_per_batch=4096, + eval_num_rays_per_batch=4096, + ), model=InstantNGPModelConfig(eval_num_rays_per_chunk=8192), ), optimizers={ "fields": { "optimizer": AdamOptimizerConfig(lr=1e-2, eps=1e-15), - "scheduler": None, + "scheduler": ExponentialDecaySchedulerConfig(lr_final=0.0001, max_steps=200000), } }, - viewer=ViewerConfig(num_rays_per_chunk=64000), + viewer=ViewerConfig(num_rays_per_chunk=1 << 12), vis="viewer", ) @@ -271,9 +274,10 @@ method_configs["instant-ngp-bounded"] = TrainerConfig( datamanager=VanillaDataManagerConfig(dataparser=InstantNGPDataParserConfig(), train_num_rays_per_batch=8192), model=InstantNGPModelConfig( eval_num_rays_per_chunk=8192, - contraction_type=ContractionType.AABB, - render_step_size=0.001, - max_num_samples_per_ray=48, + grid_levels=1, + alpha_thre=0.0, + cone_angle=0.0, + disable_scene_contraction=True, near_plane=0.01, background_color="black", ), @@ -281,10 +285,10 @@ method_configs["instant-ngp-bounded"] = TrainerConfig( optimizers={ "fields": { "optimizer": AdamOptimizerConfig(lr=1e-2, eps=1e-15), - "scheduler": None, + "scheduler": ExponentialDecaySchedulerConfig(lr_final=0.0001, max_steps=200000), } }, - viewer=ViewerConfig(num_rays_per_chunk=64000), + viewer=ViewerConfig(num_rays_per_chunk=1 << 12), vis="viewer", ) @@ -752,9 +756,10 @@ method_configs["nerfplayer-ngp"] = TrainerConfig( datamanager=DepthDataManagerConfig(dataparser=DycheckDataParserConfig(), train_num_rays_per_batch=8192), model=NerfplayerNGPModelConfig( eval_num_rays_per_chunk=8192, - contraction_type=ContractionType.AABB, + grid_levels=1, + alpha_thre=0.0, render_step_size=0.001, - max_num_samples_per_ray=48, + disable_scene_contraction=True, near_plane=0.01, ), ), diff --git a/nerfstudio/data/datamanagers/base_datamanager.py b/nerfstudio/data/datamanagers/base_datamanager.py index 7e2ebedeb..d696519af 100644 --- a/nerfstudio/data/datamanagers/base_datamanager.py +++ b/nerfstudio/data/datamanagers/base_datamanager.py @@ -179,6 +179,7 @@ class DataManager(nn.Module): eval_count (int): the step number of our eval iteration, needs to be incremented manually train_dataset (Dataset): the dataset for the train dataset eval_dataset (Dataset): the dataset for the eval dataset + includes_time (bool): whether the dataset includes time information Additional attributes specific to each subclass are defined in the setup_train and setup_eval functions. @@ -189,6 +190,7 @@ class DataManager(nn.Module): eval_dataset: Optional[Dataset] = None train_sampler: Optional[DistributedSampler] = None eval_sampler: Optional[DistributedSampler] = None + includes_time: bool = False def __init__(self): """Constructor for the DataManager class. @@ -407,6 +409,7 @@ class VanillaDataManager(DataManager): # pylint: disable=abstract-method else: self.config.data = self.config.dataparser.data self.dataparser = self.dataparser_config.setup() + self.includes_time = self.dataparser.includes_time self.train_dataparser_outputs = self.dataparser.get_dataparser_outputs(split="train") self.train_dataset = self.create_train_dataset() diff --git a/nerfstudio/data/dataparsers/base_dataparser.py b/nerfstudio/data/dataparsers/base_dataparser.py index 12367e51f..f92b34172 100644 --- a/nerfstudio/data/dataparsers/base_dataparser.py +++ b/nerfstudio/data/dataparsers/base_dataparser.py @@ -127,9 +127,14 @@ class DataParser: Args: config: datasetparser config containing all information needed to instantiate dataset + + Attributes: + config: datasetparser config containing all information needed to instantiate dataset + includes_time: Does the dataset include time information in the camera poses. """ config: DataParserConfig + includes_time: bool = False def __init__(self, config: DataParserConfig): super().__init__() diff --git a/nerfstudio/data/dataparsers/dnerf_dataparser.py b/nerfstudio/data/dataparsers/dnerf_dataparser.py index c26621e92..d7411b4c6 100644 --- a/nerfstudio/data/dataparsers/dnerf_dataparser.py +++ b/nerfstudio/data/dataparsers/dnerf_dataparser.py @@ -53,6 +53,7 @@ class DNeRF(DataParser): """DNeRF Dataset""" config: DNeRFDataParserConfig + includes_time: bool = True def __init__(self, config: DNeRFDataParserConfig): super().__init__(config=config) diff --git a/nerfstudio/data/dataparsers/dycheck_dataparser.py b/nerfstudio/data/dataparsers/dycheck_dataparser.py index 92cca35ff..7ddf051e1 100644 --- a/nerfstudio/data/dataparsers/dycheck_dataparser.py +++ b/nerfstudio/data/dataparsers/dycheck_dataparser.py @@ -202,6 +202,7 @@ class Dycheck(DataParser): """Dycheck (https://arxiv.org/abs/2210.13445) Dataset `iphone` subset""" config: DycheckDataParserConfig + includes_time: bool = True def __init__(self, config: DycheckDataParserConfig): super().__init__(config=config) diff --git a/nerfstudio/data/dataparsers/instant_ngp_dataparser.py b/nerfstudio/data/dataparsers/instant_ngp_dataparser.py index 2fdc2df85..8b4bf724c 100644 --- a/nerfstudio/data/dataparsers/instant_ngp_dataparser.py +++ b/nerfstudio/data/dataparsers/instant_ngp_dataparser.py @@ -65,6 +65,7 @@ class InstantNGP(DataParser): data_dir = self.config.data image_filenames = [] + mask_filenames = [] poses = [] num_skipped_image_filenames = 0 for frame in meta["frames"]: @@ -87,6 +88,9 @@ class InstantNGP(DataParser): meta["h"] = h image_filenames.append(fname) poses.append(np.array(frame["transform_matrix"])) + if "mask_path" in frame: + mask_fname = data_dir / Path(frame["mask_path"]) + mask_filenames.append(mask_fname) if num_skipped_image_filenames >= 0: CONSOLE.print(f"Skipping {num_skipped_image_filenames} files in dataset split {split}.") assert ( @@ -144,6 +148,7 @@ class InstantNGP(DataParser): image_filenames=image_filenames, cameras=cameras, scene_box=scene_box, + mask_filenames=mask_filenames if len(mask_filenames) > 0 else None, dataparser_scale=self.config.scene_scale, ) diff --git a/nerfstudio/data/pixel_samplers.py b/nerfstudio/data/pixel_samplers.py index 244498df8..b58d99fe3 100644 --- a/nerfstudio/data/pixel_samplers.py +++ b/nerfstudio/data/pixel_samplers.py @@ -96,6 +96,7 @@ class PixelSampler: # pylint: disable=too-few-public-methods indices = self.sample_method(num_rays_per_batch, num_images, image_height, image_width, device=device) c, y, x = (i.flatten() for i in torch.split(indices, 1, dim=-1)) + c, y, x = c.cpu(), y.cpu(), x.cpu() collated_batch = { key: value[c, y, x] for key, value in batch.items() if key != "image_idx" and value is not None } diff --git a/nerfstudio/engine/trainer.py b/nerfstudio/engine/trainer.py index bd34c965d..8925cc93e 100644 --- a/nerfstudio/engine/trainer.py +++ b/nerfstudio/engine/trainer.py @@ -75,6 +75,8 @@ class TrainerConfig(ExperimentConfig): """Maximum number of iterations to run.""" mixed_precision: bool = False """Whether or not to use mixed precision for training.""" + use_grad_scaler: bool = False + """Use gradient scaler even if the automatic mixed precision is disabled.""" save_only_latest_checkpoint: bool = True """Whether to only save the latest checkpoint or all checkpoints.""" # optional parameters if we want to resume training @@ -104,7 +106,7 @@ class Trainer: pipeline: The pipeline object. optimizers: The optimizers object. callbacks: The callbacks object. - is_training: Whether the model is training. + training_state: Current model training state. """ pipeline: VanillaPipeline @@ -118,13 +120,15 @@ class Trainer: self.world_size = world_size self.device: TORCH_DEVICE = "cpu" if world_size == 0 else f"cuda:{local_rank}" self.mixed_precision: bool = self.config.mixed_precision - self.is_training: bool = True + self.use_grad_scaler: bool = self.mixed_precision or self.config.use_grad_scaler + self.training_state: Literal["training", "paused", "completed"] = "training" + if self.device == "cpu": self.mixed_precision = False CONSOLE.print("Mixed precision is disabled for CPU training.") self._start_step: int = 0 # optimizers - self.grad_scaler = GradScaler(enabled=self.mixed_precision) + self.grad_scaler = GradScaler(enabled=self.use_grad_scaler) self.base_dir: Path = config.get_base_dir() # directory to save checkpoints @@ -186,7 +190,7 @@ class Trainer: self.config.logging, max_iter=self.config.max_num_iterations, banner_messages=banner_messages ) writer.put_config(name="config", config_dict=dataclasses.asdict(self.config), step=0) - profiler.setup_profiler(self.config.logging) + profiler.setup_profiler(self.config.logging, writer_log_path) def setup_optimizers(self) -> Optimizers: """Helper to set up the optimizers @@ -219,7 +223,7 @@ class Trainer: num_iterations = self.config.max_num_iterations step = 0 for step in range(self._start_step, self._start_step + num_iterations): - while not self.is_training: + while self.training_state == "paused": time.sleep(0.01) with self.train_lock: with TimeWriter(writer, EventName.ITER_TRAIN_TIME, step=step) as train_t: @@ -283,6 +287,8 @@ class Trainer: CONSOLE.rule() CONSOLE.print("[bold green]:tada: :tada: :tada: Training Finished :tada: :tada: :tada:", justify="center") if not self.config.viewer.quit_on_train_completion: + self.training_state = "completed" + self._train_complete_viewer() CONSOLE.print("Use ctrl+c to quit", justify="center") while True: time.sleep(0.01) @@ -307,7 +313,7 @@ class Trainer: assert self.viewer_state and self.pipeline.datamanager.train_dataset self.viewer_state.init_scene( dataset=self.pipeline.datamanager.train_dataset, - start_train=True, + train_state="training", ) @check_viewer_enabled @@ -326,6 +332,16 @@ class Trainer: time.sleep(0.03) # sleep to allow buffer to reset CONSOLE.log("Viewer failed. Continuing training.") + @check_viewer_enabled + def _train_complete_viewer(self) -> None: + """Let the viewer know that the training is complete""" + assert self.viewer_state is not None + try: + self.viewer_state.training_complete() + except RuntimeError: + time.sleep(0.03) # sleep to allow buffer to reset + CONSOLE.log("Viewer failed. Continuing training.") + @check_viewer_enabled def _update_viewer_rays_per_sec(self, train_t: TimeWriter, vis_t: TimeWriter, step: int) -> None: """Performs update on rays/sec calculation for training diff --git a/nerfstudio/fields/instant_ngp_field.py b/nerfstudio/fields/instant_ngp_field.py index d1b3e2fa7..e205c73f0 100644 --- a/nerfstudio/fields/instant_ngp_field.py +++ b/nerfstudio/fields/instant_ngp_field.py @@ -21,7 +21,6 @@ from typing import Dict, Optional, Tuple import numpy as np import torch -from nerfacc import ContractionType, contract from torch.nn.parameter import Parameter from torchtyping import TensorType @@ -30,6 +29,10 @@ from nerfstudio.data.scene_box import SceneBox from nerfstudio.field_components.activations import trunc_exp from nerfstudio.field_components.embedding import Embedding from nerfstudio.field_components.field_heads import FieldHeadNames +from nerfstudio.field_components.spatial_distortions import ( + SceneContraction, + SpatialDistortion, +) from nerfstudio.fields.base_field import Field, shift_directions_for_tcnn try: @@ -69,16 +72,16 @@ class TCNNInstantNGPField(Field): use_appearance_embedding: Optional[bool] = False, num_images: Optional[int] = None, appearance_embedding_dim: int = 32, - contraction_type: ContractionType = ContractionType.UN_BOUNDED_SPHERE, num_levels: int = 16, log2_hashmap_size: int = 19, max_res: int = 2048, + spatial_distortion: Optional[SpatialDistortion] = SceneContraction(), ) -> None: super().__init__() self.aabb = Parameter(aabb, requires_grad=False) self.geo_feat_dim = geo_feat_dim - self.contraction_type = contraction_type + self.spatial_distortion = spatial_distortion self.use_appearance_embedding = use_appearance_embedding if use_appearance_embedding: @@ -88,7 +91,8 @@ class TCNNInstantNGPField(Field): # TODO: set this properly based on the aabb base_res: int = 16 - per_level_scale = np.exp((np.log(max_res) - np.log(base_res)) / (num_levels - 1)) + features_per_level: int = 2 + growth_factor = np.exp((np.log(max_res) - np.log(base_res)) / (num_levels - 1)) self.direction_encoding = tcnn.Encoding( n_input_dims=3, @@ -104,10 +108,10 @@ class TCNNInstantNGPField(Field): encoding_config={ "otype": "HashGrid", "n_levels": num_levels, - "n_features_per_level": 2, + "n_features_per_level": features_per_level, "log2_hashmap_size": log2_hashmap_size, "base_resolution": base_res, - "per_level_scale": per_level_scale, + "per_level_scale": growth_factor, }, network_config={ "otype": "FullyFusedMLP", @@ -134,9 +138,16 @@ class TCNNInstantNGPField(Field): ) def get_density(self, ray_samples: RaySamples) -> Tuple[TensorType, TensorType]: - positions = ray_samples.frustums.get_positions() + if self.spatial_distortion is not None: + positions = ray_samples.frustums.get_positions() + positions = self.spatial_distortion(positions) + positions = (positions + 2.0) / 4.0 + else: + positions = SceneBox.get_normalized_positions(ray_samples.frustums.get_positions(), self.aabb) + # Make sure the tcnn gets inputs between 0 and 1. + selector = ((positions > 0.0) & (positions < 1.0)).all(dim=-1) + positions = positions * selector[..., None] positions_flat = positions.view(-1, 3) - positions_flat = contract(x=positions_flat, roi=self.aabb, type=self.contraction_type) h = self.mlp_base(positions_flat).view(*ray_samples.frustums.shape, -1) density_before_activation, base_mlp_out = torch.split(h, [1, self.geo_feat_dim], dim=-1) @@ -145,20 +156,18 @@ class TCNNInstantNGPField(Field): # softplus, because it enables high post-activation (float32) density outputs # from smaller internal (float16) parameters. density = trunc_exp(density_before_activation.to(positions)) + density = density * selector[..., None] return density, base_mlp_out def get_outputs( self, ray_samples: RaySamples, density_embedding: Optional[TensorType] = None ) -> Dict[FieldHeadNames, TensorType]: + assert density_embedding is not None directions = shift_directions_for_tcnn(ray_samples.frustums.directions) directions_flat = directions.view(-1, 3) - d = self.direction_encoding(directions_flat) - if density_embedding is None: - positions = SceneBox.get_normalized_positions(ray_samples.frustums.get_positions(), self.aabb) - h = torch.cat([d, positions.view(-1, 3)], dim=-1) - else: - h = torch.cat([d, density_embedding.view(-1, self.geo_feat_dim)], dim=-1) + + h = torch.cat([d, density_embedding.view(-1, self.geo_feat_dim)], dim=-1) if self.use_appearance_embedding: if ray_samples.camera_indices is None: @@ -183,20 +192,5 @@ class TCNNInstantNGPField(Field): step_size: the step size to use for the opacity evaluation. """ density = self.density_fn(positions) - ## TODO: We should scale step size based on the distortion. Currently it uses too much memory. - # aabb_min, aabb_max = self.aabb[0], self.aabb[1] - # if self.contraction_type is not ContractionType.AABB: - # x = (positions - aabb_min) / (aabb_max - aabb_min) - # x = x * 2 - 1 # aabb is at [-1, 1] - # mag = x.norm(dim=-1, keepdim=True) - # mask = mag.squeeze(-1) > 1 - - # dev = (2 * mag - 1) / mag**2 + 2 * x**2 * (1 / mag**3 - (2 * mag - 1) / mag**4) - # dev[~mask] = 1.0 - # dev = torch.clamp(dev, min=1e-6) - # step_size = step_size / dev.norm(dim=-1, keepdim=True) - # else: - # step_size = step_size * (aabb_max - aabb_min) - opacity = density * step_size return opacity diff --git a/nerfstudio/fields/nerfacto_field.py b/nerfstudio/fields/nerfacto_field.py index de0fd7c7d..acc868648 100644 --- a/nerfstudio/fields/nerfacto_field.py +++ b/nerfstudio/fields/nerfacto_field.py @@ -103,7 +103,7 @@ class TCNNNerfactoField(Field): pass_semantic_gradients: bool = False, use_pred_normals: bool = False, use_average_appearance_embedding: bool = False, - spatial_distortion: SpatialDistortion = None, + spatial_distortion: Optional[SpatialDistortion] = None, ) -> None: super().__init__() diff --git a/nerfstudio/fields/nerfplayer_ngp_field.py b/nerfstudio/fields/nerfplayer_ngp_field.py index 2952a4369..bbf7b72e6 100644 --- a/nerfstudio/fields/nerfplayer_ngp_field.py +++ b/nerfstudio/fields/nerfplayer_ngp_field.py @@ -20,7 +20,6 @@ NeRFPlayer (https://arxiv.org/abs/2210.15947) field implementations with Instant from typing import Dict, Optional, Tuple import torch -from nerfacc import ContractionType, contract from torch.nn.parameter import Parameter from torchtyping import TensorType @@ -29,6 +28,10 @@ from nerfstudio.data.scene_box import SceneBox from nerfstudio.field_components.activations import trunc_exp from nerfstudio.field_components.embedding import Embedding from nerfstudio.field_components.field_heads import FieldHeadNames +from nerfstudio.field_components.spatial_distortions import ( + SceneContraction, + SpatialDistortion, +) from nerfstudio.field_components.temporal_grid import TemporalGridEncoder from nerfstudio.fields.base_field import Field, shift_directions_for_tcnn @@ -82,13 +85,13 @@ class NerfplayerNGPField(Field): disable_viewing_dependent: bool = False, num_images: Optional[int] = None, appearance_embedding_dim: int = 32, - contraction_type: ContractionType = ContractionType.UN_BOUNDED_SPHERE, + spatial_distortion: Optional[SpatialDistortion] = SceneContraction(), ) -> None: super().__init__() self.aabb = Parameter(aabb, requires_grad=False) self.geo_feat_dim = geo_feat_dim - self.contraction_type = contraction_type + self.spatial_distortion = spatial_distortion self.use_appearance_embedding = use_appearance_embedding if use_appearance_embedding: @@ -144,9 +147,16 @@ class NerfplayerNGPField(Field): ) def get_density(self, ray_samples: RaySamples) -> Tuple[TensorType, TensorType]: - positions = ray_samples.frustums.get_positions() + if self.spatial_distortion is not None: + positions = ray_samples.frustums.get_positions() + positions = self.spatial_distortion(positions) + positions = (positions + 2.0) / 4.0 + else: + positions = SceneBox.get_normalized_positions(ray_samples.frustums.get_positions(), self.aabb) + # Make sure the tcnn gets inputs between 0 and 1. + selector = ((positions > 0.0) & (positions < 1.0)).all(dim=-1) + positions = positions * selector[..., None] positions_flat = positions.view(-1, 3) - positions_flat = contract(x=positions_flat, roi=self.aabb, type=self.contraction_type) assert ray_samples.times is not None, "Time should be included in the input for NeRFPlayer" times_flat = ray_samples.times.view(-1, 1) @@ -158,28 +168,22 @@ class NerfplayerNGPField(Field): # softplus, because it enables high post-activation (float32) density outputs # from smaller internal (float16) parameters. density = trunc_exp(density_before_activation.to(positions)) + density = density * selector[..., None] return density, base_mlp_out def get_outputs( self, ray_samples: RaySamples, density_embedding: Optional[TensorType] = None ) -> Dict[FieldHeadNames, TensorType]: + assert density_embedding is not None directions = shift_directions_for_tcnn(ray_samples.frustums.directions) directions_flat = directions.view(-1, 3) if self.direction_encoding is not None: d = self.direction_encoding(directions_flat) - if density_embedding is None: - positions = SceneBox.get_normalized_positions(ray_samples.frustums.get_positions(), self.aabb) - h = torch.cat([d, positions.view(-1, 3)], dim=-1) - else: - h = torch.cat([d, density_embedding.view(-1, self.geo_feat_dim)], dim=-1) + h = torch.cat([d, density_embedding.view(-1, self.geo_feat_dim)], dim=-1) else: # viewing direction is disabled - if density_embedding is None: - positions = SceneBox.get_normalized_positions(ray_samples.frustums.get_positions(), self.aabb) - h = positions.view(-1, 3) - else: - h = density_embedding.view(-1, self.geo_feat_dim) + h = density_embedding.view(-1, self.geo_feat_dim) if self.use_appearance_embedding: if ray_samples.camera_indices is None: diff --git a/nerfstudio/model_components/ray_samplers.py b/nerfstudio/model_components/ray_samplers.py index 641da83e1..ff2c62f64 100644 --- a/nerfstudio/model_components/ray_samplers.py +++ b/nerfstudio/model_components/ray_samplers.py @@ -19,9 +19,8 @@ Collection of sampling strategies from abc import abstractmethod from typing import Callable, List, Optional, Tuple, Union -import nerfacc import torch -from nerfacc import OccupancyGrid +from nerfacc import OccGridEstimator from torch import nn from torchtyping import TensorType @@ -382,24 +381,22 @@ class VolumetricSampler(Sampler): def __init__( self, - occupancy_grid: Optional[OccupancyGrid] = None, + occupancy_grid: OccGridEstimator, density_fn: Optional[Callable[[TensorType[..., 3]], TensorType[..., 1]]] = None, - scene_aabb: Optional[TensorType[2, 3]] = None, ) -> None: super().__init__() - self.scene_aabb = scene_aabb + assert occupancy_grid is not None self.density_fn = density_fn self.occupancy_grid = occupancy_grid - if self.scene_aabb is not None: - self.scene_aabb = self.scene_aabb.to("cuda").flatten() - def get_sigma_fn(self, origins, directions) -> Optional[Callable]: + def get_sigma_fn(self, origins, directions, times=None) -> Optional[Callable]: """Returns a function that returns the density of a point. Args: origins: Origins of rays directions: Directions of rays + times: Times at which rays are sampled Returns: Function that returns the density of a point or None if a density function is not provided. """ @@ -412,8 +409,10 @@ class VolumetricSampler(Sampler): def sigma_fn(t_starts, t_ends, ray_indices): t_origins = origins[ray_indices] t_dirs = directions[ray_indices] - positions = t_origins + t_dirs * (t_starts + t_ends) / 2.0 - return density_fn(positions) + positions = t_origins + t_dirs * (t_starts + t_ends)[:, None] / 2.0 + if times is None: + return density_fn(positions).squeeze(-1) + return density_fn(positions, times[ray_indices]).squeeze(-1) return sigma_fn @@ -429,6 +428,7 @@ class VolumetricSampler(Sampler): render_step_size: float, near_plane: float = 0.0, far_plane: Optional[float] = None, + alpha_thre: float = 0.01, cone_angle: float = 0.0, ) -> Tuple[RaySamples, TensorType["total_samples",]]: """Generate ray samples in a bounding box. @@ -438,6 +438,7 @@ class VolumetricSampler(Sampler): render_step_size: Minimum step size to use for rendering near_plane: Near plane for raymarching far_plane: Far plane for raymarching + alpha_thre: Opacity threshold skipping samples. cone_angle: Cone angle for raymarching, set to 0 for uniform marching. Returns: @@ -448,6 +449,7 @@ class VolumetricSampler(Sampler): rays_o = ray_bundle.origins.contiguous() rays_d = ray_bundle.directions.contiguous() + times = ray_bundle.times if ray_bundle.nears is not None and ray_bundle.fars is not None: t_min = ray_bundle.nears.contiguous().reshape(-1) @@ -457,34 +459,33 @@ class VolumetricSampler(Sampler): t_min = None t_max = None + if far_plane is None: + far_plane = 1e10 + if ray_bundle.camera_indices is not None: camera_indices = ray_bundle.camera_indices.contiguous() else: camera_indices = None - - ray_indices, starts, ends = nerfacc.ray_marching( + ray_indices, starts, ends = self.occupancy_grid.sampling( rays_o=rays_o, rays_d=rays_d, t_min=t_min, t_max=t_max, - scene_aabb=self.scene_aabb, - grid=self.occupancy_grid, - # this is a workaround - using density causes crash and damage quality. should be fixed - sigma_fn=None, # self.get_sigma_fn(rays_o, rays_d), + sigma_fn=self.get_sigma_fn(rays_o, rays_d, times), render_step_size=render_step_size, near_plane=near_plane, far_plane=far_plane, stratified=self.training, cone_angle=cone_angle, - alpha_thre=1e-2, + alpha_thre=alpha_thre, ) num_samples = starts.shape[0] if num_samples == 0: # create a single fake sample and update packed_info accordingly # this says the last ray in packed_info has 1 sample, which starts and ends at 1 ray_indices = torch.zeros((1,), dtype=torch.long, device=rays_o.device) - starts = torch.ones((1, 1), dtype=starts.dtype, device=rays_o.device) - ends = torch.ones((1, 1), dtype=ends.dtype, device=rays_o.device) + starts = torch.ones((1,), dtype=starts.dtype, device=rays_o.device) + ends = torch.ones((1,), dtype=ends.dtype, device=rays_o.device) origins = rays_o[ray_indices] dirs = rays_d[ray_indices] @@ -496,8 +497,8 @@ class VolumetricSampler(Sampler): frustums=Frustums( origins=origins, directions=dirs, - starts=starts, - ends=ends, + starts=starts[..., None], + ends=ends[..., None], pixel_area=zeros, ), camera_indices=camera_indices, diff --git a/nerfstudio/model_components/renderers.py b/nerfstudio/model_components/renderers.py index 41c2ec812..9a855bdc9 100644 --- a/nerfstudio/model_components/renderers.py +++ b/nerfstudio/model_components/renderers.py @@ -91,8 +91,12 @@ class RGBRenderer(nn.Module): # Necessary for packed samples from volumetric ray sampler if background_color == "last_sample": raise NotImplementedError("Background color 'last_sample' not implemented for packed samples.") - comp_rgb = nerfacc.accumulate_along_rays(weights, ray_indices, rgb, num_rays) - accumulated_weight = nerfacc.accumulate_along_rays(weights, ray_indices, None, num_rays) + comp_rgb = nerfacc.accumulate_along_rays( + weights[..., 0], values=rgb, ray_indices=ray_indices, n_rays=num_rays + ) + accumulated_weight = nerfacc.accumulate_along_rays( + weights[..., 0], values=None, ray_indices=ray_indices, n_rays=num_rays + ) else: comp_rgb = torch.sum(weights * rgb, dim=-2) accumulated_weight = torch.sum(weights, dim=-2) @@ -217,7 +221,9 @@ class AccumulationRenderer(nn.Module): if ray_indices is not None and num_rays is not None: # Necessary for packed samples from volumetric ray sampler - accumulation = nerfacc.accumulate_along_rays(weights, ray_indices, None, num_rays) + accumulation = nerfacc.accumulate_along_rays( + weights[..., 0], values=None, ray_indices=ray_indices, n_rays=num_rays + ) else: accumulation = torch.sum(weights, dim=-2) return accumulation @@ -274,8 +280,12 @@ class DepthRenderer(nn.Module): if ray_indices is not None and num_rays is not None: # Necessary for packed samples from volumetric ray sampler - depth = nerfacc.accumulate_along_rays(weights, ray_indices, steps, num_rays) - accumulation = nerfacc.accumulate_along_rays(weights, ray_indices, None, num_rays) + depth = nerfacc.accumulate_along_rays( + weights[..., 0], values=steps, ray_indices=ray_indices, n_rays=num_rays + ) + accumulation = nerfacc.accumulate_along_rays( + weights[..., 0], values=None, ray_indices=ray_indices, n_rays=num_rays + ) depth = depth / (accumulation + eps) else: depth = torch.sum(weights * steps, dim=-2) / (torch.sum(weights, -2) + eps) diff --git a/nerfstudio/models/base_model.py b/nerfstudio/models/base_model.py index d6c55856e..24927c532 100644 --- a/nerfstudio/models/base_model.py +++ b/nerfstudio/models/base_model.py @@ -181,12 +181,12 @@ class Model(nn.Module): ray_bundle = camera_ray_bundle.get_row_major_sliced_ray_bundle(start_idx, end_idx) outputs = self.forward(ray_bundle=ray_bundle) for output_name, output in outputs.items(): # type: ignore + if not torch.is_tensor(output): + # TODO: handle lists of tensors as well + continue outputs_lists[output_name].append(output) outputs = {} for output_name, outputs_list in outputs_lists.items(): - if not torch.is_tensor(outputs_list[0]): - # TODO: handle lists of tensors as well - continue outputs[output_name] = torch.cat(outputs_list).view(image_height, image_width, -1) # type: ignore return outputs diff --git a/nerfstudio/models/instant_ngp.py b/nerfstudio/models/instant_ngp.py index 2bdf85b6a..d8119db5c 100644 --- a/nerfstudio/models/instant_ngp.py +++ b/nerfstudio/models/instant_ngp.py @@ -23,7 +23,6 @@ from typing import Dict, List, Optional, Tuple, Type import nerfacc import torch -from nerfacc import ContractionType from torch.nn import Parameter from torchmetrics import PeakSignalNoiseRatio from torchmetrics.functional import structural_similarity_index_measure @@ -37,7 +36,8 @@ from nerfstudio.engine.callbacks import ( TrainingCallbackLocation, ) from nerfstudio.field_components.field_heads import FieldHeadNames -from nerfstudio.fields.instant_ngp_field import TCNNInstantNGPField +from nerfstudio.field_components.spatial_distortions import SceneContraction +from nerfstudio.fields.nerfacto_field import TCNNNerfactoField from nerfstudio.model_components.losses import MSELoss from nerfstudio.model_components.ray_samplers import VolumetricSampler from nerfstudio.model_components.renderers import ( @@ -61,19 +61,19 @@ class InstantNGPModelConfig(ModelConfig): """Whether to create a scene collider to filter rays.""" collider_params: Optional[Dict[str, float]] = None """Instant NGP doesn't use a collider.""" - max_num_samples_per_ray: int = 24 - """Number of samples in field evaluation.""" grid_resolution: int = 128 """Resolution of the grid used for the field.""" + grid_levels: int = 4 + """Levels of the grid used for the field.""" max_res: int = 2048 """Maximum resolution of the hashmap for the base mlp.""" log2_hashmap_size: int = 19 """Size of the hashmap for the base mlp""" - contraction_type: ContractionType = ContractionType.UN_BOUNDED_SPHERE - """Contraction type used for spatial deformation of the field.""" + alpha_thre: float = 0.01 + """Threshold for opacity skipping.""" cone_angle: float = 0.004 """Should be set to 0.0 for blender scenes but 1./256 for real scenes.""" - render_step_size: float = 0.01 + render_step_size: Optional[float] = None """Minimum step size for rendering.""" near_plane: float = 0.05 """How far along ray to start sampling.""" @@ -83,6 +83,8 @@ class InstantNGPModelConfig(ModelConfig): """Whether to use an appearance embedding.""" background_color: Literal["random", "black", "white"] = "random" """The color that is given to untrained areas.""" + disable_scene_contraction: bool = False + """Whether to disable scene contraction or not.""" class NGPModel(Model): @@ -93,7 +95,7 @@ class NGPModel(Model): """ config: InstantNGPModelConfig - field: TCNNInstantNGPField + field: TCNNNerfactoField def __init__(self, config: InstantNGPModelConfig, **kwargs) -> None: super().__init__(config=config, **kwargs) @@ -102,28 +104,33 @@ class NGPModel(Model): """Set the fields and modules.""" super().populate_modules() - self.field = TCNNInstantNGPField( + if self.config.disable_scene_contraction: + scene_contraction = None + else: + scene_contraction = SceneContraction(order=float("inf")) + + self.field = TCNNNerfactoField( aabb=self.scene_box.aabb, - contraction_type=self.config.contraction_type, - use_appearance_embedding=self.config.use_appearance_embedding, num_images=self.num_train_data, log2_hashmap_size=self.config.log2_hashmap_size, max_res=self.config.max_res, + spatial_distortion=scene_contraction, ) self.scene_aabb = Parameter(self.scene_box.aabb.flatten(), requires_grad=False) - # Occupancy Grid - self.occupancy_grid = nerfacc.OccupancyGrid( + if self.config.render_step_size is None: + # auto step size: ~1000 samples in the base level grid + self.config.render_step_size = ((self.scene_aabb[3:] - self.scene_aabb[:3]) ** 2).sum().sqrt().item() / 1000 + # Occupancy Grid. + self.occupancy_grid = nerfacc.OccGridEstimator( roi_aabb=self.scene_aabb, resolution=self.config.grid_resolution, - contraction_type=self.config.contraction_type, + levels=self.config.grid_levels, ) # Sampler - vol_sampler_aabb = self.scene_box.aabb if self.config.contraction_type == ContractionType.AABB else None self.sampler = VolumetricSampler( - scene_aabb=vol_sampler_aabb, occupancy_grid=self.occupancy_grid, density_fn=self.field.density_fn, ) @@ -145,11 +152,9 @@ class NGPModel(Model): self, training_callback_attributes: TrainingCallbackAttributes ) -> List[TrainingCallback]: def update_occupancy_grid(step: int): - # TODO: needs to get access to the sampler, on how the step size is determinated at each x. See - # https://github.com/KAIR-BAIR/nerfacc/blob/127223b11401125a9fce5ce269bb0546ee4de6e8/examples/train_ngp_nerf.py#L190-L213 - self.occupancy_grid.every_n_step( + self.occupancy_grid.update_every_n_steps( step=step, - occ_eval_fn=lambda x: self.field.get_opacity(x, self.config.render_step_size), + occ_eval_fn=lambda x: self.field.density_fn(x) * self.config.render_step_size, ) return [ @@ -177,6 +182,7 @@ class NGPModel(Model): near_plane=self.config.near_plane, far_plane=self.config.far_plane, render_step_size=self.config.render_step_size, + alpha_thre=self.config.alpha_thre, cone_angle=self.config.cone_angle, ) @@ -185,11 +191,12 @@ class NGPModel(Model): # accumulation packed_info = nerfacc.pack_info(ray_indices, num_rays) weights = nerfacc.render_weight_from_density( + t_starts=ray_samples.frustums.starts[..., 0], + t_ends=ray_samples.frustums.ends[..., 0], + sigmas=field_outputs[FieldHeadNames.DENSITY][..., 0], packed_info=packed_info, - sigmas=field_outputs[FieldHeadNames.DENSITY], - t_starts=ray_samples.frustums.starts, - t_ends=ray_samples.frustums.ends, - ) + )[0] + weights = weights[..., None] rgb = self.renderer_rgb( rgb=field_outputs[FieldHeadNames.RGB], @@ -201,13 +208,11 @@ class NGPModel(Model): weights=weights, ray_samples=ray_samples, ray_indices=ray_indices, num_rays=num_rays ) accumulation = self.renderer_accumulation(weights=weights, ray_indices=ray_indices, num_rays=num_rays) - alive_ray_mask = accumulation.squeeze(-1) > 0 outputs = { "rgb": rgb, "accumulation": accumulation, "depth": depth, - "alive_ray_mask": alive_ray_mask, # the rays we kept from sampler "num_samples_per_ray": packed_info[:, 1], } return outputs @@ -221,8 +226,7 @@ class NGPModel(Model): def get_loss_dict(self, outputs, batch, metrics_dict=None): image = batch["image"].to(self.device) - mask = outputs["alive_ray_mask"] - rgb_loss = self.rgb_loss(image[mask], outputs["rgb"][mask]) + rgb_loss = self.rgb_loss(image, outputs["rgb"]) loss_dict = {"rgb_loss": rgb_loss} return loss_dict @@ -237,12 +241,10 @@ class NGPModel(Model): outputs["depth"], accumulation=outputs["accumulation"], ) - alive_ray_mask = colormaps.apply_colormap(outputs["alive_ray_mask"]) combined_rgb = torch.cat([image, rgb], dim=1) combined_acc = torch.cat([acc], dim=1) combined_depth = torch.cat([depth], dim=1) - combined_alive_ray_mask = torch.cat([alive_ray_mask], dim=1) # Switch images from [H, W, C] to [1, C, H, W] for metrics computations image = torch.moveaxis(image, -1, 0)[None, ...] @@ -260,7 +262,6 @@ class NGPModel(Model): "img": combined_rgb, "accumulation": combined_acc, "depth": combined_depth, - "alive_ray_mask": combined_alive_ray_mask, } return metrics_dict, images_dict diff --git a/nerfstudio/models/nerfplayer_ngp.py b/nerfstudio/models/nerfplayer_ngp.py index 09f68d67f..4f884674f 100644 --- a/nerfstudio/models/nerfplayer_ngp.py +++ b/nerfstudio/models/nerfplayer_ngp.py @@ -19,11 +19,10 @@ Implementation of NeRFPlayer (https://arxiv.org/abs/2210.15947) with InstantNGP from __future__ import annotations from dataclasses import dataclass, field -from typing import Type +from typing import List, Type import nerfacc import torch -from nerfacc import ContractionType from torch.nn import Parameter from torchmetrics import PeakSignalNoiseRatio from torchmetrics.functional import structural_similarity_index_measure @@ -31,7 +30,13 @@ from torchmetrics.image.lpip import LearnedPerceptualImagePatchSimilarity from typing_extensions import Literal from nerfstudio.cameras.rays import RayBundle +from nerfstudio.engine.callbacks import ( + TrainingCallback, + TrainingCallbackAttributes, + TrainingCallbackLocation, +) from nerfstudio.field_components.field_heads import FieldHeadNames +from nerfstudio.field_components.spatial_distortions import SceneContraction from nerfstudio.fields.nerfplayer_ngp_field import NerfplayerNGPField from nerfstudio.model_components.losses import MSELoss from nerfstudio.model_components.ray_samplers import VolumetricSampler @@ -80,6 +85,8 @@ class NerfplayerNGPModelConfig(InstantNGPModelConfig): """The training background color that is given to untrained areas.""" disable_viewing_dependent: bool = True """Disable viewing dependent effects.""" + disable_scene_contraction: bool = False + """Whether to disable scene contraction or not.""" class NerfplayerNGPModel(NGPModel): @@ -96,9 +103,13 @@ class NerfplayerNGPModel(NGPModel): """Set the fields and modules.""" Model.populate_modules(self) + if self.config.disable_scene_contraction: + scene_contraction = None + else: + scene_contraction = SceneContraction(order=float("inf")) + self.field = NerfplayerNGPField( aabb=self.scene_box.aabb, - contraction_type=self.config.contraction_type, use_appearance_embedding=self.config.use_appearance_embedding, num_images=self.num_train_data, temporal_dim=self.config.temporal_dim, @@ -107,21 +118,20 @@ class NerfplayerNGPModel(NGPModel): log2_hashmap_size=self.config.log2_hashmap_size, base_resolution=self.config.base_resolution, disable_viewing_dependent=self.config.disable_viewing_dependent, + spatial_distortion=scene_contraction, ) self.scene_aabb = Parameter(self.scene_box.aabb.flatten(), requires_grad=False) # Occupancy Grid - self.occupancy_grid = nerfacc.OccupancyGrid( + self.occupancy_grid = nerfacc.OccGridEstimator( roi_aabb=self.scene_aabb, resolution=self.config.grid_resolution, - contraction_type=self.config.contraction_type, + levels=self.config.grid_levels, ) # Sampler - vol_sampler_aabb = self.scene_box.aabb if self.config.contraction_type == ContractionType.AABB else None self.sampler = VolumetricSampler( - scene_aabb=vol_sampler_aabb, occupancy_grid=self.occupancy_grid, density_fn=self.field.density_fn, ) # need to update the density_fn later during forward (for input time) @@ -140,11 +150,26 @@ class NerfplayerNGPModel(NGPModel): self.lpips = LearnedPerceptualImagePatchSimilarity(normalize=True) self.temporal_distortion = True # for viewer + def get_training_callbacks( + self, training_callback_attributes: TrainingCallbackAttributes + ) -> List[TrainingCallback]: + def update_occupancy_grid(step: int): + self.occupancy_grid.update_every_n_steps( + step=step, + occ_eval_fn=lambda x: self.field.get_opacity(x, self.config.render_step_size), + ) + + return [ + TrainingCallback( + where_to_run=[TrainingCallbackLocation.BEFORE_TRAIN_ITERATION], + update_every_num_iters=1, + func=update_occupancy_grid, + ), + ] + def get_outputs(self, ray_bundle: RayBundle): num_rays = len(ray_bundle) - # update the density_fn of the sampler so that the density is time aware - self.sampler.density_fn = lambda x: self.field.density_fn(x, ray_bundle.times) with torch.no_grad(): ray_samples, ray_indices = self.sampler( ray_bundle=ray_bundle, @@ -152,6 +177,7 @@ class NerfplayerNGPModel(NGPModel): far_plane=self.config.far_plane, render_step_size=self.config.render_step_size, cone_angle=self.config.cone_angle, + alpha_thre=self.config.alpha_thre, ) field_outputs = self.field(ray_samples) @@ -160,10 +186,11 @@ class NerfplayerNGPModel(NGPModel): packed_info = nerfacc.pack_info(ray_indices, num_rays) weights = nerfacc.render_weight_from_density( packed_info=packed_info, - sigmas=field_outputs[FieldHeadNames.DENSITY], - t_starts=ray_samples.frustums.starts, - t_ends=ray_samples.frustums.ends, - ) + sigmas=field_outputs[FieldHeadNames.DENSITY][..., 0], + t_starts=ray_samples.frustums.starts[..., 0], + t_ends=ray_samples.frustums.ends[..., 0], + )[0] + weights = weights[..., None] # update bgcolor in the renderer; usually random color for training and fixed color for inference if self.training: @@ -180,13 +207,11 @@ class NerfplayerNGPModel(NGPModel): weights=weights, ray_samples=ray_samples, ray_indices=ray_indices, num_rays=num_rays ) accumulation = self.renderer_accumulation(weights=weights, ray_indices=ray_indices, num_rays=num_rays) - alive_ray_mask = accumulation.squeeze(-1) > 0 outputs = { "rgb": rgb, "accumulation": accumulation, "depth": depth, - "alive_ray_mask": alive_ray_mask, # the rays we kept from sampler "num_samples_per_ray": packed_info[:, 1], } # adding them to outputs for calculating losses @@ -199,10 +224,9 @@ class NerfplayerNGPModel(NGPModel): def get_loss_dict(self, outputs, batch, metrics_dict=None): image = batch["image"].to(self.device) - mask = outputs["alive_ray_mask"] - rgb_loss = self.rgb_loss(image[mask], outputs["rgb"][mask]) + rgb_loss = self.rgb_loss(image, outputs["rgb"]) loss_dict = {"rgb_loss": rgb_loss} - if "depth_image" in batch.keys() and self.config.depth_weight > 0: + if "depth_image" in batch.keys() and self.config.depth_weight > 0 and self.training: mask = batch["depth_image"] != 0 # First we calculate the depth value, just like most of the papers. loss_dict["depth_loss"] = (outputs["depth"][mask] - batch["depth_image"][mask]).abs().mean() @@ -228,6 +252,6 @@ class NerfplayerNGPModel(NGPModel): density_min_mask = (gt_depth_packed - steps > margin) & (gt_depth_packed != 0) loss_dict["depth_loss"] += (outputs["sigmas"][density_min_mask[..., 0]].pow(2)).mean() * 1e-2 loss_dict["depth_loss"] *= self.config.depth_weight - if self.config.temporal_tv_weight > 0: + if self.config.temporal_tv_weight > 0 and self.training: loss_dict["temporal_tv_loss"] = self.config.temporal_tv_weight * self.field.mlp_base.get_temporal_tv_loss() return loss_dict diff --git a/nerfstudio/process_data/base_converter_to_nerfstudio_dataset.py b/nerfstudio/process_data/base_converter_to_nerfstudio_dataset.py new file mode 100644 index 000000000..a1578cc05 --- /dev/null +++ b/nerfstudio/process_data/base_converter_to_nerfstudio_dataset.py @@ -0,0 +1,46 @@ +# Copyright 2022 The Nerfstudio Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Base class to process images or video into a nerfstudio dataset +""" + +from abc import ABC, abstractmethod +from dataclasses import dataclass +from pathlib import Path + + +@dataclass +class BaseConverterToNerfstudioDataset(ABC): + """Base class to process images or video into a nerfstudio dataset""" + + data: Path + """Path the data, either a video file or a directory of images.""" + output_dir: Path + """Path to the output directory.""" + verbose: bool = False + """If True, print extra logging.""" + + def __post_init__(self) -> None: + self.output_dir.mkdir(parents=True, exist_ok=True) + self.image_dir.mkdir(parents=True, exist_ok=True) + + @property + def image_dir(self) -> Path: # pylint: disable=missing-function-docstring + return self.output_dir / "images" + + @abstractmethod + def main(self) -> None: + """This method implements the conversion logic for each type of data""" + raise NotImplementedError("the main method for conversion needs to be implemented") diff --git a/nerfstudio/process_data/colmap_converter_to_nerfstudio_dataset.py b/nerfstudio/process_data/colmap_converter_to_nerfstudio_dataset.py new file mode 100644 index 000000000..4effdc377 --- /dev/null +++ b/nerfstudio/process_data/colmap_converter_to_nerfstudio_dataset.py @@ -0,0 +1,228 @@ +# Copyright 2022 The Nerfstudio Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Base class to processes a video or image sequence to a nerfstudio compatible dataset.""" + +from dataclasses import dataclass +from pathlib import Path +from typing import Dict, List, Optional, Tuple + +from rich.console import Console +from typing_extensions import Literal + +from nerfstudio.process_data import colmap_utils, hloc_utils, process_data_utils +from nerfstudio.process_data.base_converter_to_nerfstudio_dataset import ( + BaseConverterToNerfstudioDataset, +) +from nerfstudio.process_data.process_data_utils import CAMERA_MODELS +from nerfstudio.utils import install_checks + +CONSOLE = Console(width=120) + + +@dataclass +class ColmapConverterToNerfstudioDataset(BaseConverterToNerfstudioDataset): + """Base class to process images or video into a nerfstudio dataset using colmap""" + + camera_type: Literal["perspective", "fisheye", "equirectangular"] = "perspective" + """Camera model to use.""" + matching_method: Literal["exhaustive", "sequential", "vocab_tree"] = "vocab_tree" + """Feature matching method to use. Vocab tree is recommended for a balance of speed + and accuracy. Exhaustive is slower but more accurate. Sequential is faster but + should only be used for videos.""" + sfm_tool: Literal["any", "colmap", "hloc"] = "any" + """Structure from motion tool to use. Colmap will use sift features, hloc can use + many modern methods such as superpoint features and superglue matcher""" + refine_pixsfm: bool = False + """If True, runs refinement using Pixel Perfect SFM. + Only works with hloc sfm_tool""" + feature_type: Literal[ + "any", + "sift", + "superpoint", + "superpoint_aachen", + "superpoint_max", + "superpoint_inloc", + "r2d2", + "d2net-ss", + "sosnet", + "disk", + ] = "any" + """Type of feature to use.""" + matcher_type: Literal[ + "any", + "NN", + "superglue", + "superglue-fast", + "NN-superpoint", + "NN-ratio", + "NN-mutual", + "adalam", + ] = "any" + """Matching algorithm.""" + num_downscales: int = 3 + """Number of times to downscale the images. Downscales by 2 each time. For example a value of 3 will downscale the + images by 2x, 4x, and 8x.""" + skip_colmap: bool = False + """If True, skips COLMAP and generates transforms.json if possible.""" + skip_image_processing: bool = False + """If True, skips copying and downscaling of images and only runs COLMAP if possible and enabled""" + colmap_model_path: Path = Path("colmap/sparse/0") + """Optionally sets the path of the colmap model. Used only when --skip-colmap is set to True. The path is relative + to the output directory. + """ + colmap_cmd: str = "colmap" + """How to call the COLMAP executable.""" + images_per_equirect: Literal[8, 14] = 8 + """Number of samples per image to take from each equirectangular image. + Used only when camera-type is equirectangular. + """ + crop_factor: Tuple[float, float, float, float] = (0.0, 0.0, 0.0, 0.0) + """Portion of the image to crop. All values should be in [0,1]. (top, bottom, left, right)""" + crop_bottom: float = 0.0 + """Portion of the image to crop from the bottom. + Can be used instead of `crop-factor 0.0 [num] 0.0 0.0` Should be in [0,1]. + """ + gpu: bool = True + """If True, use GPU.""" + use_sfm_depth: bool = False + """If True, export and use depth maps induced from SfM points.""" + include_depth_debug: bool = False + """If --use-sfm-depth and this flag is True, also export debug images showing Sf overlaid upon input images.""" + + @staticmethod + def default_colmap_path() -> Path: # pylint: disable=missing-function-docstring + return Path("colmap/sparse/0") + + @property + def absolute_colmap_model_path(self) -> Path: # pylint: disable=missing-function-docstring + return self.output_dir / self.colmap_model_path + + @property + def absolute_colmap_path(self) -> Path: # pylint: disable=missing-function-docstring + return self.output_dir / "colmap" + + def _save_transforms( + self, + num_frames: int, + image_id_to_depth_path: Optional[Dict[int, Path]] = None, + camera_mask_path: Optional[Path] = None, + image_rename_map: Optional[Dict[str, str]] = None, + ) -> List[str]: + """Save colmap transforms into the output folder + + Args: + image_id_to_depth_path: When including sfm-based depth, embed these depth file paths in the exported json + image_rename_map: Use these image names instead of the names embedded in the COLMAP db + """ + summary_log = [] + if (self.absolute_colmap_model_path / "cameras.bin").exists(): + with CONSOLE.status("[bold yellow]Saving results to transforms.json", spinner="balloon"): + num_matched_frames = colmap_utils.colmap_to_json( + recon_dir=self.absolute_colmap_model_path, + output_dir=self.output_dir, + image_id_to_depth_path=image_id_to_depth_path, + camera_mask_path=camera_mask_path, + image_rename_map=image_rename_map, + ) + summary_log.append(f"Colmap matched {num_matched_frames} images") + summary_log.append(colmap_utils.get_matching_summary(num_frames, num_matched_frames)) + + else: + CONSOLE.log( + "[bold yellow]Warning: Could not find existing COLMAP results. " "Not generating transforms.json" + ) + return summary_log + + def _export_depth(self) -> Tuple[Optional[Dict[int, Path]], List[str]]: + """If SFM is used for creating depth image, this method will create the depth images from image in + `self.image_dir`. + + Returns: + Depth file paths indexed by COLMAP image id, logs + """ + summary_log = [] + if self.use_sfm_depth: + depth_dir = self.output_dir / "depth" + depth_dir.mkdir(parents=True, exist_ok=True) + image_id_to_depth_path = colmap_utils.create_sfm_depth( + recon_dir=self.output_dir / self.default_colmap_path(), + output_dir=depth_dir, + include_depth_debug=self.include_depth_debug, + input_images_dir=self.image_dir, + verbose=self.verbose, + ) + summary_log.append( + process_data_utils.downscale_images( + depth_dir, + self.num_downscales, + folder_name="depths", + nearest_neighbor=True, + verbose=self.verbose, + ) + ) + return image_id_to_depth_path, summary_log + return None, summary_log + + def _run_colmap(self, mask_path: Optional[Path] = None): + """ + Args: + mask_path: Path to the camera mask. Defaults to None. + """ + self.absolute_colmap_path.mkdir(parents=True, exist_ok=True) + + (sfm_tool, feature_type, matcher_type,) = process_data_utils.find_tool_feature_matcher_combination( + self.sfm_tool, self.feature_type, self.matcher_type + ) + # check that sfm_tool is hloc if using refine_pixsfm + if self.refine_pixsfm: + assert sfm_tool == "hloc", "refine_pixsfm only works with sfm_tool hloc" + + if sfm_tool == "colmap": + colmap_utils.run_colmap( + image_dir=self.image_dir, + colmap_dir=self.absolute_colmap_path, + camera_model=CAMERA_MODELS[self.camera_type], + camera_mask_path=mask_path, + gpu=self.gpu, + verbose=self.verbose, + matching_method=self.matching_method, + colmap_cmd=self.colmap_cmd, + ) + elif sfm_tool == "hloc": + if mask_path is not None: + raise RuntimeError("Cannot use a mask with hloc. Please remove the cropping options " "and try again.") + hloc_utils.run_hloc( + image_dir=self.image_dir, + colmap_dir=self.absolute_colmap_path, + camera_model=CAMERA_MODELS[self.camera_type], + verbose=self.verbose, + matching_method=self.matching_method, + feature_type=feature_type, + matcher_type=matcher_type, + refine_pixsfm=self.refine_pixsfm, + ) + else: + raise RuntimeError("Invalid combination of sfm_tool, feature_type, and matcher_type, " "exiting") + + def __post_init__(self) -> None: + super().__post_init__() + install_checks.check_ffmpeg_installed() + install_checks.check_colmap_installed() + + if self.crop_bottom < 0.0 or self.crop_bottom > 1: + raise RuntimeError("crop_bottom must be set between 0 and 1.") + + if self.crop_bottom > 0.0: + self.crop_factor = (0.0, self.crop_bottom, 0.0, 0.0) diff --git a/nerfstudio/process_data/images_to_nerstudio_dataset.py b/nerfstudio/process_data/images_to_nerstudio_dataset.py new file mode 100644 index 000000000..477c94c20 --- /dev/null +++ b/nerfstudio/process_data/images_to_nerstudio_dataset.py @@ -0,0 +1,106 @@ +# Copyright 2022 The Nerfstudio Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Processes an image sequence to a nerfstudio compatible dataset.""" + +from dataclasses import dataclass +from typing import Optional + +from rich.console import Console + +from nerfstudio.process_data import equirect_utils, process_data_utils +from nerfstudio.process_data.colmap_converter_to_nerfstudio_dataset import ( + ColmapConverterToNerfstudioDataset, +) + +CONSOLE = Console(width=120) + + +@dataclass +class ImagesToNerfstudioDataset(ColmapConverterToNerfstudioDataset): + """Process images into a nerfstudio dataset. + + 1. Scales images to a specified size. + 2. Calculates the camera poses for each image using `COLMAP `_. + """ + + def main(self) -> None: # pylint: disable=R0915 + """Process images into a nerfstudio dataset.""" + + # pylint: disable=too-many-statements + require_cameras_exist = False + if self.colmap_model_path != ColmapConverterToNerfstudioDataset.default_colmap_path(): + if not self.skip_colmap: + raise RuntimeError("The --colmap-model-path can only be used when --skip-colmap is not set.") + if not (self.output_dir / self.colmap_model_path).exists(): + raise RuntimeError(f"The colmap-model-path {self.output_dir / self.colmap_model_path} does not exist.") + require_cameras_exist = True + + image_rename_map: Optional[dict[str, str]] = None + + # Generate planar projections if equirectangular + if self.camera_type == "equirectangular": + pers_size = equirect_utils.compute_resolution_from_equirect(self.data, self.images_per_equirect) + CONSOLE.log(f"Generating {self.images_per_equirect} {pers_size} sized images per equirectangular image") + self.data = equirect_utils.generate_planar_projections_from_equirectangular( + self.data, pers_size, self.images_per_equirect, crop_factor=self.crop_factor + ) + + summary_log = [] + + # Copy and downscale images + if not self.skip_image_processing: + # Copy images to output directory + image_rename_map_paths = process_data_utils.copy_images( + self.data, image_dir=self.image_dir, crop_factor=self.crop_factor, verbose=self.verbose + ) + image_rename_map = dict((a.name, b.name) for a, b in image_rename_map_paths.items()) + num_frames = len(image_rename_map) + summary_log.append(f"Starting with {num_frames} images") + + # Downscale images + summary_log.append( + process_data_utils.downscale_images(self.image_dir, self.num_downscales, verbose=self.verbose) + ) + else: + num_frames = len(process_data_utils.list_images(self.data)) + if num_frames == 0: + raise RuntimeError("No usable images in the data folder.") + summary_log.append(f"Starting with {num_frames} images") + + # Run COLMAP + if not self.skip_colmap: + require_cameras_exist = True + self._run_colmap() + # Colmap uses renamed images + image_rename_map = None + + # Export depth maps + image_id_to_depth_path, log_tmp = self._export_depth() + summary_log += log_tmp + + if require_cameras_exist and not (self.absolute_colmap_model_path / "cameras.bin").exists(): + raise RuntimeError(f"Could not find existing COLMAP results ({self.colmap_model_path / 'cameras.bin'}).") + + summary_log += self._save_transforms( + num_frames, + image_id_to_depth_path, + None, + image_rename_map, + ) + + CONSOLE.log("[bold green]:tada: :tada: :tada: All DONE :tada: :tada: :tada:") + + for summary in summary_log: + CONSOLE.log(summary) diff --git a/nerfstudio/process_data/video_to_nerfstudio_dataset.py b/nerfstudio/process_data/video_to_nerfstudio_dataset.py new file mode 100644 index 000000000..405231fcf --- /dev/null +++ b/nerfstudio/process_data/video_to_nerfstudio_dataset.py @@ -0,0 +1,122 @@ +# Copyright 2022 The Nerfstudio Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Processes a video to a nerfstudio compatible dataset.""" + +import shutil +from dataclasses import dataclass + +from rich.console import Console + +from nerfstudio.process_data import equirect_utils, process_data_utils +from nerfstudio.process_data.colmap_converter_to_nerfstudio_dataset import ( + ColmapConverterToNerfstudioDataset, +) + +CONSOLE = Console(width=120) + + +@dataclass +class VideoToNerfstudioDataset(ColmapConverterToNerfstudioDataset): + """Process videos into a nerfstudio dataset. + + This script does the following: + + 1. Converts the video into images. + 2. Scales images to a specified size. + 3. Calculates the camera poses for each image using `COLMAP `_. + """ + + num_frames_target: int = 300 + """Target number of frames to use for the dataset, results may not be exact.""" + percent_radius_crop: float = 1.0 + """Create circle crop mask. The radius is the percent of the image diagonal.""" + + def main(self) -> None: # pylint: disable=R0915 + """Process video into a nerfstudio dataset.""" + + summary_log = [] + # Convert video to images + if self.camera_type == "equirectangular": + # create temp images folder to store the equirect and perspective images + temp_image_dir = self.output_dir / "temp_images" + temp_image_dir.mkdir(parents=True, exist_ok=True) + summary_log, num_extracted_frames = process_data_utils.convert_video_to_images( + self.data, + image_dir=temp_image_dir, + num_frames_target=self.num_frames_target, + crop_factor=(0.0, 0.0, 0.0, 0.0), + verbose=self.verbose, + ) + else: + summary_log, num_extracted_frames = process_data_utils.convert_video_to_images( + self.data, + image_dir=self.image_dir, + num_frames_target=self.num_frames_target, + crop_factor=self.crop_factor, + verbose=self.verbose, + ) + + # Generate planar projections if equirectangular + if self.camera_type == "equirectangular": + perspective_image_size = equirect_utils.compute_resolution_from_equirect( + self.output_dir / "temp_images", self.images_per_equirect + ) + + equirect_utils.generate_planar_projections_from_equirectangular( + self.output_dir / "temp_images", + perspective_image_size, + self.images_per_equirect, + crop_factor=self.crop_factor, + ) + + # copy the perspective images to the image directory + process_data_utils.copy_images( + self.output_dir / "temp_images" / "planar_projections", + image_dir=self.output_dir / "images", + verbose=False, + ) + + # remove the temp_images folder + shutil.rmtree(self.output_dir / "temp_images", ignore_errors=True) + + # # Create mask + mask_path = process_data_utils.save_mask( + image_dir=self.image_dir, + num_downscales=self.num_downscales, + crop_factor=(0.0, 0.0, 0.0, 0.0), + percent_radius=self.percent_radius_crop, + ) + if mask_path is not None: + summary_log.append(f"Saved mask to {mask_path}") + + # # Downscale images + summary_log.append( + process_data_utils.downscale_images(self.image_dir, self.num_downscales, verbose=self.verbose) + ) + + # Run Colmap + if not self.skip_colmap: + self._run_colmap(mask_path) + + # Export depth maps + image_id_to_depth_path, log_tmp = self._export_depth() + summary_log += log_tmp + + summary_log += self._save_transforms(num_extracted_frames, image_id_to_depth_path, mask_path) + + CONSOLE.log("[bold green]:tada: :tada: :tada: All DONE :tada: :tada: :tada:") + + for summary in summary_log: + CONSOLE.log(summary) diff --git a/nerfstudio/utils/decorators.py b/nerfstudio/utils/decorators.py index 6f92ac679..356369f82 100644 --- a/nerfstudio/utils/decorators.py +++ b/nerfstudio/utils/decorators.py @@ -42,7 +42,7 @@ def check_profiler_enabled(func: Callable) -> Callable: def wrapper(self, *args, **kwargs): ret = None - if self.config.enable_profiler: + if self.config.profiler != "none": ret = func(self, *args, **kwargs) return ret diff --git a/nerfstudio/utils/math.py b/nerfstudio/utils/math.py index f54b6509a..89a4d3250 100644 --- a/nerfstudio/utils/math.py +++ b/nerfstudio/utils/math.py @@ -262,7 +262,10 @@ def intersect_aabb( try: import nerfacc # pylint: disable=import-outside-toplevel - t_min, t_max = nerfacc.ray_aabb_intersect(origins, directions, aabb) + t_min, t_max, _ = nerfacc.ray_aabb_intersect( + origins, directions, aabb[None, :], near_plane=0, far_plane=1e10, miss_value=1e10 + ) + t_min, t_max = t_min.squeeze(-1), t_max.squeeze(-1) except: # pylint: disable=bare-except t_min, t_max = _intersect_aabb(origins, directions, aabb, max_bound=1e10, invalid_value=1e10) _USE_NERFACC = False diff --git a/nerfstudio/utils/profiler.py b/nerfstudio/utils/profiler.py index c1fc96dcb..5d5599de6 100644 --- a/nerfstudio/utils/profiler.py +++ b/nerfstudio/utils/profiler.py @@ -17,10 +17,17 @@ Profiler base class and functionality """ from __future__ import annotations +import functools +import os import time -from typing import Callable +import typing +from collections import deque +from contextlib import ContextDecorator, contextmanager +from pathlib import Path +from typing import Callable, Dict, List, Optional, Tuple, Union from rich.console import Console +from torch.profiler import ProfilerActivity, profile, record_function from nerfstudio.configs import base_config as cfg from nerfstudio.utils import comms @@ -33,32 +40,124 @@ from nerfstudio.utils.decorators import ( CONSOLE = Console(width=120) PROFILER = [] +PYTORCH_PROFILER = None -def time_function(func: Callable) -> Callable: - """Decorator: time a function call""" +class time_function(ContextDecorator): # pylint: disable=invalid-name + """Decorator/Context manager: time a function call or a block of code""" - def wrapper(*args, **kwargs): - start = time.time() - ret = func(*args, **kwargs) + def __init__(self, name: Union[str, Callable]): + # NOTE: This is a workaround for the fact that the __new__ method of a ContextDecorator + # is not picked up by VSCode intellisense + self.name: str = typing.cast(str, name) + self.start = None + self._profiler_contexts = deque() + self._function_call_args: Optional[Tuple[Tuple, Dict]] = None + + def __new__(cls, func: Union[str, Callable]): + instance = super().__new__(cls) + if isinstance(func, str): + instance.__init__(func) + return instance + if callable(func): + instance.__init__(func.__qualname__) + return instance(func) + raise ValueError(f"Argument func of type {type(func)} is not a string or a callable.") + + def __enter__(self): + self.start = time.time() + if PYTORCH_PROFILER is not None: + args, kwargs = tuple(), {} + if self._function_call_args is not None: + args, kwargs = self._function_call_args + ctx = PYTORCH_PROFILER.record_function(self.name, *args, **kwargs) + ctx.__enter__() # pylint: disable=no-member + self._profiler_contexts.append(ctx) + if self._function_call_args is None: + ctx = record_function(self.name) + ctx.__enter__() + self._profiler_contexts.append(ctx) + + def __exit__(self, *args, **kwargs): + while self._profiler_contexts: + context = self._profiler_contexts.pop() + context.__exit__(*args, **kwargs) if PROFILER: - class_str = func.__qualname__ - PROFILER[0].update_time(class_str, start, time.time()) - return ret + PROFILER[0].update_time(self.name, self.start, time.time()) - return wrapper + def __call__(self, func: Callable): + @functools.wraps(func) + def inner(*args, **kwargs): + self._function_call_args = (args, kwargs) + with self: + out = func(*args, **kwargs) + self._function_call_args = None + return out + + return inner def flush_profiler(config: cfg.LoggingConfig): """Method that checks if profiler is enabled before flushing""" - if config.enable_profiler and PROFILER: + if config.profiler != "none" and PROFILER: PROFILER[0].print_profile() -def setup_profiler(config: cfg.LoggingConfig): +def setup_profiler(config: cfg.LoggingConfig, log_dir: Path): """Initialization of profilers""" + global PYTORCH_PROFILER # pylint: disable=global-statement if comms.is_main_process(): PROFILER.append(Profiler(config)) + if config.profiler == "pytorch": + PYTORCH_PROFILER = PytorchProfiler(log_dir) + + +class PytorchProfiler: # pylint: disable=too-few-public-methods + """ + Wrapper for Pytorch Profiler + """ + + def __init__(self, output_path: Path, trace_steps: Optional[List[int]] = None): + self.output_path = output_path / "profiler_traces" + if trace_steps is None: + # Some arbitrary steps which likely do not overlap with steps usually chosen to run callbacks + trace_steps = [12, 17] + self.trace_steps = trace_steps + + @contextmanager + def record_function(self, function: str, *args, **_kwargs): + """ + Context manager that records a function call and saves the trace to a json file. + Traced functions are: train_iteration, eval_iteration + """ + if function.endswith("train_iteration") or function.endswith("eval_iteration"): + step = args[1] + assert isinstance(step, int) + assert len(args) == 2 + stage = function.split(".")[-1].split("_")[0] + if step in self.trace_steps: + launch_kernel_blocking = self.trace_steps.index(step) % 2 == 0 + backup_lb_var = "" + if launch_kernel_blocking: + backup_lb_var = os.environ.get("CUDA_LAUNCH_BLOCKING", "") + os.environ["CUDA_LAUNCH_BLOCKING"] = "1" + with profile( + activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], + record_shapes=True, + with_stack=True, + profile_memory=True, + ) as prof: + yield None + if launch_kernel_blocking: + os.environ["CUDA_LAUNCH_BLOCKING"] = backup_lb_var + self.output_path.mkdir(parents=True, exist_ok=True) + prof.export_chrome_trace( + str(self.output_path / f"trace_{stage}_{step}{'_blocking' if launch_kernel_blocking else ''}.json") + ) + return + # Functions are recorded automatically + yield None + return @decorate_all([check_profiler_enabled, check_main_thread]) diff --git a/nerfstudio/viewer/app/package.json b/nerfstudio/viewer/app/package.json index 349b25249..a232ef355 100644 --- a/nerfstudio/viewer/app/package.json +++ b/nerfstudio/viewer/app/package.json @@ -1,7 +1,7 @@ { "name": "viewer", "homepage": ".", - "version": "23-04-10-0", + "version": "23-04-18-0", "private": true, "dependencies": { "@emotion/react": "^11.10.4", @@ -84,6 +84,13 @@ "wait-on": "^6.0.1" }, "resolutions": { - "nth-check": "^2.0.1" + "babel-loader": "^9.1.2", + "json5": "^2.2.3", + "loader-utils": "^3.2.1", + "nth-check": "^2.0.1", + "@pmmmwh/react-refresh-webpack-plugin": "^0.5.10", + "recursive-readdir": "^2.2.3", + "resolve-url-loader": "^5.0.0", + "yaml": "^2.2.2" } } diff --git a/nerfstudio/viewer/app/src/index.scss b/nerfstudio/viewer/app/src/index.scss index a3af6c244..83e09b372 100644 --- a/nerfstudio/viewer/app/src/index.scss +++ b/nerfstudio/viewer/app/src/index.scss @@ -17,6 +17,7 @@ code { .App { position: relative; height: 100vh; + background-color: #1d222a; } $base-color1: #292d39; @@ -358,8 +359,32 @@ $sidebar-width: 100% !default; border: $custom-border; } -.RenderWindow { - z-index: 1; +#not-connected-overlay { + position: absolute; + background-color: #1d222a; + height: 100%; + width: 100%; + text-align: center; + margin: 0; + padding: 0; + z-index: 10; + opacity: 0.8; +} + +#not-connected-overlay-text { + position: absolute; + left: 50%; + top: 50%; + transform: translate(-50%, -50%); + background-color: #393939; + border-radius: 10px; + padding-left: 15px; + padding-right: 15px; + padding-top: 5px; + padding-bottom: 5px; + color: #ff6969; + font-size: 18px; + z-index: 10; } .ViewerWindow-render-crop-container { diff --git a/nerfstudio/viewer/app/src/modules/RenderWindow/RenderWindow.jsx b/nerfstudio/viewer/app/src/modules/RenderWindow/RenderWindow.jsx deleted file mode 100644 index 598a13298..000000000 --- a/nerfstudio/viewer/app/src/modules/RenderWindow/RenderWindow.jsx +++ /dev/null @@ -1,14 +0,0 @@ -import * as React from 'react'; - -export default function RenderWindow() { - return ( -
- Render window -
- ); -} diff --git a/nerfstudio/viewer/app/src/modules/SidePanel/CameraPanel/CameraPanel.jsx b/nerfstudio/viewer/app/src/modules/SidePanel/CameraPanel/CameraPanel.jsx index 8a727be27..720501734 100644 --- a/nerfstudio/viewer/app/src/modules/SidePanel/CameraPanel/CameraPanel.jsx +++ b/nerfstudio/viewer/app/src/modules/SidePanel/CameraPanel/CameraPanel.jsx @@ -48,6 +48,7 @@ import { get_curve_object_from_cameras, get_transform_matrix } from './curve'; import { ViserWebSocketContext, sendWebsocketMessage, + makeThrottledMessageSender, } from '../../WebSocket/ViserWebSocket'; import RenderModal from '../../RenderModal'; import LoadPathModal from '../../LoadPathModal'; @@ -65,6 +66,100 @@ function set_camera_position(camera, matrix) { mat.decompose(camera.position, camera.quaternion, camera.scale); } +function RenderTimeSelector(props) { + const throttled_time_message_sender = props.throttled_time_message_sender; + const disabled = props.disabled; + const isGlobal = props.isGlobal; + const camera = props.camera; + const globalRenderTime = props.globalRenderTime; + const setGlobalRenderTime = props.setGlobalRenderTime; + const applyAll = props.applyAll; + const setAllCameraRenderTime = props.setAllCameraRenderTime; + + const getRenderTimeLabel = () => { + if (!isGlobal) { + return camera.renderTime; + } + camera.renderTime = globalRenderTime; + return globalRenderTime; + }; + + const [UIRenderTime, setUIRenderTime] = React.useState( + isGlobal ? globalRenderTime : getRenderTimeLabel(), + ); + + const [valid, setValid] = React.useState(true); + + useEffect(() => { + setUIRenderTime(getRenderTimeLabel()); + throttled_time_message_sender({ + type: 'TimeConditionMessage', + time: camera.renderTime, + }); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [camera, globalRenderTime]); + + const setRndrTime = (val) => { + if (!isGlobal) { + camera.renderTime = val; + } else { + camera.renderTime = val; + setGlobalRenderTime(val); + } + + if (applyAll) { + setAllCameraRenderTime(val); + } + throttled_time_message_sender({ + type: 'TimeConditionMessage', + time: camera.renderTime, + }); + }; + + const handleValidation = (e) => { + const valueFloat = parseFloat(e.target.value); + let valueStr = String(valueFloat); + if (e.target.value >= 0 && e.target.value <= 1) { + setValid(true); + if (valueFloat === 1.0) { + valueStr = '1.0'; + } + if (valueFloat === 0.0) { + valueStr = '0.0'; + } + setUIRenderTime(parseFloat(valueStr)); + setRndrTime(parseFloat(valueStr)); + } else { + setValid(false); + } + }; + + return ( + setUIRenderTime(e.target.value)} + onBlur={(e) => handleValidation(e)} + disabled={disabled} + sx={{ + input: { + WebkitTextFillColor: `${disabled ? '#24B6FF' : '#EBEBEB'} !important`, + color: `${disabled ? '#24B6FF' : '#EBEBEB'} !important`, + }, + }} + value={UIRenderTime} + error={!valid} + helperText={!valid ? 'RenderTime should be between 0.0 and 1.0' : ''} + variant="standard" + /> + ); +} + function FovSelector(props) { const fovLabel = props.fovLabel; const setFovLabel = props.setFovLabel; @@ -200,6 +295,7 @@ function FovSelector(props) { } function CameraList(props) { + const throttled_time_message_sender = props.throttled_time_message_sender; const sceneTree = props.sceneTree; const cameras = props.cameras; const camera_main = props.camera_main; @@ -211,7 +307,6 @@ function CameraList(props) { const cameraProperties = props.cameraProperties; const setCameraProperties = props.setCameraProperties; const isAnimated = props.isAnimated; - const dispatch = props.dispatch; // eslint-disable-next-line no-unused-vars const slider_value = props.slider_value; const set_slider_value = props.set_slider_value; @@ -411,13 +506,19 @@ function CameraList(props) { fovLabel={fovLabel} setFovLabel={setFovLabel} camera={camera} - dispatch={dispatch} disabled={!isAnimated('FOV')} isGlobal={false} - changeMain={false} /> )} - {!isAnimated('FOV') && ( + {isAnimated('RenderTime') && ( + + )} + {!isAnimated('FOV') && !isAnimated('RenderTime') && (

Animated camera properties will show up here!

@@ -455,7 +556,10 @@ export default function CameraPanel(props) { ); const viser_websocket = useContext(ViserWebSocketContext); + const throttled_time_message_sender = + makeThrottledMessageSender(viser_websocket); const DEFAULT_FOV = 50; + const DEFAULT_RENDER_TIME = 0.0; // react state const [cameras, setCameras] = React.useState([]); @@ -471,6 +575,8 @@ export default function CameraPanel(props) { const [load_path_modal_open, setLoadPathModalOpen] = React.useState(false); const [animate, setAnimate] = React.useState(new Set()); const [globalFov, setGlobalFov] = React.useState(DEFAULT_FOV); + const [globalRenderTime, setGlobalRenderTime] = + React.useState(DEFAULT_RENDER_TIME); // leva store const cameraPropsStore = useCreateStore(); @@ -506,6 +612,10 @@ export default function CameraPanel(props) { const crop_center = useSelector((state) => state.renderingState.crop_center); const crop_scale = useSelector((state) => state.renderingState.crop_scale); + const display_render_time = useSelector( + (state) => state.renderingState.use_time_conditioning, + ); + const setRenderHeight = (value) => { dispatch({ type: 'write', @@ -561,7 +671,7 @@ export default function CameraPanel(props) { const camera_main_copy = camera_main.clone(); camera_main_copy.aspect = 1.0; camera_main_copy.fov = globalFov; - camera_main_copy.renderTime = 0.0; + camera_main_copy.renderTime = globalRenderTime; const new_camera_properties = new Map(); camera_main_copy.properties = new_camera_properties; new_camera_properties.set('FOV', globalFov); @@ -800,14 +910,18 @@ export default function CameraPanel(props) { let lookat = null; let up = null; let fov = null; + let render_time = null; position = curve_object.curve_positions.getPoint(point); lookat = curve_object.curve_lookats.getPoint(point); up = curve_object.curve_ups.getPoint(point); fov = curve_object.curve_fovs.getPoint(point).z; + render_time = curve_object.curve_render_times.getPoint(point).z; + render_time = Math.max(Math.min(render_time, 1.0), 0.0); // clamp time values to [0, 1] const mat = get_transform_matrix(position, lookat, up); set_camera_position(camera_render, mat); setFieldOfView(fov); setGlobalFov(fov); + setGlobalRenderTime(render_time); } // eslint-disable-next-line react-hooks/exhaustive-deps }, [slider_value, render_height, render_width]); @@ -847,11 +961,21 @@ export default function CameraPanel(props) { const mat = get_transform_matrix(position, lookat, up); - camera_path.push({ - camera_to_world: mat.transpose().elements, // convert from col-major to row-major matrix - fov, - aspect: camera_render.aspect, - }); + if (display_render_time) { + const renderTime = curve_object.curve_render_times.getPoint(pt).z; + camera_path.push({ + camera_to_world: mat.transpose().elements, // convert from col-major to row-major matrix + fov, + aspect: camera_render.aspect, + render_time: Math.max(Math.min(renderTime, 1.0), 0.0), // clamp time values to [0, 1] + }); + } else { + camera_path.push({ + camera_to_world: mat.transpose().elements, // convert from col-major to row-major matrix + fov, + aspect: camera_render.aspect, + }); + } } const keyframes = []; @@ -1026,6 +1150,12 @@ export default function CameraPanel(props) { } }; + const setAllCameraRenderTime = (val) => { + for (let i = 0; i < cameras.length; i += 1) { + cameras[i].renderTime = val; + } + }; + return (
@@ -1090,6 +1220,52 @@ export default function CameraPanel(props) { />
+ {display_render_time && ( +
+ + + + +
+ )} {camera_type !== 'equirectangular' && (
@@ -1345,6 +1521,7 @@ export default function CameraPanel(props) {
state.renderingState.isTraining); + const training_state = useSelector( + (state) => state.renderingState.training_state, + ); const sceneTree = props.sceneTree; const isWebsocketConnected = useSelector( @@ -53,23 +56,38 @@ export default function StatusPanel(props: StatusPanelProps) { }, [camera_choice, is_scene_visible]); const handlePlayChange = () => { + let new_state = null; + if (training_state === 'training') { + new_state = 'paused'; + } else if (training_state === 'paused') { + new_state = 'training'; + } else { + return; + } dispatch({ type: 'write', - path: 'renderingState/isTraining', - data: !isTraining, + path: 'renderingState/training_state', + data: new_state, }); sendWebsocketMessage(viser_websocket, { - type: 'IsTrainingMessage', - is_training: !isTraining, + type: 'TrainingStateMessage', + training_state: new_state, }); }; - const is_training_text = isTraining ? 'Pause Training' : 'Resume Training'; - const training_icon = isTraining ? : ; - - const websocket_connected_text = isWebsocketConnected - ? 'Renderer Connected' - : 'Renderer Disconnected'; - const websocket_connected_color = isWebsocketConnected ? 'success' : 'error'; + let is_training_text = ''; + let training_icon = null; + let color = 'secondary'; + if (training_state === 'training') { + is_training_text = 'Pause Training'; + training_icon = ; + } else if (training_state === 'paused') { + is_training_text = 'Resume Training'; + training_icon = ; + } else { + is_training_text = 'Training Complete'; + color = 'success'; + training_icon = ; + } return (
@@ -77,7 +95,7 @@ export default function StatusPanel(props: StatusPanelProps) {
-
); } diff --git a/nerfstudio/viewer/app/src/modules/ViewerWindow/ViewerWindow.jsx b/nerfstudio/viewer/app/src/modules/ViewerWindow/ViewerWindow.jsx index 38c6cee0e..7415f09a0 100644 --- a/nerfstudio/viewer/app/src/modules/ViewerWindow/ViewerWindow.jsx +++ b/nerfstudio/viewer/app/src/modules/ViewerWindow/ViewerWindow.jsx @@ -12,7 +12,6 @@ import SyncOutlinedIcon from '@mui/icons-material/SyncOutlined'; import ThreeDRotationIcon from '@mui/icons-material/ThreeDRotation'; import VideoCameraBackIcon from '@mui/icons-material/VideoCameraBackOutlined'; import { isEqual } from 'lodash'; -import RenderWindow from '../RenderWindow/RenderWindow'; import { makeThrottledMessageSender, ViserWebSocketContext, @@ -187,6 +186,19 @@ export default function ViewerWindow(props) { // start the three.js rendering loop // when the DOM is ready useEffect(() => { + document.getElementById("background-image").onload = function () { + if (scene) { + const oldBackground = scene.background; + const texture = new THREE.Texture(this); + texture.minFilter = THREE.LinearFilter; + texture.magFilter = THREE.NearestFilter; + texture.needsUpdate = true; + scene.background = texture; + if (oldBackground) { + oldBackground.dispose(); + } + } + } myRef.current.append(renderer.domElement); render(); // eslint-disable-next-line react-hooks/exhaustive-deps @@ -203,17 +215,17 @@ export default function ViewerWindow(props) { let crop_h; const render_aspect = render_width / render_height; const viewport_aspect = viewport_width / viewport_height; - let render_viewport_apsect_ratio = null; + let render_viewport_aspect_ratio = null; if (render_aspect > viewport_aspect) { // render width is the limiting factor crop_w = viewport_width; crop_h = viewport_width / render_aspect; - render_viewport_apsect_ratio = viewport_aspect / render_aspect; + render_viewport_aspect_ratio = viewport_aspect / render_aspect; } else { // render height is the limiting factor crop_w = viewport_height * render_aspect; crop_h = viewport_height; - render_viewport_apsect_ratio = 1.0; + render_viewport_aspect_ratio = 1.0; } let display = null; @@ -233,7 +245,7 @@ export default function ViewerWindow(props) { // such that the rendered video will match correctly if (camera_choice !== 'Main Camera') { const fl = 1.0 / Math.tan((field_of_view * Math.PI) / 360); - const fl_new = fl * render_viewport_apsect_ratio; + const fl_new = fl * render_viewport_aspect_ratio; const fov = Math.atan(1 / fl_new) / (Math.PI / 360); sceneTree.metadata.camera.fov = fov; } else { @@ -303,7 +315,17 @@ export default function ViewerWindow(props) { return ( <> - +
+ +
+
diff --git a/nerfstudio/viewer/app/src/modules/WebSocket/ViserMessages.tsx b/nerfstudio/viewer/app/src/modules/WebSocket/ViserMessages.tsx index 52328f50f..e0e713727 100644 --- a/nerfstudio/viewer/app/src/modules/WebSocket/ViserMessages.tsx +++ b/nerfstudio/viewer/app/src/modules/WebSocket/ViserMessages.tsx @@ -78,9 +78,9 @@ interface DatasetImageMessage { idx: string; json: any; } -interface IsTrainingMessage { - type: 'IsTrainingMessage'; - is_training: boolean; +interface TrainingStateMessage { + type: 'TrainingStateMessage'; + training_state: 'training' | 'paused' | 'completed'; } interface CameraPathPayloadMessage { type: 'CameraPathPayloadMessage'; @@ -109,6 +109,13 @@ interface StatusMessage { interface SaveCheckpointMessage { type: 'SaveCheckpointMessage'; } +interface UseTimeConditioningMessage { + type: 'UseTimeConditioningMessage'; +} +interface TimeConditionMessage { + type: 'TimeConditionMessage'; + time: number; +} export type Message = | BackgroundImageMessage @@ -122,10 +129,12 @@ export type Message = | CameraMessage | SceneBoxMessage | DatasetImageMessage - | IsTrainingMessage + | TrainingStateMessage | CameraPathPayloadMessage | CameraPathOptionsRequest | CameraPathsMessage | CropParamsMessage | StatusMessage - | SaveCheckpointMessage; + | SaveCheckpointMessage + | UseTimeConditioningMessage + | TimeConditionMessage; diff --git a/nerfstudio/viewer/app/src/modules/WebSocket/ViserWebSocket.tsx b/nerfstudio/viewer/app/src/modules/WebSocket/ViserWebSocket.tsx index 7816815e6..6bc1e312d 100644 --- a/nerfstudio/viewer/app/src/modules/WebSocket/ViserWebSocket.tsx +++ b/nerfstudio/viewer/app/src/modules/WebSocket/ViserWebSocket.tsx @@ -184,11 +184,11 @@ function handleMessage( break; } // Set training value. - case 'IsTrainingMessage': { + case 'TrainingStateMessage': { dispatch({ type: 'write', - path: 'renderingState/isTraining', - data: message.is_training, + path: 'renderingState/training_state', + data: message.training_state, }); break; } @@ -258,6 +258,24 @@ function handleMessage( }); break; } + // Handle time conditioning messages. + case 'UseTimeConditioningMessage': { + console.log('HERERERE'); + dispatch({ + type: 'write', + path: 'renderingState/use_time_conditioning', + data: true, + }); + break; + } + case 'TimeConditionMessage': { + dispatch({ + type: 'write', + path: 'renderingState/time_condition', + data: message.time, + }); + break; + } default: { console.log('Received message did not match any known types:', message); break; @@ -269,7 +287,7 @@ export function ViserWebSocket({ children }: { children: React.ReactNode }) { const dispatch = useDispatch(); const store = useStore(); - const ws = React.useRef(); + const ws = React.useRef(null); const websocket_url = useSelector( (state) => state.websocketState.websocket_url, @@ -286,8 +304,19 @@ export function ViserWebSocket({ children }: { children: React.ReactNode }) { ws.current = new WebSocket(websocket_url); + const connecting_timeout = setTimeout(() => { + if ( + ws.current != null && + ws.current.readyState === WebSocket.CONNECTING + ) { + ws.current.close(); + console.log('WebSocket connection timed out'); + } + }, 5000); // timeout after 5 seconds + ws.current.onopen = () => { console.log(`Viser connected! ${websocket_url}`); + clearTimeout(connecting_timeout); dispatch({ type: 'write', path: 'websocketState/isConnected', @@ -324,6 +353,11 @@ export function ViserWebSocket({ children }: { children: React.ReactNode }) { } } }; + + // add websocket error handling + ws.current.onerror = (err) => { + console.log('Websocket error: ', err); + }; } let timeout = setTimeout(tryConnect, 500); diff --git a/nerfstudio/viewer/app/src/modules/WebSocketUrlField.jsx b/nerfstudio/viewer/app/src/modules/WebSocketUrlField.jsx index f93b528fc..d25b8ba2a 100644 --- a/nerfstudio/viewer/app/src/modules/WebSocketUrlField.jsx +++ b/nerfstudio/viewer/app/src/modules/WebSocketUrlField.jsx @@ -29,6 +29,8 @@ export default function WebSocketUrlField() { } }; + const currentHost = `${window.location.protocol}//${window.location.host}`; + return (
- viewer.nerf.studio?websocket_url={websocket_url} + {currentHost}?websocket_url={websocket_url}
); diff --git a/nerfstudio/viewer/app/src/reducer.js b/nerfstudio/viewer/app/src/reducer.js index 1fd0f740d..65cb2f810 100644 --- a/nerfstudio/viewer/app/src/reducer.js +++ b/nerfstudio/viewer/app/src/reducer.js @@ -40,7 +40,7 @@ const initialState = { field_of_view: 50, camera_type: 'perspective', - isTraining: true, + training_state: 'training', step: 0, eval_res: '?', @@ -55,6 +55,9 @@ const initialState = { crop_bg_color: { r: 38, g: 42, b: 55 }, crop_scale: [2.0, 2.0, 2.0], crop_center: [0.0, 0.0, 0.0], + + // Time options + use_time_conditioning: false, }, // the scene state sceneState: { diff --git a/nerfstudio/viewer/app/src/themes/theme.ts b/nerfstudio/viewer/app/src/themes/theme.ts index b789c3899..0f6f2044c 100644 --- a/nerfstudio/viewer/app/src/themes/theme.ts +++ b/nerfstudio/viewer/app/src/themes/theme.ts @@ -4,6 +4,7 @@ export const appTheme = createTheme({ palette: { primary: { main: '#EEEEEE' }, secondary: { main: '#FFD369' }, + success: { main: '#5daf5d' }, text: { primary: '#EEEEEE', secondary: '#FFD369', diff --git a/nerfstudio/viewer/app/yarn.lock b/nerfstudio/viewer/app/yarn.lock index 484c9a3be..24d156e1f 100644 --- a/nerfstudio/viewer/app/yarn.lock +++ b/nerfstudio/viewer/app/yarn.lock @@ -1604,7 +1604,7 @@ "@jridgewell/sourcemap-codec" "^1.4.10" "@jridgewell/trace-mapping" "^0.3.9" -"@jridgewell/resolve-uri@^3.0.3": +"@jridgewell/resolve-uri@3.1.0", "@jridgewell/resolve-uri@^3.0.3": version "3.1.0" resolved "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.0.tgz" integrity sha512-F2msla3tad+Mfht5cJq7LSXcdudKTWCVYUgw6pLFOOHSTtZlj6SWNYAp+AhuqLmWdBO2X5hPrLcu8cVP8fy28w== @@ -1622,11 +1622,19 @@ "@jridgewell/gen-mapping" "^0.3.0" "@jridgewell/trace-mapping" "^0.3.9" -"@jridgewell/sourcemap-codec@^1.4.10": +"@jridgewell/sourcemap-codec@1.4.14", "@jridgewell/sourcemap-codec@^1.4.10": version "1.4.14" resolved "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.4.14.tgz" integrity sha512-XPSJHWmi394fuUuzDnGz1wiKqWfo1yXecHQMRf2l6hztTO+nPru658AyDngaBe7isIxEkRsPR3FZh+s7iVa4Uw== +"@jridgewell/trace-mapping@^0.3.17": + version "0.3.18" + resolved "https://registry.yarnpkg.com/@jridgewell/trace-mapping/-/trace-mapping-0.3.18.tgz#25783b2086daf6ff1dcb53c9249ae480e4dd4cd6" + integrity sha512-w+niJYzMHdd7USdiH2U6869nqhD2nbfZXND5Yp93qIbEmnDNk7PD48o+YchRVpzMU7M6jVCbenTR7PA1FLQ9pA== + dependencies: + "@jridgewell/resolve-uri" "3.1.0" + "@jridgewell/sourcemap-codec" "1.4.14" + "@jridgewell/trace-mapping@^0.3.7", "@jridgewell/trace-mapping@^0.3.9": version "0.3.14" resolved "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.14.tgz" @@ -1897,18 +1905,18 @@ "@nodelib/fs.scandir" "2.1.5" fastq "^1.6.0" -"@pmmmwh/react-refresh-webpack-plugin@^0.5.3": - version "0.5.7" - resolved "https://registry.npmjs.org/@pmmmwh/react-refresh-webpack-plugin/-/react-refresh-webpack-plugin-0.5.7.tgz" - integrity sha512-bcKCAzF0DV2IIROp9ZHkRJa6O4jy7NlnHdWL3GmcUxYWNjLXkK5kfELELwEfSP5hXPfVL/qOGMAROuMQb9GG8Q== +"@pmmmwh/react-refresh-webpack-plugin@^0.5.10", "@pmmmwh/react-refresh-webpack-plugin@^0.5.3": + version "0.5.10" + resolved "https://registry.yarnpkg.com/@pmmmwh/react-refresh-webpack-plugin/-/react-refresh-webpack-plugin-0.5.10.tgz#2eba163b8e7dbabb4ce3609ab5e32ab63dda3ef8" + integrity sha512-j0Ya0hCFZPd4x40qLzbhGsh9TMtdb+CJQiso+WxLOPNasohq9cc5SNUcwsZaRH6++Xh91Xkm/xHCkuIiIu0LUA== dependencies: ansi-html-community "^0.0.8" common-path-prefix "^3.0.0" - core-js-pure "^3.8.1" + core-js-pure "^3.23.3" error-stack-parser "^2.0.6" find-up "^5.0.0" html-entities "^2.1.0" - loader-utils "^2.0.0" + loader-utils "^2.0.4" schema-utils "^3.0.0" source-map "^0.7.3" @@ -2176,9 +2184,9 @@ "@hapi/hoek" "^9.0.0" "@sideway/formula@^3.0.0": - version "3.0.0" - resolved "https://registry.npmjs.org/@sideway/formula/-/formula-3.0.0.tgz" - integrity sha512-vHe7wZ4NOXVfkoRb8T5otiENVlT7a3IAiw7H5M2+GO+9CDgcVUUsX1zalAztCmwyOr2RUTGJdgB+ZvSVqmdHmg== + version "3.0.1" + resolved "https://registry.yarnpkg.com/@sideway/formula/-/formula-3.0.1.tgz#80fcbcbaf7ce031e0ef2dd29b1bfc7c3f583611f" + integrity sha512-/poHZJJVjx3L+zVD6g9KgHfYnb443oi7wLu/XKojDviHy6HOEOA6z1Trk5aR1dGcmPenJEgb2sK2I80LeS3MIg== "@sideway/pinpoint@^2.0.0": version "2.0.0" @@ -2474,10 +2482,10 @@ resolved "https://registry.npmjs.org/@types/estree/-/estree-0.0.39.tgz" integrity sha512-EYNwp3bU+98cpU4lAWYYL7Zz+2gryWH1qbdDTidVd6hkiR6weksdbMadyXKXNPEkQFhXM+hVO9ZygomHXp+AIw== -"@types/estree@^0.0.51": - version "0.0.51" - resolved "https://registry.npmjs.org/@types/estree/-/estree-0.0.51.tgz" - integrity sha512-CuPgU6f3eT/XgKKPqKd/gLZV1Xmvf1a2R5POBOGQa6uv82xpls89HU5zKeVoyR8XzHd1RGNOlQlvUe3CFkjWNQ== +"@types/estree@^1.0.0": + version "1.0.1" + resolved "https://registry.yarnpkg.com/@types/estree/-/estree-1.0.1.tgz#aa22750962f3bf0e79d753d3cc067f010c95f194" + integrity sha512-LG4opVs2ANWZ1TJoKc937iMmNstM/d0ae1vNbnBvBhqCSezgVUOzcLCqbI5elV8Vy6WKwKjaqR+zO9VKirBBCA== "@types/express-serve-static-core@*", "@types/express-serve-static-core@^4.17.18": version "4.17.30" @@ -2818,125 +2826,125 @@ dependencies: "@use-gesture/core" "10.2.17" -"@webassemblyjs/ast@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/ast/-/ast-1.11.1.tgz" - integrity sha512-ukBh14qFLjxTQNTXocdyksN5QdM28S1CxHt2rdskFyL+xFV7VremuBLVbmCePj+URalXBENx/9Lm7lnhihtCSw== +"@webassemblyjs/ast@1.11.5", "@webassemblyjs/ast@^1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/ast/-/ast-1.11.5.tgz#6e818036b94548c1fb53b754b5cae3c9b208281c" + integrity sha512-LHY/GSAZZRpsNQH+/oHqhRQ5FT7eoULcBqgfyTB5nQHogFnK3/7QoN7dLnwSE/JkUAF0SrRuclT7ODqMFtWxxQ== dependencies: - "@webassemblyjs/helper-numbers" "1.11.1" - "@webassemblyjs/helper-wasm-bytecode" "1.11.1" + "@webassemblyjs/helper-numbers" "1.11.5" + "@webassemblyjs/helper-wasm-bytecode" "1.11.5" -"@webassemblyjs/floating-point-hex-parser@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/floating-point-hex-parser/-/floating-point-hex-parser-1.11.1.tgz" - integrity sha512-iGRfyc5Bq+NnNuX8b5hwBrRjzf0ocrJPI6GWFodBFzmFnyvrQ83SHKhmilCU/8Jv67i4GJZBMhEzltxzcNagtQ== +"@webassemblyjs/floating-point-hex-parser@1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/floating-point-hex-parser/-/floating-point-hex-parser-1.11.5.tgz#e85dfdb01cad16b812ff166b96806c050555f1b4" + integrity sha512-1j1zTIC5EZOtCplMBG/IEwLtUojtwFVwdyVMbL/hwWqbzlQoJsWCOavrdnLkemwNoC/EOwtUFch3fuo+cbcXYQ== -"@webassemblyjs/helper-api-error@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/helper-api-error/-/helper-api-error-1.11.1.tgz" - integrity sha512-RlhS8CBCXfRUR/cwo2ho9bkheSXG0+NwooXcc3PAILALf2QLdFyj7KGsKRbVc95hZnhnERon4kW/D3SZpp6Tcg== +"@webassemblyjs/helper-api-error@1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/helper-api-error/-/helper-api-error-1.11.5.tgz#1e82fa7958c681ddcf4eabef756ce09d49d442d1" + integrity sha512-L65bDPmfpY0+yFrsgz8b6LhXmbbs38OnwDCf6NpnMUYqa+ENfE5Dq9E42ny0qz/PdR0LJyq/T5YijPnU8AXEpA== -"@webassemblyjs/helper-buffer@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/helper-buffer/-/helper-buffer-1.11.1.tgz" - integrity sha512-gwikF65aDNeeXa8JxXa2BAk+REjSyhrNC9ZwdT0f8jc4dQQeDQ7G4m0f2QCLPJiMTTO6wfDmRmj/pW0PsUvIcA== +"@webassemblyjs/helper-buffer@1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/helper-buffer/-/helper-buffer-1.11.5.tgz#91381652ea95bb38bbfd270702351c0c89d69fba" + integrity sha512-fDKo1gstwFFSfacIeH5KfwzjykIE6ldh1iH9Y/8YkAZrhmu4TctqYjSh7t0K2VyDSXOZJ1MLhht/k9IvYGcIxg== -"@webassemblyjs/helper-numbers@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/helper-numbers/-/helper-numbers-1.11.1.tgz" - integrity sha512-vDkbxiB8zfnPdNK9Rajcey5C0w+QJugEglN0of+kmO8l7lDb77AnlKYQF7aarZuCrv+l0UvqL+68gSDr3k9LPQ== +"@webassemblyjs/helper-numbers@1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/helper-numbers/-/helper-numbers-1.11.5.tgz#23380c910d56764957292839006fecbe05e135a9" + integrity sha512-DhykHXM0ZABqfIGYNv93A5KKDw/+ywBFnuWybZZWcuzWHfbp21wUfRkbtz7dMGwGgT4iXjWuhRMA2Mzod6W4WA== dependencies: - "@webassemblyjs/floating-point-hex-parser" "1.11.1" - "@webassemblyjs/helper-api-error" "1.11.1" + "@webassemblyjs/floating-point-hex-parser" "1.11.5" + "@webassemblyjs/helper-api-error" "1.11.5" "@xtuc/long" "4.2.2" -"@webassemblyjs/helper-wasm-bytecode@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/helper-wasm-bytecode/-/helper-wasm-bytecode-1.11.1.tgz" - integrity sha512-PvpoOGiJwXeTrSf/qfudJhwlvDQxFgelbMqtq52WWiXC6Xgg1IREdngmPN3bs4RoO83PnL/nFrxucXj1+BX62Q== +"@webassemblyjs/helper-wasm-bytecode@1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/helper-wasm-bytecode/-/helper-wasm-bytecode-1.11.5.tgz#e258a25251bc69a52ef817da3001863cc1c24b9f" + integrity sha512-oC4Qa0bNcqnjAowFn7MPCETQgDYytpsfvz4ujZz63Zu/a/v71HeCAAmZsgZ3YVKec3zSPYytG3/PrRCqbtcAvA== -"@webassemblyjs/helper-wasm-section@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/helper-wasm-section/-/helper-wasm-section-1.11.1.tgz" - integrity sha512-10P9No29rYX1j7F3EVPX3JvGPQPae+AomuSTPiF9eBQeChHI6iqjMIwR9JmOJXwpnn/oVGDk7I5IlskuMwU/pg== +"@webassemblyjs/helper-wasm-section@1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/helper-wasm-section/-/helper-wasm-section-1.11.5.tgz#966e855a6fae04d5570ad4ec87fbcf29b42ba78e" + integrity sha512-uEoThA1LN2NA+K3B9wDo3yKlBfVtC6rh0i4/6hvbz071E8gTNZD/pT0MsBf7MeD6KbApMSkaAK0XeKyOZC7CIA== dependencies: - "@webassemblyjs/ast" "1.11.1" - "@webassemblyjs/helper-buffer" "1.11.1" - "@webassemblyjs/helper-wasm-bytecode" "1.11.1" - "@webassemblyjs/wasm-gen" "1.11.1" + "@webassemblyjs/ast" "1.11.5" + "@webassemblyjs/helper-buffer" "1.11.5" + "@webassemblyjs/helper-wasm-bytecode" "1.11.5" + "@webassemblyjs/wasm-gen" "1.11.5" -"@webassemblyjs/ieee754@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/ieee754/-/ieee754-1.11.1.tgz" - integrity sha512-hJ87QIPtAMKbFq6CGTkZYJivEwZDbQUgYd3qKSadTNOhVY7p+gfP6Sr0lLRVTaG1JjFj+r3YchoqRYxNH3M0GQ== +"@webassemblyjs/ieee754@1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/ieee754/-/ieee754-1.11.5.tgz#b2db1b33ce9c91e34236194c2b5cba9b25ca9d60" + integrity sha512-37aGq6qVL8A8oPbPrSGMBcp38YZFXcHfiROflJn9jxSdSMMM5dS5P/9e2/TpaJuhE+wFrbukN2WI6Hw9MH5acg== dependencies: "@xtuc/ieee754" "^1.2.0" -"@webassemblyjs/leb128@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/leb128/-/leb128-1.11.1.tgz" - integrity sha512-BJ2P0hNZ0u+Th1YZXJpzW6miwqQUGcIHT1G/sf72gLVD9DZ5AdYTqPNbHZh6K1M5VmKvFXwGSWZADz+qBWxeRw== +"@webassemblyjs/leb128@1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/leb128/-/leb128-1.11.5.tgz#482e44d26b6b949edf042a8525a66c649e38935a" + integrity sha512-ajqrRSXaTJoPW+xmkfYN6l8VIeNnR4vBOTQO9HzR7IygoCcKWkICbKFbVTNMjMgMREqXEr0+2M6zukzM47ZUfQ== dependencies: "@xtuc/long" "4.2.2" -"@webassemblyjs/utf8@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/utf8/-/utf8-1.11.1.tgz" - integrity sha512-9kqcxAEdMhiwQkHpkNiorZzqpGrodQQ2IGrHHxCy+Ozng0ofyMA0lTqiLkVs1uzTRejX+/O0EOT7KxqVPuXosQ== +"@webassemblyjs/utf8@1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/utf8/-/utf8-1.11.5.tgz#83bef94856e399f3740e8df9f63bc47a987eae1a" + integrity sha512-WiOhulHKTZU5UPlRl53gHR8OxdGsSOxqfpqWeA2FmcwBMaoEdz6b2x2si3IwC9/fSPLfe8pBMRTHVMk5nlwnFQ== -"@webassemblyjs/wasm-edit@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/wasm-edit/-/wasm-edit-1.11.1.tgz" - integrity sha512-g+RsupUC1aTHfR8CDgnsVRVZFJqdkFHpsHMfJuWQzWU3tvnLC07UqHICfP+4XyL2tnr1amvl1Sdp06TnYCmVkA== +"@webassemblyjs/wasm-edit@^1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/wasm-edit/-/wasm-edit-1.11.5.tgz#93ee10a08037657e21c70de31c47fdad6b522b2d" + integrity sha512-C0p9D2fAu3Twwqvygvf42iGCQ4av8MFBLiTb+08SZ4cEdwzWx9QeAHDo1E2k+9s/0w1DM40oflJOpkZ8jW4HCQ== dependencies: - "@webassemblyjs/ast" "1.11.1" - "@webassemblyjs/helper-buffer" "1.11.1" - "@webassemblyjs/helper-wasm-bytecode" "1.11.1" - "@webassemblyjs/helper-wasm-section" "1.11.1" - "@webassemblyjs/wasm-gen" "1.11.1" - "@webassemblyjs/wasm-opt" "1.11.1" - "@webassemblyjs/wasm-parser" "1.11.1" - "@webassemblyjs/wast-printer" "1.11.1" + "@webassemblyjs/ast" "1.11.5" + "@webassemblyjs/helper-buffer" "1.11.5" + "@webassemblyjs/helper-wasm-bytecode" "1.11.5" + "@webassemblyjs/helper-wasm-section" "1.11.5" + "@webassemblyjs/wasm-gen" "1.11.5" + "@webassemblyjs/wasm-opt" "1.11.5" + "@webassemblyjs/wasm-parser" "1.11.5" + "@webassemblyjs/wast-printer" "1.11.5" -"@webassemblyjs/wasm-gen@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/wasm-gen/-/wasm-gen-1.11.1.tgz" - integrity sha512-F7QqKXwwNlMmsulj6+O7r4mmtAlCWfO/0HdgOxSklZfQcDu0TpLiD1mRt/zF25Bk59FIjEuGAIyn5ei4yMfLhA== +"@webassemblyjs/wasm-gen@1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/wasm-gen/-/wasm-gen-1.11.5.tgz#ceb1c82b40bf0cf67a492c53381916756ef7f0b1" + integrity sha512-14vteRlRjxLK9eSyYFvw1K8Vv+iPdZU0Aebk3j6oB8TQiQYuO6hj9s4d7qf6f2HJr2khzvNldAFG13CgdkAIfA== dependencies: - "@webassemblyjs/ast" "1.11.1" - "@webassemblyjs/helper-wasm-bytecode" "1.11.1" - "@webassemblyjs/ieee754" "1.11.1" - "@webassemblyjs/leb128" "1.11.1" - "@webassemblyjs/utf8" "1.11.1" + "@webassemblyjs/ast" "1.11.5" + "@webassemblyjs/helper-wasm-bytecode" "1.11.5" + "@webassemblyjs/ieee754" "1.11.5" + "@webassemblyjs/leb128" "1.11.5" + "@webassemblyjs/utf8" "1.11.5" -"@webassemblyjs/wasm-opt@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/wasm-opt/-/wasm-opt-1.11.1.tgz" - integrity sha512-VqnkNqnZlU5EB64pp1l7hdm3hmQw7Vgqa0KF/KCNO9sIpI6Fk6brDEiX+iCOYrvMuBWDws0NkTOxYEb85XQHHw== +"@webassemblyjs/wasm-opt@1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/wasm-opt/-/wasm-opt-1.11.5.tgz#b52bac29681fa62487e16d3bb7f0633d5e62ca0a" + integrity sha512-tcKwlIXstBQgbKy1MlbDMlXaxpucn42eb17H29rawYLxm5+MsEmgPzeCP8B1Cl69hCice8LeKgZpRUAPtqYPgw== dependencies: - "@webassemblyjs/ast" "1.11.1" - "@webassemblyjs/helper-buffer" "1.11.1" - "@webassemblyjs/wasm-gen" "1.11.1" - "@webassemblyjs/wasm-parser" "1.11.1" + "@webassemblyjs/ast" "1.11.5" + "@webassemblyjs/helper-buffer" "1.11.5" + "@webassemblyjs/wasm-gen" "1.11.5" + "@webassemblyjs/wasm-parser" "1.11.5" -"@webassemblyjs/wasm-parser@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/wasm-parser/-/wasm-parser-1.11.1.tgz" - integrity sha512-rrBujw+dJu32gYB7/Lup6UhdkPx9S9SnobZzRVL7VcBH9Bt9bCBLEuX/YXOOtBsOZ4NQrRykKhffRWHvigQvOA== +"@webassemblyjs/wasm-parser@1.11.5", "@webassemblyjs/wasm-parser@^1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/wasm-parser/-/wasm-parser-1.11.5.tgz#7ba0697ca74c860ea13e3ba226b29617046982e2" + integrity sha512-SVXUIwsLQlc8srSD7jejsfTU83g7pIGr2YYNb9oHdtldSxaOhvA5xwvIiWIfcX8PlSakgqMXsLpLfbbJ4cBYew== dependencies: - "@webassemblyjs/ast" "1.11.1" - "@webassemblyjs/helper-api-error" "1.11.1" - "@webassemblyjs/helper-wasm-bytecode" "1.11.1" - "@webassemblyjs/ieee754" "1.11.1" - "@webassemblyjs/leb128" "1.11.1" - "@webassemblyjs/utf8" "1.11.1" + "@webassemblyjs/ast" "1.11.5" + "@webassemblyjs/helper-api-error" "1.11.5" + "@webassemblyjs/helper-wasm-bytecode" "1.11.5" + "@webassemblyjs/ieee754" "1.11.5" + "@webassemblyjs/leb128" "1.11.5" + "@webassemblyjs/utf8" "1.11.5" -"@webassemblyjs/wast-printer@1.11.1": - version "1.11.1" - resolved "https://registry.npmjs.org/@webassemblyjs/wast-printer/-/wast-printer-1.11.1.tgz" - integrity sha512-IQboUWM4eKzWW+N/jij2sRatKMh99QEelo3Eb2q0qXkvPRISAj8Qxtmw5itwqK+TTkBuUIE45AxYPToqPtL5gg== +"@webassemblyjs/wast-printer@1.11.5": + version "1.11.5" + resolved "https://registry.yarnpkg.com/@webassemblyjs/wast-printer/-/wast-printer-1.11.5.tgz#7a5e9689043f3eca82d544d7be7a8e6373a6fa98" + integrity sha512-f7Pq3wvg3GSPUPzR0F6bmI89Hdb+u9WXrSKc4v+N0aV0q6r42WoF92Jp2jEorBEBRoRNXgjp53nBniDXcqZYPA== dependencies: - "@webassemblyjs/ast" "1.11.1" + "@webassemblyjs/ast" "1.11.5" "@xtuc/long" "4.2.2" "@welldone-software/why-did-you-render@^6.2.3": @@ -3018,7 +3026,7 @@ address@^1.0.1, address@^1.1.2: adjust-sourcemap-loader@^4.0.0: version "4.0.0" - resolved "https://registry.npmjs.org/adjust-sourcemap-loader/-/adjust-sourcemap-loader-4.0.0.tgz" + resolved "https://registry.yarnpkg.com/adjust-sourcemap-loader/-/adjust-sourcemap-loader-4.0.0.tgz#fc4a0fd080f7d10471f30a7320f25560ade28c99" integrity sha512-OXwN5b9pCUXNQHJpwwD2qP40byEmSgzj8B4ydSN0uMNYWiFmJ6x6KwUllMmfk8Rwu/HJDFR7U8ubsWBoN0Xp0A== dependencies: loader-utils "^2.0.0" @@ -3294,15 +3302,13 @@ babel-jest@^27.4.2, babel-jest@^27.5.1: graceful-fs "^4.2.9" slash "^3.0.0" -babel-loader@^8.2.3: - version "8.2.5" - resolved "https://registry.npmjs.org/babel-loader/-/babel-loader-8.2.5.tgz" - integrity sha512-OSiFfH89LrEMiWd4pLNqGz4CwJDtbs2ZVc+iGu2HrkRfPxId9F2anQj38IxWpmRfsUY0aBZYi1EFcd3mhtRMLQ== +babel-loader@^8.2.3, babel-loader@^9.1.2: + version "9.1.2" + resolved "https://registry.yarnpkg.com/babel-loader/-/babel-loader-9.1.2.tgz#a16a080de52d08854ee14570469905a5fc00d39c" + integrity sha512-mN14niXW43tddohGl8HPu5yfQq70iUThvFL/4QzESA7GcZoC0eVOhvWdQ8+3UlSjaDE9MVtsW9mxDY07W7VpVA== dependencies: - find-cache-dir "^3.3.1" - loader-utils "^2.0.0" - make-dir "^3.1.0" - schema-utils "^2.6.5" + find-cache-dir "^3.3.2" + schema-utils "^4.0.0" babel-plugin-dynamic-import-node@^2.3.3: version "2.3.3" @@ -3443,11 +3449,6 @@ bfj@^7.0.2: hoopy "^0.1.4" tryer "^1.0.1" -big.js@^5.2.2: - version "5.2.2" - resolved "https://registry.npmjs.org/big.js/-/big.js-5.2.2.tgz" - integrity sha512-vyL2OymJxmarO8gxMr0mhChsO9QGwhynfuu4+MHTAW6czfq9humCB7rKpUjDd9YUiDPU4mzpyupFSvOClAwbmQ== - binary-extensions@^2.0.0: version "2.2.0" resolved "https://registry.npmjs.org/binary-extensions/-/binary-extensions-2.2.0.tgz" @@ -3590,7 +3591,7 @@ camelcase-css@^2.0.1: camelcase@^5.3.1: version "5.3.1" - resolved "https://registry.npmjs.org/camelcase/-/camelcase-5.3.1.tgz" + resolved "https://registry.yarnpkg.com/camelcase/-/camelcase-5.3.1.tgz#e3c9b31569e106811df242f715725a1f4c494320" integrity sha512-L28STB170nwWS63UjtlEOE3dldQApaJXZkOI1uMFfzf3rRuPegHaHesyee+YxQ+W6SvRDQV6UrdOdRiR153wJg== camelcase@^6.2.0, camelcase@^6.2.1: @@ -3625,7 +3626,7 @@ case-sensitive-paths-webpack-plugin@^2.4.0: chalk@^2.0.0, chalk@^2.4.1: version "2.4.2" - resolved "https://registry.npmjs.org/chalk/-/chalk-2.4.2.tgz" + resolved "https://registry.yarnpkg.com/chalk/-/chalk-2.4.2.tgz#cd42541677a54333cf541a49108c1432b44c9424" integrity sha512-Mti+f9lpJNcwF4tWV8/OrTTtF1gZi+f8FqlyAdouralcFWFQWF2+NgCHShjkCb+IFBLq9buZwE1xckQU4peSuQ== dependencies: ansi-styles "^3.2.1" @@ -3896,11 +3897,16 @@ core-js-compat@^3.21.0, core-js-compat@^3.22.1: browserslist "^4.21.3" semver "7.0.0" -core-js-pure@^3.20.2, core-js-pure@^3.8.1: +core-js-pure@^3.20.2: version "3.24.1" resolved "https://registry.npmjs.org/core-js-pure/-/core-js-pure-3.24.1.tgz" integrity sha512-r1nJk41QLLPyozHUUPmILCEMtMw24NG4oWK6RbsDdjzQgg9ZvrUsPBj1MnG0wXXp1DCDU6j+wUvEmBSrtRbLXg== +core-js-pure@^3.23.3: + version "3.30.1" + resolved "https://registry.yarnpkg.com/core-js-pure/-/core-js-pure-3.30.1.tgz#7d93dc89e7d47b8ef05d7e79f507b0e99ea77eec" + integrity sha512-nXBEVpmUnNRhz83cHd9JRQC52cTMcuXAmR56+9dSMpRdpeA4I1PX6yjmhd71Eyc/wXNsdBdUDIj1QTIeZpU5Tg== + core-js@^3.19.2: version "3.24.1" resolved "https://registry.npmjs.org/core-js/-/core-js-3.24.1.tgz" @@ -4220,9 +4226,9 @@ decimal.js@^10.2.1: integrity sha512-V0pfhfr8suzyPGOx3nmq4aHqabehUZn6Ch9kyFpV79TGDTWFmHqUqXdabR7QHqxzrYolF4+tVmJhUG4OURg5dQ== decode-uri-component@^0.2.0: - version "0.2.0" - resolved "https://registry.npmjs.org/decode-uri-component/-/decode-uri-component-0.2.0.tgz" - integrity sha512-hjf+xovcEn31w/EUYdTXQh/8smFL/dzYjohQGEIgjyNavaJfBY2p5F527Bo1VPATxv0VYTUC2bOcXvqFwk78Og== + version "0.2.2" + resolved "https://registry.yarnpkg.com/decode-uri-component/-/decode-uri-component-0.2.2.tgz#e69dbe25d37941171dd540e024c444cd5188e1e9" + integrity sha512-FqUYQ+8o158GyGTrMFJms9qh3CqTKvAqgqsTnkLI8sKu0028orqBhxNMFkFen0zGyg6epACD32pjVk58ngIErQ== dedent@^0.7.0: version "0.7.0" @@ -4507,11 +4513,6 @@ emoji-regex@^9.2.2: resolved "https://registry.npmjs.org/emoji-regex/-/emoji-regex-9.2.2.tgz" integrity sha512-L18DaJsXSUk2+42pv8mLs5jJT2hqFkFE4j21wOmgbUqsZ2hL72NsUU785g9RXgo3s0ZNgVl42TiHp3ZtOv/Vyg== -emojis-list@^3.0.0: - version "3.0.0" - resolved "https://registry.npmjs.org/emojis-list/-/emojis-list-3.0.0.tgz" - integrity sha512-/kyM18EfinwXZbno9FyUGeFh87KC8HRQBQGildHZbEuRyWFOmv1U10o9BBp8XVZDVNNuQKyIGIu5ZYAAXJ0V2Q== - encodeurl@~1.0.2: version "1.0.2" resolved "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz" @@ -4533,10 +4534,10 @@ engine.io-parser@~5.0.3: resolved "https://registry.npmjs.org/engine.io-parser/-/engine.io-parser-5.0.4.tgz" integrity sha512-+nVFp+5z1E3HcToEnO7ZIj3g+3k9389DvWtvJZz0T6/eOCPIyyxehFcedoYrZQrp0LgQbD9pPXhpMBKMd5QURg== -enhanced-resolve@^5.10.0: - version "5.10.0" - resolved "https://registry.npmjs.org/enhanced-resolve/-/enhanced-resolve-5.10.0.tgz" - integrity sha512-T0yTFjdpldGY8PmuXXR0PyQ1ufZpEGiHVrp7zHKB7jdR4qlmZHhONVM5AQOAWXuF/w3dnHbEQVrNptJgt7F+cQ== +enhanced-resolve@^5.13.0: + version "5.13.0" + resolved "https://registry.yarnpkg.com/enhanced-resolve/-/enhanced-resolve-5.13.0.tgz#26d1ecc448c02de997133217b5c1053f34a0a275" + integrity sha512-eyV8f0y1+bzyfh8xAwW/WTSZpLbjhqc4ne9eGSH4Zo2ejdyiNG9pU6mf9DG8a7+Auk6MFTlNOT4Y2y/9k8GKVg== dependencies: graceful-fs "^4.2.4" tapable "^2.2.0" @@ -4594,10 +4595,10 @@ es-array-method-boxes-properly@^1.0.0: resolved "https://registry.npmjs.org/es-array-method-boxes-properly/-/es-array-method-boxes-properly-1.0.0.tgz" integrity sha512-wd6JXUmyHmt8T5a2xreUwKcGPq6f1f+WwIJkijUqiGcJz1qqnZgP6XIK+QyIWU5lT7imeNxUll48bziG+TSYcA== -es-module-lexer@^0.9.0: - version "0.9.3" - resolved "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-0.9.3.tgz" - integrity sha512-1HQ2M2sPtxwnvOvT1ZClHyQDiggdNjURWpY2we6aMKCQiUVxTmVs2UYPLIrD84sS+kMdUwfBSylbJPwNnBrnHQ== +es-module-lexer@^1.2.1: + version "1.2.1" + resolved "https://registry.yarnpkg.com/es-module-lexer/-/es-module-lexer-1.2.1.tgz#ba303831f63e6a394983fde2f97ad77b22324527" + integrity sha512-9978wrXM50Y4rTMmW5kXIC09ZdXQZqkE4mxhwkd8VbzsGkXGPgV4zWuqQJgCEzYngdo2dYDa0l8xhX4fkSwJSg== es-shim-unscopables@^1.0.0: version "1.0.0" @@ -4626,7 +4627,7 @@ es5-ext@^0.10.35, es5-ext@^0.10.50: es6-iterator@^2.0.3: version "2.0.3" - resolved "https://registry.npmjs.org/es6-iterator/-/es6-iterator-2.0.3.tgz" + resolved "https://registry.yarnpkg.com/es6-iterator/-/es6-iterator-2.0.3.tgz#a7de889141a05a94b0854403b2d0a0fbfa98f3b7" integrity sha512-zw4SRzoUkd+cl+ZoE15A9o1oQd920Bb0iOJMQkQhl3jNc03YqVjAhG7scf9C5KWRU/R13Orf588uCC6525o02g== dependencies: d "1" @@ -5182,9 +5183,9 @@ finalhandler@1.2.0: statuses "2.0.1" unpipe "~1.0.0" -find-cache-dir@^3.3.1: +find-cache-dir@^3.3.2: version "3.3.2" - resolved "https://registry.npmjs.org/find-cache-dir/-/find-cache-dir-3.3.2.tgz" + resolved "https://registry.yarnpkg.com/find-cache-dir/-/find-cache-dir-3.3.2.tgz#b30c5b6eff0730731aea9bbd9dbecbd80256d64b" integrity sha512-wXZV5emFEjrridIgED11OoUKLxiYjAcqot/NJdAkOhlJ+vGzwhOAfcG5OX1jP+S0PcjEn8bdMJv+g2jwQ3Onig== dependencies: commondir "^1.0.1" @@ -6677,17 +6678,10 @@ json-stable-stringify-without-jsonify@^1.0.1: resolved "https://registry.npmjs.org/json-stable-stringify-without-jsonify/-/json-stable-stringify-without-jsonify-1.0.1.tgz" integrity sha512-Bdboy+l7tA3OGW6FjyFHWkP5LuByj1Tk33Ljyq0axyzdk9//JSi2u3fP1QSmd1KNwq6VOKYGlAu87CisVir6Pw== -json5@^1.0.1: - version "1.0.2" - resolved "https://registry.npmjs.org/json5/-/json5-1.0.2.tgz" - integrity sha512-g1MWMLBiz8FKi1e4w0UyVL3w+iJceWAFBAaBnnGKOpNa5f8TLktkbre1+s6oICydWAm+HRUGTmI+//xv2hvXYA== - dependencies: - minimist "^1.2.0" - -json5@^2.1.2, json5@^2.2.0, json5@^2.2.1: - version "2.2.1" - resolved "https://registry.npmjs.org/json5/-/json5-2.2.1.tgz" - integrity sha512-1hqLFMSrGHRHxav9q9gNjJ5EXznIxGVO09xQRrwplcS8qs28pZ8s8hupZAmqDwZUmVZ2Qb2jnyPOWcDH8m8dlA== +json5@^1.0.1, json5@^2.2.0, json5@^2.2.1, json5@^2.2.3: + version "2.2.3" + resolved "https://registry.yarnpkg.com/json5/-/json5-2.2.3.tgz#78cd6f1a19bdc12b73db5ad0c61efd66c1e29283" + integrity sha512-XmOWe7eyHYH14cLdVPoyg+GOH3rYX++KpzrylJwSW98t3Nk+U8XOl8FWKOgwtzdb8lXGf6zYwDUzeHMWfxasyg== jsonfile@^6.0.1: version "6.1.0" @@ -6792,19 +6786,10 @@ loader-runner@^4.2.0: resolved "https://registry.npmjs.org/loader-runner/-/loader-runner-4.3.0.tgz" integrity sha512-3R/1M+yS3j5ou80Me59j7F9IMs4PXs3VqRrm0TU3AbKPxlmpoY1TNscJV/oGJXo8qCatFGTfDbY6W6ipGOYXfg== -loader-utils@^2.0.0: - version "2.0.4" - resolved "https://registry.npmjs.org/loader-utils/-/loader-utils-2.0.4.tgz" - integrity sha512-xXqpXoINfFhgua9xiqD8fPFHgkoq1mmmpE92WlDbm9rNRd/EbRb+Gqf908T2DMfuHjjJlksiK2RbHVOdD/MqSw== - dependencies: - big.js "^5.2.2" - emojis-list "^3.0.0" - json5 "^2.1.2" - -loader-utils@^3.2.0: - version "3.2.0" - resolved "https://registry.npmjs.org/loader-utils/-/loader-utils-3.2.0.tgz" - integrity sha512-HVl9ZqccQihZ7JM85dco1MvO9G+ONvxoGa9rkhzFsneGLKSUg1gJf9bWzhRhcvm2qChhWpebQhP44qxjKIUCaQ== +loader-utils@^2.0.0, loader-utils@^2.0.4, loader-utils@^3.2.0, loader-utils@^3.2.1: + version "3.2.1" + resolved "https://registry.yarnpkg.com/loader-utils/-/loader-utils-3.2.1.tgz#4fb104b599daafd82ef3e1a41fb9265f87e1f576" + integrity sha512-ZvFw1KWS3GVyYBYb7qkmRM/WwL2TQQBxgCK62rlvm4WpVQ23Nb4tYjApUlfjrEGvOs7KHEsmyUn75OHZrJMWPw== locate-path@^2.0.0: version "2.0.0" @@ -6899,7 +6884,7 @@ magic-string@^0.25.0, magic-string@^0.25.7: dependencies: sourcemap-codec "^1.4.8" -make-dir@^3.0.0, make-dir@^3.0.2, make-dir@^3.1.0: +make-dir@^3.0.0, make-dir@^3.0.2: version "3.1.0" resolved "https://registry.npmjs.org/make-dir/-/make-dir-3.1.0.tgz" integrity sha512-g3FeP20LNwhALb/6Cz6Dd4F2ngze0jz7tbzrD2wAV+o9FeNHe4rL+yK2md0J/fiSf1sa1ADhXqi5+oVwOM/eGw== @@ -7017,14 +7002,7 @@ minimalistic-assert@^1.0.0: resolved "https://registry.npmjs.org/minimalistic-assert/-/minimalistic-assert-1.0.1.tgz" integrity sha512-UtJcAD4yEaGtjPezWuO9wC4nwUnVH/8/Im3yEHQP4b67cXlD/Qr9hdITCU1xDbSEXg2XKNaP8jsReV7vQd00/A== -minimatch@3.0.4: - version "3.0.4" - resolved "https://registry.npmjs.org/minimatch/-/minimatch-3.0.4.tgz" - integrity sha512-yJHVQEhyqPLUTgt9B83PXu6W3rx4MvvHvSUvToogpwoGDOUQ+yDrR0HRot+yOCdCO7u4hX3pWft6kWBBcqh0UA== - dependencies: - brace-expansion "^1.1.7" - -minimatch@^3.0.4, minimatch@^3.1.1, minimatch@^3.1.2: +minimatch@^3.0.4, minimatch@^3.0.5, minimatch@^3.1.1, minimatch@^3.1.2: version "3.1.2" resolved "https://registry.npmjs.org/minimatch/-/minimatch-3.1.2.tgz" integrity sha512-J7p63hRiAjw1NDEww1W7i37+ByIrOWO5XQQAzZ3VOcL0PNybwpfmV/N05zFAzwQ9USyEcX6t3UO+K5aqBQOIHw== @@ -7038,7 +7016,7 @@ minimatch@^5.0.1: dependencies: brace-expansion "^2.0.1" -minimist@^1.2.0, minimist@^1.2.5, minimist@^1.2.6: +minimist@^1.2.5, minimist@^1.2.6: version "1.2.7" resolved "https://registry.npmjs.org/minimist/-/minimist-1.2.7.tgz" integrity sha512-bzfL1YUZsP41gmu/qjrEk0Q6i2ix/cVeAhbCbqH9u3zYutS1cLg00qhrD0M2MVdCcx4Sc0UpP2eBWo9rotpq6g== @@ -7117,6 +7095,11 @@ nanoid@^3.3.4: resolved "https://registry.npmjs.org/nanoid/-/nanoid-3.3.4.tgz" integrity sha512-MqBkQh/OHTS2egovRtLk45wEyNXwF+cokD+1YPf9u5VfJiRdAiRwB2froX5Co9Rh20xs4siNPm8naNotSD6RBw== +nanoid@^3.3.6: + version "3.3.6" + resolved "https://registry.yarnpkg.com/nanoid/-/nanoid-3.3.6.tgz#443380c856d6e9f9824267d960b4236ad583ea4c" + integrity sha512-BGcqMMJuToF7i1rt+2PWSNVnWIkGCU78jBG3RxO/bZlnZPK2Cmi2QaffxGO/2RvWi9sL+FAiRiXMgsyxQ1DIDA== + natural-compare@^1.4.0: version "1.4.0" resolved "https://registry.npmjs.org/natural-compare/-/natural-compare-1.4.0.tgz" @@ -7493,11 +7476,6 @@ performance-now@^2.1.0: resolved "https://registry.npmjs.org/performance-now/-/performance-now-2.1.0.tgz" integrity sha512-7EAHlyLHI56VEIdK57uwHdHKIaAGbnXPiw0yWbarQZOKaKpvUIgW0jWRVLiatnM+XXlSwsanIBH/hzGMJulMow== -picocolors@^0.2.1: - version "0.2.1" - resolved "https://registry.npmjs.org/picocolors/-/picocolors-0.2.1.tgz" - integrity sha512-cMlDqaLEqfSaW8Z7N5Jw+lyIW869EzT73/F5lhtY9cLGoVxSXznfgfXMO0Z5K0o0Q2TkTXq+0KFsdnSe3jDViA== - picocolors@^1.0.0: version "1.0.0" resolved "https://registry.npmjs.org/picocolors/-/picocolors-1.0.0.tgz" @@ -8065,13 +8043,14 @@ postcss-value-parser@^4.0.0, postcss-value-parser@^4.1.0, postcss-value-parser@^ resolved "https://registry.npmjs.org/postcss-value-parser/-/postcss-value-parser-4.2.0.tgz" integrity sha512-1NNCs6uurfkVbeXG4S8JFT9t19m45ICnif8zWLd5oPSZ50QnwMfK+H3jv408d4jw/7Bttv5axS5IiHoLaVNHeQ== -postcss@^7.0.35: - version "7.0.39" - resolved "https://registry.npmjs.org/postcss/-/postcss-7.0.39.tgz" - integrity sha512-yioayjNbHn6z1/Bywyb2Y4s3yvDAeXGOyxqD+LnVOinq6Mdmd++SW2wUNVzavyyHxd6+DxzWGIuosg6P1Rj8uA== +postcss@^8.2.14: + version "8.4.23" + resolved "https://registry.yarnpkg.com/postcss/-/postcss-8.4.23.tgz#df0aee9ac7c5e53e1075c24a3613496f9e6552ab" + integrity sha512-bQ3qMcpF6A/YjR55xtoTr0jGOlnPOKAIMdOWiv0EIT6HVPEaJiJB4NLljSbiHoC2RX7DN5Uvjtpbg1NPdwv1oA== dependencies: - picocolors "^0.2.1" - source-map "^0.6.1" + nanoid "^3.3.6" + picocolors "^1.0.0" + source-map-js "^1.0.2" postcss@^8.3.5, postcss@^8.4.14, postcss@^8.4.4, postcss@^8.4.7: version "8.4.14" @@ -8463,12 +8442,12 @@ readdirp@~3.6.0: dependencies: picomatch "^2.2.1" -recursive-readdir@^2.2.2: - version "2.2.2" - resolved "https://registry.npmjs.org/recursive-readdir/-/recursive-readdir-2.2.2.tgz" - integrity sha512-nRCcW9Sj7NuZwa2XvH9co8NPeXUBhZP7CRKJtU+cS6PW9FpCIFoI5ib0NT1ZrbNuPoRy0ylyCaUL8Gih4LSyFg== +recursive-readdir@^2.2.2, recursive-readdir@^2.2.3: + version "2.2.3" + resolved "https://registry.yarnpkg.com/recursive-readdir/-/recursive-readdir-2.2.3.tgz#e726f328c0d69153bcabd5c322d3195252379372" + integrity sha512-8HrF5ZsXk5FAH9dgsx3BlUer73nIhuj+9OrQwEbLTPOBzGkL1lsFCR01am+v+0m2Cmbs1nP12hLDl5FA7EszKA== dependencies: - minimatch "3.0.4" + minimatch "^3.0.5" redent@^3.0.0: version "3.0.0" @@ -8615,15 +8594,15 @@ resolve-from@^5.0.0: resolved "https://registry.npmjs.org/resolve-from/-/resolve-from-5.0.0.tgz" integrity sha512-qYg9KP24dD5qka9J47d0aVky0N+b4fTU89LN9iDnjB5waksiC49rvMB0PrUJQGoTmH50XPiqOvAjDfaijGxYZw== -resolve-url-loader@^4.0.0: - version "4.0.0" - resolved "https://registry.npmjs.org/resolve-url-loader/-/resolve-url-loader-4.0.0.tgz" - integrity sha512-05VEMczVREcbtT7Bz+C+96eUO5HDNvdthIiMB34t7FcF8ehcu4wC0sSgPUubs3XW2Q3CNLJk/BJrCU9wVRymiA== +resolve-url-loader@^4.0.0, resolve-url-loader@^5.0.0: + version "5.0.0" + resolved "https://registry.yarnpkg.com/resolve-url-loader/-/resolve-url-loader-5.0.0.tgz#ee3142fb1f1e0d9db9524d539cfa166e9314f795" + integrity sha512-uZtduh8/8srhBoMx//5bwqjQ+rfYOUq8zC9NrMUGtjBiGTtFJM42s58/36+hTqeqINcnYe08Nj3LkK9lW4N8Xg== dependencies: adjust-sourcemap-loader "^4.0.0" convert-source-map "^1.7.0" loader-utils "^2.0.0" - postcss "^7.0.35" + postcss "^8.2.14" source-map "0.6.1" resolve.exports@^1.1.0: @@ -8767,16 +8746,7 @@ schema-utils@2.7.0: ajv "^6.12.2" ajv-keywords "^3.4.1" -schema-utils@^2.6.5: - version "2.7.1" - resolved "https://registry.npmjs.org/schema-utils/-/schema-utils-2.7.1.tgz" - integrity sha512-SHiNtMOUGWBQJwzISiVYKu82GiV4QYGePp3odlY1tuKO7gPtphAT5R/py0fA6xtbgLL/RvtJZnU9b8s0F1q0Xg== - dependencies: - "@types/json-schema" "^7.0.5" - ajv "^6.12.4" - ajv-keywords "^3.5.2" - -schema-utils@^3.0.0, schema-utils@^3.1.0, schema-utils@^3.1.1: +schema-utils@^3.0.0, schema-utils@^3.1.1: version "3.1.1" resolved "https://registry.npmjs.org/schema-utils/-/schema-utils-3.1.1.tgz" integrity sha512-Y5PQxS4ITlC+EahLuXaY86TXfR7Dc5lw294alXOq86JAHCihAIZfqv8nNCWvaEJvaC51uN9hbLGeV0cFBdH+Fw== @@ -8785,6 +8755,15 @@ schema-utils@^3.0.0, schema-utils@^3.1.0, schema-utils@^3.1.1: ajv "^6.12.5" ajv-keywords "^3.5.2" +schema-utils@^3.1.2: + version "3.1.2" + resolved "https://registry.yarnpkg.com/schema-utils/-/schema-utils-3.1.2.tgz#36c10abca6f7577aeae136c804b0c741edeadc99" + integrity sha512-pvjEHOgWc9OWA/f/DE3ohBWTD6EleVLf7iFUkoSwAxttdBhB9QUebQgxER2kWueOvRJXPHNnyrvvh9eZINB8Eg== + dependencies: + "@types/json-schema" "^7.0.8" + ajv "^6.12.5" + ajv-keywords "^3.5.2" + schema-utils@^4.0.0: version "4.0.0" resolved "https://registry.npmjs.org/schema-utils/-/schema-utils-4.0.0.tgz" @@ -8857,6 +8836,13 @@ serialize-javascript@^6.0.0: dependencies: randombytes "^2.1.0" +serialize-javascript@^6.0.1: + version "6.0.1" + resolved "https://registry.yarnpkg.com/serialize-javascript/-/serialize-javascript-6.0.1.tgz#b206efb27c3da0b0ab6b52f48d170b7996458e5c" + integrity sha512-owoXEFjWRllis8/M1Q+Cw5k8ZH40e3zhp/ovX+Xr/vi1qj6QesbyXXViFbpNvWvPNAD62SutwEXavefrLJWj7w== + dependencies: + randombytes "^2.1.0" + serve-index@^1.9.1: version "1.9.1" resolved "https://registry.npmjs.org/serve-index/-/serve-index-1.9.1.tgz" @@ -9388,7 +9374,7 @@ terminal-link@^2.0.0: ansi-escapes "^4.2.1" supports-hyperlinks "^2.0.0" -terser-webpack-plugin@^5.1.3, terser-webpack-plugin@^5.2.5: +terser-webpack-plugin@^5.2.5: version "5.3.3" resolved "https://registry.npmjs.org/terser-webpack-plugin/-/terser-webpack-plugin-5.3.3.tgz" integrity sha512-Fx60G5HNYknNTNQnzQ1VePRuu89ZVYWfjRAeT5rITuCY/1b08s49e5kSQwHDirKZWuoKOBRFS98EUUoZ9kLEwQ== @@ -9399,6 +9385,17 @@ terser-webpack-plugin@^5.1.3, terser-webpack-plugin@^5.2.5: serialize-javascript "^6.0.0" terser "^5.7.2" +terser-webpack-plugin@^5.3.7: + version "5.3.7" + resolved "https://registry.yarnpkg.com/terser-webpack-plugin/-/terser-webpack-plugin-5.3.7.tgz#ef760632d24991760f339fe9290deb936ad1ffc7" + integrity sha512-AfKwIktyP7Cu50xNjXF/6Qb5lBNzYaWpU6YfoX3uZicTx0zTy0stDDCsvjDapKsSDvOeWo5MEq4TmdBy2cNoHw== + dependencies: + "@jridgewell/trace-mapping" "^0.3.17" + jest-worker "^27.4.5" + schema-utils "^3.1.1" + serialize-javascript "^6.0.1" + terser "^5.16.5" + terser@^5.0.0, terser@^5.10.0, terser@^5.7.2: version "5.14.2" resolved "https://registry.npmjs.org/terser/-/terser-5.14.2.tgz" @@ -9409,6 +9406,16 @@ terser@^5.0.0, terser@^5.10.0, terser@^5.7.2: commander "^2.20.0" source-map-support "~0.5.20" +terser@^5.16.5: + version "5.17.1" + resolved "https://registry.yarnpkg.com/terser/-/terser-5.17.1.tgz#948f10830454761e2eeedc6debe45c532c83fd69" + integrity sha512-hVl35zClmpisy6oaoKALOpS0rDYLxRFLHhRuDlEGTKey9qHjS1w9GMORjuwIMt70Wan4lwsLYyWDVnWgF+KUEw== + dependencies: + "@jridgewell/source-map" "^0.3.2" + acorn "^8.5.0" + commander "^2.20.0" + source-map-support "~0.5.20" + test-exclude@^6.0.0: version "6.0.0" resolved "https://registry.npmjs.org/test-exclude/-/test-exclude-6.0.0.tgz" @@ -9880,21 +9887,21 @@ webpack-sources@^3.2.3: integrity sha512-/DyMEOrDgLKKIG0fmvtz+4dUX/3Ghozwgm6iPp8KRhvn+eQf9+Q7GWxVNMk3+uCPWfdXYC4ExGBckIXdFEfH1w== webpack@^5.64.4: - version "5.74.0" - resolved "https://registry.npmjs.org/webpack/-/webpack-5.74.0.tgz" - integrity sha512-A2InDwnhhGN4LYctJj6M1JEaGL7Luj6LOmyBHjcI8529cm5p6VXiTIW2sn6ffvEAKmveLzvu4jrihwXtPojlAA== + version "5.80.0" + resolved "https://registry.yarnpkg.com/webpack/-/webpack-5.80.0.tgz#3e660b4ab572be38c5e954bdaae7e2bf76010fdc" + integrity sha512-OIMiq37XK1rWO8mH9ssfFKZsXg4n6klTEDL7S8/HqbAOBBaiy8ABvXvz0dDCXeEF9gqwxSvVk611zFPjS8hJxA== dependencies: "@types/eslint-scope" "^3.7.3" - "@types/estree" "^0.0.51" - "@webassemblyjs/ast" "1.11.1" - "@webassemblyjs/wasm-edit" "1.11.1" - "@webassemblyjs/wasm-parser" "1.11.1" + "@types/estree" "^1.0.0" + "@webassemblyjs/ast" "^1.11.5" + "@webassemblyjs/wasm-edit" "^1.11.5" + "@webassemblyjs/wasm-parser" "^1.11.5" acorn "^8.7.1" acorn-import-assertions "^1.7.6" browserslist "^4.14.5" chrome-trace-event "^1.0.2" - enhanced-resolve "^5.10.0" - es-module-lexer "^0.9.0" + enhanced-resolve "^5.13.0" + es-module-lexer "^1.2.1" eslint-scope "5.1.1" events "^3.2.0" glob-to-regexp "^0.4.1" @@ -9903,9 +9910,9 @@ webpack@^5.64.4: loader-runner "^4.2.0" mime-types "^2.1.27" neo-async "^2.6.2" - schema-utils "^3.1.0" + schema-utils "^3.1.2" tapable "^2.1.1" - terser-webpack-plugin "^5.1.3" + terser-webpack-plugin "^5.3.7" watchpack "^2.4.0" webpack-sources "^3.2.3" @@ -10250,10 +10257,10 @@ yallist@^4.0.0: resolved "https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz" integrity sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A== -yaml@^1.10.0, yaml@^1.10.2, yaml@^1.7.2: - version "1.10.2" - resolved "https://registry.npmjs.org/yaml/-/yaml-1.10.2.tgz" - integrity sha512-r3vXyErRCYJ7wg28yvBY5VSoAF8ZvlcW9/BwUzEtUsjvX/DKs24dIkuwjtuprwJJHsbyUbLApepYTR1BN4uHrg== +yaml@^1.10.0, yaml@^1.10.2, yaml@^1.7.2, yaml@^2.2.2: + version "2.2.2" + resolved "https://registry.yarnpkg.com/yaml/-/yaml-2.2.2.tgz#ec551ef37326e6d42872dad1970300f8eb83a073" + integrity sha512-CBKFWExMn46Foo4cldiChEzn7S7SRV+wqiluAb6xmueD/fGyRHIhX8m14vVGgeFWjN540nKCNVj6P21eQjgTuA== yargs-parser@^20.2.2: version "20.2.9" diff --git a/nerfstudio/viewer/server/control_panel.py b/nerfstudio/viewer/server/control_panel.py index 343ebd1ae..3856fc70e 100644 --- a/nerfstudio/viewer/server/control_panel.py +++ b/nerfstudio/viewer/server/control_panel.py @@ -35,12 +35,14 @@ class ControlPanel: """ Initializes the control panel with all the elements Args: + time_enabled: whether or not the time slider should be enabled rerender_cb: a callback that will be called when the user changes a parameter that requires a rerender (eg train speed, max res, etc) crop_update_cb: a callback that will be called when the user changes the crop parameters + update_output_cb: a callback that will be called when the user changes the output render """ - def __init__(self, rerender_cb: Callable, crop_update_cb: Callable, update_output_cb: Callable): + def __init__(self, time_enabled: bool, rerender_cb: Callable, crop_update_cb: Callable, update_output_cb: Callable): # elements holds a mapping from tag: [elements] self._elements_by_tag: DefaultDict[str, List[ViewerElement]] = defaultdict(lambda: []) @@ -51,7 +53,7 @@ class ControlPanel: "Output Render", "not set", ["not set"], - cb_hook=lambda han: [self.update_control_panel(), rerender_cb(han), update_output_cb(han)], + cb_hook=lambda han: [self.update_control_panel(), update_output_cb(han), rerender_cb(han)], ) self._colormap = ViewerDropdown("Colormap", "default", ["default"], cb_hook=rerender_cb) self._invert = ViewerCheckbox("| Invert", False, cb_hook=rerender_cb) @@ -63,11 +65,13 @@ class ControlPanel: self._crop_viewport = ViewerCheckbox( "Crop Viewport", False, - cb_hook=lambda han: [self.update_control_panel(), rerender_cb(han), crop_update_cb(han)], + cb_hook=lambda han: [self.update_control_panel(), crop_update_cb(han), rerender_cb(han)], ) self._background_color = ViewerRGB("| Background color", (38, 42, 55), cb_hook=crop_update_cb) self._crop_min = ViewerVec3("| Crop Min", (-1, -1, -1), 0.05, cb_hook=crop_update_cb) self._crop_max = ViewerVec3("| Crop Max", (1, 1, 1), 0.05, cb_hook=crop_update_cb) + self._time = ViewerSlider("Time", 0.0, 0.0, 1.0, 0.01, cb_hook=rerender_cb) + self._time_enabled = time_enabled self.add_element(self._train_speed) self.add_element(self._output_render) @@ -86,6 +90,8 @@ class ControlPanel: self.add_element(self._crop_min, additional_tags=("crop",)) self.add_element(self._crop_max, additional_tags=("crop",)) + self.add_element(self._time, additional_tags=("time",)) + def _train_speed_cb(self) -> None: """Callback for when the train speed is changed""" if self.train_speed == "Fast": @@ -135,6 +141,7 @@ class ControlPanel: e.set_hidden(self.output_render == "rgb") for e in self._elements_by_tag["crop"]: e.set_hidden(not self.crop_viewport) + self._time.set_hidden(not self._time_enabled) def update_colormap_options(self, dimensions: int, dtype: type) -> None: """update the colormap options based on the current render @@ -235,3 +242,13 @@ class ControlPanel: def background_color(self, value: Tuple[int, int, int]): """Sets the background color""" self._background_color.value = value + + @property + def time(self) -> float: + """Returns the current background color""" + return self._time.value + + @time.setter + def time(self, value: float): + """Sets the background color""" + self._time.value = value diff --git a/nerfstudio/viewer/server/gui_utils.py b/nerfstudio/viewer/server/gui_utils.py index d901bff47..792d3f2b4 100644 --- a/nerfstudio/viewer/server/gui_utils.py +++ b/nerfstudio/viewer/server/gui_utils.py @@ -20,9 +20,6 @@ from typing import Any, List, Tuple from torch import nn -from nerfstudio.pipelines.base_pipeline import Pipeline -from nerfstudio.viewer.server.viewer_elements import ViewerElement - def parse_object( obj: Any, @@ -68,14 +65,3 @@ def parse_object( for ts, o in lower_rets: add(ret, ts, o) return ret - - -def get_viewer_elements(pipeline: Pipeline) -> List[Tuple[str, ViewerElement]]: - """ - Recursively parse the pipeline object and return a tree of all the ViewerElements contained - - returns a list of (path/to/object, param), which represents the path down the object tree as well as - the ViewerElement itself - """ - ret = parse_object(pipeline, ViewerElement, "Custom") - return ret diff --git a/nerfstudio/viewer/server/render_state_machine.py b/nerfstudio/viewer/server/render_state_machine.py index 835181782..bd601bfd4 100644 --- a/nerfstudio/viewer/server/render_state_machine.py +++ b/nerfstudio/viewer/server/render_state_machine.py @@ -157,7 +157,7 @@ class RenderStateMachine(threading.Thread): cy=intrinsics_matrix[1, 2], camera_type=camera_type, camera_to_worlds=camera_to_world[None, ...], - times=torch.tensor([0.0]), + times=torch.tensor([self.viewer.control_panel.time], dtype=torch.float32), ) camera = camera.to(self.viewer.get_model().device) diff --git a/nerfstudio/viewer/server/viewer_state.py b/nerfstudio/viewer/server/viewer_state.py index beee00ba7..8bccd3587 100644 --- a/nerfstudio/viewer/server/viewer_state.py +++ b/nerfstudio/viewer/server/viewer_state.py @@ -25,6 +25,7 @@ from rich import box, style from rich.console import Console from rich.panel import Panel from rich.table import Table +from typing_extensions import Literal from nerfstudio.configs import base_config as cfg from nerfstudio.data.datasets.base_dataset import InputDataset @@ -36,7 +37,7 @@ from nerfstudio.utils.io import load_from_json, write_to_json from nerfstudio.utils.writer import GLOBAL_BUFFER, EventName from nerfstudio.viewer.server import viewer_utils from nerfstudio.viewer.server.control_panel import ControlPanel -from nerfstudio.viewer.server.gui_utils import get_viewer_elements +from nerfstudio.viewer.server.gui_utils import parse_object from nerfstudio.viewer.server.render_state_machine import ( RenderAction, RenderStateMachine, @@ -48,9 +49,10 @@ from nerfstudio.viewer.viser.messages import ( CameraPathOptionsRequest, CameraPathPayloadMessage, CropParamsMessage, - IsTrainingMessage, NerfstudioMessage, SaveCheckpointMessage, + TimeConditionMessage, + TrainingStateMessage, ) if TYPE_CHECKING: @@ -110,28 +112,31 @@ class ViewerState: CONSOLE.print(Panel(table, title="[bold][yellow]Viewer[/bold]", expand=False)) + self.include_time = self.pipeline.datamanager.includes_time + # viewer specific variables - self.prev_moving = False self.output_type_changed = True - self.check_done_render = True self.step = 0 - self.camera_moving = False - self.prev_camera_timestamp = 0 - self.static_fps = 1.0 - self.train_btn_state = True + self.train_btn_state: Literal["training", "paused", "completed"] = "training" + self._prev_train_state: Literal["training", "paused", "completed"] = "training" self.camera_message = None - self.viser_server = ViserServer(host="localhost", port=websocket_port) + self.viser_server = ViserServer(host=config.websocket_host, port=websocket_port) - self.viser_server.register_handler(IsTrainingMessage, self._handle_is_training) + self.viser_server.register_handler(TrainingStateMessage, self._handle_training_state_message) self.viser_server.register_handler(SaveCheckpointMessage, self._handle_save_checkpoint) self.viser_server.register_handler(CameraMessage, self._handle_camera_update) self.viser_server.register_handler(CameraPathOptionsRequest, self._handle_camera_path_option_request) self.viser_server.register_handler(CameraPathPayloadMessage, self._handle_camera_path_payload) self.viser_server.register_handler(CropParamsMessage, self._handle_crop_params_message) + if self.include_time: + self.viser_server.use_time_conditioning() + self.viser_server.register_handler(TimeConditionMessage, self._handle_time_condition_message) - self.control_panel = ControlPanel(self._interrupt_render, self._crop_params_update, self._output_type_change) + self.control_panel = ControlPanel( + self.include_time, self._interrupt_render, self._crop_params_update, self._output_type_change + ) self.control_panel.install(self.viser_server) def nested_folder_install(folder_labels: List[str], element: ViewerElement): @@ -144,7 +149,11 @@ class ViewerState: with self.viser_server.gui_folder(folder_labels[0]): nested_folder_install(folder_labels[1:], element) - self.viewer_elements = get_viewer_elements(self.pipeline) + self.viewer_elements = [] + if self.trainer is not None: + self.viewer_elements.extend(parse_object(self.trainer, ViewerElement, "Trainer")) + else: + self.viewer_elements.extend(parse_object(pipeline, ViewerElement, "Pipeline")) for param_path, element in self.viewer_elements: folder_labels = param_path.split("/")[:-1] nested_folder_install(folder_labels, element) @@ -176,15 +185,15 @@ class ViewerState: crop_center=tuple(crop_center.tolist()), ) - def _handle_is_training(self, message: NerfstudioMessage) -> None: - """Handle is_training message from viewer.""" - assert isinstance(message, IsTrainingMessage) - self.is_training = message.is_training - self.train_btn_state = message.is_training - self.viser_server.set_is_training(message.is_training) + def _handle_training_state_message(self, message: NerfstudioMessage) -> None: + """Handle training state message from viewer.""" + assert isinstance(message, TrainingStateMessage) + self.train_btn_state = message.training_state + self.training_state = message.training_state + self.viser_server.set_training_state(message.training_state) def _handle_save_checkpoint(self, message: NerfstudioMessage) -> None: - """Handle is_training message from viewer.""" + """Handle save checkpoint message from viewer.""" assert isinstance(message, SaveCheckpointMessage) if self.trainer is not None: self.trainer.save_checkpoint(self.step) @@ -193,13 +202,13 @@ class ViewerState: """Handle camera update message from viewer.""" assert isinstance(message, CameraMessage) self.camera_message = message - self.camera_moving = message.is_moving if message.is_moving: self.render_statemachine.action(RenderAction("move", self.camera_message)) - self.is_training = False + if self.training_state == "training": + self.training_state = "paused" else: self.render_statemachine.action(RenderAction("static", self.camera_message)) - self.is_training = self.train_btn_state + self.training_state = self.train_btn_state def _handle_camera_path_option_request(self, message: NerfstudioMessage) -> None: """Handle camera path option request message from viewer.""" @@ -233,18 +242,23 @@ class ViewerState: self.control_panel.crop_min = tuple(crop_min.tolist()) self.control_panel.crop_max = tuple(crop_max.tolist()) - @property - def is_training(self) -> bool: - """Get is_training flag from viewer.""" - if self.trainer is not None: - return self.trainer.is_training - return False + def _handle_time_condition_message(self, message: NerfstudioMessage) -> None: + """Handle time conditioning message from viewer.""" + assert isinstance(message, TimeConditionMessage) + self.control_panel.time = message.time - @is_training.setter - def is_training(self, is_training: bool) -> None: - """Set is_training flag in viewer.""" + @property + def training_state(self) -> Literal["training", "paused", "completed"]: + """Get training state flag.""" if self.trainer is not None: - self.trainer.is_training = is_training + return self.trainer.training_state + return self.train_btn_state + + @training_state.setter + def training_state(self, training_state: Literal["training", "paused", "completed"]) -> None: + """Set training state flag.""" + if self.trainer is not None: + self.trainer.training_state = training_state def _pick_drawn_image_idxs(self, total_num: int) -> list[int]: """Determine indicies of images to display in viewer. @@ -262,13 +276,12 @@ class ViewerState: # draw indices, roughly evenly spaced return np.linspace(0, total_num - 1, num_display_images, dtype=np.int32).tolist() - def init_scene(self, dataset: InputDataset, start_train=True) -> None: + def init_scene(self, dataset: InputDataset, train_state=Literal["training", "paused", "completed"]) -> None: """Draw some images and the scene aabb in the viewer. Args: dataset: dataset to render in the scene - start_train: whether to start train when viewer init; - if False, only displays dataset until resume train is toggled + train_state: Current status of training """ self.viser_server.send_file_path_info( config_base_dir=self.log_filename.parents[0], @@ -288,8 +301,8 @@ class ViewerState: self.viser_server.update_scene_box(dataset.scene_box) # set the initial state whether to train or not - self.train_btn_state = start_train - self.viser_server.set_is_training(start_train) + self.train_btn_state = train_state + self.viser_server.set_training_state(train_state) def update_scene(self, step: int, num_rays_per_batch: Optional[int] = None) -> None: """updates the scene based on the graph weights @@ -303,7 +316,11 @@ class ViewerState: if self.camera_message is None: return - if self.trainer is not None and self.trainer.is_training: + if ( + self.trainer is not None + and self.trainer.training_state == "training" + and self.control_panel.train_util != 1 + ): if ( EventName.TRAIN_RAYS_PER_SEC.value in GLOBAL_BUFFER["events"] and EventName.VIS_RAYS_PER_SEC.value in GLOBAL_BUFFER["events"] @@ -337,3 +354,8 @@ class ViewerState: def get_model(self) -> Model: """Returns the model.""" return self.pipeline.model + + def training_complete(self) -> None: + """Called when training is complete.""" + self.training_state = "completed" + self.viser_server.set_training_state("completed") diff --git a/nerfstudio/viewer/viser/message_api.py b/nerfstudio/viewer/viser/message_api.py index c0471af69..9c7a56ddd 100644 --- a/nerfstudio/viewer/viser/message_api.py +++ b/nerfstudio/viewer/viser/message_api.py @@ -439,13 +439,13 @@ class MessageApi(abc.ABC): """ self._queue(messages.DatasetImageMessage(idx=idx, json=json)) - def set_is_training(self, is_training: bool) -> None: + def set_training_state(self, training_state: Literal["training", "paused", "completed"]) -> None: """Set the training mode. Args: - is_training: The training mode. + training_state: The training mode. """ - self._queue(messages.IsTrainingMessage(is_training=is_training)) + self._queue(messages.TrainingStateMessage(training_state=training_state)) def send_camera_paths(self, camera_paths: Dict[str, Any]) -> None: """Send camera paths to the scene. @@ -532,6 +532,10 @@ class MessageApi(abc.ABC): ) return GuiHandle(handle_state) + def use_time_conditioning(self) -> None: + """Use time conditioning.""" + self._queue(messages.UseTimeConditioningMessage()) + def _handle_gui_updates( self: MessageApi, client_id: ClientId, diff --git a/nerfstudio/viewer/viser/messages.py b/nerfstudio/viewer/viser/messages.py index dda627d36..de4245d14 100644 --- a/nerfstudio/viewer/viser/messages.py +++ b/nerfstudio/viewer/viser/messages.py @@ -45,7 +45,7 @@ class GuiAddMessage(NerfstudioMessage): """Sent server->client to add a new GUI input.""" name: str - folder_labels: Tuple[str] + folder_labels: Tuple[str, ...] leva_conf: Any @override @@ -171,10 +171,10 @@ class DatasetImageMessage(NerfstudioMessage): @dataclasses.dataclass -class IsTrainingMessage(NerfstudioMessage): +class TrainingStateMessage(NerfstudioMessage): """Wheather the scene is in training mode or not.""" - is_training: bool + training_state: Literal["training", "paused", "completed"] """True if the model is currently trianing, False otherwise""" @@ -228,3 +228,16 @@ class StatusMessage(NerfstudioMessage): @dataclasses.dataclass class SaveCheckpointMessage(NerfstudioMessage): """Save checkpoint message.""" + + +@dataclasses.dataclass +class UseTimeConditioningMessage(NerfstudioMessage): + """Use time conditioning message.""" + + +@dataclasses.dataclass +class TimeConditionMessage(NerfstudioMessage): + """Time conditioning message.""" + + time: float + """ Time conditioning value """ diff --git a/pyproject.toml b/pyproject.toml index 583ab01bd..f7bb86ebf 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "nerfstudio" -version = "0.2.0" +version = "0.2.1" description = "All-in-one repository for state-of-the-art NeRFs" readme = "README.md" license = { text="Apache 2.0"} @@ -30,7 +30,7 @@ dependencies = [ "mediapy>=1.1.0", "msgpack>=1.0.4", "msgpack_numpy>=0.4.8", - "nerfacc==0.3.5", + "nerfacc==0.5.2", "open3d>=0.16.0", "opencv-python==4.6.0.66", "Pillow>=9.3.0", @@ -51,6 +51,7 @@ dependencies = [ "torchmetrics[image]>=0.9.3", "torchtyping>=0.1.4", "torchvision>=0.13.0", + "typing_extensions>=4.4.0", "viser>=0.0.5", "nuscenes-devkit>=1.1.1", "wandb>=0.13.3", @@ -121,7 +122,7 @@ dependency_links = [ include = ["nerfstudio*","scripts*"] [tool.setuptools.package-data] -"*" = ["*.json", "py.typed", "setup.bash", "setup.zsh"] +"*" = ["*.cu", "*.json", "py.typed", "setup.bash", "setup.zsh"] # black [tool.black] @@ -144,6 +145,7 @@ disable = [ "too-many-branches", "too-many-instance-attributes", "too-many-locals", + "too-many-statements", "unnecessary-ellipsis", ] diff --git a/scripts/completions/install.py b/scripts/completions/install.py index 067bea2bd..a7acc862c 100644 --- a/scripts/completions/install.py +++ b/scripts/completions/install.py @@ -31,6 +31,7 @@ HEADER_LINE = "# Source nerfstudio autocompletions." def _get_all_entry_points() -> List[str]: + # TODO: we should filter out entrypoints that are not tyro CLIs. entry_points = importlib_metadata.distribution("nerfstudio").entry_points return [x.name for x in entry_points] diff --git a/scripts/datasets/process_nuscenes_masks.py b/scripts/datasets/process_nuscenes_masks.py index cb9328f95..c86baec03 100644 --- a/scripts/datasets/process_nuscenes_masks.py +++ b/scripts/datasets/process_nuscenes_masks.py @@ -79,8 +79,11 @@ class ProcessNuScenesMasks: # Instead use BoxVisibility.NONE and make sure to rasterize box faces correctly mask = np.ones((900, 1600), dtype=np.uint8) - for box in boxes: + # If is backcam, mask the truck of the ego vehicle + if camera == "CAM_BACK": + mask[-100:] = 0 + for box in boxes: # Dont mask out static objects (static in all frames) instance_token = nusc.get("sample_annotation", box.token)["instance_token"] if not instances_is_dynamic[instance_token]: diff --git a/scripts/process_data.py b/scripts/process_data.py index e11541024..f1f873ae3 100644 --- a/scripts/process_data.py +++ b/scripts/process_data.py @@ -2,465 +2,37 @@ """Processes a video or image sequence to a nerfstudio compatible dataset.""" -import shutil import sys import zipfile from dataclasses import dataclass from pathlib import Path -from typing import Dict, Optional, Tuple, Union +from typing import Union import numpy as np import tyro from rich.console import Console -from typing_extensions import Annotated, Literal +from typing_extensions import Annotated from nerfstudio.process_data import ( - colmap_utils, - equirect_utils, - hloc_utils, metashape_utils, polycam_utils, process_data_utils, realitycapture_utils, record3d_utils, ) -from nerfstudio.process_data.process_data_utils import CAMERA_MODELS -from nerfstudio.utils import install_checks +from nerfstudio.process_data.colmap_converter_to_nerfstudio_dataset import ( + BaseConverterToNerfstudioDataset, +) +from nerfstudio.process_data.images_to_nerstudio_dataset import ( + ImagesToNerfstudioDataset, +) +from nerfstudio.process_data.video_to_nerfstudio_dataset import VideoToNerfstudioDataset CONSOLE = Console(width=120) -DEFAULT_COLMAP_PATH = Path("colmap/sparse/0") @dataclass -class ProcessImages: - """Process images into a nerfstudio dataset. - - This script does the following: - - 1. Scales images to a specified size. - 2. Calculates the camera poses for each image using `COLMAP `_. - """ - - data: Path - """Path the data, either a video file or a directory of images.""" - output_dir: Path - """Path to the output directory.""" - camera_type: Literal["perspective", "fisheye", "equirectangular"] = "perspective" - """Camera model to use.""" - matching_method: Literal["exhaustive", "sequential", "vocab_tree"] = "vocab_tree" - """Feature matching method to use. Vocab tree is recommended for a balance of speed and - accuracy. Exhaustive is slower but more accurate. Sequential is faster but should only be used for videos.""" - sfm_tool: Literal["any", "colmap", "hloc"] = "any" - """Structure from motion tool to use. Colmap will use sift features, hloc can use many modern methods - such as superpoint features and superglue matcher""" - refine_pixsfm: bool = False - """If True, runs refinement using Pixel Perfect SFM. Only works with hloc sfm_tool""" - feature_type: Literal[ - "any", - "sift", - "superpoint", - "superpoint_aachen", - "superpoint_max", - "superpoint_inloc", - "r2d2", - "d2net-ss", - "sosnet", - "disk", - ] = "any" - """Type of feature to use.""" - matcher_type: Literal[ - "any", "NN", "superglue", "superglue-fast", "NN-superpoint", "NN-ratio", "NN-mutual", "adalam" - ] = "any" - """Matching algorithm.""" - num_downscales: int = 3 - """Number of times to downscale the images. Downscales by 2 each time. For example a value of 3 - will downscale the images by 2x, 4x, and 8x.""" - skip_colmap: bool = False - """If True, skips COLMAP and generates transforms.json if possible.""" - skip_image_processing: bool = False - """If True, skips copying and downscaling of images and only runs COLMAP if possible and enabled""" - colmap_model_path: Path = DEFAULT_COLMAP_PATH - """Optionally sets the path of the colmap model. Used only when --skip-colmap is set to True. - The path is relative to the output directory. - """ - colmap_cmd: str = "colmap" - """How to call the COLMAP executable.""" - images_per_equirect: Literal[8, 14] = 8 - """Number of samples per image to take from each equirectangular image. - Used only when camera-type is equirectangular. - """ - crop_factor: Tuple[float, float, float, float] = (0.0, 0.0, 0.0, 0.0) - """Portion of the image to crop. All values should be in [0,1]. (top, bottom, left, right)""" - gpu: bool = True - """If True, use GPU.""" - use_sfm_depth: bool = False - """If True, export and use depth maps induced from SfM points.""" - include_depth_debug: bool = False - """If --use-sfm-depth and this flag is True, also export debug images showing SfM overlaid upon input images.""" - verbose: bool = False - """If True, print extra logging.""" - - def main(self) -> None: # pylint: disable=R0915 - """Process images into a nerfstudio dataset.""" - # pylint: disable=too-many-statements - require_cameras_exist = False - colmap_model_path = self.output_dir / Path(self.colmap_model_path) - if self.colmap_model_path != DEFAULT_COLMAP_PATH: - if not self.skip_colmap: - CONSOLE.log("[bold red]The --colmap-model-path can only be used when --skip-colmap is not set.") - sys.exit(1) - elif not (self.output_dir / self.colmap_model_path).exists(): - CONSOLE.log( - f"[bold red]The colmap-model-path {self.output_dir / self.colmap_model_path} does not exist." - ) - sys.exit(1) - require_cameras_exist = True - - install_checks.check_ffmpeg_installed() - install_checks.check_colmap_installed() - - image_rename_map: Optional[Dict[str, str]] = None - self.output_dir.mkdir(parents=True, exist_ok=True) - image_dir = self.output_dir / "images" - image_dir.mkdir(parents=True, exist_ok=True) - - # Generate planar projections if equirectangular - if self.camera_type == "equirectangular": - pers_size = equirect_utils.compute_resolution_from_equirect(self.data, self.images_per_equirect) - CONSOLE.log(f"Generating {self.images_per_equirect} {pers_size} sized images per equirectangular image") - self.data = equirect_utils.generate_planar_projections_from_equirectangular( - self.data, pers_size, self.images_per_equirect, crop_factor=self.crop_factor - ) - - summary_log = [] - - # Copy and downscale images - if not self.skip_image_processing: - # Copy images to output directory - image_rename_map_paths = process_data_utils.copy_images( - self.data, image_dir=image_dir, crop_factor=self.crop_factor, verbose=self.verbose - ) - image_rename_map = dict((a.name, b.name) for a, b in image_rename_map_paths.items()) - num_frames = len(image_rename_map) - summary_log.append(f"Starting with {num_frames} images") - - # Downscale images - summary_log.append( - process_data_utils.downscale_images(image_dir, self.num_downscales, verbose=self.verbose) - ) - else: - num_frames = len(process_data_utils.list_images(self.data)) - if num_frames == 0: - CONSOLE.log("[bold red]:skull: No usable images in the data folder.") - sys.exit(1) - summary_log.append(f"Starting with {num_frames} images") - - # Run COLMAP - colmap_dir = self.output_dir / "colmap" - if not self.skip_colmap: - colmap_dir.mkdir(parents=True, exist_ok=True) - colmap_model_path = colmap_dir / "sparse" / "0" - require_cameras_exist = True - - self._run_colmap(image_dir, colmap_dir) - - # Colmap uses renamed images - image_rename_map = None - - # Export depth maps - if self.use_sfm_depth: - depth_dir = self.output_dir / "depth" - depth_dir.mkdir(parents=True, exist_ok=True) - image_id_to_depth_path = colmap_utils.create_sfm_depth( - recon_dir=colmap_dir / "sparse" / "0", - output_dir=depth_dir, - include_depth_debug=self.include_depth_debug, - input_images_dir=image_dir, - verbose=self.verbose, - ) - summary_log.append( - process_data_utils.downscale_images( - depth_dir, self.num_downscales, folder_name="depths", nearest_neighbor=True, verbose=self.verbose - ) - ) - else: - image_id_to_depth_path = None - - # Save transforms.json - if (colmap_model_path / "cameras.bin").exists(): - with CONSOLE.status("[bold yellow]Saving results to transforms.json", spinner="balloon"): - num_matched_frames = colmap_utils.colmap_to_json( - recon_dir=colmap_model_path, - output_dir=self.output_dir, - image_id_to_depth_path=image_id_to_depth_path, - image_rename_map=image_rename_map, - ) - summary_log.append(f"Colmap matched {num_matched_frames} images") - summary_log.append(colmap_utils.get_matching_summary(num_frames, num_matched_frames)) - elif require_cameras_exist: - CONSOLE.log(f"[bold red]Could not find existing COLMAP results ({colmap_model_path / 'cameras.bin'}).") - sys.exit(1) - else: - CONSOLE.log("[bold yellow]Warning: could not find existing COLMAP results. Not generating transforms.json") - - CONSOLE.rule("[bold green]:tada: :tada: :tada: All DONE :tada: :tada: :tada:") - - for summary in summary_log: - CONSOLE.print(summary, justify="center") - CONSOLE.rule() - - def _run_colmap(self, image_dir, colmap_dir): - (sfm_tool, feature_type, matcher_type) = process_data_utils.find_tool_feature_matcher_combination( - self.sfm_tool, self.feature_type, self.matcher_type - ) - # check that sfm_tool is hloc if using refine_pixsfm - if self.refine_pixsfm: - assert sfm_tool == "hloc", "refine_pixsfm only works with sfm_tool hloc" - - if sfm_tool == "colmap": - colmap_utils.run_colmap( - image_dir=image_dir, - colmap_dir=colmap_dir, - camera_model=CAMERA_MODELS[self.camera_type], - gpu=self.gpu, - verbose=self.verbose, - matching_method=self.matching_method, - colmap_cmd=self.colmap_cmd, - ) - elif sfm_tool == "hloc": - hloc_utils.run_hloc( - image_dir=image_dir, - colmap_dir=colmap_dir, - camera_model=CAMERA_MODELS[self.camera_type], - verbose=self.verbose, - matching_method=self.matching_method, - feature_type=feature_type, - matcher_type=matcher_type, - refine_pixsfm=self.refine_pixsfm, - ) - else: - CONSOLE.log("[bold red]Invalid combination of sfm_tool, feature_type, and matcher_type, exiting") - sys.exit(1) - - -@dataclass -class ProcessVideo: - """Process videos into a nerfstudio dataset. - - This script does the following: - - 1. Converts the video into images. - 2. Scales images to a specified size. - 3. Calculates the camera poses for each image using `COLMAP `_. - """ - - data: Path - """Path the data, either a video file or a directory of images.""" - output_dir: Path - """Path to the output directory.""" - num_frames_target: int = 300 - """Target number of frames to use for the dataset, results may not be exact.""" - camera_type: Literal["perspective", "fisheye", "equirectangular"] = "perspective" - """Camera model to use.""" - matching_method: Literal["exhaustive", "sequential", "vocab_tree"] = "vocab_tree" - """Feature matching method to use. Vocab tree is recommended for a balance of speed and - accuracy. Exhaustive is slower but more accurate. Sequential is faster but should only be used for videos.""" - sfm_tool: Literal["any", "colmap", "hloc"] = "any" - """Structure from motion tool to use. Colmap will use sift features, hloc can use many modern methods - such as superpoint features and superglue matcher""" - refine_pixsfm: bool = False - """If True, runs refinement using Pixel Perfect SFM. Only has an effect if sfm_tool is hloc""" - feature_type: Literal[ - "any", - "sift", - "superpoint", - "superpoint_aachen", - "superpoint_max", - "superpoint_inloc", - "r2d2", - "d2net-ss", - "sosnet", - "disk", - ] = "any" - """Type of feature to use.""" - matcher_type: Literal[ - "any", "NN", "superglue", "superglue-fast", "NN-superpoint", "NN-ratio", "NN-mutual", "adalam" - ] = "any" - """Matching algorithm.""" - num_downscales: int = 3 - """Number of times to downscale the images. Downscales by 2 each time. For example a value of 3 - will downscale the images by 2x, 4x, and 8x.""" - skip_colmap: bool = False - """If True, skips COLMAP and generates transforms.json if possible.""" - colmap_cmd: str = "colmap" - """How to call the COLMAP executable.""" - images_per_equirect: Literal[8, 14] = 8 - """Number of samples per image to take from each equirectangular image. - Used only when camera-type is equirectangular. - """ - percent_radius_crop: float = 1.0 - """Create circle crop mask. The radius is the percent of the image diagonal.""" - crop_factor: Tuple[float, float, float, float] = (0.0, 0.0, 0.0, 0.0) - """Portion of the image to crop. All values should be in [0,1]. (top, bottom, left, right)""" - use_sfm_depth: bool = False - """If True, export and use depth maps induced from SfM points.""" - include_depth_debug: bool = False - """If --use-sfm-depth and this flag is True, also export debug images showing SfM overlaid upon input images.""" - gpu: bool = True - """If True, use GPU.""" - verbose: bool = False - """If True, print extra logging.""" - - def main(self) -> None: # pylint: disable=R0915 - """Process video into a nerfstudio dataset.""" - install_checks.check_ffmpeg_installed() - install_checks.check_colmap_installed() - - self.output_dir.mkdir(parents=True, exist_ok=True) - image_dir = self.output_dir / "images" - image_dir.mkdir(parents=True, exist_ok=True) - - summary_log = [] - # Convert video to images - if self.camera_type == "equirectangular": - # create temp images folder to store the equirect and perspective images - temp_image_dir = self.output_dir / "temp_images" - temp_image_dir.mkdir(parents=True, exist_ok=True) - summary_log, num_extracted_frames = process_data_utils.convert_video_to_images( - self.data, - image_dir=temp_image_dir, - num_frames_target=self.num_frames_target, - crop_factor=(0.0, 0.0, 0.0, 0.0), - verbose=self.verbose, - ) - else: - summary_log, num_extracted_frames = process_data_utils.convert_video_to_images( - self.data, - image_dir=image_dir, - num_frames_target=self.num_frames_target, - crop_factor=self.crop_factor, - verbose=self.verbose, - ) - - # Generate planar projections if equirectangular - if self.camera_type == "equirectangular": - perspective_image_size = equirect_utils.compute_resolution_from_equirect( - self.output_dir / "temp_images", self.images_per_equirect - ) - image_dir = equirect_utils.generate_planar_projections_from_equirectangular( - self.output_dir / "temp_images", - perspective_image_size, - self.images_per_equirect, - crop_factor=self.crop_factor, - ) - - # copy the perspective images to the image directory - process_data_utils.copy_images( - self.output_dir / "temp_images" / "planar_projections", - image_dir=self.output_dir / "images", - verbose=False, - ) - image_dir = self.output_dir / "images" - - # remove the temp_images folder - shutil.rmtree(self.output_dir / "temp_images", ignore_errors=True) - - # # Create mask - mask_path = process_data_utils.save_mask( - image_dir=image_dir, - num_downscales=self.num_downscales, - crop_factor=(0.0, 0.0, 0.0, 0.0), - percent_radius=self.percent_radius_crop, - ) - if mask_path is not None: - summary_log.append(f"Saved mask to {mask_path}") - - # # Downscale images - summary_log.append(process_data_utils.downscale_images(image_dir, self.num_downscales, verbose=self.verbose)) - - # Run Colmap - colmap_dir = self.output_dir / "colmap" - if not self.skip_colmap: - colmap_dir.mkdir(parents=True, exist_ok=True) - - (sfm_tool, feature_type, matcher_type) = process_data_utils.find_tool_feature_matcher_combination( - self.sfm_tool, self.feature_type, self.matcher_type - ) - - if sfm_tool == "colmap": - colmap_utils.run_colmap( - image_dir=image_dir, - colmap_dir=colmap_dir, - camera_model=CAMERA_MODELS[self.camera_type], - camera_mask_path=mask_path, - gpu=self.gpu, - verbose=self.verbose, - matching_method=self.matching_method, - colmap_cmd=self.colmap_cmd, - ) - elif sfm_tool == "hloc": - if mask_path is not None: - CONSOLE.log( - "[bold red]Cannot use a mask with hloc. Please remove the cropping options and try again." - ) - sys.exit(1) - hloc_utils.run_hloc( - image_dir=image_dir, - colmap_dir=colmap_dir, - camera_model=CAMERA_MODELS[self.camera_type], - verbose=self.verbose, - matching_method=self.matching_method, - feature_type=feature_type, - matcher_type=matcher_type, - refine_pixsfm=self.refine_pixsfm, - ) - else: - CONSOLE.log("[bold red]Invalid combination of sfm_tool, feature_type, and matcher_type, exiting") - sys.exit(1) - - # Export depth maps - if self.use_sfm_depth: - depth_dir = self.output_dir / "depth" - depth_dir.mkdir(parents=True, exist_ok=True) - image_id_to_depth_path = colmap_utils.create_sfm_depth( - recon_dir=colmap_dir / "sparse" / "0", - output_dir=depth_dir, - include_depth_debug=self.include_depth_debug, - input_images_dir=image_dir, - verbose=self.verbose, - ) - summary_log.append( - process_data_utils.downscale_images( - depth_dir, self.num_downscales, folder_name="depths", nearest_neighbor=True, verbose=self.verbose - ) - ) - else: - image_id_to_depth_path = None - - # Save transforms.json - if (colmap_dir / "sparse" / "0" / "cameras.bin").exists(): - with CONSOLE.status("[bold yellow]Saving results to transforms.json", spinner="balloon"): - num_matched_frames = colmap_utils.colmap_to_json( - recon_dir=colmap_dir / "sparse" / "0", - output_dir=self.output_dir, - image_id_to_depth_path=image_id_to_depth_path, - camera_mask_path=mask_path, - image_rename_map=None, - ) - summary_log.append(f"Colmap matched {num_matched_frames} images") - summary_log.append(colmap_utils.get_matching_summary(num_extracted_frames, num_matched_frames)) - else: - CONSOLE.log("[bold yellow]Warning: could not find existing COLMAP results. Not generating transforms.json") - - CONSOLE.rule("[bold green]:tada: :tada: :tada: All DONE :tada: :tada: :tada:") - - for summary in summary_log: - CONSOLE.print(summary, justify="center") - CONSOLE.rule() - - -@dataclass -class ProcessRecord3D: +class ProcessRecord3D(BaseConverterToNerfstudioDataset): """Process Record3D data into a nerfstudio dataset. This script does the following: @@ -469,10 +41,6 @@ class ProcessRecord3D: 2. Converts Record3D poses into the nerfstudio format. """ - data: Path - """Path to the record3D data.""" - output_dir: Path - """Path to the output directory.""" num_downscales: int = 3 """Number of times to downscale the images. Downscales by 2 each time. For example a value of 3 will downscale the images by 2x, 4x, and 8x.""" @@ -480,9 +48,6 @@ class ProcessRecord3D: """Max number of images to train on. If the dataset has more, images will be sampled approximately evenly. If -1, use all images.""" - verbose: bool = False - """If True, print extra logging.""" - def main(self) -> None: """Process images into a nerfstudio dataset.""" @@ -537,7 +102,7 @@ class ProcessRecord3D: @dataclass -class ProcessPolycam: +class ProcessPolycam(BaseConverterToNerfstudioDataset): """Process Polycam data into a nerfstudio dataset. To capture data, use the Polycam app on an iPhone or iPad with LiDAR. The capture must be in LiDAR or ROOM mode. @@ -550,10 +115,6 @@ class ProcessPolycam: 2. Converts Polycam poses into the nerfstudio format. """ - data: Path - """Path the polycam export data folder. Can be .zip file or folder.""" - output_dir: Path - """Path to the output directory.""" num_downscales: int = 3 """Number of times to downscale the images. Downscales by 2 each time. For example a value of 3 will downscale the images by 2x, 4x, and 8x.""" @@ -568,8 +129,6 @@ class ProcessPolycam: """Number of pixels to crop from each border of the image. Useful as borders may be black due to undistortion.""" use_depth: bool = False """If True, processes the generated depth maps from Polycam""" - verbose: bool = False - """If True, print extra logging.""" def main(self) -> None: """Process images into a nerfstudio dataset.""" @@ -648,7 +207,15 @@ class ProcessPolycam: @dataclass -class ProcessMetashape: +class _NoDefaultProcessMetashape: + """Private class to order the parameters of ProcessMetashape in the right order for default values.""" + + xml: Path + """Path to the Metashape xml file.""" + + +@dataclass +class ProcessMetashape(BaseConverterToNerfstudioDataset, _NoDefaultProcessMetashape): """Process Metashape data into a nerfstudio dataset. This script assumes that cameras have been aligned using Metashape. After alignment, it is necessary to export the @@ -660,20 +227,12 @@ class ProcessMetashape: 2. Converts Metashape poses into the nerfstudio format. """ - data: Path - """Path to a folder of images.""" - xml: Path - """Path to the Metashape xml file.""" - output_dir: Path - """Path to the output directory.""" num_downscales: int = 3 """Number of times to downscale the images. Downscales by 2 each time. For example a value of 3 will downscale the images by 2x, 4x, and 8x.""" max_dataset_size: int = 600 """Max number of images to train on. If the dataset has more, images will be sampled approximately evenly. If -1, use all images.""" - verbose: bool = False - """If True, print extra logging.""" def main(self) -> None: """Process images into a nerfstudio dataset.""" @@ -735,7 +294,15 @@ class ProcessMetashape: @dataclass -class ProcessRealityCapture: +class _NoDefaultProcessRealityCapture: + """Private class to order the parameters of ProcessRealityCapture in the right order for default values.""" + + csv: Path + """Path to the RealityCapture cameras CSV file.""" + + +@dataclass +class ProcessRealityCapture(BaseConverterToNerfstudioDataset, _NoDefaultProcessRealityCapture): """Process RealityCapture data into a nerfstudio dataset. This script assumes that cameras have been aligned using RealityCapture. After alignment, it is necessary to @@ -747,20 +314,12 @@ class ProcessRealityCapture: 2. Converts RealityCapture poses into the nerfstudio format. """ - data: Path - """Path to a folder of images.""" - csv: Path - """Path to the RealityCapture cameras CSV file.""" - output_dir: Path - """Path to the output directory.""" num_downscales: int = 3 """Number of times to downscale the images. Downscales by 2 each time. For example a value of 3 will downscale the images by 2x, 4x, and 8x.""" max_dataset_size: int = 600 """Max number of images to train on. If the dataset has more, images will be sampled approximately evenly. If -1, use all images.""" - verbose: bool = False - """If True, print extra logging.""" def main(self) -> None: """Process images into a nerfstudio dataset.""" @@ -821,8 +380,8 @@ class ProcessRealityCapture: Commands = Union[ - Annotated[ProcessImages, tyro.conf.subcommand(name="images")], - Annotated[ProcessVideo, tyro.conf.subcommand(name="video")], + Annotated[ImagesToNerfstudioDataset, tyro.conf.subcommand(name="images")], + Annotated[VideoToNerfstudioDataset, tyro.conf.subcommand(name="video")], Annotated[ProcessPolycam, tyro.conf.subcommand(name="polycam")], Annotated[ProcessMetashape, tyro.conf.subcommand(name="metashape")], Annotated[ProcessRealityCapture, tyro.conf.subcommand(name="realitycapture")], @@ -833,7 +392,10 @@ Commands = Union[ def entrypoint(): """Entrypoint for use with pyproject scripts.""" tyro.extras.set_accent_color("bright_yellow") - tyro.cli(Commands).main() + try: + tyro.cli(Commands).main() + except RuntimeError as e: + CONSOLE.log("[bold red]" + e.args[0]) if __name__ == "__main__": diff --git a/scripts/render.py b/scripts/render.py index 9ea9e57ce..214c5612a 100644 --- a/scripts/render.py +++ b/scripts/render.py @@ -17,7 +17,9 @@ import mediapy as media import numpy as np import torch import tyro +from rich import box, style from rich.console import Console +from rich.panel import Panel from rich.progress import ( BarColumn, Progress, @@ -25,6 +27,7 @@ from rich.progress import ( TextColumn, TimeRemainingColumn, ) +from rich.table import Table from torchtyping import TensorType from typing_extensions import Literal, assert_never @@ -80,8 +83,8 @@ def _render_trajectory_video( ItersPerSecColumn(suffix="fps"), TimeRemainingColumn(elapsed_when_finished=True, compact=True), ) + output_image_dir = output_filename.parent / output_filename.stem if output_format == "images": - output_image_dir = output_filename.parent / output_filename.stem output_image_dir.mkdir(parents=True, exist_ok=True) if output_format == "video": # make the folder if it doesn't exist @@ -143,9 +146,19 @@ def _render_trajectory_video( ) writer.add_image(render_image) + table = Table( + title=None, + show_header=False, + box=box.MINIMAL, + title_style=style.Style(bold=True), + ) if output_format == "video": if camera_type == CameraType.EQUIRECTANGULAR: insert_spherical_metadata_into_file(output_filename) + table.add_row("Video", str(output_filename)) + else: + table.add_row("Images", str(output_image_dir)) + CONSOLE.print(Panel(table, title="[bold][green]:tada: Render Complete :tada:[/bold]", expand=False)) def insert_spherical_metadata_into_file( diff --git a/scripts/viewer/run_viewer.py b/scripts/viewer/run_viewer.py index 1c38cf10f..67d96ae2a 100644 --- a/scripts/viewer/run_viewer.py +++ b/scripts/viewer/run_viewer.py @@ -132,9 +132,9 @@ def _start_viewer(config: TrainerConfig, pipeline: Pipeline, step: int): assert viewer_state and pipeline.datamanager.train_dataset viewer_state.init_scene( dataset=pipeline.datamanager.train_dataset, - start_train=False, + train_state="completed", ) - viewer_state.viser_server.set_is_training(False) + viewer_state.viser_server.set_training_state("completed") viewer_state.update_scene(step=step) while True: time.sleep(0.01) diff --git a/scripts/viewer/sync_viser_message_defs.py b/scripts/viewer/sync_viser_message_defs.py index ef0b380d7..369836acb 100644 --- a/scripts/viewer/sync_viser_message_defs.py +++ b/scripts/viewer/sync_viser_message_defs.py @@ -1,14 +1,15 @@ """Generate viser message definitions for TypeScript, by parsing Python dataclasses.""" - +import json import pathlib -import subprocess +from datetime import datetime +import tyro from viser.infra import generate_typescript_interfaces from nerfstudio.viewer.viser import NerfstudioMessage -def entrypoint() -> None: +def main() -> None: """Generate viser message definitions for TypeScript, by parsing Python dataclasses.""" # Generate typescript source. @@ -23,13 +24,30 @@ def entrypoint() -> None: "nerfstudio/viewer/app/src/modules/WebSocket/ViserMessages.tsx" ) assert target_path.exists() - target_path.write_text(defs, encoding="utf_8") - print(f"Wrote to {target_path}") - # Run prettier. - # TODO: if this is not installed maybe we should print some error with installation - # instructions? - subprocess.run(args=["prettier", "-w", str(target_path)], check=False) + old_defs = target_path.read_text(encoding="utf_8") + + if old_defs != defs: + target_path.write_text(defs, encoding="utf_8") + + with open("nerfstudio/viewer/app/package.json", "r", encoding="utf_8") as f: + data = json.load(f) + + now = datetime.now() + data["version"] = now.strftime("%y-%m-%d") + "-0" + + with open("nerfstudio/viewer/app/package.json", "w", encoding="utf_8") as f: + json.dump(data, f, indent=2) + print(f"Wrote updates to {target_path}") + print(f"Current viewer version is now {data['version']}") + else: + print("No update to messages.") + + +def entrypoint() -> None: + """Entrypoint for use with pyproject scripts.""" + # All entrypoints must currently be tyro CLIs. + tyro.cli(main) if __name__ == "__main__": diff --git a/tests/process_data/test_process_images.py b/tests/process_data/test_process_images.py index 1b86a5adb..7f27e5cf5 100644 --- a/tests/process_data/test_process_images.py +++ b/tests/process_data/test_process_images.py @@ -16,7 +16,9 @@ from nerfstudio.data.utils.colmap_parsing_utils import ( write_cameras_binary, write_images_binary, ) -from scripts.process_data import ProcessImages +from nerfstudio.process_data.images_to_nerstudio_dataset import ( + ImagesToNerfstudioDataset, +) def random_quaternion(num_poses: int): @@ -79,8 +81,10 @@ def test_process_images_skip_colmap(tmp_path: Path): (tmp_path / "mocked_bin" / "colmap").touch(mode=0o777) (tmp_path / "mocked_bin" / "ffmpeg").touch(mode=0o777) - # Run ProcessImages - cmd = ProcessImages(tmp_path / "images", tmp_path / "nerfstudio", colmap_model_path=sparse_path, skip_colmap=True) + # Convert images into a NerfStudio dataset + cmd = ImagesToNerfstudioDataset( + data=tmp_path / "images", output_dir=tmp_path / "nerfstudio", colmap_model_path=sparse_path, skip_colmap=True + ) cmd.main() os.environ["PATH"] = old_path diff --git a/tests/utils/test_aabb_intersection.py b/tests/utils/test_aabb_intersection.py index 1d6ba34b9..ab8e2ce0e 100644 --- a/tests/utils/test_aabb_intersection.py +++ b/tests/utils/test_aabb_intersection.py @@ -168,7 +168,11 @@ def test_equall_nerfacc(): # time2 = time.time() # time3 = time.time() - t_min_nerfacc, t_max_nerfacc = nerfacc.ray_aabb_intersect(origins, directions, aabb) + t_min_nerfacc, t_max_nerfacc, _ = nerfacc.ray_aabb_intersect( + origins, directions, aabb[None, :], near_plane=0, far_plane=1e10, miss_value=1e10 + ) + t_max_nerfacc = t_max_nerfacc.squeeze(-1) + t_min_nerfacc = t_min_nerfacc.squeeze(-1) # time4 = time.time() # print("pytorch ", time2-time1)