Commit Graph
78 Commits
Author SHA1 Message Date
drewgilliam 06e59ed1ec camera/plugin docs (#2412)
* api docs for cameras & plugins

* fix docstrings
2023-09-08 09:14:16 -07:00
85ef567682 Flesh out viewer beta (#2086)
* Increase max_res of first proposal network

* fix render statemachine logic for viewer beta

* clicking on cameras in the viewer

* fix fov calculation, update camera positions while training

* cleanup

* backup

* fix configure theme

* Fix theme configure for latest dev version of viser

* Fix runtime errors

* add set up direction button

* add pause resume button

* Export panel

* Render tab skeleton

* make responsiveness more snappy by stopping training

* depth composite

* start implementing viewercontrol in new viewer

* refactor background img to use depth and add depth layering option

* show/hide images

* speed up rendering

* fix crop color background

* crop

* rotated crops

* rotated crops

* transform handle on crop box

* fix xyzw for crop

* oriented crop export for pointclouds, not for meshes yet

* Initial render panel

* Fix export button

* Theme tweaks

* fix background issue in cropping

* export rotated poisson meshe

* double normals definition in export panel

* Visual nits

* remove tsdf from export panel

* Add licenses

* Fix some CI errors

* Type fixes

* Fix tests

* Fix numpy broken

* various bugfix

---------

Co-authored-by: Brent Yi <yibrenth@gmail.com>
Co-authored-by: terrancewang <terrance.3512@gmail.com>
2023-09-08 05:10:00 +00:00
Matias Turkulainen 4fe8e657d2 Rename num_cameras to camera_indices (#2326)
rename num_cameras -> camera_indices
2023-08-10 15:36:01 -07:00
Cyrus Vachha 8682e481b1 Render VR180 video (#2261)
* Render VR180 video

Adding the VR180 camera model to render VR180 video and images. I condensed the omnidirectional stereo and VR180 render post processing in the render script. I have updated the documentation for VR180 support.

* Condensing formatting of camera types

Condensing the if statement for camera type intrinsics.
2023-07-24 07:26:20 -07:00
Cyrus Vachha 554dec77f2 Omnidirectional Stereo Update - Alignment and Compositing (#2217)
Update to omnidirectional stereo rendering (VR 360) to fix slight positional alignment to accurately composite with Blender objects. Modified documentation for ODS and the Blender add-on to include details on Blender camera settings to composite with the Nerfstudio ODS camera model.
2023-07-13 07:59:10 -07:00
Matias TurkulainenandMatthew Tancik e0e66ce9fb Add non-trainable camera indices to CameraOptimizer (#2080)
* add non-trainable indices to camera optimizer

* improve comment

* create on device

---------

Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-06-18 12:55:13 -04:00
b17268e83b [WIP] dreamfusion implementation (#1008)
* starting the framework for dreamfusion

* Removed view direction from RGB to calculate albedo

* Started diffusion

* Added dreamfusion to CLI, removed direction embeddings

* Normals in dreamfusion

* Lambertian shading added

* Train normals in Dreamfusion

* Stable diffusion from Stable Dreamfusion

* Fix normal orientation loss

* Configurable light direction sampling

* adding density initalization at origin

* V1 datamanager

* Don't train normals for first 1k iters

* SDS loss added to dreamfusion

* Dreamfusion pipeline

* Background MLP and SphereCollider

* Normals prediction training fixed

* Sigmoid on background, prompt in config

* Shading hyperparameter adjustements

* Shading hyperparameter tuning

* Location based prompting, density initialization fades, LR change

* Cleaned imports, more config options

* Tuning, more config changes

* remove unused sampler, normalize predicted normals from network

* change position encoding to direction encoding for background color mlp

* Updating dreamfusion to use custom trainer, fixed bug, multi-gpu support (put diffusion on one specific device), merged with main

* Hard coding less, offloading to config

* Appeasing the linter

* First pass opacity / sparsity losses

* Small patches and configs

* Accidentally pushed logs

* Removing logs for real this time

* Transmittance loss bug fix

* 12 GB Dreamfusion!

* Lint

* Remove unecissary collider changes

* Start SD cleanup

* Cleanup SD code

* adding overhead location prompting

* Use lambertian renderer

* turn grad accum off by default, decrease background mlp size, add overhead text embedding

* Cleanup

* Fix normals vis

* change uniform sampler to prediction network, add option to jit unet for faster sds_loss

* Use shaders

* cleaning up jit code

* add warning if using non jit unet

* add proposal network optimizer to config

* speed up render and cleanup code

* Clean up datamanager

* Add distortion loss

* fix wanb

* adding interpolated location sampling

* Update configs

* interpolated location prompting option, lowering init density, transmittance loss interpolation flipped

* rename alphas mult to alphas loss mult

* Mess around with normals

* add option to use random bg color + bg mlp

* Remove Dreamfusion Trainer (#1423)

* Dreamfusion trainer no longer needed

* Added typing for grad scaler

* SDS return loss

* Fix CFG

* Cleanup

* Update configs

* Fix opacity loss, use unform sampling first round of prop

* add prompt edditing to viewer

* Choose your own diffusion

* remove dreamfusion pipeline file

* merging + remove dreamfusion pipeline file

* Update configs

* add stable diff utils

* cleanup

* cleanup

* Fix test

* positional prompting bugfix

* Warmups

* tune

* lint

* Remove dataparser

* merging + text embedding bugfix

* Set data path to launch viewer for DreamFusion (#1776)

* Set data path to launch viewer for DreamFusion

* Apply formatter

* Fallback to 'base_dir' if data manager has no path

* Make if statement non-inline

* Reformat

---------

Co-authored-by: Matthew Tancik <mattt4001@gmail.com>

* add deepfloyd option first try

* merging

* merging from main

* merging

* removing comments, cleaning code, making stable diff utils usable for deepfloyd, remove deepfloyd text encoder after use

* undo chagnes in data folder

* cleaning code, removing unneccesary changes from main, removed normals prediction network

* remove old scripts files, rename generative pipeline flag to no_input_data, revert to default histogram padding in proposal sampler

* changing pipeline flag

* move no input data flag from pipeline to datamanager

* undo config and merge in deepfloyd from main

* merging grad scaler pipeline addition

* merging + setting proposal net scheduler

* revert nerfacto config change

* typing + linting

* rename dreamfusion datamanager to random camras datamanager, check datamanager typing in pipeline and trainer

* fix test import

* fix test import

* changing datamanager check in trainer, add typing in random cameras datamanager

* add Tensor to randomcamerasdatamanager

* remove old comments

* resolve tyro warning

* update fields to use tcnn

* lint

* add packages for deepfloyd

* add versions

* docs first draft

* rename to generfacto, clean docs

* fix import error

* minor changes

* fix orientation loss bug, retune params

* cleanup

* typing

* trying docs fix

* fixing generative package imports

* fix pyright

* docs fix

* merge w main

* docs minor change

* tuning + bugfixes

* remove torch.backends

---------

Co-authored-by: Abhik Ahuja <ahujaabhik@berkeley.edu>
Co-authored-by: David McAllister <davidrmc2@gmail.com>
Co-authored-by: Jake Austin <jake-austin@berkeley.edu>
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
Co-authored-by: Matt Tancik <mattt4001@gmail.com>
Co-authored-by: SeungWoo Yoo <dreamy1534@gmail.com>
Co-authored-by: ayaanzhaque <ayaanzhaque@gmail.com>
Co-authored-by: Ayaan Haque <ayaanzhaque@berkeley.edu>
2023-06-17 23:34:04 -07:00
Matias Turkulainen 742dfd96a7 Add antialias parameter as default None (#2033)
add antialias parameter as default None
2023-06-04 19:19:18 -07:00
Cyrus Vachha 6993f75dc8 Render Omni-directional stereo video (360 3D VR) (#1986)
* Render Omni-directional stereo (360 3D VR)

Adding the omni-directional stereo camera model to render 3D 360 video and images for VR. This includes the camera models for the left and right eyes and additions to the render script to stack the left and right renders for the final render.
I have also updated the documentation on the custom_dataset page to have information on ODS but let me know if there is a better place to put it.

* Updated ODS camera model and render paths

Updates based on suggested code changes including adding a helper function for ODS and improving path creation.
I also added a part where if a user tries to render a video with an output path that doesn't have an extension it will automatically add .mp4 to the end. Otherwise this usually results in an error when there is no file extension when rendering a video.

* Updated formatting for render and cam path scripts

Updated python formatting for render and camera_paths script

* Adding type checking for ODS function and reformatting camera type code

Added the type checking and doc string for the ODS camera rays function and reformatted the cam_type ray if statements.
2023-05-26 18:01:13 -07:00
Milan van Wouden d413ca4204 fix large number of typos in repository (#1977)
* fix large number of typos in repository

* fix typos in tests
2023-05-23 10:41:49 -07:00
Jonáš Kulhánekandmachenmusik 3bc160cc7d Backward compatibility for torch 1.x and fix for windows (#1962)
* backward compatibility for torch 1.x and fix for windows

* Update pyproject.toml

* Update README.md

* Update README.md

(As per request)

---------

Co-authored-by: machenmusik <machenmusik@users.noreply.github.com>
2023-05-22 08:58:08 -07:00
Jonáš Kulhánek 08e3a1f1a6 Drop backward compatibility for Torch 1.13 (#1948)
* Drop backward compatibility for Torch 1.13

* Fix lint errors
2023-05-18 09:12:51 +02:00
Jonáš KulhánekandBrent Yi f481f5ebec Fix some typing problems (#1940)
* Fix some typing problems

* nerfstudio.data typing error fixes

* Add tiny fixes

* Remove ABC from pipeline

* Prevent circular imports

* Tweak ModuleList typing

* TrainingCallbackAttributes cleanup

* Various minor nits

* More nits

* Run `ruff`

---------

Co-authored-by: Brent Yi <yibrenth@gmail.com>
2023-05-17 17:36:56 -07:00
e898f562e6 Switching to Torch 2.0 by default. (#1922)
* Update requirements for torch 2.0

* Update docs

* Fix missing _LRScheduler import

* Fix pylint issues

* Update docs, drop 1.13

* Fix pip postfix version

* fix docs

* Allow python 1.13 (for now)

* Minor fix for torch.compile

* Switch to torch.compile

* ruff

* Fix test

---------

Co-authored-by: Matt Tancik <mattt4001@gmail.com>
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-05-17 16:03:17 -07:00
Brent YiandMatt Tancik d542646c72 Switch to ruff (#1939)
* `isort` -> `ruff`

* `pylint` -> `ruff`

* Remove line

* More minor cleanup

* Better documentation for rules we are enabling

* Fix gitignore, run black

* vscode settings

---------

Co-authored-by: Matt Tancik <mattt4001@gmail.com>
2023-05-17 10:08:14 -07:00
Brent Yi 814db32ec4 More typing fixes (#1941)
* Fix more types

* Fix tests

* Minor cleaning

* Address comments

* Resolve linting errors

* Re-ignore pymeshlab type

* Add extra assert / error message to mipnerf model
2023-05-17 07:17:40 -07:00
Matthew Tancik 8b9574bb5a Type fixes and cleanup (#1937) 2023-05-16 18:47:46 -07:00
Brent YiandMatt Tancik 6ba2a18b2b torchtyping => jaxtyping (#1838)
* Run `ruff --fix`

* Add jaxtyping migration script

* Run migration scripts + ruff + black

* Merge remote-tracking branch 'origin/main' into brent/jaxtyping

* black

* Fix lint

* fix lint

* Cleanup and more specific types

* Fixed annotation errors

* Fix more errors

* Fix annotation

* Delete _torchtyping_to_jaxtyping.py

---------

Co-authored-by: Matt Tancik <mattt4001@gmail.com>
2023-05-16 15:57:47 -07:00
Matthew Tancik 531f9d0c4c Split up render commands (#1923)
* Split up render commands

* bump viewer
2023-05-15 11:34:40 -07:00
Ethan Weber 13e0fce122 Update copyright (#1917)
* updated copyright

* including txt file

* updated black

* removing license

* fixing license
2023-05-12 16:16:13 -07:00
bc2651963c Improved interpolate render (#1911)
* [WIP] Speed is adjusted depending on camera distance

* [WIP] Speed is adjusted depending on camera distance

* Speed is adjusted depending on camera distance

* Constant time between poses and ordered poses

* Removed old speed adjust

* remove prints

* renamed variables

* added launch.json back

* added back blank lines

* Trying to solve blank line git  issue in launch.json

* Added docstring and type annotations

---------

Co-authored-by: jperezh <jperezh@arquimea.com>
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-05-12 15:19:13 -07:00
Brent Yi 63c6303736 Bump minimum Python version to 3.8 (#1876)
* Bump minimum Python version to 3.8

* typing_extensions.Literal -> typing.Literal

* ns-dev-test

* typing_extensions.get_args => typing.get_args

* Format with older version of black

* Bump `black`

* Run black
2023-05-06 13:29:48 -07:00
Matthew Tancik 9ace02b3ae Remove nerfacc aabb (#1874)
* Remove nerfacc aabb

* remove custom nerfacc logic

* lint
2023-05-04 10:46:51 -07:00
Ruilong Li(李瑞龙) ea2993d5b1 Update NGP related models with nerfacc 0.5.2 (#1809)
* profiling

* set pose_optimizer to None if it's mode is "off"

* radial_and_tangential_undistort in nerfacc

* fix camera distortions

* update to nerfacc 0.5.1. Upgrade instant-ngp model.

* revert nerfacc to 0.5.1

* format

* allows to fall back to the pytorch undistortion

* format

* fix linting

* fix test

* fix test

* fix nerfplayer-ngp

* enable_collider=False for NGP

* nerfplayer verified.

* zero cone_angle for ngp bounded

* fix format

* fix pylint
2023-04-24 17:11:30 -07:00
Jake Austin db03d5792e Metadata for cameras (#1623)
Adding metadata field to cameras explicitly, will be broadcast to rays generated
2023-04-14 20:56:25 -07:00
Jingchen Ye d59c1bdc2e Remove useless code in camera_paths (#1717)
* Remove useless code

* Minor typo fix in docstrings
2023-04-10 09:26:09 -07:00
Pablo Vela ddfa463f4d NeuS Implementation (#1444)
* add initial method configs and data manager

* add neus scheduler

* add initial surface model and neus

* add Neus ray sampler

* add sdf field

* move scale/shift calc to math utils

* add debug and fix sdf_dataset typo

* update vscode neus debug

* fix dataset and get initial working version

* add magic number from original repo

* black fixes

* clean up get weights/transmittance from alphas

* clean up typos in sdf datamanager

* type annotations on neus scheduler and remove unused params

* clean up and annotate sdf field along w/ other fixes

* more clean up

* remove unnessary directions norm

* resolve comments

* update neus scheduler to follow refactor

* fix linter errors

* fix some typos

* remove get_weights_from_alpha

* more comment fixes

* more linter error fixes

* clean up and add comments

* more little fixes for types/comments

* more comments

* rename NeuS Scheduler to DelayedScheduler

* rename DelayedScheduler to CosineDecayScheduler
2023-03-12 16:01:18 -04:00
Jonáš Kulhánek d9a391f79a Fixed: MinimalDataParser add options, allow Nerfstudio data format to specify test indices (#1586)
* MinimalDataParser add options, allow Nerfstudio data format to specify test indices

* Switch from indices to filenames and add tests

* Fix  #1581

* Fix changed interface
2023-03-11 22:14:45 +01:00
Frédéric DevernayandFrédéric Devernay a277cd4886 camera_utils: add "vertical" orientation and "focus" center methods (#1543)
* camera_utils: add "vertical" orientation and "focus" center methods

* camera_utils: -Z is the look-at direction

* Update nerfstudio_dataparser.py

* missing import

* fix pylint issues

* camera_utils: handle degenerate case for "vertical" orientation method

* camera_utils: split focus_of_attention computation

---------

Co-authored-by: Frédéric Devernay <f-dy@users.noreply.github.com>
2023-03-10 11:22:21 -08:00
Matthew Tancik 510188b1e6 Revert "MinimalDataParser add options, allow Nerfstudio data format to specify test indices" (#1583)
Revert "MinimalDataParser add options, allow Nerfstudio data format to specify test indices (#1541)"

This reverts commit c61d3e9387.
2023-03-10 10:23:17 -08:00
Jonáš Kulhánek c61d3e9387 MinimalDataParser add options, allow Nerfstudio data format to specify test indices (#1541)
* MinimalDataParser add options, allow Nerfstudio data format to specify test indices

* Switch from indices to filenames and add tests
2023-03-09 19:14:24 +01:00
Georgy Ponimatkin 7366456835 Adding conversion to torch.float32 in get_interpolated_poses_many. (#1532) 2023-02-28 09:22:22 -08:00
Saurav Maheshkar 4545dcb6b3 feat: improve type hints (#1515)
* feat(plugins/registry): Improve Type Hints

* feat(exporter/texture_utils): Improve Type Hints

* feat(exporter/tsdf_utils): Improve Type Hints

* feat(cameras/camera_utils): Improve Type Hints

* feat(cameras/cameras): Improve Type Hints

* feat(cameras/rays): Improve Type Hints

* fix: add default value for device

* fix: ArrayLike type hint
2023-02-26 12:22:45 -08:00
Georgy Ponimatkin 7da588c5d2 Extending ns-render to support loop rendering using interpolated cameras (#1508)
* Extending ns-render to support loop rendering using interpolated cameras.

* Adding docstring for render.py. Applying formating.

* Expanding docstrings for class. Renaming trajectory type.

* Removing mentions of loop from render.py.
2023-02-26 09:30:19 -08:00
Paul Wais 5c73766653 SFM: sfm-based depth, some breadcrumbs for pycolmap (#1457)
* save

* save

* more cleanup

* save

* save

* update phototourism

* save

* works

* works

* cleanup

* cleanup

* lint

* python3 default

* cleanup
2023-02-25 21:10:53 -08:00
Matthew Tancik f3a34bbf95 Refactor schedulers (#1466)
* Refactor schedulers

* lint

* lint
2023-02-21 12:51:51 -08:00
LSongandLcDog b653eef91c add time (#1391)
* add time

* fix black

---------

Co-authored-by: LcDog <12553253+LcDog@users.noreply.github.com>
2023-02-09 16:43:21 -08:00
NCJ 92cadf5ce4 Fix Multiple Typos (#1376)
fix multiple typos
2023-02-08 10:33:23 -08:00
Matthew Tancik 567268854b Cleanup aabb intersection logic (#1329) 2023-02-01 12:18:09 -08:00
Matthew Tancik 79395ecd8d Fix crop robustness (#1321) 2023-01-31 18:25:30 -08:00
Matthew Tancik b5031dd814 Enable rendering with crop (#1295) 2023-01-26 14:22:43 -08:00
Matthew Tancik 90489b5a6d Allow users to modify background color when cropping (#1289) 2023-01-25 17:47:45 -08:00
2120bd5dd9 CropBox GUI implementation (#1205)
* fixed an exception bug in case of num_samples is 0

fixed an exception bug when num samples is 0 , 
"IndexError tensors used as indices must be long, byte or bool tensors"

change type to long ( was float32)

* implemented crop box in gui. For now works on instant-ngp-bounded

* fixed import bugs

* update aabb every time and not only on training

* prevent crash when chosing accomulation and crop box

* Checkbox

* max_box is bounded by scevne box in the gui

* implemented altenative intesection to nerfacc

* convert str to bool correctly

* fixed isort bug

* fixed isort bug2

* update viewer version

* remove json bug

* Update UI

* Force black background

* Fix formatting

Co-authored-by: dror <you@example.com>
Co-authored-by: Matt Tancik <tancik@berkeley.edu>
Co-authored-by: Matthew Tancik <mattt4001@gmail.com>
2023-01-19 17:39:37 -08:00
matsurenandMatthew Tancik 408ab709ec Fix typo (#1231)
* Fix typo COLAMP->COLMAP

* Fix more typo

Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-01-16 12:12:55 -08:00
Liam SchoneveldandLiam Schoneveld 31be97d59c Render cameras with render times for dynamic nerfs (#1199)
* Render cameras with render times for dynamic nerfs

* Add --dynamic flag to cmd. Clamp render times.

* Fix for issues with clamping renderTime

* Only add render_time to camera path if dynamic nerf

* change DEFAULT_TIME to DEFAULT_RENDER_TIME

Co-authored-by: Liam Schoneveld <liam.schoneveld@woven-planet.global>
2023-01-03 21:23:20 -08:00
Michal Pandy 3da877050d Depth supervision for Nerfacto (#1173)
* Implement depth supervision

* Fixes

* Change normalize dtype

* refactor

* Change depth weight

* Small refactor

* Change docstring

* Removu unused import

* Add assumption to depth comment

* Implement URF loss

* Remove default depth loss type param

* URF fixes

* Fix depth loss scale

* Visualize gt depths

* Fixes & documentation

* Documentation and formatting

* Fix formatting problems

* Fix linter problems

* Fix tests

* Fix tests
2023-01-02 10:13:38 +01:00
Jake Austin 3ffee522e0 TrainerConfig inherit from InstantiateConfig (#1175)
Making TrainerConfig an InstantiateConfig so it can be swapped in and out with different methods. Also made the TrainerConfig a subclass of ExperimentConfig (which was changed from a printableconfig to an instantiateconfig), since we were already instantiating the Trainer class with the ExperimentConfig

Removed the "config" field from the TrainerCallbackAttributes struct to eliminate circular dependencies when types are checked.

Minor scripts and docs changes to accommodate the update to the config struct, meaning the flags that were associated with the TrainerConfig were moved up to the top level of the config.
2022-12-27 18:19:34 -10:00
Matthew Tancik c338de4aa4 Support non-pinhole rendering (#1141) 2022-12-18 10:46:39 -08:00
Matthew Tancik b41ce8060a Save out dataparser transforms (#1105)
* Save out dataparser transforms

* Make missing folder
2022-12-11 13:02:50 -08:00
François Darmon 90447ab223 Bug correction auto_orient_and_center_poses (#1098)
When using auto_orient_and_center_poses with method="none" the returned pose if 4x4 (contrary to up and pca) which triggers an assertion error later. This modification solves it.
2022-12-07 09:40:53 -08:00