Commit Graph
101 Commits
Author SHA1 Message Date
71c6b9cf3d Monodepth Regularization (#2284)
* monodepth implemented

* device

* pair pixel sampler

* docstring

* removing unused imports

* fix bugs

* cache in cache dir, no longer edit json file

* moving to depth nerfacto

* fixed bug using median depth by accident

* removing depthdataset from nerfacto

* added pixelsamplerconfig

* added equirectangular warning back

* bug fix

* delete sample_config.py + small fixes

* rename to pixel_sampler

* cleaning

* test

* passing num_rays_per_batch properly

* fixed transforms typo

* renaming in nerfacto

* remove depth files from test data

* removed defaulting to base pixel sampler with mask

* adding patch_size back

---------

Co-authored-by: AdamRashid96 <abrashid@berkeley.edu>
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-08-01 18:52:47 -07:00
Fernando Rivas-Manzanequeandjperezh e6cd95109b Moved Volinga as an external method (#2282)
* Moved Volinga as an external method

* Update external_methods.py for black

* Formatted with black

---------

Co-authored-by: jperezh <jperezh@arquimea.com>
2023-07-31 07:34:38 -07:00
hyperzy e7fd1aefa2 Load schedulers from ckpt in a backward-compatible manner (#2236) 2023-07-18 18:02:15 +00:00
Yueh-Cheng Liu 劉岳承andMatthew Tancik 8cd9beccd5 Fix arguments that are not passed from config to NerfactoField (#2186)
* Fix arguments that are not passed from config to NerfactoField

* Add comments

* Reset base_res and features_per_level config of nerfacto-big and nerfacto-huge to default to match the original behavior

---------

Co-authored-by: Matthew Tancik <matt@lumalabs.ai>
2023-07-15 14:24:37 +00:00
Matthew Tancik 54d26814d9 Update torchmetrics (#2223) 2023-07-13 18:02:19 -07:00
Jonáš Kulhánek aa6c052660 Jkulhanek/colmap dataparser (#2115)
* Add COLMAP dataparser implementation

* Fix downscale_images input

* Add help to COLMAP dataparser
2023-06-25 17:08:17 +00:00
b17268e83b [WIP] dreamfusion implementation (#1008)
* starting the framework for dreamfusion

* Removed view direction from RGB to calculate albedo

* Started diffusion

* Added dreamfusion to CLI, removed direction embeddings

* Normals in dreamfusion

* Lambertian shading added

* Train normals in Dreamfusion

* Stable diffusion from Stable Dreamfusion

* Fix normal orientation loss

* Configurable light direction sampling

* adding density initalization at origin

* V1 datamanager

* Don't train normals for first 1k iters

* SDS loss added to dreamfusion

* Dreamfusion pipeline

* Background MLP and SphereCollider

* Normals prediction training fixed

* Sigmoid on background, prompt in config

* Shading hyperparameter adjustements

* Shading hyperparameter tuning

* Location based prompting, density initialization fades, LR change

* Cleaned imports, more config options

* Tuning, more config changes

* remove unused sampler, normalize predicted normals from network

* change position encoding to direction encoding for background color mlp

* Updating dreamfusion to use custom trainer, fixed bug, multi-gpu support (put diffusion on one specific device), merged with main

* Hard coding less, offloading to config

* Appeasing the linter

* First pass opacity / sparsity losses

* Small patches and configs

* Accidentally pushed logs

* Removing logs for real this time

* Transmittance loss bug fix

* 12 GB Dreamfusion!

* Lint

* Remove unecissary collider changes

* Start SD cleanup

* Cleanup SD code

* adding overhead location prompting

* Use lambertian renderer

* turn grad accum off by default, decrease background mlp size, add overhead text embedding

* Cleanup

* Fix normals vis

* change uniform sampler to prediction network, add option to jit unet for faster sds_loss

* Use shaders

* cleaning up jit code

* add warning if using non jit unet

* add proposal network optimizer to config

* speed up render and cleanup code

* Clean up datamanager

* Add distortion loss

* fix wanb

* adding interpolated location sampling

* Update configs

* interpolated location prompting option, lowering init density, transmittance loss interpolation flipped

* rename alphas mult to alphas loss mult

* Mess around with normals

* add option to use random bg color + bg mlp

* Remove Dreamfusion Trainer (#1423)

* Dreamfusion trainer no longer needed

* Added typing for grad scaler

* SDS return loss

* Fix CFG

* Cleanup

* Update configs

* Fix opacity loss, use unform sampling first round of prop

* add prompt edditing to viewer

* Choose your own diffusion

* remove dreamfusion pipeline file

* merging + remove dreamfusion pipeline file

* Update configs

* add stable diff utils

* cleanup

* cleanup

* Fix test

* positional prompting bugfix

* Warmups

* tune

* lint

* Remove dataparser

* merging + text embedding bugfix

* Set data path to launch viewer for DreamFusion (#1776)

* Set data path to launch viewer for DreamFusion

* Apply formatter

* Fallback to 'base_dir' if data manager has no path

* Make if statement non-inline

* Reformat

---------

Co-authored-by: Matthew Tancik <mattt4001@gmail.com>

* add deepfloyd option first try

* merging

* merging from main

* merging

* removing comments, cleaning code, making stable diff utils usable for deepfloyd, remove deepfloyd text encoder after use

* undo chagnes in data folder

* cleaning code, removing unneccesary changes from main, removed normals prediction network

* remove old scripts files, rename generative pipeline flag to no_input_data, revert to default histogram padding in proposal sampler

* changing pipeline flag

* move no input data flag from pipeline to datamanager

* undo config and merge in deepfloyd from main

* merging grad scaler pipeline addition

* merging + setting proposal net scheduler

* revert nerfacto config change

* typing + linting

* rename dreamfusion datamanager to random camras datamanager, check datamanager typing in pipeline and trainer

* fix test import

* fix test import

* changing datamanager check in trainer, add typing in random cameras datamanager

* add Tensor to randomcamerasdatamanager

* remove old comments

* resolve tyro warning

* update fields to use tcnn

* lint

* add packages for deepfloyd

* add versions

* docs first draft

* rename to generfacto, clean docs

* fix import error

* minor changes

* fix orientation loss bug, retune params

* cleanup

* typing

* trying docs fix

* fixing generative package imports

* fix pyright

* docs fix

* merge w main

* docs minor change

* tuning + bugfixes

* remove torch.backends

---------

Co-authored-by: Abhik Ahuja <ahujaabhik@berkeley.edu>
Co-authored-by: David McAllister <davidrmc2@gmail.com>
Co-authored-by: Jake Austin <jake-austin@berkeley.edu>
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
Co-authored-by: Matt Tancik <mattt4001@gmail.com>
Co-authored-by: SeungWoo Yoo <dreamy1534@gmail.com>
Co-authored-by: ayaanzhaque <ayaanzhaque@gmail.com>
Co-authored-by: Ayaan Haque <ayaanzhaque@berkeley.edu>
2023-06-17 23:34:04 -07:00
machenmusik 1066f8f433 don't force downscale factor to 1 for nerfacto-huge (#2088)
* per @kerrj, nerfacto-huge doesn't need downscale_factor=1
2023-06-16 23:49:56 -07:00
Justin Kerr f9d06d1f9c reduce nerfacto-huge memory (#2082)
reduce memory to 24G by reducing appearance embed dim
2023-06-15 16:33:30 -07:00
Justin Kerr 8fad92c39a nerfacto-huge (#2003)
An even larger version of nerfacto with scaled hashgrid resolution, MLP sizes, batch sizes, appearance embeddings, proposal sampler resolution, and more.
Also includes some minor tweaks to nerfacto-big configs
2023-06-15 11:05:53 -07:00
LSong 10dab142d0 Move out NeRFPlayer: clean files and add method link (#2072)
* clean files and add method link

* fix black

* update
2023-06-14 13:51:09 -07:00
Ethan WeberandEthan Weber bb6925bac6 Ability to use vanilla pytorch instead of tcnn for Nerfacto (#1936)
* started a package to easily swap out tcnn for pytorch

* trains with pytorch and not tcnn

* cleaned up imports

* making it easy to install on mac

* removed/reverted files

* merged main

* reverted changes

* updating config

* starting to add nerfacto to tests

* fixxing default value

* fixed default implementation

* defaults to torch if tcnn and not installed

* update log

* add comment

* resolved pr comments

* fix

* update default

---------

Co-authored-by: Ethan Weber <ethanweber@eecs-128-32-175-250.dhcp.berkeley.edu>
2023-05-25 14:55:42 -07:00
Matthew Tancik efd83abb6d Basic viser frontend (#1973)
* Use viser frontend

* Rendering but poses incorrect

* Fix cameras
2023-05-23 12:21:38 -07:00
Milan van Wouden d413ca4204 fix large number of typos in repository (#1977)
* fix large number of typos in repository

* fix typos in tests
2023-05-23 10:41:49 -07:00
Giodiroandgmeanti be291970db Documentation for k-planes add-on (#1935)
* Add documentation for kplanes add-on

* Fix all integration spots with nerfstudio

---------

Co-authored-by: gmeanti <giacomo.meanti@gmail.com>
2023-05-17 11:48:08 -07:00
Brent YiandMatt Tancik d542646c72 Switch to ruff (#1939)
* `isort` -> `ruff`

* `pylint` -> `ruff`

* Remove line

* More minor cleanup

* Better documentation for rules we are enabling

* Fix gitignore, run black

* vscode settings

---------

Co-authored-by: Matt Tancik <mattt4001@gmail.com>
2023-05-17 10:08:14 -07:00
Brent Yi 814db32ec4 More typing fixes (#1941)
* Fix more types

* Fix tests

* Minor cleaning

* Address comments

* Resolve linting errors

* Re-ignore pymeshlab type

* Add extra assert / error message to mipnerf model
2023-05-17 07:17:40 -07:00
Jonáš Kulhánek 9b6290ea43 Discover external methods from CLI (#1934)
* Discover external methods from CLI

* Fix external method registration

* External method add docstrings

* Update docs: register custom method in external_methods.py
2023-05-16 22:15:47 +02:00
Brent Yi f1b60d4641 Fix static type for dataparser configs (#1924) 2023-05-15 14:15:51 -07:00
Ethan Weber 13e0fce122 Update copyright (#1917)
* updated copyright

* including txt file

* updated black

* removing license

* fixing license
2023-05-12 16:16:13 -07:00
wuziruiandMatthew Tancik bc110d7bd9 extract dataparser configs to a stand-alone file, support custom dataparsers (#1909)
* extract dataparser configs to a stand-alone file, support custom dataparsers

* add license

* fix pylint issues

* fix type error issue

* update naming; add documentations;

* print f string

* fix pylint issue

---------

Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-05-12 15:11:46 -07:00
Matthew Tancik 4242a3c982 Make rich console global (#1916) 2023-05-12 14:20:40 -07:00
Yan XiaoandYan Xiao 38cbb0f358 implementing l1 and tv reg in the original tensorf paper (#1855)
* implementing l1 and tv reg in the original tensorf paper

* added documentation for tv loss

* added test for tv loss

* fix formating

* fix formatting

---------

Co-authored-by: Yan Xiao <yax010@ucsd.edu>
2023-05-09 09:32:51 -07:00
Brent Yi 63c6303736 Bump minimum Python version to 3.8 (#1876)
* Bump minimum Python version to 3.8

* typing_extensions.Literal -> typing.Literal

* ns-dev-test

* typing_extensions.get_args => typing.get_args

* Format with older version of black

* Bump `black`

* Run black
2023-05-06 13:29:48 -07:00
Ethan Weber 7537551f38 Avoids running test_train.py on external methods (#1882)
removes testing on external methods
2023-05-06 12:19:26 -07:00
Jonáš Kulhánek 1147fe4d5c Make VanillaDataManager generic (#1868)
* Make VanillaDataManager generic

* Remove removed classes from docs

* Simplify generic datamanager

* Simplify code

* Add serialization tests
2023-05-06 09:54:52 -07:00
Ethan Weber 081bf82bf5 Wandb/experiment improvements (#1879)
exposing experiment settings
2023-05-05 18:28:52 -07:00
Ruilong Li(李瑞龙) 7887b14ca0 Reduce the GPU memory required in viewer for NGP methods (#1837)
Fix the issue #1835:

- change the default `viewer.num_rays_per_chunk` from `1 << 15` to `1 << 12`.
- remove the `render_step_size=0.001` for `instant-ngp-bounded` so that it is automatically calculated based on the scene aabb (target on roughly 1000 samples).
2023-04-28 13:44:34 -07:00
Ruilong Li(李瑞龙) ea2993d5b1 Update NGP related models with nerfacc 0.5.2 (#1809)
* profiling

* set pose_optimizer to None if it's mode is "off"

* radial_and_tangential_undistort in nerfacc

* fix camera distortions

* update to nerfacc 0.5.1. Upgrade instant-ngp model.

* revert nerfacc to 0.5.1

* format

* allows to fall back to the pytorch undistortion

* format

* fix linting

* fix test

* fix test

* fix nerfplayer-ngp

* enable_collider=False for NGP

* nerfplayer verified.

* zero cone_angle for ngp bounded

* fix format

* fix pylint
2023-04-24 17:11:30 -07:00
Matthew TancikandBrent Yi 8d9c4005d4 Configurable ip address (#1779)
* Configurable ip address

* Rename `ip_address` => `websocket_host`

---------

Co-authored-by: Brent Yi <yibrenth@gmail.com>
2023-04-20 07:36:32 -07:00
Jonáš Kulhánek ea14152ae6 Add PyTorch profiler to support tracing of forward/backward calls (#1761)
* Add PyTorch profiler support

* Add support for wrapped code, fix linting errors

* Update docs
2023-04-18 20:29:09 +02:00
6a7c8820b8 Viewer backend refactor (#1702)
* Copy in `viser` communication infrastructure

* Update background image

* Add build directory to fix viser HTTP server
May want to remove these later

* Copy in `viser` communication infrastructure

* Update background image

* Pass dataset vis using viser

* initial impl of ViewerParameter

* idk what i did but brent and matt know

* Suppress eslint typescript warnings

* implement ViewerElement base class, then ViewerButton, ViewerSlider, etc

* client -> server message setup

* multi folder stuff

* Stream controls over viser

* import

* add types to viewer_param.py

* Move another message to viser

* Update folder color

* Use non throttled messages

* multiple folders works

* Cleanup, more porting

* starting impl of control panel in viser

* More porting

* scenebox to viser

* Cleanup

* Fix render updates

* More cleanup, docs, typing

* start control_panel.py

* set_hidden logic but not merged in

* control_panel replicates behavior of existing control panel (visually), except for setting the list of output options. not yet integrated with viewer_utils.py for logic flow. also implemented set_disabled and set_hidden in javscript/ViewerElement side

* Remove some original leva components

* add some train util logic, attempt set_value, doesn't work on javascript side fully yet (value updates but not slider visually

* bye zmq

* missing call to decoder

* 'gui_set' works properly, control panel now replicates behavior of original

* remove print

* Fix status viewer message

* More cleanup

* support set_options in dropdown

* hiding buttons works correctly

* fix set_options, switch from cur_value to get_value()

* Fix slowdown

* output options

* colormap logic converted

* add handle to cb_hook

* fix some lambda bugs

* cleanup

* Save checkpoint logic readded

* Cleanup and minor fixes

* snappier ui

* Refactoring

* Fix state update bugs

* Bug fixes

* Write less to redux

* added comments to reducer code

* Render state machine (#1727)

* state machine works, but train step throttling not implemented

* more step logic, cleanup

* remove random try catcH

* mutex logic for render

* Update lock logic

* types and docs

* added some state machine logic to help rerenders

* merge

---------

Co-authored-by: Justin <justin.g.kerr@gmail.com>

* Refactor and cleanup

* More cleanup and docs

* docstring to gui_utils

* docstrings, rename viewer_param.py to viewer_elements.py

* Fix viewer, remove bridge stuff

* Render on connect

* Fix lint errors

* More eslint fixes

* render_freq logic

* better rerender logic for crop

* More cleanup

* Purge

* Cleanup

* Lint fixes

* Fix cache

* better parsing logic

* Fix infinite test

* More cleanup

* typo

* Update webdocs

* Update docs

* rerender on custom hooks

* add more docs

* Use shared utilities in `viser.infra`

* rework callback to use elements instead of handles

* Fix ESlint errors

* change naming of handle to element, add setter for value

* cleanup

---------

Co-authored-by: Matt Tancik <mattt4001@gmail.com>
Co-authored-by: Justin <justin.g.kerr@gmail.com>
Co-authored-by: Ethan Weber <eweb0124@gmail.com>
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-04-14 14:53:10 -07:00
Pablo VelaandEthan Weber 8126d47d8c NeusFacto (#1654)
* add neus_facto method

* fix missing pixsfm from video process

* fix dev errors and small typos

---------

Co-authored-by: Ethan Weber <eweb0124@gmail.com>
2023-04-12 19:08:53 -05:00
835a0151f2 Create nerfacto-big config for nerfacto with sampling, network size, resolution, and train steps cranked up (#1668)
* Increase max_res of first proposal network

* nerfacto-big params

* skip nerfacto_big in test

---------

Co-authored-by: Matt Tancik <mattt4001@gmail.com>
Co-authored-by: Ethan Weber <ethanweber@berkeley.edu>
2023-04-05 07:43:39 -07:00
Pythonix Huang bde853bd14 Update scheduler for nerfacto (#1662)
* Update scheduler for nerfacto

Add scheduler (lr decay) in nerfacto is also effective for preventing long-time training nan and good for convergence

* refactor

refactor

* refactor

refactor

refactor

* refactor

refactor

* Update method_configs.py

* Revert "Update method_configs.py"

This reverts commit feaff61819.

* Revert "refactor"

This reverts commit cf7c3153a2.

* Revert "Merge branch 'update-scheduler' of https://github.com/HrsPythonix/nerfstudio into update-scheduler"

This reverts commit b31575385c, reversing
changes made to 6b78039059.

* Revert "refactor"

This reverts commit 6b78039059.

* Update method_configs.py
2023-03-31 13:39:49 -07:00
machenmusik 83d0d75635 realitycapture nerfacto variable resolution (#1651)
* allow periods within filename, not only for file extension

* don't assume all images are the same resolution

* black reformatting

* Move variable resolution collate logic into vanilla datamanager

* good catch pylint

* now that vanilla supports variable resolution, remove VariableResDataManager

* remove comment

* CONSOLE.print

* avoid `IndexError: tensors used as indices must be long, byte or bool tensors`

* use large dataset parameters for phototourism
2023-03-23 21:45:40 -04:00
machenmusik 6db1d314cc add quality setting; allow png for lossless compression (#1639)
* add quality setting; allow png for lossless compression

* black reformatting

* JPEG default, 90 quality
2023-03-21 19:04:12 -04:00
Ethan WeberandMatthew Tancik b3f1153a78 Add option to auto select a websocket port (#1628)
* option to auto select a websocket port

* keep default port

* add docs

* default port

* Update base_config.py

---------

Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-03-17 15:53:27 -07:00
David McAllisterandMatthew Tancik 4c2e200098 TensoRF go brrrrrrr (#1609)
* pdfsampler no longer includes uniform samples

* Lint

* Typo fix

---------

Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-03-15 20:27:37 -07:00
Pablo Vela ddfa463f4d NeuS Implementation (#1444)
* add initial method configs and data manager

* add neus scheduler

* add initial surface model and neus

* add Neus ray sampler

* add sdf field

* move scale/shift calc to math utils

* add debug and fix sdf_dataset typo

* update vscode neus debug

* fix dataset and get initial working version

* add magic number from original repo

* black fixes

* clean up get weights/transmittance from alphas

* clean up typos in sdf datamanager

* type annotations on neus scheduler and remove unused params

* clean up and annotate sdf field along w/ other fixes

* more clean up

* remove unnessary directions norm

* resolve comments

* update neus scheduler to follow refactor

* fix linter errors

* fix some typos

* remove get_weights_from_alpha

* more comment fixes

* more linter error fixes

* clean up and add comments

* more little fixes for types/comments

* more comments

* rename NeuS Scheduler to DelayedScheduler

* rename DelayedScheduler to CosineDecayScheduler
2023-03-12 16:01:18 -04:00
Matthew Tancik 63af268495 Remove webrtc (#1579)
* Remove webrtc

* cleanup

* lint

* cleanup

* Lower quailty

* Update viewer version

* Update docs

* remove cryptography req
2023-03-10 12:00:35 -08:00
Matthew Tancik bfaf11cd72 Remove dataparser dependency (#1529) 2023-02-27 18:32:40 -08:00
Matthew Tancik 5158cc36ff Fix logic when using filename datasets (#1510) 2023-02-25 16:06:41 -08:00
Fernando Rivas-ManzanequeandMatthew Tancik 9053643932 Volinga model and NVOL documentation (#1486)
* added volinga model to NeRFStudio

* Removed unused dependencies

* Modified dockerfile to match main

* Corrected typo in core_code_checks.yml

* Corrected typo in core_code_checks.yml

* updated test with Volinga

* Added conversion to NVOL documentation

* [WIP] Refactor documentation and removing Volinga config

* Volinga use existing NerfactoModelConfig

* Added Unreal Engine documentation to extensions

* removed unused field class import

* Updated docs

* Documentation ready

* Update Dockerfile

---------

Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-02-25 14:49:53 -08:00
Jingchen Ye 871eb0f7d3 Rename friends dataset to sitcoms3d (#1503)
* Rename friends dataset with official name sitcoms3d

* Rename downloaded folder

* Fix pylint
2023-02-25 14:38:23 -08:00
Matthew Tancik 27eb33f491 Enable viewer with eval (#1474) 2023-02-22 10:21:55 -08:00
Matthew Tancik f3a34bbf95 Refactor schedulers (#1466)
* Refactor schedulers

* lint

* lint
2023-02-21 12:51:51 -08:00
Jonáš Kulhánek 59f71eda0e Allow methods configs to be discovered from external packages. (#1357)
* Allow method_config to be registered from external packages

* Add license

* Add MethodSpecification class + fix linter errors

---------
2023-02-06 13:25:33 -08:00
machenmusik bf64769128 add viewer.local config property when viewing local instance (#1358)
If running local server instance, avoid using relays to communicate with the viewer.
2023-02-06 15:53:27 -05:00
machenmusik 319a2b965c New config parameter --viewer.codec defaults to VP8, but allows H264 (#1343)
* New config parameter --viewer.codec defaults to H264 rather than VP8

* make quantize clearer

* change codec to Literal

default is VP8 due to libx264 log spam, see https://github.com/nerfstudio-project/nerfstudio/pull/1343#issuecomment-1416235500

* isort

* pylint, including python 3.7 compatibility
2023-02-03 13:38:40 -08:00