* NERFSTUDIO_METHOD_CONFIGS allow method specification as callabale (function or class)
* docs - NERFSTUDIO_METHOD_CONFIGS accepts function or derived class
* nerfbusters dataparser
* nerfbuster dataparser and visualization of eval views
* resolved some errors from pyright
* enable ns-process-data to take two videos as input and use one for eval and the other for train
* resolve some pyright issues
* instead of data being a tuple, I added eval_data as another argument
* implemented train eval split for all cases
* implemented train eval split for all cases
* reverting equi changes
* reverting reality capture changes
* raise valueerror
* revert comment
* added split override code back to dataparser
* reverted one change
* using image border now
* adding docs
---------
Co-authored-by: Ethan Weber <eweb0124@gmail.com>
* Moved Volinga as an external method
* Update external_methods.py for black
* Formatted with black
---------
Co-authored-by: jperezh <jperezh@arquimea.com>
* Implement nerfacto alpha transparency training
* Stop adding all ones alpha channel and clean up blend_background
* Make background_color a required argument in blend_background
* Enable alpha transparent training by default
* Improve background_color typing
* Blend uniform background color for metric computation
* Fix typing error
* Clean up typing
* Fix background_color to outputs assignment
* Model outputs returns unblended RGB (black bg) for random bg
* Update renderers
* Fix comments
* Add support for transparent BG for other methods
* Minor refac
* Fix get_background_color
* Fix typing error for blend_background
* Fix comments
* Export utils support transparency
* Pylint fix
* Fix pcd export masking
* Filter pcd view directions based on opacity
* Clean up get_rgba_image
---------
Co-authored-by: Jonas Kulhanek <jonas.kulhanek@live.com>
* starting the framework for dreamfusion
* Removed view direction from RGB to calculate albedo
* Started diffusion
* Added dreamfusion to CLI, removed direction embeddings
* Normals in dreamfusion
* Lambertian shading added
* Train normals in Dreamfusion
* Stable diffusion from Stable Dreamfusion
* Fix normal orientation loss
* Configurable light direction sampling
* adding density initalization at origin
* V1 datamanager
* Don't train normals for first 1k iters
* SDS loss added to dreamfusion
* Dreamfusion pipeline
* Background MLP and SphereCollider
* Normals prediction training fixed
* Sigmoid on background, prompt in config
* Shading hyperparameter adjustements
* Shading hyperparameter tuning
* Location based prompting, density initialization fades, LR change
* Cleaned imports, more config options
* Tuning, more config changes
* remove unused sampler, normalize predicted normals from network
* change position encoding to direction encoding for background color mlp
* Updating dreamfusion to use custom trainer, fixed bug, multi-gpu support (put diffusion on one specific device), merged with main
* Hard coding less, offloading to config
* Appeasing the linter
* First pass opacity / sparsity losses
* Small patches and configs
* Accidentally pushed logs
* Removing logs for real this time
* Transmittance loss bug fix
* 12 GB Dreamfusion!
* Lint
* Remove unecissary collider changes
* Start SD cleanup
* Cleanup SD code
* adding overhead location prompting
* Use lambertian renderer
* turn grad accum off by default, decrease background mlp size, add overhead text embedding
* Cleanup
* Fix normals vis
* change uniform sampler to prediction network, add option to jit unet for faster sds_loss
* Use shaders
* cleaning up jit code
* add warning if using non jit unet
* add proposal network optimizer to config
* speed up render and cleanup code
* Clean up datamanager
* Add distortion loss
* fix wanb
* adding interpolated location sampling
* Update configs
* interpolated location prompting option, lowering init density, transmittance loss interpolation flipped
* rename alphas mult to alphas loss mult
* Mess around with normals
* add option to use random bg color + bg mlp
* Remove Dreamfusion Trainer (#1423)
* Dreamfusion trainer no longer needed
* Added typing for grad scaler
* SDS return loss
* Fix CFG
* Cleanup
* Update configs
* Fix opacity loss, use unform sampling first round of prop
* add prompt edditing to viewer
* Choose your own diffusion
* remove dreamfusion pipeline file
* merging + remove dreamfusion pipeline file
* Update configs
* add stable diff utils
* cleanup
* cleanup
* Fix test
* positional prompting bugfix
* Warmups
* tune
* lint
* Remove dataparser
* merging + text embedding bugfix
* Set data path to launch viewer for DreamFusion (#1776)
* Set data path to launch viewer for DreamFusion
* Apply formatter
* Fallback to 'base_dir' if data manager has no path
* Make if statement non-inline
* Reformat
---------
Co-authored-by: Matthew Tancik <mattt4001@gmail.com>
* add deepfloyd option first try
* merging
* merging from main
* merging
* removing comments, cleaning code, making stable diff utils usable for deepfloyd, remove deepfloyd text encoder after use
* undo chagnes in data folder
* cleaning code, removing unneccesary changes from main, removed normals prediction network
* remove old scripts files, rename generative pipeline flag to no_input_data, revert to default histogram padding in proposal sampler
* changing pipeline flag
* move no input data flag from pipeline to datamanager
* undo config and merge in deepfloyd from main
* merging grad scaler pipeline addition
* merging + setting proposal net scheduler
* revert nerfacto config change
* typing + linting
* rename dreamfusion datamanager to random camras datamanager, check datamanager typing in pipeline and trainer
* fix test import
* fix test import
* changing datamanager check in trainer, add typing in random cameras datamanager
* add Tensor to randomcamerasdatamanager
* remove old comments
* resolve tyro warning
* update fields to use tcnn
* lint
* add packages for deepfloyd
* add versions
* docs first draft
* rename to generfacto, clean docs
* fix import error
* minor changes
* fix orientation loss bug, retune params
* cleanup
* typing
* trying docs fix
* fixing generative package imports
* fix pyright
* docs fix
* merge w main
* docs minor change
* tuning + bugfixes
* remove torch.backends
---------
Co-authored-by: Abhik Ahuja <ahujaabhik@berkeley.edu>
Co-authored-by: David McAllister <davidrmc2@gmail.com>
Co-authored-by: Jake Austin <jake-austin@berkeley.edu>
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
Co-authored-by: Matt Tancik <mattt4001@gmail.com>
Co-authored-by: SeungWoo Yoo <dreamy1534@gmail.com>
Co-authored-by: ayaanzhaque <ayaanzhaque@gmail.com>
Co-authored-by: Ayaan Haque <ayaanzhaque@berkeley.edu>
* Fix model loading collate_fn non-static
* Fix serialization
* Add tests to reproduce the error
* Comment the change to let the tests fail
* Finish changes
* started a package to easily swap out tcnn for pytorch
* trains with pytorch and not tcnn
* cleaned up imports
* making it easy to install on mac
* removed/reverted files
* merged main
* reverted changes
* updating config
* starting to add nerfacto to tests
* fixxing default value
* fixed default implementation
* defaults to torch if tcnn and not installed
* update log
* add comment
* resolved pr comments
* fix
* update default
---------
Co-authored-by: Ethan Weber <ethanweber@eecs-128-32-175-250.dhcp.berkeley.edu>
* knocking out a couple
* Fix more errors
* Add pyright
* Remove some unnecessary cv2 type ignores, warning on missing imports
(for missing tinycudann)
* Minor annotation fixes
* Update docs
* Drop `None` from temporal distortion output
---------
Co-authored-by: Brent Yi <yibrenth@gmail.com>
* `isort` -> `ruff`
* `pylint` -> `ruff`
* Remove line
* More minor cleanup
* Better documentation for rules we are enabling
* Fix gitignore, run black
* vscode settings
---------
Co-authored-by: Matt Tancik <mattt4001@gmail.com>
* implementing l1 and tv reg in the original tensorf paper
* added documentation for tv loss
* added test for tv loss
* fix formating
* fix formatting
---------
Co-authored-by: Yan Xiao <yax010@ucsd.edu>
* Refactor of image and video dataset conversion
The classes to convert images and video to a NerfStudio
dataset are moved to the main repo and refactor:
* Code duplications are removed
* Import of the class is now done from nerfstudio package instead of
scripts
* Doc formatting nits
* Doc formatting
---------
Co-authored-by: Brent Yi <yibrenth@gmail.com>
* profiling
* set pose_optimizer to None if it's mode is "off"
* radial_and_tangential_undistort in nerfacc
* fix camera distortions
* update to nerfacc 0.5.1. Upgrade instant-ngp model.
* revert nerfacc to 0.5.1
* format
* allows to fall back to the pytorch undistortion
* format
* fix linting
* fix test
* fix test
* fix nerfplayer-ngp
* enable_collider=False for NGP
* nerfplayer verified.
* zero cone_angle for ngp bounded
* fix format
* fix pylint
* Copy in `viser` communication infrastructure
* Update background image
* Add build directory to fix viser HTTP server
May want to remove these later
* Copy in `viser` communication infrastructure
* Update background image
* Pass dataset vis using viser
* initial impl of ViewerParameter
* idk what i did but brent and matt know
* Suppress eslint typescript warnings
* implement ViewerElement base class, then ViewerButton, ViewerSlider, etc
* client -> server message setup
* multi folder stuff
* Stream controls over viser
* import
* add types to viewer_param.py
* Move another message to viser
* Update folder color
* Use non throttled messages
* multiple folders works
* Cleanup, more porting
* starting impl of control panel in viser
* More porting
* scenebox to viser
* Cleanup
* Fix render updates
* More cleanup, docs, typing
* start control_panel.py
* set_hidden logic but not merged in
* control_panel replicates behavior of existing control panel (visually), except for setting the list of output options. not yet integrated with viewer_utils.py for logic flow. also implemented set_disabled and set_hidden in javscript/ViewerElement side
* Remove some original leva components
* add some train util logic, attempt set_value, doesn't work on javascript side fully yet (value updates but not slider visually
* bye zmq
* missing call to decoder
* 'gui_set' works properly, control panel now replicates behavior of original
* remove print
* Fix status viewer message
* More cleanup
* support set_options in dropdown
* hiding buttons works correctly
* fix set_options, switch from cur_value to get_value()
* Fix slowdown
* output options
* colormap logic converted
* add handle to cb_hook
* fix some lambda bugs
* cleanup
* Save checkpoint logic readded
* Cleanup and minor fixes
* snappier ui
* Refactoring
* Fix state update bugs
* Bug fixes
* Write less to redux
* added comments to reducer code
* Render state machine (#1727)
* state machine works, but train step throttling not implemented
* more step logic, cleanup
* remove random try catcH
* mutex logic for render
* Update lock logic
* types and docs
* added some state machine logic to help rerenders
* merge
---------
Co-authored-by: Justin <justin.g.kerr@gmail.com>
* Refactor and cleanup
* More cleanup and docs
* docstring to gui_utils
* docstrings, rename viewer_param.py to viewer_elements.py
* Fix viewer, remove bridge stuff
* Render on connect
* Fix lint errors
* More eslint fixes
* render_freq logic
* better rerender logic for crop
* More cleanup
* Purge
* Cleanup
* Lint fixes
* Fix cache
* better parsing logic
* Fix infinite test
* More cleanup
* typo
* Update webdocs
* Update docs
* rerender on custom hooks
* add more docs
* Use shared utilities in `viser.infra`
* rework callback to use elements instead of handles
* Fix ESlint errors
* change naming of handle to element, add setter for value
* cleanup
---------
Co-authored-by: Matt Tancik <mattt4001@gmail.com>
Co-authored-by: Justin <justin.g.kerr@gmail.com>
Co-authored-by: Ethan Weber <eweb0124@gmail.com>
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
* add neus_facto method
* fix missing pixsfm from video process
* fix dev errors and small typos
---------
Co-authored-by: Ethan Weber <eweb0124@gmail.com>
* Increase max_res of first proposal network
* nerfacto-big params
* skip nerfacto_big in test
---------
Co-authored-by: Matt Tancik <mattt4001@gmail.com>
Co-authored-by: Ethan Weber <ethanweber@berkeley.edu>
* fix checkpoint generated by ddp training cannot be loaded
* fix typo
* reformatted nerfstudio/pipelines/base_pipeline.py
* rewritten as for loop
* add test case
* reformatted
* fix pylint test failed
* add initial method configs and data manager
* add neus scheduler
* add initial surface model and neus
* add Neus ray sampler
* add sdf field
* move scale/shift calc to math utils
* add debug and fix sdf_dataset typo
* update vscode neus debug
* fix dataset and get initial working version
* add magic number from original repo
* black fixes
* clean up get weights/transmittance from alphas
* clean up typos in sdf datamanager
* type annotations on neus scheduler and remove unused params
* clean up and annotate sdf field along w/ other fixes
* more clean up
* remove unnessary directions norm
* resolve comments
* update neus scheduler to follow refactor
* fix linter errors
* fix some typos
* remove get_weights_from_alpha
* more comment fixes
* more linter error fixes
* clean up and add comments
* more little fixes for types/comments
* more comments
* rename NeuS Scheduler to DelayedScheduler
* rename DelayedScheduler to CosineDecayScheduler
* MinimalDataParser add options, allow Nerfstudio data format to specify test indices
* Switch from indices to filenames and add tests
* Fix #1581
* Fix changed interface
* added volinga model to NeRFStudio
* Removed unused dependencies
* Modified dockerfile to match main
* Corrected typo in core_code_checks.yml
* Corrected typo in core_code_checks.yml
* updated test with Volinga
* Added conversion to NVOL documentation
* [WIP] Refactor documentation and removing Volinga config
* Volinga use existing NerfactoModelConfig
* Added Unreal Engine documentation to extensions
* removed unused field class import
* Updated docs
* Documentation ready
* Update Dockerfile
---------
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
* Allow black/white background color in Nerfacto
* Use white background in Nerfacto Blender Benchmark
* Fixed unused import
* Updated index for license headers script
* fixed an exception bug in case of num_samples is 0
fixed an exception bug when num samples is 0 ,
"IndexError tensors used as indices must be long, byte or bool tensors"
change type to long ( was float32)
* implemented crop box in gui. For now works on instant-ngp-bounded
* fixed import bugs
* update aabb every time and not only on training
* prevent crash when chosing accomulation and crop box
* Checkbox
* max_box is bounded by scevne box in the gui
* implemented altenative intesection to nerfacc
* convert str to bool correctly
* fixed isort bug
* fixed isort bug2
* update viewer version
* remove json bug
* Update UI
* Force black background
* Fix formatting
Co-authored-by: dror <you@example.com>
Co-authored-by: Matt Tancik <tancik@berkeley.edu>
Co-authored-by: Matthew Tancik <mattt4001@gmail.com>
* fix notebook
* samples
* run all cells
* box center
* try saving again
* import order
* cube with different sides
* black
* black
* update metadata
* fix test
Co-authored-by: Katrin Schmid <kabischmid@gmail.com>
Co-authored-by: Matt Tancik <tancik@berkeley.edu>
