* implementing l1 and tv reg in the original tensorf paper
* added documentation for tv loss
* added test for tv loss
* fix formating
* fix formatting
---------
Co-authored-by: Yan Xiao <yax010@ucsd.edu>
* Refactor of image and video dataset conversion
The classes to convert images and video to a NerfStudio
dataset are moved to the main repo and refactor:
* Code duplications are removed
* Import of the class is now done from nerfstudio package instead of
scripts
* Doc formatting nits
* Doc formatting
---------
Co-authored-by: Brent Yi <yibrenth@gmail.com>
* profiling
* set pose_optimizer to None if it's mode is "off"
* radial_and_tangential_undistort in nerfacc
* fix camera distortions
* update to nerfacc 0.5.1. Upgrade instant-ngp model.
* revert nerfacc to 0.5.1
* format
* allows to fall back to the pytorch undistortion
* format
* fix linting
* fix test
* fix test
* fix nerfplayer-ngp
* enable_collider=False for NGP
* nerfplayer verified.
* zero cone_angle for ngp bounded
* fix format
* fix pylint
* Copy in `viser` communication infrastructure
* Update background image
* Add build directory to fix viser HTTP server
May want to remove these later
* Copy in `viser` communication infrastructure
* Update background image
* Pass dataset vis using viser
* initial impl of ViewerParameter
* idk what i did but brent and matt know
* Suppress eslint typescript warnings
* implement ViewerElement base class, then ViewerButton, ViewerSlider, etc
* client -> server message setup
* multi folder stuff
* Stream controls over viser
* import
* add types to viewer_param.py
* Move another message to viser
* Update folder color
* Use non throttled messages
* multiple folders works
* Cleanup, more porting
* starting impl of control panel in viser
* More porting
* scenebox to viser
* Cleanup
* Fix render updates
* More cleanup, docs, typing
* start control_panel.py
* set_hidden logic but not merged in
* control_panel replicates behavior of existing control panel (visually), except for setting the list of output options. not yet integrated with viewer_utils.py for logic flow. also implemented set_disabled and set_hidden in javscript/ViewerElement side
* Remove some original leva components
* add some train util logic, attempt set_value, doesn't work on javascript side fully yet (value updates but not slider visually
* bye zmq
* missing call to decoder
* 'gui_set' works properly, control panel now replicates behavior of original
* remove print
* Fix status viewer message
* More cleanup
* support set_options in dropdown
* hiding buttons works correctly
* fix set_options, switch from cur_value to get_value()
* Fix slowdown
* output options
* colormap logic converted
* add handle to cb_hook
* fix some lambda bugs
* cleanup
* Save checkpoint logic readded
* Cleanup and minor fixes
* snappier ui
* Refactoring
* Fix state update bugs
* Bug fixes
* Write less to redux
* added comments to reducer code
* Render state machine (#1727)
* state machine works, but train step throttling not implemented
* more step logic, cleanup
* remove random try catcH
* mutex logic for render
* Update lock logic
* types and docs
* added some state machine logic to help rerenders
* merge
---------
Co-authored-by: Justin <justin.g.kerr@gmail.com>
* Refactor and cleanup
* More cleanup and docs
* docstring to gui_utils
* docstrings, rename viewer_param.py to viewer_elements.py
* Fix viewer, remove bridge stuff
* Render on connect
* Fix lint errors
* More eslint fixes
* render_freq logic
* better rerender logic for crop
* More cleanup
* Purge
* Cleanup
* Lint fixes
* Fix cache
* better parsing logic
* Fix infinite test
* More cleanup
* typo
* Update webdocs
* Update docs
* rerender on custom hooks
* add more docs
* Use shared utilities in `viser.infra`
* rework callback to use elements instead of handles
* Fix ESlint errors
* change naming of handle to element, add setter for value
* cleanup
---------
Co-authored-by: Matt Tancik <mattt4001@gmail.com>
Co-authored-by: Justin <justin.g.kerr@gmail.com>
Co-authored-by: Ethan Weber <eweb0124@gmail.com>
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
* add neus_facto method
* fix missing pixsfm from video process
* fix dev errors and small typos
---------
Co-authored-by: Ethan Weber <eweb0124@gmail.com>
* Increase max_res of first proposal network
* nerfacto-big params
* skip nerfacto_big in test
---------
Co-authored-by: Matt Tancik <mattt4001@gmail.com>
Co-authored-by: Ethan Weber <ethanweber@berkeley.edu>
* fix checkpoint generated by ddp training cannot be loaded
* fix typo
* reformatted nerfstudio/pipelines/base_pipeline.py
* rewritten as for loop
* add test case
* reformatted
* fix pylint test failed
* add initial method configs and data manager
* add neus scheduler
* add initial surface model and neus
* add Neus ray sampler
* add sdf field
* move scale/shift calc to math utils
* add debug and fix sdf_dataset typo
* update vscode neus debug
* fix dataset and get initial working version
* add magic number from original repo
* black fixes
* clean up get weights/transmittance from alphas
* clean up typos in sdf datamanager
* type annotations on neus scheduler and remove unused params
* clean up and annotate sdf field along w/ other fixes
* more clean up
* remove unnessary directions norm
* resolve comments
* update neus scheduler to follow refactor
* fix linter errors
* fix some typos
* remove get_weights_from_alpha
* more comment fixes
* more linter error fixes
* clean up and add comments
* more little fixes for types/comments
* more comments
* rename NeuS Scheduler to DelayedScheduler
* rename DelayedScheduler to CosineDecayScheduler
* MinimalDataParser add options, allow Nerfstudio data format to specify test indices
* Switch from indices to filenames and add tests
* Fix #1581
* Fix changed interface
* added volinga model to NeRFStudio
* Removed unused dependencies
* Modified dockerfile to match main
* Corrected typo in core_code_checks.yml
* Corrected typo in core_code_checks.yml
* updated test with Volinga
* Added conversion to NVOL documentation
* [WIP] Refactor documentation and removing Volinga config
* Volinga use existing NerfactoModelConfig
* Added Unreal Engine documentation to extensions
* removed unused field class import
* Updated docs
* Documentation ready
* Update Dockerfile
---------
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
* Allow black/white background color in Nerfacto
* Use white background in Nerfacto Blender Benchmark
* Fixed unused import
* Updated index for license headers script
* fixed an exception bug in case of num_samples is 0
fixed an exception bug when num samples is 0 ,
"IndexError tensors used as indices must be long, byte or bool tensors"
change type to long ( was float32)
* implemented crop box in gui. For now works on instant-ngp-bounded
* fixed import bugs
* update aabb every time and not only on training
* prevent crash when chosing accomulation and crop box
* Checkbox
* max_box is bounded by scevne box in the gui
* implemented altenative intesection to nerfacc
* convert str to bool correctly
* fixed isort bug
* fixed isort bug2
* update viewer version
* remove json bug
* Update UI
* Force black background
* Fix formatting
Co-authored-by: dror <you@example.com>
Co-authored-by: Matt Tancik <tancik@berkeley.edu>
Co-authored-by: Matthew Tancik <mattt4001@gmail.com>
* fix notebook
* samples
* run all cells
* box center
* try saving again
* import order
* cube with different sides
* black
* black
* update metadata
* fix test
Co-authored-by: Katrin Schmid <kabischmid@gmail.com>
Co-authored-by: Matt Tancik <tancik@berkeley.edu>
Making TrainerConfig an InstantiateConfig so it can be swapped in and out with different methods. Also made the TrainerConfig a subclass of ExperimentConfig (which was changed from a printableconfig to an instantiateconfig), since we were already instantiating the Trainer class with the ExperimentConfig
Removed the "config" field from the TrainerCallbackAttributes struct to eliminate circular dependencies when types are checked.
Minor scripts and docs changes to accommodate the update to the config struct, meaning the flags that were associated with the TrainerConfig were moved up to the top level of the config.
* inerted a new model configuration instant-ngp-bounded
* added more remarks and small fixes
* went back to old near plane
* added the oprtion to get a file as well as directory in nerfstudio data parser
* pretify prev commit
* review fixes
* more clear explanation in README.md
* reverted readme changes
* fixed merge bug
* review changes. update config.data docs support for json files inputs
Co-authored-by: dror <you@example.com>
This branch gives us the flexibility to use Cameras as a tensordataclass, giving arbitrary batch dimensions, slicing, etc all for free. It tests this with the phototourism dataset.
Tensordataclass changes:
Originally, the tensordataclass would look at tensors in an object and take everything in front of the last dimension to be the batch dimensions, but in the case of the camera_to_world matrices, everything except for the last TWO are the important dimensions, so we needed to have a way for the tensordataclass to acknowledge these specific fields where there are multiple data dimensions behind the batch dimensions. This is important for things like reshaping where we need to know which dimensions get reshaped and which dimensions get left alone.
Cameras changes:
Now subclass of tensordataclass, which required a change to the tensordataclass itself.
The possibility of extra batch dimensions is the hairy part that requires the majority of these changes. The extra batch dimensions aren't a hassle in every function except in ray generation. The ray generation is where this operates different relative to the old cameras. The reason for this is before, for each ray we wanted to generate, we could specify a single index into our batch of cameras, but if you have multiple camera dimensions, for each ray, you need to now specify one index per camera batch dimension, meaning our camera_indices we specify need to have one extra dimension at the end, a dimension of size Cameras.ndim (number of batch dimensions in the camera object).
I also added some neat logic in generate_rays where if you want your output raybundle to be batched, if you specify batched camera_indices, you will receive the same batch size for the output raybundle, meaning you can easily generate raybundles corresponding to things like image patches and whatnot.
Phototourism changes:
The difficulty here is that images in the phototourism dataset are of different sizes, which is difficult because we can no longer batch the image tensors. We also need to better utilize caching in the CacheDataloader since the vanilladatamanager defaults don't really use it as well as they could, and without properly utilizing it, the rays / sec are maybe 25% of the standard nerfstudio dataset speeds (which are 10% of the size of the phototourism datasets). To do this, we specify a collate function in the existing datamanager configs, and we make a subclass of the vanilladatamanagerconfig named VariableResDataManagerConfig which will have a custom collate function storing the images in a list, and using better defaults for caching in the dataloader, caching ~40 images at a time. The slowest part of using a list is pixel sampling, where we added a custom if case for if our pixel sampler receives a batch of images that are in a list, and the lack of ability to vectorize operations over a list means that caching in the dataloader is even more important.
* Add D-NeRF temporal distortion and D-NeRF model
This allows for temporal distortion to be enabled on a variety of models through config
parameters, without an explicit model. Temporal distortion and MIP-NeRF are not currently
jointly runnable but can be changed in the future.
* Simplifying changes
* Simplifying changes
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
