Commit Graph
167 Commits
Author SHA1 Message Date
Justin Kerr ccf1dd5dc7 dataloader memory leak 2023-10-12 16:03:55 -07:00
Justin Kerr 17e95e1ba2 more bugfix, number of gaussians doesnt go crazy anymore 2023-10-10 10:29:44 -07:00
Justin Kerr c0d5e19451 inhouse impl start 2023-09-26 16:26:05 -07:00
Justin Kerr 3c40314b62 merge main 2023-09-22 16:29:03 -07:00
Justin Kerr c2fef380e9 inhouse impl start 2023-09-22 16:28:20 -07:00
Jonáš Kulhánek 76f8bcf953 Disable loading colmap points by default (#2451) 2023-09-21 13:51:35 -07:00
Jake Austin jake-austin@berkeley.edu 35be6fd421 Accounting for camera distortion, caching undistorted images 2023-09-18 10:13:43 +00:00
anc2001andBrent Yi 805e6e702d fix pixel sampling with masks issue when data is list (#2369)
Co-authored-by: Brent Yi <yibrenth@gmail.com>
2023-09-14 16:11:15 +00:00
Jonáš Kulhánek e9330d515d Colmap parser: fix downscale for complex paths (#2425) 2023-09-13 10:55:24 -07:00
Brent Yi 9c061f2a99 Config fixes 2023-09-12 18:18:38 -07:00
Jake Austin jake-austin@berkeley.edu 1a214ba129 Temp save 2023-09-12 23:19:47 +00:00
Jake Austin jake-austin@berkeley.edu bb90b94fb0 Basic working, still needs to be profiled and tuned, some preliminary TODOs labeled 'jake-austin' need to be addressed 2023-09-12 03:15:11 +00:00
Jonáš Kulhánek 570ccf3c0a Rename COLMAP 3D point features (#2417) 2023-09-11 09:00:44 -07:00
Jonáš Kulhánek ead9b37c65 Allow colmap parser to load 3D points (#2408) 2023-09-11 07:50:18 +00:00
Jake Austin jake-austin@berkeley.edu 749059d4fb First pass at dataloader, set up file structure (minus the actual rendering code) 2023-09-08 17:55:46 +00:00
85ef567682 Flesh out viewer beta (#2086)
* Increase max_res of first proposal network

* fix render statemachine logic for viewer beta

* clicking on cameras in the viewer

* fix fov calculation, update camera positions while training

* cleanup

* backup

* fix configure theme

* Fix theme configure for latest dev version of viser

* Fix runtime errors

* add set up direction button

* add pause resume button

* Export panel

* Render tab skeleton

* make responsiveness more snappy by stopping training

* depth composite

* start implementing viewercontrol in new viewer

* refactor background img to use depth and add depth layering option

* show/hide images

* speed up rendering

* fix crop color background

* crop

* rotated crops

* rotated crops

* transform handle on crop box

* fix xyzw for crop

* oriented crop export for pointclouds, not for meshes yet

* Initial render panel

* Fix export button

* Theme tweaks

* fix background issue in cropping

* export rotated poisson meshe

* double normals definition in export panel

* Visual nits

* remove tsdf from export panel

* Add licenses

* Fix some CI errors

* Type fixes

* Fix tests

* Fix numpy broken

* various bugfix

---------

Co-authored-by: Brent Yi <yibrenth@gmail.com>
Co-authored-by: terrancewang <terrance.3512@gmail.com>
2023-09-08 05:10:00 +00:00
elenacliuandAdamRashid96 ae5d569376 Fix: typo in PairPixelSamplerConfig's description (#2372)
Co-authored-by: AdamRashid96 <71362382+AdamRashid96@users.noreply.github.com>
2023-08-26 03:09:44 -07:00
6e37b2b542 Enable SparseNerf depth ranking loss to be used with GT/Sensor depth (#2312)
* add ranking loss to depth-nerfacto with GT depth

* formating

* pair pixel sampler bug fix from pr #2315

---------

Co-authored-by: Sasha Sax <asax@stanford.edu>
Co-authored-by: Ethan Weber <ethanweber@berkeley.edu>
Co-authored-by: AdamRashid96 <abrashid@berkeley.edu>
Co-authored-by: AdamRashid96 <71362382+AdamRashid96@users.noreply.github.com>
2023-08-26 02:56:52 -07:00
Sasha SaxandMatthew Tancik 9cd957f776 Update pixel_samplers.py (#2327)
* Update pixel_samplers.py

Now uses the specified batch_size param in sample_method()

* black

---------

Co-authored-by: Matthew Tancik <matt@lumalabs.ai>
2023-08-17 18:04:51 +00:00
elenacliu bc8341378a fix #2318: add train/test/eval split for instant ngp dataparser (#2335)
* fix #2318: add train/test/eval split for instant ngp dataparser

* fix: instant_ngp_dataparser.py format
2023-08-14 04:24:34 +00:00
Haoyuan Wang 3455fb749f fix: ColmapDataParser downscaling images error. (#2337) 2023-08-13 21:16:47 -07:00
machenmusik cb2405d3ea When generating depth images, use torch_compile with ZoeDepth for compatibility (#2303)
* Use torch_compile with Zoe for compatibility

torch_compile module support

Previously the compatibility utility torch.compile worked for functions, but not for modules as torch.compile was originally employed here.

* torch_compile -> Any
2023-08-09 17:08:17 -07:00
Ethan Weberandmachenmusik 9b6010ea0d Adding support for PairPixelSampler and PatchPixelSampler masks (#2305)
* changes

* pair pixel sampling with a mask

* reverting changes

* revert changes

* remove line in erode

* support for patch pixel samplerr masks

* fix

---------

Co-authored-by: machenmusik <machenmusik@users.noreply.github.com>
2023-08-05 10:40:23 -04:00
7e90cb7989 PixelSampler Fixes (#2298)
* monodepth implemented

* device

* pair pixel sampler

* docstring

* removing unused imports

* fix bugs

* cache in cache dir, no longer edit json file

* moving to depth nerfacto

* fixed bug using median depth by accident

* removing depthdataset from nerfacto

* added pixelsamplerconfig

* added equirectangular warning back

* bug fix

* delete sample_config.py + small fixes

* rename to pixel_sampler

* cleaning

* test

* passing num_rays_per_batch properly

* fixed transforms typo

* renaming in nerfacto

* remove depth files from test data

* removed defaulting to base pixel sampler with mask

* adding patch_size back

* Warning for notimplemented error and patch_size fix

* fix config.pixel_sampler check

---------

Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
Co-authored-by: Ethan Weber <ethanweber@berkeley.edu>
Co-authored-by: Ethan Weber <eweb0124@gmail.com>
2023-08-04 10:27:35 -04:00
Frederik WarburgandEthan Weber 9b03299d40 Train and eval data splits (Nerfbusters) (#2207)
* nerfbusters dataparser

* nerfbuster dataparser and visualization of eval views

* resolved some errors from  pyright

* enable ns-process-data to take two videos as input and use one for eval and the other for train

* resolve some pyright issues

* instead of data being a tuple, I added eval_data as another argument

* implemented train eval split for all cases

* implemented train eval split for all cases

* reverting equi changes

* reverting reality capture changes

* raise valueerror

* revert comment

* added split override code back to dataparser

* reverted one change

* using image border now

* adding docs

---------

Co-authored-by: Ethan Weber <eweb0124@gmail.com>
2023-08-04 10:11:55 -04:00
71c6b9cf3d Monodepth Regularization (#2284)
* monodepth implemented

* device

* pair pixel sampler

* docstring

* removing unused imports

* fix bugs

* cache in cache dir, no longer edit json file

* moving to depth nerfacto

* fixed bug using median depth by accident

* removing depthdataset from nerfacto

* added pixelsamplerconfig

* added equirectangular warning back

* bug fix

* delete sample_config.py + small fixes

* rename to pixel_sampler

* cleaning

* test

* passing num_rays_per_batch properly

* fixed transforms typo

* renaming in nerfacto

* remove depth files from test data

* removed defaulting to base pixel sampler with mask

* adding patch_size back

---------

Co-authored-by: AdamRashid96 <abrashid@berkeley.edu>
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-08-01 18:52:47 -07:00
AX-I 98d36ed7d9 Option to process images on GPU for speed at the expense of memory (#2288)
* Fix pixel sampler cpu bottleneck

cpu usage reduced 1700% -> 100%, and even more if using multi-gpu training

* Convert to config option

* Change optional to bool
2023-08-01 11:12:30 -07:00
Sasha Sax 7e8d67f8a8 Add depth_unit_scale_factor to sdfstudio_dataparser (#2285)
Depth supervision for `depth-nerfacto` was added in #1173. This patch extends that support to `sdfstudio` datasets ([as originally intended](https://github.com/nerfstudio-project/nerfstudio/issues/1173#issuecomment-1494633338_)) by adding `depth_unit_scale_factor` to `sdfstudio` metadata.
2023-08-01 09:03:01 -07:00
Nicholas PfaffandJonas Kulhanek 9f487a464f Implement nerfacto alpha transparency training (#2165)
* Implement nerfacto alpha transparency training

* Stop adding all ones alpha channel and clean up blend_background

* Make background_color a required argument in blend_background

* Enable alpha transparent training by default

* Improve background_color typing

* Blend uniform background color for metric computation

* Fix typing error

* Clean up typing

* Fix background_color to outputs assignment

* Model outputs returns unblended RGB (black bg) for random bg

* Update renderers

* Fix comments

* Add support for transparent BG for other methods

* Minor refac

* Fix get_background_color

* Fix typing error for blend_background

* Fix comments

* Export utils support transparency

* Pylint fix

* Fix pcd export masking

* Filter pcd view directions based on opacity

* Clean up get_rgba_image

---------

Co-authored-by: Jonas Kulhanek <jonas.kulhanek@live.com>
2023-07-18 12:54:52 +02:00
Jonáš KulhánekandMatthew Tancik b3df46314c Remove breakpoint from colmap_dataparser (#2141)
Co-authored-by: Matthew Tancik <matt@lumalabs.ai>
2023-06-29 17:03:29 +00:00
Jonáš Kulhánek aa6c052660 Jkulhanek/colmap dataparser (#2115)
* Add COLMAP dataparser implementation

* Fix downscale_images input

* Add help to COLMAP dataparser
2023-06-25 17:08:17 +00:00
Jonáš Kulhánek ac6a83cc53 Fix pickling of VanillaDataManager (#2129)
* Fix pickling of VanillaDataManager

* Fix vanilladatamanager init
2023-06-25 18:46:07 +02:00
machenmusik ccb8a25d4a Add option to process masks on GPU for speed at the expense of memory (#2110)
--pipeline.datamanager.masks-on-gpu True
2023-06-21 09:16:14 -07:00
b17268e83b [WIP] dreamfusion implementation (#1008)
* starting the framework for dreamfusion

* Removed view direction from RGB to calculate albedo

* Started diffusion

* Added dreamfusion to CLI, removed direction embeddings

* Normals in dreamfusion

* Lambertian shading added

* Train normals in Dreamfusion

* Stable diffusion from Stable Dreamfusion

* Fix normal orientation loss

* Configurable light direction sampling

* adding density initalization at origin

* V1 datamanager

* Don't train normals for first 1k iters

* SDS loss added to dreamfusion

* Dreamfusion pipeline

* Background MLP and SphereCollider

* Normals prediction training fixed

* Sigmoid on background, prompt in config

* Shading hyperparameter adjustements

* Shading hyperparameter tuning

* Location based prompting, density initialization fades, LR change

* Cleaned imports, more config options

* Tuning, more config changes

* remove unused sampler, normalize predicted normals from network

* change position encoding to direction encoding for background color mlp

* Updating dreamfusion to use custom trainer, fixed bug, multi-gpu support (put diffusion on one specific device), merged with main

* Hard coding less, offloading to config

* Appeasing the linter

* First pass opacity / sparsity losses

* Small patches and configs

* Accidentally pushed logs

* Removing logs for real this time

* Transmittance loss bug fix

* 12 GB Dreamfusion!

* Lint

* Remove unecissary collider changes

* Start SD cleanup

* Cleanup SD code

* adding overhead location prompting

* Use lambertian renderer

* turn grad accum off by default, decrease background mlp size, add overhead text embedding

* Cleanup

* Fix normals vis

* change uniform sampler to prediction network, add option to jit unet for faster sds_loss

* Use shaders

* cleaning up jit code

* add warning if using non jit unet

* add proposal network optimizer to config

* speed up render and cleanup code

* Clean up datamanager

* Add distortion loss

* fix wanb

* adding interpolated location sampling

* Update configs

* interpolated location prompting option, lowering init density, transmittance loss interpolation flipped

* rename alphas mult to alphas loss mult

* Mess around with normals

* add option to use random bg color + bg mlp

* Remove Dreamfusion Trainer (#1423)

* Dreamfusion trainer no longer needed

* Added typing for grad scaler

* SDS return loss

* Fix CFG

* Cleanup

* Update configs

* Fix opacity loss, use unform sampling first round of prop

* add prompt edditing to viewer

* Choose your own diffusion

* remove dreamfusion pipeline file

* merging + remove dreamfusion pipeline file

* Update configs

* add stable diff utils

* cleanup

* cleanup

* Fix test

* positional prompting bugfix

* Warmups

* tune

* lint

* Remove dataparser

* merging + text embedding bugfix

* Set data path to launch viewer for DreamFusion (#1776)

* Set data path to launch viewer for DreamFusion

* Apply formatter

* Fallback to 'base_dir' if data manager has no path

* Make if statement non-inline

* Reformat

---------

Co-authored-by: Matthew Tancik <mattt4001@gmail.com>

* add deepfloyd option first try

* merging

* merging from main

* merging

* removing comments, cleaning code, making stable diff utils usable for deepfloyd, remove deepfloyd text encoder after use

* undo chagnes in data folder

* cleaning code, removing unneccesary changes from main, removed normals prediction network

* remove old scripts files, rename generative pipeline flag to no_input_data, revert to default histogram padding in proposal sampler

* changing pipeline flag

* move no input data flag from pipeline to datamanager

* undo config and merge in deepfloyd from main

* merging grad scaler pipeline addition

* merging + setting proposal net scheduler

* revert nerfacto config change

* typing + linting

* rename dreamfusion datamanager to random camras datamanager, check datamanager typing in pipeline and trainer

* fix test import

* fix test import

* changing datamanager check in trainer, add typing in random cameras datamanager

* add Tensor to randomcamerasdatamanager

* remove old comments

* resolve tyro warning

* update fields to use tcnn

* lint

* add packages for deepfloyd

* add versions

* docs first draft

* rename to generfacto, clean docs

* fix import error

* minor changes

* fix orientation loss bug, retune params

* cleanup

* typing

* trying docs fix

* fixing generative package imports

* fix pyright

* docs fix

* merge w main

* docs minor change

* tuning + bugfixes

* remove torch.backends

---------

Co-authored-by: Abhik Ahuja <ahujaabhik@berkeley.edu>
Co-authored-by: David McAllister <davidrmc2@gmail.com>
Co-authored-by: Jake Austin <jake-austin@berkeley.edu>
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
Co-authored-by: Matt Tancik <mattt4001@gmail.com>
Co-authored-by: SeungWoo Yoo <dreamy1534@gmail.com>
Co-authored-by: ayaanzhaque <ayaanzhaque@gmail.com>
Co-authored-by: Ayaan Haque <ayaanzhaque@berkeley.edu>
2023-06-17 23:34:04 -07:00
Matthew Tancik 26fc1c95dc Update data_utils doc (#2076) 2023-06-14 17:57:33 -07:00
Weihang Li f2fdd10d9b fix for scannet dataset camera pose loading (#2056)
* fix scannet dataparser pose issue

* camera pose corrected
2023-06-10 09:07:57 -07:00
Frédéric DevernayandFrédéric Devernay e2e5637d05 nerfstudio dataparser: avoid loading images at inference time (#2051)
Also remove the image skipping logic, which seems to be a remnant of some old code (all images should exist)

Co-authored-by: Frédéric Devernay <f-dy@users.noreply.github.com>
2023-06-09 21:29:32 -04:00
Matias Turkulainen 88b253f506 Add kwargs to dataparser outputs (#2053)
add kwargs to dataparser outputs
2023-06-09 21:21:17 -04:00
Jonáš Kulhánek 3b0f7581d8 Reduce GPU memory usage in CacheDataloader (#1730)
* Reduce GPU memory usage in CacheDataloader

* Move also masks and seg.masks, depths to cpu

* Fix linter issues
2023-06-07 14:14:24 -04:00
Jonáš Kulhánek a0d5307dd7 [WIP] Fix model loading collate_fn non-static (#1990)
* Fix model loading collate_fn non-static

* Fix serialization

* Add tests to reproduce the error

* Comment the change to let the tests fail

* Finish changes
2023-05-29 00:56:41 -07:00
Mingwu ZhengandMatthew Tancik ba08a8f530 Add Metadata Handling to Cameras Collation Function (#2001)
* Add metadata collate for cameras

* checks passed

---------

Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-05-27 21:00:40 +00:00
Milan van Wouden d413ca4204 fix large number of typos in repository (#1977)
* fix large number of typos in repository

* fix typos in tests
2023-05-23 10:41:49 -07:00
Daniel Crispell ccdcc3951e generalize mask collation to other image-like data (#1959) 2023-05-22 18:10:39 -07:00
Lukas Hoellein d8f2446122 Fix mask check in PatchPixelSampler (#1971)
* fix mask check in PatchPixelSampler

* change torch.Tensor to Tensor
2023-05-22 10:43:14 -07:00
Matthew TancikandBrent Yi 2b0831b6d4 Final typing fixes + add Pyright to CI (#1944)
* knocking out a couple

* Fix more errors

* Add pyright

* Remove some unnecessary cv2 type ignores, warning on missing imports
(for missing tinycudann)

* Minor annotation fixes

* Update docs

* Drop `None` from temporal distortion output

---------

Co-authored-by: Brent Yi <yibrenth@gmail.com>
2023-05-18 02:25:34 -07:00
Brent YiandMatt Tancik d542646c72 Switch to ruff (#1939)
* `isort` -> `ruff`

* `pylint` -> `ruff`

* Remove line

* More minor cleanup

* Better documentation for rules we are enabling

* Fix gitignore, run black

* vscode settings

---------

Co-authored-by: Matt Tancik <mattt4001@gmail.com>
2023-05-17 10:08:14 -07:00
Brent Yi 814db32ec4 More typing fixes (#1941)
* Fix more types

* Fix tests

* Minor cleaning

* Address comments

* Resolve linting errors

* Re-ignore pymeshlab type

* Add extra assert / error message to mipnerf model
2023-05-17 07:17:40 -07:00
Matthew Tancik 8b9574bb5a Type fixes and cleanup (#1937) 2023-05-16 18:47:46 -07:00
Brent YiandMatt Tancik 6ba2a18b2b torchtyping => jaxtyping (#1838)
* Run `ruff --fix`

* Add jaxtyping migration script

* Run migration scripts + ruff + black

* Merge remote-tracking branch 'origin/main' into brent/jaxtyping

* black

* Fix lint

* fix lint

* Cleanup and more specific types

* Fixed annotation errors

* Fix more errors

* Fix annotation

* Delete _torchtyping_to_jaxtyping.py

---------

Co-authored-by: Matt Tancik <mattt4001@gmail.com>
2023-05-16 15:57:47 -07:00
Ethan Weber 13e0fce122 Update copyright (#1917)
* updated copyright

* including txt file

* updated black

* removing license

* fixing license
2023-05-12 16:16:13 -07:00