Commit Graph
52 Commits
Author SHA1 Message Date
Afloat 98a459e626 Keep SceneContraction derivatives finite at the origin 2026-10-08 06:21:00 -07:00
Frédéric DevernayandBrent Yi 83fe2f6cd4 fix core tests (#3687)
* fix core tests

* Try alternative fix for pyright errors

* Run ruff, bump core code checks to 3.11

* Run correct ruff version

* Use the previous pyright version 👀

* cv2 ignore, pin dev dependency to newer pytorch version

* ruff

* Remove pyright error suppressions

---------

Co-authored-by: Brent Yi <yibrenth@gmail.com>
2025-07-15 16:35:57 -07:00
Justin Kerr 66cd6817f9 rerun formatting with ruff 0.9.1 (#3569)
rerun with ruff 0.9.1
2025-01-13 10:46:11 -08:00
Alexander Kristoffersen a8888e7d9e Move util functions out of splatfacto (#3538)
* Move util functions out of splatfacto

Nothing else currently uses some of the SH utils, but it might make sense to get them out of splatfacto.

I also moved the k nearest neighbors to utils since it doesn't depend on the model class.

* fix assert

* fix sh test

* convert to using degrees, not levels
2024-12-05 13:25:55 -05:00
Justin Kerr d6df39b70d Compatibility for numpy 2.0 (#3265)
* compat for numpy 2.0

* lint changes
2024-06-28 08:07:10 +09:00
Matthew Hoffman f0aadf9de8 Use ruff as linter and formatter (#2765)
* Switch from black to ruff format

* Sort dependencies with ruff check

* Run ruff format
2024-01-15 13:43:14 -08:00
Ethan WeberandBrent Yi 4c627edbac Encoder + MLP combo (#2063)
* adding encoder + mlp combo

* update documentation

* minor fixes

* fixed issue when tcnn isn't installed

---------

Co-authored-by: Brent Yi <yibrenth@gmail.com>
2023-12-01 03:16:42 +00:00
blacksino 747dd557f6 fix include_input for NeRFEncoding (#2642) 2023-11-30 23:07:26 +00:00
79297c7c61 (minor) Align upsampling in TensorCPEncoding with TensorVMEncoding (#2493)
* (minor) Align upsampling in TensorCPEncoding with TensorVMEncoding

* Fix black

---------

Co-authored-by: Matthew Tancik <mattt4001@gmail.com>
Co-authored-by: Matthew Tancik <matt@lumalabs.ai>
2023-10-08 21:22:53 -07:00
Maxim BonnaerensandJonáš Kulhánek 0d746b1363 Fourier Feature encodings and polyhedron encodings (#2463)
* Fourier Feature encodings and polyhedron encodings

Rework RFFEncoding to be a subclass of the more general Fourier Feature
Encodings and introduce Polyhedron encodings as introduced in
mipnerf360.

* b_matrix -> basis

* Add typing

* fighting pyright

* use scale argument

* continue the fight

* ignore

* ignore em all

* add docstring and rename generate_basis to generate_polyhedron_basis

* Try to please pyright with assert

* Immediately allocate tensor on correct device

* private functions and docstrings update

* doc fix continued

---------

Co-authored-by: Jonáš Kulhánek <jonas.kulhanek@live.com>
2023-10-02 21:07:34 +00:00
Jonáš Kulhánek ef2fd3dbe1 Fix import error when tcnn installed but CUDA runtime not available (#2429) 2023-09-15 17:18:23 -04:00
Riley 6cb10ae682 Adding input handling for RFFEncoding's get_out_dim(self) (#2201)
fixed RFFEncoding's get_out_dim(self) to incude dims of input if include_input
2023-07-11 22:56:12 +00:00
Georg HessandMatthew Tancik 492fa0e67b Remove pytorch hash table if tcnn implementation is used (#2175)
* fixes div by 1 error

* removes pytorch hash_table if tcnn is used

* formatting

* test fixes

---------

Co-authored-by: Matthew Tancik <matt@lumalabs.ai>
2023-07-06 17:55:22 +00:00
Matthew Tancik 061d691d43 Default to tcnn (#2159) 2023-07-02 21:36:28 -07:00
Ethan WeberandMatias Turkulainen d9ee5752d2 Fix to fall back to torch if tcnn not found (#2124)
fix for falling back to torch if tcnn not found

Co-authored-by: Matias Turkulainen <30566358+maturk@users.noreply.github.com>
2023-06-27 09:33:28 -07:00
b17268e83b [WIP] dreamfusion implementation (#1008)
* starting the framework for dreamfusion

* Removed view direction from RGB to calculate albedo

* Started diffusion

* Added dreamfusion to CLI, removed direction embeddings

* Normals in dreamfusion

* Lambertian shading added

* Train normals in Dreamfusion

* Stable diffusion from Stable Dreamfusion

* Fix normal orientation loss

* Configurable light direction sampling

* adding density initalization at origin

* V1 datamanager

* Don't train normals for first 1k iters

* SDS loss added to dreamfusion

* Dreamfusion pipeline

* Background MLP and SphereCollider

* Normals prediction training fixed

* Sigmoid on background, prompt in config

* Shading hyperparameter adjustements

* Shading hyperparameter tuning

* Location based prompting, density initialization fades, LR change

* Cleaned imports, more config options

* Tuning, more config changes

* remove unused sampler, normalize predicted normals from network

* change position encoding to direction encoding for background color mlp

* Updating dreamfusion to use custom trainer, fixed bug, multi-gpu support (put diffusion on one specific device), merged with main

* Hard coding less, offloading to config

* Appeasing the linter

* First pass opacity / sparsity losses

* Small patches and configs

* Accidentally pushed logs

* Removing logs for real this time

* Transmittance loss bug fix

* 12 GB Dreamfusion!

* Lint

* Remove unecissary collider changes

* Start SD cleanup

* Cleanup SD code

* adding overhead location prompting

* Use lambertian renderer

* turn grad accum off by default, decrease background mlp size, add overhead text embedding

* Cleanup

* Fix normals vis

* change uniform sampler to prediction network, add option to jit unet for faster sds_loss

* Use shaders

* cleaning up jit code

* add warning if using non jit unet

* add proposal network optimizer to config

* speed up render and cleanup code

* Clean up datamanager

* Add distortion loss

* fix wanb

* adding interpolated location sampling

* Update configs

* interpolated location prompting option, lowering init density, transmittance loss interpolation flipped

* rename alphas mult to alphas loss mult

* Mess around with normals

* add option to use random bg color + bg mlp

* Remove Dreamfusion Trainer (#1423)

* Dreamfusion trainer no longer needed

* Added typing for grad scaler

* SDS return loss

* Fix CFG

* Cleanup

* Update configs

* Fix opacity loss, use unform sampling first round of prop

* add prompt edditing to viewer

* Choose your own diffusion

* remove dreamfusion pipeline file

* merging + remove dreamfusion pipeline file

* Update configs

* add stable diff utils

* cleanup

* cleanup

* Fix test

* positional prompting bugfix

* Warmups

* tune

* lint

* Remove dataparser

* merging + text embedding bugfix

* Set data path to launch viewer for DreamFusion (#1776)

* Set data path to launch viewer for DreamFusion

* Apply formatter

* Fallback to 'base_dir' if data manager has no path

* Make if statement non-inline

* Reformat

---------

Co-authored-by: Matthew Tancik <mattt4001@gmail.com>

* add deepfloyd option first try

* merging

* merging from main

* merging

* removing comments, cleaning code, making stable diff utils usable for deepfloyd, remove deepfloyd text encoder after use

* undo chagnes in data folder

* cleaning code, removing unneccesary changes from main, removed normals prediction network

* remove old scripts files, rename generative pipeline flag to no_input_data, revert to default histogram padding in proposal sampler

* changing pipeline flag

* move no input data flag from pipeline to datamanager

* undo config and merge in deepfloyd from main

* merging grad scaler pipeline addition

* merging + setting proposal net scheduler

* revert nerfacto config change

* typing + linting

* rename dreamfusion datamanager to random camras datamanager, check datamanager typing in pipeline and trainer

* fix test import

* fix test import

* changing datamanager check in trainer, add typing in random cameras datamanager

* add Tensor to randomcamerasdatamanager

* remove old comments

* resolve tyro warning

* update fields to use tcnn

* lint

* add packages for deepfloyd

* add versions

* docs first draft

* rename to generfacto, clean docs

* fix import error

* minor changes

* fix orientation loss bug, retune params

* cleanup

* typing

* trying docs fix

* fixing generative package imports

* fix pyright

* docs fix

* merge w main

* docs minor change

* tuning + bugfixes

* remove torch.backends

---------

Co-authored-by: Abhik Ahuja <ahujaabhik@berkeley.edu>
Co-authored-by: David McAllister <davidrmc2@gmail.com>
Co-authored-by: Jake Austin <jake-austin@berkeley.edu>
Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
Co-authored-by: Matt Tancik <mattt4001@gmail.com>
Co-authored-by: SeungWoo Yoo <dreamy1534@gmail.com>
Co-authored-by: ayaanzhaque <ayaanzhaque@gmail.com>
Co-authored-by: Ayaan Haque <ayaanzhaque@berkeley.edu>
2023-06-17 23:34:04 -07:00
LSong 10dab142d0 Move out NeRFPlayer: clean files and add method link (#2072)
* clean files and add method link

* fix black

* update
2023-06-14 13:51:09 -07:00
Ethan WeberandEthan Weber bb6925bac6 Ability to use vanilla pytorch instead of tcnn for Nerfacto (#1936)
* started a package to easily swap out tcnn for pytorch

* trains with pytorch and not tcnn

* cleaned up imports

* making it easy to install on mac

* removed/reverted files

* merged main

* reverted changes

* updating config

* starting to add nerfacto to tests

* fixxing default value

* fixed default implementation

* defaults to torch if tcnn and not installed

* update log

* add comment

* resolved pr comments

* fix

* update default

---------

Co-authored-by: Ethan Weber <ethanweber@eecs-128-32-175-250.dhcp.berkeley.edu>
2023-05-25 14:55:42 -07:00
Matthew Tancik 27e2c874cb Report tcnn import errors (#1969) 2023-05-22 09:36:00 -07:00
Matthew TancikandBrent Yi 2b0831b6d4 Final typing fixes + add Pyright to CI (#1944)
* knocking out a couple

* Fix more errors

* Add pyright

* Remove some unnecessary cv2 type ignores, warning on missing imports
(for missing tinycudann)

* Minor annotation fixes

* Update docs

* Drop `None` from temporal distortion output

---------

Co-authored-by: Brent Yi <yibrenth@gmail.com>
2023-05-18 02:25:34 -07:00
Jonáš KulhánekandBrent Yi f481f5ebec Fix some typing problems (#1940)
* Fix some typing problems

* nerfstudio.data typing error fixes

* Add tiny fixes

* Remove ABC from pipeline

* Prevent circular imports

* Tweak ModuleList typing

* TrainingCallbackAttributes cleanup

* Various minor nits

* More nits

* Run `ruff`

---------

Co-authored-by: Brent Yi <yibrenth@gmail.com>
2023-05-17 17:36:56 -07:00
Brent YiandMatt Tancik d542646c72 Switch to ruff (#1939)
* `isort` -> `ruff`

* `pylint` -> `ruff`

* Remove line

* More minor cleanup

* Better documentation for rules we are enabling

* Fix gitignore, run black

* vscode settings

---------

Co-authored-by: Matt Tancik <mattt4001@gmail.com>
2023-05-17 10:08:14 -07:00
Brent Yi 814db32ec4 More typing fixes (#1941)
* Fix more types

* Fix tests

* Minor cleaning

* Address comments

* Resolve linting errors

* Re-ignore pymeshlab type

* Add extra assert / error message to mipnerf model
2023-05-17 07:17:40 -07:00
Brent YiandMatt Tancik 6ba2a18b2b torchtyping => jaxtyping (#1838)
* Run `ruff --fix`

* Add jaxtyping migration script

* Run migration scripts + ruff + black

* Merge remote-tracking branch 'origin/main' into brent/jaxtyping

* black

* Fix lint

* fix lint

* Cleanup and more specific types

* Fixed annotation errors

* Fix more errors

* Fix annotation

* Delete _torchtyping_to_jaxtyping.py

---------

Co-authored-by: Matt Tancik <mattt4001@gmail.com>
2023-05-16 15:57:47 -07:00
Ethan Weber 13e0fce122 Update copyright (#1917)
* updated copyright

* including txt file

* updated black

* removing license

* fixing license
2023-05-12 16:16:13 -07:00
Brent Yi 63c6303736 Bump minimum Python version to 3.8 (#1876)
* Bump minimum Python version to 3.8

* typing_extensions.Literal -> typing.Literal

* ns-dev-test

* typing_extensions.get_args => typing.get_args

* Format with older version of black

* Bump `black`

* Run black
2023-05-06 13:29:48 -07:00
Daniel Crispell cf74119adc SceneContraction maps to range [-2, 2], not [-1, 1]. (#1872) 2023-05-04 10:04:30 -07:00
hturki 2f18e323cd return unscaled input in positional encoding (#1736)
* return unscaled input in positional encoding

* Update encodings.py
2023-04-17 13:08:31 -07:00
Giodiroandgmeanti 78fbe6513e Add KPlanesEncoding encoding (#1658)
Co-authored-by: gmeanti <giacomo.meanti@gmail.com>
2023-03-24 08:18:57 -07:00
Pablo Vela ddfa463f4d NeuS Implementation (#1444)
* add initial method configs and data manager

* add neus scheduler

* add initial surface model and neus

* add Neus ray sampler

* add sdf field

* move scale/shift calc to math utils

* add debug and fix sdf_dataset typo

* update vscode neus debug

* fix dataset and get initial working version

* add magic number from original repo

* black fixes

* clean up get weights/transmittance from alphas

* clean up typos in sdf datamanager

* type annotations on neus scheduler and remove unused params

* clean up and annotate sdf field along w/ other fixes

* more clean up

* remove unnessary directions norm

* resolve comments

* update neus scheduler to follow refactor

* fix linter errors

* fix some typos

* remove get_weights_from_alpha

* more comment fixes

* more linter error fixes

* clean up and add comments

* more little fixes for types/comments

* more comments

* rename NeuS Scheduler to DelayedScheduler

* rename DelayedScheduler to CosineDecayScheduler
2023-03-12 16:01:18 -04:00
Jake AustinandMatthew Tancik 50dc5d6cd7 Triplanes (#1456)
* Triplanes working

* Doing the TODOs, better variable names

---------

Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-02-22 16:23:12 -08:00
LSongandLcDog b653eef91c add time (#1391)
* add time

* fix black

---------

Co-authored-by: LcDog <12553253+LcDog@users.noreply.github.com>
2023-02-09 16:43:21 -08:00
LSongandmachenmusik e73692e599 nerfplayer with ngp and nerfacto (#1277)
* nerfplayer with ngp and nerfacto

* update comments

* update cpp extension build

* avoid Windows repeated CUDA compile

---------

Co-authored-by: machenmusik <machenmusik@users.noreply.github.com>
2023-01-31 11:56:28 -08:00
LSong b170e175ed NeRFPlayer sliding window implementation (#1255)
* NeRFPlayer sliding window implementation

* add comments
2023-01-18 18:57:55 -08:00
matsurenandMatthew Tancik 408ab709ec Fix typo (#1231)
* Fix typo COLAMP->COLMAP

* Fix more typo

Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2023-01-16 12:12:55 -08:00
Ikko Ashimine e01f5caa53 Fix typo in mlp.py (#1185)
Ouput -> Output
2022-12-29 10:17:08 -07:00
Penn bc690632bb Use torch.where for faster contract (#1132)
use where for faster contract
2022-12-15 13:24:31 -08:00
Alexander DemidkoandAlexander Demidko bcbddf42d6 Interpolation config for tcnn hashgrid (#1058)
* Interpolation config for tcnn hashgrid

* Fixed ns-dev-test issues

Co-authored-by: Alexander Demidko <ademidko@matterport.com>
2022-12-01 07:38:38 -08:00
Julian Knodt 0d3cfe2d38 bugfix: AABB normalized_positions to [-1, 1] (#998)
AABB normalized_positions to [-1, 1]

Previously, SceneBox.get_normalized_positions() returned a value in 0,1.
This was fine for InstantNGP and Nerfacto, which use these values directly in an MLP.
For TensoRF though, this was an error because grid sample expects values in the range [-1,1].
Thus, I think only 1/4th of the encoding planes were actually being used, and half of the
encoding lines were.
2022-11-21 12:15:01 -08:00
b829499c12 Computing and predicting normals (#905)
* staring normals PR

* code runs but visualization looks bad

* Camera and world coordinate conventions (#889)

* documented coordinate conventions without a figure for now

* another sentence

* update frames docs

* explain intrinsics and extrinsics

* more comments

* fix build

* Start viewer with more accurate speed calculation (#899)

Start viewer with more accurate speed calc

* Minor doc updates (#903)

* fix conflicts

* code runs but visualization looks bad

* normals implemented and vis fixed. still debugging quality

* minor name change

* comment

* fix black

* lint

* predicting normals

* adding detach

* predicted normals

* improved normals prediction performance

* cleaning code

* cleaning code

* cleaning code

* adding type to function param

* fixing tensorf forward func

* reverting tensorf changes

* fix style with tensorf field

* adding normals check to tensorffield

Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
Co-authored-by: terrance <terrance.3512@gmail.com>
2022-11-20 13:22:37 -08:00
Julian KnodtandMatthew Tancik 9ff167d705 Add temporal distortion and D-NeRF model (#825)
* Add D-NeRF temporal distortion and D-NeRF model

This allows for temporal distortion to be enabled on a variety of models through config
parameters, without an explicit model. Temporal distortion and MIP-NeRF are not currently
jointly runnable but can be changed in the future.

* Simplifying changes

* Simplifying changes

Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2022-11-15 14:29:29 -08:00
Kevin Chen 3e52aebd9a Do not apply activation for last MLP layer (#932) 2022-11-10 10:08:02 -08:00
Matthew Tancik 3833e4de56 Missing mlp output activation (#930) 2022-11-09 22:54:04 -08:00
Nikhil Mohan 1a24f3e58c Refactor Data Pipeline (#896)
* add none check

* black formatting

* cleanup

* cleanup

* fix tests

* remove new line

* init

* toy example working

* semantic nerf working

* simplify pixel samplers

* cleanup + fix tests

* comments

* fix tests

* fix tests
2022-11-06 07:40:43 -08:00
Terrance WangandMatthew Tancik 7106f42fcc tensorf (#586)
* tensorf wip

* model changes

* tuning model

* changing imports

* merging

* changing rff encoding to nerfencoding, basis matrix should be properly updated, changed density activation, lr scheduler

* debugging tensorf

* debugging tensorf

* tensorf debugging

* tensorf debugging

* tensorf debugging

* tensorf debug

* debugging tensorf, fixed bug with collider

* tuning model, ray sampling

* adding thresholding for ray computations

* removing unused imports

* making more params configurable in config, adding spherical harmonics

* removing scene contraction, adding arg docstrings

* fixing style

* Fixing bug

Co-authored-by: Matthew Tancik <tancik@berkeley.edu>
2022-11-02 14:32:03 -07:00
Matthew Tancik 9bed69601f Support python 3.7 (#802)
* Support python 3.7

* Update docs
2022-10-18 16:49:27 -07:00
Ethan Weber 05ba4b6b00 semantic nerfw (#640)
transients not working right now
2022-10-03 20:36:32 -07:00
evonneng 968959bdc9 quickstart updates + (other nits+links added to repo) (#633)
* small changes

* rewrite quickstart + readme

* fixed all licensing headers

* nits
2022-10-03 16:01:16 -07:00
Alexander Kristoffersen 60bf31844f Camera Extrinsics Optimization (#610)
* init commit.

* cleanup.

* docstring cleanup.
2022-10-02 17:56:25 -07:00
Ethan Weber c20fed83bb moving around files (#616)
moved around files
2022-10-02 16:54:34 -07:00