From 7d0ab6aca86540c11afe4c9dabeb40b71d35d26e Mon Sep 17 00:00:00 2001 From: Justin Kerr Date: Wed, 17 Jan 2024 11:44:29 -0800 Subject: [PATCH] Docs for gaussian splatting (#2776) * fill in doc page for GS --- docs/index.md | 1 + docs/nerfology/methods/index.md | 1 + docs/nerfology/methods/splat.md | 29 +++++++++++++++++++++++++++++ 3 files changed, 31 insertions(+) create mode 100644 docs/nerfology/methods/splat.md diff --git a/docs/index.md b/docs/index.md index 9e6ac0fac..2f3093834 100644 --- a/docs/index.md +++ b/docs/index.md @@ -140,6 +140,7 @@ This documentation is organized into 3 parts: - [NeRF](nerfology/methods/nerf.md): OG Neural Radiance Fields - [Mip-NeRF](nerfology/methods/mipnerf.md): A Multiscale Representation for Anti-Aliasing Neural Radiance Fields - [TensoRF](nerfology/methods/tensorf.md): Tensorial Radiance Fields +- [Gaussian Splatting](nerfology/methods/splat.md): 3D Gaussian Splatting (third_party_methods)= diff --git a/docs/nerfology/methods/index.md b/docs/nerfology/methods/index.md index 980fa5335..ab15c1a18 100644 --- a/docs/nerfology/methods/index.md +++ b/docs/nerfology/methods/index.md @@ -27,6 +27,7 @@ The following methods are supported in nerfstudio: ```{toctree} :maxdepth: 1 Instant-NGP + 3D Gaussian Splatting Instruct-NeRF2NeRF K-Planes LERF diff --git a/docs/nerfology/methods/splat.md b/docs/nerfology/methods/splat.md new file mode 100644 index 000000000..802dbe861 --- /dev/null +++ b/docs/nerfology/methods/splat.md @@ -0,0 +1,29 @@ +# Gaussian Splatting +[3D Gaussian Splatting](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/) was proposed in SIGGRAPH 2023 from INRIA, and is a completely +different method of representing radiance fields by explicitly storing a collection of 3D volumetric gaussians. These can be "splatted", or projected, onto a 2D image +provided a camera pose, and rasterized to obtain per-pixel colors. Because rasterization is very fast on GPUs, this method can render much faster than neural representations +of radiance fields. + +### Installation +Nerfstudio uses [gsplat](https://github.com/nerfstudio-project/gsplat) as its gaussian rasterization backend, an in-house re-implementation which is meant to be more developer friendly. This can be installed with `pip install gsplat`. The associated CUDA code will be compiled the first time gaussian splatting is executed. Some users with PyTorch 2.0 have experienced issues with this, which can be resolved by either installing gsplat from source, or upgrading torch to 2.1. + +### Data +Gaussian Splatting works much better if you initialize it from pre-existing geometry, such as SfM points rom COLMAP. COLMAP datasets or datasets from `ns-process-data` will automatically save these points and initialize gaussians on them. Other datasets currently do not support initialization, and will initialize gaussians randomly. Initializing from other data inputs (i.e. depth from phone app scanners) may be supported in the future. + +Because gaussian splatting trains on *full images* instead of bundles of rays, there is a new datamanager in `full_images_datamanager.py` which undistorts input images, caches them, and provides single images at each train step. + + +### Running the Method +To run gaussian splatting, run `ns-train gaussian-splatting --data `. Just like NeRF methods, the splat can be interactively viewed in the web-viewer, rendered, and exported. + +### Details +For more details on the method, see the [original paper](https://arxiv.org/abs/2308.04079). Additionally, for a detailed derivation of the gradients used in the gsplat library, see [here](https://arxiv.org/abs/2312.02121). + +### Exporting splats +Gaussian splats can be exported as a `.ply` file which are ingestable by a variety of online web viewers. You can do this via the viewer, or `ns-export gaussian-splat`. Currently splats can only be exported from trained splats, not from nerfacto. + +### FAQ +- Can I export a mesh or pointcloud? +Currently these export options are not supported, but may in the future and contributions are always welcome! +- Can I render fisheye, equirectangular, orthographic images? +Currently, no. Gaussian splatting assumes a perspective camera for its rasterization pipeline. Implementing other camera models is of interest but not currently planned.