From be291970dbf1f36c76ba7e261b0007b7944ec349 Mon Sep 17 00:00:00 2001 From: Giodiro Date: Wed, 17 May 2023 20:48:08 +0200 Subject: [PATCH] Documentation for k-planes add-on (#1935) * Add documentation for kplanes add-on * Fix all integration spots with nerfstudio --------- Co-authored-by: gmeanti --- docs/index.md | 1 + docs/nerfology/methods/index.md | 1 + docs/nerfology/methods/kplanes.md | 87 ++++++++++++++++++++++++++ nerfstudio/configs/external_methods.py | 15 +++++ 4 files changed, 104 insertions(+) create mode 100644 docs/nerfology/methods/kplanes.md diff --git a/docs/index.md b/docs/index.md index abddd1f82..e34650018 100644 --- a/docs/index.md +++ b/docs/index.md @@ -112,6 +112,7 @@ This documentation is organized into 3 parts: ### Third-party Methods - [Instruct-NeRF2NeRF](nerfology/methods/in2n.md): Editing 3D Scenes with Instructions +- [K-Planes](nerfology/methods/kplanes.md): Unified 3D and 4D Radiance Fields - [LERF](nerfology/methods/lerf.md): Language Embedded Radiance Fields - [Tetra-NeRF](nerfology/methods/tetranerf.md): Representing Neural Radiance Fields Using Tetrahedra diff --git a/docs/nerfology/methods/index.md b/docs/nerfology/methods/index.md index 6d746a267..07eccb4b0 100644 --- a/docs/nerfology/methods/index.md +++ b/docs/nerfology/methods/index.md @@ -28,6 +28,7 @@ The following methods are supported in nerfstudio: :maxdepth: 1 Instant-NGP Instruct-NeRF2NeRF + K-Planes LERF Mip-NeRF NeRF diff --git a/docs/nerfology/methods/kplanes.md b/docs/nerfology/methods/kplanes.md new file mode 100644 index 000000000..65c6c1044 --- /dev/null +++ b/docs/nerfology/methods/kplanes.md @@ -0,0 +1,87 @@ +# K-Planes + +

Explicit Radiance Fields in Space, Time, and Appearance

+ + +```{button-link} https://sarafridov.github.io/K-Planes/ +:color: primary +:outline: +Paper Website +``` + +```{button-link} https://github.com/sarafridov/K-Plane +:color: primary +:outline: +Official Code +``` + +```{button-link} https://github.com/Giodiro/kplanes_nerfstudio +:color: primary +:outline: +Nerfstudio add-on code +``` + + + +**A unified model for static, dynamic and variable appearance NeRFs.** + + +## Installation + +First, install nerfstudio and its dependencies. Then install the K-Planes add-on +``` +pip install kplanes-nerfstudio +``` + +## Running K-Planes + +There are two default configurations provided which use the blender and DNeRF dataloaders. However, you can easily extend them to create a new configuration for different datasets. + +The default configurations provided are +| Method | Description | Scene type | Memory | +| ----------------- | -------------------------| ------------------------------ | ------ | +| `kplanes` | Tuned for blender scenes | static, synthetic | ~4GB | +| `kplanes-dynamic` | Tuned for DNeRF dataset | dynamic (monocular), synthetic | ~5GB | + + +for training with these two configurations you should run +```bash +ns-train kplanes --data +``` +or +```bash +ns-train kplanes-dynamic --data +``` + +:::{admonition} Note +:class: warning + +`kplanes` is set up to use blender data, (download it running `ns-download-data blender`), +`kplanes-dynamic` is set up to use DNeRF data, (download it running `ns-download-data dnerf`). +::: + + +## Method + +![method overview](https://sarafridov.github.io/K-Planes/assets/intro_figure.jpg)
+K-planes represents a scene in k dimensions -- where k can be 3 for static 3-dimensional scenes or 4 for scenes which change in time -- +using k-choose-2 planes (or grids). After ray-sampling, the querying the field at a certain position consists in querying each plane (with interpolation), and combining the resulting features through multiplication. +This factorization of space and time keeps memory usage low, and is very flexible in the kinds of priors and regularizers that can be added.
+
+ +We support hybrid models with a small MLP (left) and fully explicit models (right), through the `linear_decoder` [configuration key](https://github.com/Giodiro/kplanes_nerfstudio/blob/db4130605159dadaf180228be5d0335d2c4d21f9/kplanes/kplanes.py#L87) +
+ + + +The model also supports decomposing a scene into its space and time components. For more information on how to do this see the [official code repo](https://github.com/sarafridov/K-Plane) +
+ diff --git a/nerfstudio/configs/external_methods.py b/nerfstudio/configs/external_methods.py index 11d4541c6..0510d6ec2 100644 --- a/nerfstudio/configs/external_methods.py +++ b/nerfstudio/configs/external_methods.py @@ -58,6 +58,21 @@ To enable Instruct-NeRF2NeRF, you must install it first by running: ) ) +# K-Planes +external_methods.append( + ExternalMethod( + """[bold yellow]K-Planes[/bold yellow] +For more information visit https://docs.nerf.studio/en/latest/nerfology/methods/kplanes.html + +To enable K-Planes, you must install it first by running: + [grey]pip install kplanes-nerfstudio[/grey]""", + configurations=[ + ("kplanes", "K-Planes model tuned to static blender scenes"), + ("kplanes-dynamic", "K-Planes model tuned to dynamic DNeRF scenes"), + ], + pip_package="kplanes-nerfstudio", + ) +) # LERF external_methods.append(