From f9a52278286b3a0191c0a9eed562cf3490257245 Mon Sep 17 00:00:00 2001 From: psetinek <122691000+psetinek@users.noreply.github.com> Date: Sun, 28 Jan 2024 20:01:46 +0100 Subject: [PATCH] External Method: seathru_nerf (#2829) * add seathru_nerf as external method --- docs/index.md | 1 + docs/nerfology/methods/index.md | 1 + docs/nerfology/methods/seathru_nerf.md | 105 +++++++++++++++++++++++++ nerfstudio/configs/external_methods.py | 16 ++++ 4 files changed, 123 insertions(+) create mode 100644 docs/nerfology/methods/seathru_nerf.md diff --git a/docs/index.md b/docs/index.md index 4922385a7..685dc635e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -155,6 +155,7 @@ This documentation is organized into 3 parts: - [Tetra-NeRF](nerfology/methods/tetranerf.md): Representing Neural Radiance Fields Using Tetrahedra - [Instruct-GS2GS](nerfology/methods/igs2gs.md): Editing 3DGS Scenes with Instructions - [PyNeRF](nerfology/methods/pynerf.md): Pyramidal Neural Radiance Fields +- [SeaThru-NeRF](nerfology/methods/seathru_nerf.md): Neural Radiance Field for subsea scenes **Eager to contribute a method?** We'd love to see you use nerfstudio in implementing new (or even existing) methods! Please view our {ref}`guide` for more details about how to add to this list! diff --git a/docs/nerfology/methods/index.md b/docs/nerfology/methods/index.md index 6a7b296ab..d29477221 100644 --- a/docs/nerfology/methods/index.md +++ b/docs/nerfology/methods/index.md @@ -41,6 +41,7 @@ The following methods are supported in nerfstudio: Generfacto Instruct-GS2GS PyNeRF + SeaThru-NeRF ``` (own_method_docs)= diff --git a/docs/nerfology/methods/seathru_nerf.md b/docs/nerfology/methods/seathru_nerf.md new file mode 100644 index 000000000..a5b3b590c --- /dev/null +++ b/docs/nerfology/methods/seathru_nerf.md @@ -0,0 +1,105 @@ +# SeaThru-NeRF +```{button-link} https://sea-thru-nerf.github.io +:color: primary +:outline: +Official Paper Website +``` + +```{button-link} https://github.com/AkerBP/seathru_nerf +:color: primary +:outline: +Code (nerfstudio implementation) +``` + +

+ Example Render +

+ +**A Neural Radiance Field for subsea scenes.** + +## Requirements + +We provide the following two model configurations: + +| Method | Description | Memory | Quality | +| ------------------- | ------------- | ------ | ------- | +| `seathru-nerf` | Larger model | ~23 GB | Best | +| `seathru-nerf-lite` | Smaller model | ~7 GB | Good | + +`seathru-nerf-lite` should run on a single desktop/laptop GPU with 8GB VRAM. + +## Installation + +After installing nerfstudio and its dependencies, run: + +```bash +pip install git+https://github.com/AkerBP/seathru_nerf +``` + +## Running SeaThru-NeRF + +To check your installation and to see the hyperparameters of the method, run: + +```bash +ns-train seathru-nerf-lite --help +``` + +If you see the help message with all the training options, you are good to go and ready to train your first susbea NeRF! 🚀🚀🚀 + +For a detailed tutorial of a training process, please see the docs provided [here](https://akerbp.github.io/seathru_nerf/). + + +## Method +This method is an unofficial extension that adapts the official [SeaThru-NeRF](https://sea-thru-nerf.github.io) publication. Since it is built ontop of nerfstudio, it allows for easy modification and experimentation. + +Compared to a classical NeRF approach, we differentiate between solid objects and the medium within a scene. Therefore both, the object colours and the medium colours of samples along a ray contribute towards the final pixel colour as follows: + +$$\boldsymbol{\hat{C}}(\mathbf{r})=\sum_{i=1}^N \boldsymbol{\hat{C}}^{\rm obj}_i(\mathbf{r})+\sum_{i=1}^N \boldsymbol{\hat{C}}^{\rm med}_i(\mathbf{r}) \,.$$ + +Those two contributions can be calculated as follows: + +$$\boldsymbol{\hat{C}}^{\rm obj}_i(\mathbf{r}) = + T^{\rm obj}_i \cdot \exp (-\boldsymbol{\sigma}^{\rm attn} t_i) + \cdot \big(1-\exp({-\sigma^{\rm obj}_i\delta_i})\big) \cdot \mathbf{c}^{\rm obj}_i \,,$$ +$$\boldsymbol{\hat{C}}^{\rm med}_i(\mathbf{r}) = + T^{\rm obj}_i \cdot \exp ( -\boldsymbol{\sigma}^{\rm bs} t_i ) + \cdot \big( 1 - \exp ( -\boldsymbol{\sigma}^{\rm bs} \delta_i ) \big) \cdot \mathbf{c}^{\rm med}\,,$$ +$$\textrm{where } \ T^{\rm obj}_i = \exp\bigg(-\sum_{j=0}^{i-1}\sigma^{\rm obj}_j\delta_j\bigg) \,. $$ + +The above equations contain five parameters that are used to describe the underlying scene: object density $\sigma^{\rm obj}_i \in \mathbb{R}^{1}$, object colour $\mathbf{c}^{obj}_i \in \mathbb{R}^{3}$, backscatter density $\boldsymbol{\sigma}^{\rm bs} \in \mathbb{R}^{3}$, attenuation density $\boldsymbol{\sigma}^{\rm attn} \in \mathbb{R}^{3}$ and medium colour $\mathbf{c}^{\rm med} \in \mathbb{R}^{3}$. + +To get a better idea of the different densities, the following figure shows an example ray with the different densities visualised: + +

+SeaThru-NeRF ray +

+ +*The image above was taken from [Levy et al. (2023)](https://arxiv.org/abs/2304.07743).* + +To predict the object and medium parameters, we use two separate networks. This subsea specific approach can be visualised as follows: (note that the third network is the proposal network, which is used to sample points in regions that contribute most to the final image) + +

+SeaThru-NeRF Architecture +

+ +If you are interested in an in depth description of the method, make sure to check out the documentation [here](https://akerbp.github.io/seathru_nerf/intro.html). + + +## Example results +Due to the underlying image formation model that allows us to seperate between the objects and the water within a scene, you can choose between different rendering options. The following options exist: + +- rgb: To render normal RGB of the scene. +- J: To render the clear scene (water effect removed). +- direct: To render the attenuated clear scene. +- bs: To render backscatter of the water within the scene. +- depth: To render depthmaps of the scene. +- accumulation: To render object weight accumulation of the scene. + +Below, you can see an original render of a scene and one with the water effects removed: + +

+RBG rendering +J rendering +

+ +*Please note that those gifs are compressed and do not do the approach justice. Please render your own videos to see the level of detail and clarity of the renders.* \ No newline at end of file diff --git a/nerfstudio/configs/external_methods.py b/nerfstudio/configs/external_methods.py index 86f237d5a..7dbae661f 100644 --- a/nerfstudio/configs/external_methods.py +++ b/nerfstudio/configs/external_methods.py @@ -176,6 +176,22 @@ To enable PyNeRF, you must install it first by running: ) ) +# SeaThru-NeRF +external_methods.append( + ExternalMethod( + """[bold yellow]Seathru-NeRF[/bold yellow] +For more information visit https://docs.nerf.studio/nerfology/methods/seathru_nerf.html + +To enable Seathru-NeRF, you must install it first by running: + [grey]pip install git+https://github.com/AkerBP/seathru_nerf[/grey]""", + configurations=[ + ("seathru-nerf", "SeaThru-NeRF for underwater scenes."), + ("seathru-nerf-lite", "SeaThru-NeRF for underwater scenes (smaller networks and batches)."), + ], + pip_package="git+https://github.com/AkerBP/seathru_nerf", + ) +) + @dataclass class ExternalMethodDummyTrainerConfig: