From 911091ccb647522f71b3c0a12e6306a527a406d9 Mon Sep 17 00:00:00 2001 From: Jan-Niklas Dihlmann Date: Mon, 8 Apr 2024 16:17:56 +0200 Subject: [PATCH] Added SIGNeRF to Nerfstudio methods (#3053) * feat: signerf as nerfstudio method * fix: formating issues and indexing --- docs/index.md | 12 ++-- docs/nerfology/methods/index.md | 3 +- docs/nerfology/methods/signerf.md | 97 ++++++++++++++++++++++++++ nerfstudio/configs/external_methods.py | 17 +++++ 4 files changed, 123 insertions(+), 6 deletions(-) create mode 100644 docs/nerfology/methods/signerf.md diff --git a/docs/index.md b/docs/index.md index fcb62c644..4e992418c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -148,12 +148,13 @@ This documentation is organized into 3 parts: ### Third-party Methods - [Instruct-NeRF2NeRF](nerfology/methods/in2n.md): Editing 3D Scenes with Instructions +- [Instruct-GS2GS](nerfology/methods/igs2gs.md): Editing 3DGS Scenes with Instructions +- [SIGNeRF](nerfology/methods/signerf.md): Controlled Generative Editing of NeRF Scenes - [K-Planes](nerfology/methods/kplanes.md): Unified 3D and 4D Radiance Fields - [LERF](nerfology/methods/lerf.md): Language Embedded Radiance Fields - [Nerfbusters](nerfology/methods/nerfbusters.md): Removing Ghostly Artifacts from Casually Captured NeRFs - [NeRFPlayer](nerfology/methods/nerfplayer.md): 4D Radiance Fields by Streaming Feature Channels - [Tetra-NeRF](nerfology/methods/tetranerf.md): Representing Neural Radiance Fields Using Tetrahedra -- [Instruct-GS2GS](nerfology/methods/igs2gs.md): Editing 3DGS Scenes with Instructions - [PyNeRF](nerfology/methods/pynerf.md): Pyramidal Neural Radiance Fields - [SeaThru-NeRF](nerfology/methods/seathru_nerf.md): Neural Radiance Field for subsea scenes - [Zip-NeRF](nerfology/methods/zipnerf.md): Anti-Aliased Grid-Based Neural Radiance Fields @@ -162,13 +163,14 @@ This documentation is organized into 3 parts: ## Quicklinks -| | | -| ---------------------------------------------------------- | ---------------------- | -| [Github](https://github.com/nerfstudio-project/nerfstudio) | Official Github Repo | -| [Discord](https://discord.gg/RyVk6w5WWP) | Join Discord Community | +| | | +| ---------------------------------------------------------- | --------------------------- | +| [Github](https://github.com/nerfstudio-project/nerfstudio) | Official Github Repo | +| [Discord](https://discord.gg/RyVk6w5WWP) | Join Discord Community | | [Feedback Form](https://forms.gle/sqN5phJN7LfQVwnP9) | Provide Nerfstudio Feedback | ## Sponsors + Sponsors of this work includes [Luma AI](https://lumalabs.ai/) and the [BAIR commons](https://bcommons.berkeley.edu/home). | | | diff --git a/docs/nerfology/methods/index.md b/docs/nerfology/methods/index.md index 5284b6887..99aa638d7 100644 --- a/docs/nerfology/methods/index.md +++ b/docs/nerfology/methods/index.md @@ -29,6 +29,8 @@ The following methods are supported in nerfstudio: Instant-NGP Splatfacto Instruct-NeRF2NeRF + Instruct-GS2GS + SIGNeRF K-Planes LERF Mip-NeRF @@ -39,7 +41,6 @@ The following methods are supported in nerfstudio: Tetra-NeRF TensoRF Generfacto - Instruct-GS2GS PyNeRF SeaThru-NeRF Zip-NeRF diff --git a/docs/nerfology/methods/signerf.md b/docs/nerfology/methods/signerf.md new file mode 100644 index 000000000..76bf14d9d --- /dev/null +++ b/docs/nerfology/methods/signerf.md @@ -0,0 +1,97 @@ +# SIGNeRF + +

Scene Integrated Generation for Neural Radiance Fields

+ +```{button-link} https://signerf.jdihlmann.com/ +:color: primary +:outline: +Website & Code +``` + + + +**Generatively edits NeRF scenes in a controlled and fast manner.** + +SIGNeRF allows for generative 3D scene editing. We present a novel approach to combine [NeRFs](https://www.matthewtancik.com/nerf) as scene representation with the image diffusion model [StableDiffusion](https://github.com/Stability-AI/stablediffusion) to allow fast and controlled 3D generation. + +## Installation + +Install nerfstudio dependencies. Then run: + +```bash +pip install git+https://github.com/cgtuebingen/SIGNeRF +``` + +SIGNeRF requires to use [Stable Diffusion Web UI](https://github.com/AUTOMATIC1111/stable-diffusion-webui). For detailed installation information follow our [installation guide](https://github.com/cgtuebingen/SIGNeRF?tab=readme-ov-file#installation). + +## Running SIGNeRF + +Details for running SIGNeRF can be found [here](https://github.com/cgtuebingen/SIGNeRF). Once installed, run: + +```bash +ns-train signerf --help +``` + +Two variants of SIGNeRF are provided: + +| Method | Description | Time | Quality | +| ------------------ | -------------------------- | ----- | ------- | +| `signerf` | Full model, used in paper | 40min | Best | +| `signerf_nerfacto` | Faster model with Nerfacto | 20min | Good | + +For more information on hardware requirements and training times, please refer to the [training section](https://github.com/cgtuebingen/SIGNeRF?tab=readme-ov-file#training-1). + +## Interface + +SIGNeRF Interface + +SIGNeRF fully integrates into the Nerfstudio [viser](https://viser.studio) interface. It allows for easy editing of NeRF scenes with a few simple clicks. The user can select the editing method, the region to edit, and the object to insert. The reference sheet can be previewed and the NeRF is fine-tuned on the edited images. If you are interested in how we fully edit the Nerfstudio interface please find our code [here](https://github.com/cgtuebingen/SIGNeRF). + +## Method + +### Overview + +SIGNeRF is a novel approach for fast and controllable NeRF scene editing and scene-integrated object generation. We introduce a new generative update strategy that ensures 3D consistency across the edited images, without requiring iterative optimization. We find that depth-conditioned diffusion models inherently possess the capability to generate 3D consistent views by requesting a grid of images instead of single views. Based on these insights, we introduce a multi-view reference sheet of modified images. Our method updates an image collection consistently based on the reference sheet and refines the original NeRF with the newly generated image set in one go. By exploiting the depth conditioning mechanism of the image diffusion model, we gain fine control over the spatial location of the edit and enforce shape guidance by a selected region or an external mesh. + +For an in-depth visual explanation and our results please watch our [videos](https://www.youtube.com/playlist?list=PL5y23CB9WmildtW3QyMEi3arXg06zB4ex) or read our [paper](https://arxiv.org/abs/2401.01647). + +## Pipeline + + + +We leverage the strengths of [ControlNet](https://github.com/lllyasviel/ControlNet), a depth condition image diffusion model, to edit an existing NeRF scene. We do so with a few simple steps in a single forward pass: + +0. We start with an original NeRF scene and select an editing method / region +1. For object generation, we place a mesh object into the scene +2. And control the precise location and shape of the edit +3. We position reference cameras in the scene +4. Render the corresponding color, depth, and mask images, and arrange them into image grids +5. These grids are used to generate the reference sheet with conditioned image diffusion +6. Generate new edited images consistent with the reference sheet by leveraging an inpainting mask. + +- Repeat step (6) for all cameras + +7. Finally, the NeRF is fine-tuned on the edited images + +### Reference Sheet Generation + +Reference Sheet Generation + +SIGNeRF uses a novel technique called reference sheet generation. We observe that the image +diffusion model [ControlNet](https://github.com/lllyasviel/ControlNet) can already generate multiview consistent images of a scene without the need for iterative refinement like [Instruct-NeRF2NeRF](https://instruct-nerf2nerf.github.io/). While generating individual views sequentially introduces too much variation to integrate them into a consistent 3D model, arranging them in a grid of images that are processed by ControlNet in one pass significantly +improves the multi-view consistency. Based on the depth maps rendered from the original NeRF +scene we employ a depth-conditioned inpainting variant of ControlNet to generate such a reference sheet of the edited scene. A mask specifies the scene region where the generation should occur. This step gives a lot of control to the user. Different appearances can be produced by generating reference sheets with different seeds or prompts. The one sheet finally selected will directly determine the look of the final 3D scene. + +If you want to learn more about the method, please read our [paper](https://arxiv.org/abs/2401.01647) or read the breakdown of the method in the [Radiance Fields article](https://radiancefields.com/controlnet-nerfs-signerf/). + +## Related Work + +SIGNeRF was inspired by the work of [Instruct-NeRF2NeRF](https://instruct-nerf2nerf.github.io/) which is also available in the [Nerfology](https://docs.nerf.studio/nerfology/methods/in2n.html) documentation. + +## Authors + +SIGNeRF was developed by [Jan-Niklas Dihlmann](https://jdihlmann.com) and [Andreas Engelhardt](https://aengelhardt.com/). diff --git a/nerfstudio/configs/external_methods.py b/nerfstudio/configs/external_methods.py index a0982f098..ff32e9f13 100644 --- a/nerfstudio/configs/external_methods.py +++ b/nerfstudio/configs/external_methods.py @@ -14,6 +14,7 @@ """This file contains the configuration for external methods which are not included in this repository.""" + import inspect import subprocess import sys @@ -208,6 +209,22 @@ To enable Zip-NeRF, you must install it first by running: ) ) +# SIGNeRF +external_methods.append( + ExternalMethod( + """[bold yellow]SIGNeRF[/bold yellow] +For more information visit: https://docs.nerf.studio/nerfology/methods/signerf.html + +To enable SIGNeRF, you must install it first by running: + [grey]pip install git+https://github.com/cgtuebingen/SIGNeRF[/grey] and install Stable Diffusion Web UI see [grey]https://github.com/cgtuebingen/SIGNeRF?tab=readme-ov-file#installation[/grey]""", + configurations=[ + ("signerf", "SIGNeRF method (high quality) used in paper"), + ("signerf_nerfacto", "SIGNeRF method combined with Nerfacto (faster training less quality)"), + ], + pip_package="git+https://github.com/cgtuebingen/SIGNeRF", + ) +) + @dataclass class ExternalMethodDummyTrainerConfig: