From ecd40dfab8be9633525160e11dd947d032799f9b Mon Sep 17 00:00:00 2001 From: "Roger Q." Date: Sat, 4 Jan 2025 03:44:04 +0800 Subject: [PATCH] Adding docs for feature-splatting as an external method (#3557) Add support for feature splatting as an external method --- docs/index.md | 1 + docs/nerfology/methods/feature_splatting.md | 87 +++++++++++++++++++++ docs/nerfology/methods/index.md | 1 + nerfstudio/configs/external_methods.py | 17 +++- 4 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 docs/nerfology/methods/feature_splatting.md diff --git a/docs/index.md b/docs/index.md index b7833ba59..71ea84e91 100644 --- a/docs/index.md +++ b/docs/index.md @@ -153,6 +153,7 @@ This documentation is organized into 3 parts: - [SIGNeRF](nerfology/methods/signerf.md): Controlled Generative Editing of NeRF Scenes - [K-Planes](nerfology/methods/kplanes.md): Unified 3D and 4D Radiance Fields - [LERF](nerfology/methods/lerf.md): Language Embedded Radiance Fields +- [Feature Splatting](nerfology/methods/feature_splatting.md): Gaussian Feature Splatting based on GSplats - [Nerfbusters](nerfology/methods/nerfbusters.md): Removing Ghostly Artifacts from Casually Captured NeRFs - [NeRFPlayer](nerfology/methods/nerfplayer.md): 4D Radiance Fields by Streaming Feature Channels - [Tetra-NeRF](nerfology/methods/tetranerf.md): Representing Neural Radiance Fields Using Tetrahedra diff --git a/docs/nerfology/methods/feature_splatting.md b/docs/nerfology/methods/feature_splatting.md new file mode 100644 index 000000000..edeb0e22b --- /dev/null +++ b/docs/nerfology/methods/feature_splatting.md @@ -0,0 +1,87 @@ +# Feature Splatting + +

Feature Splatting

+ +```{button-link} https://feature-splatting.github.io/ +:color: primary +:outline: +Paper Website +``` + +```{button-link} https://github.com/vuer-ai/feature-splatting/ +:color: primary +:outline: +Code +``` + + + +**Feature Splatting distills SAM-enhanced CLIP features into 3DGS for segmentation and editing** + +## Installation + +First install nerfstudio dependencies. Then run: + +```bash +pip install git+https://github.com/vuer-ai/feature-splatting +``` + +## Running Feature Splatting + +Details for running Feature Splatting (built with Nerfstudio!) can be found [here](https://github.com/vuer-ai/feature-splatting). +Once installed, run: + +```bash +ns-train feature-splatting --help +``` + +Currently, we provide the following variants: + +| Method | Description | Memory | Quality | +| ----------- | ----------------------------------------------- | ------ | ------- | +| `feature-splatting` | Feature Splatting with MaskCLIP ViT-L/14@336px and MobileSAMv2 | ~8 GB | Good | + +Note that the reference features used in this version are different from the version used in the paper in two ways + +- The SAM-enhanced CLIP features are computed using MobileSAMv2, which is much faster than original SAM but slightly less accurate. +- The CLIP features are computed only on the image-level. + +## Method + +Feature splatting distills CLIP features into 3DGS by view-independent rasterization, which allows open-vocabulary 2D segmentation and open-vocabulary 3D segmentation of Gaussians directly in the 3D space. This implementation supports simple editing applications by directly manipulating Gaussians. + +### Reference feature computation and joint supervision + +Feature splatting computes high-quality SAM-enhanced CLIP features as reference features. Compared to coarse CLIP features (such as those used in LERF), Feature splatting performs an object-level masked average pooling of the features to refine the boundary of objects. While the original ECCV'24 paper uses SAM for part-level masks, this implementation uses MobileSAMv2 for much faster reference features computation, which we hope would encourage downstream applications that require real-time performance. + +In addition to SAM-enhanced features, we also found that using DINOv2 features as a joint supervision helps regularize the internal structure of objects, which is similar to findings in existing work. + +### Scene Editing + +Thanks to the explicit representation of 3DGS, grouped Gaussians can be easily manipulated. While the original ECCV'24 paper proposes a series of editing primitives, to avoid introducing excessive dependencies or hacks, we support a subset of editing primitives in this implementation: + +Rigid operations +- Floor estimation (for intuitive rotation and gravity estimation) +- Translation +- Transparent (highlights segmented object and turns background Gaussians transparent) +- Rotation (yaw only w.r.t. estimated ground) + +Non-rigid operations +- Sand-like melting (based on Taichi MPM method) + + + +If you find our work helpful for your research, please consider citing + +```none +@inproceedings{qiu-2024-featuresplatting, + title={Language-Driven Physics-Based Scene Synthesis and Editing via Feature Splatting}, + author={Ri-Zhao Qiu and Ge Yang and Weijia Zeng and Xiaolong Wang}, + booktitle={European Conference on Computer Vision (ECCV)}, + year={2024} +} +``` diff --git a/docs/nerfology/methods/index.md b/docs/nerfology/methods/index.md index a8ecc7f09..320d6ae97 100644 --- a/docs/nerfology/methods/index.md +++ b/docs/nerfology/methods/index.md @@ -34,6 +34,7 @@ The following methods are supported in nerfstudio: SIGNeRF K-Planes LERF + Feature-Splatting Mip-NeRF NeRF Nerfacto diff --git a/nerfstudio/configs/external_methods.py b/nerfstudio/configs/external_methods.py index b25a9f16a..002b3299b 100644 --- a/nerfstudio/configs/external_methods.py +++ b/nerfstudio/configs/external_methods.py @@ -93,6 +93,21 @@ To enable LERF, you must install it first by running: ) ) +# Feature Splatting +external_methods.append( + ExternalMethod( + """[bold yellow]Feature-Splatting[/bold yellow] +For more information visit: https://docs.nerf.studio/nerfology/methods/feature_splatting.html + +To enable Feature Splatting, you must install it first by running: + [grey]pip install git+https://github.com/vuer-ai/feature-splatting[/grey]""", + configurations=[ + ("feature-splatting", "Feature Splatting with MaskCLIP ViT-L/14@336px, DINOv2 ViT-S/14, and MobileSAMv2"), + ], + pip_package="git+https://github.com/vuer-ai/feature-splatting", + ) +) + # Tetra-NeRF external_methods.append( ExternalMethod( @@ -213,7 +228,7 @@ external_methods.append( For more information visit https://docs.nerf.studio/nerfology/methods/zipnerf.html To enable Zip-NeRF, you must install it first by running: - [grey]pip install git+https://github.com/SuLvXiangXin/zipnerf-pytorch#subdirectory=extensions/cuda + [grey]pip install git+https://github.com/SuLvXiangXin/zipnerf-pytorch#subdirectory=extensions/cuda and pip install git+https://github.com/SuLvXiangXin/zipnerf-pytorch[/grey]""", configurations=[ ("zipnerf", "A pytorch implementation of 'Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields'")