From 9a61851bcf3859025003c77aeb6ceffea0994ccb Mon Sep 17 00:00:00 2001 From: Matthew Tancik Date: Thu, 20 Apr 2023 10:06:13 -0700 Subject: [PATCH] Update lerf docs (#1782) --- docs/developer_guides/new_methods.md | 4 ++++ docs/nerfology/methods/lerf.md | 32 +++++++++++++++------------- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/docs/developer_guides/new_methods.md b/docs/developer_guides/new_methods.md index 661629d15..26376d655 100644 --- a/docs/developer_guides/new_methods.md +++ b/docs/developer_guides/new_methods.md @@ -10,6 +10,7 @@ We recommend the following file structure: ``` ├── my_method +│ ├── __init__.py │ ├── my_config.py │ ├── custom_pipeline.py [optional] │ ├── custom_model.py [optional] @@ -56,6 +57,9 @@ dependencies = [ "nerfstudio" # you may want to consider pinning the version, ie "nerfstudio==0.1.19" ] +[tool.setuptools.packages.find] +include = ["my_method*"] + [project.entry-points.'nerfstudio.method_configs'] my-method = 'my_method.my_config:MyMethod' ``` diff --git a/docs/nerfology/methods/lerf.md b/docs/nerfology/methods/lerf.md index c2b634fb0..509c88fac 100644 --- a/docs/nerfology/methods/lerf.md +++ b/docs/nerfology/methods/lerf.md @@ -22,15 +22,11 @@ Code ## Installation -First install nerfstudio dependencies. - -Then download LERF and install it: +First install nerfstudio dependencies. Then run: ```bash -git clone https://github.com/kerrj/lerf.git -cd lerf/ && pip install -e . +pip install git+https://github.com/kerrj/lerf ``` - ## Running LERF @@ -42,39 +38,45 @@ ns-train lerf --help Three variants of LERF are provided: -| Method | Description | Memory | Quality | -| ------------ | ----------------------------------------------- | --------- | ------- | -| `lerf-big` | LERF with OpenCLIP ViT-L/14 | ~22 GB | Best | -| `lerf` | Model with OpenCLIP ViT-B/16, used in paper | ~15 GB | Good | -| `lerf-lite` | LERF with smaller network and less LERF samples | ~8 GB | Ok | +| Method | Description | Memory | Quality | +| ----------- | ----------------------------------------------- | ------ | ------- | +| `lerf-big` | LERF with OpenCLIP ViT-L/14 | ~22 GB | Best | +| `lerf` | Model with OpenCLIP ViT-B/16, used in paper | ~15 GB | Good | +| `lerf-lite` | LERF with smaller network and less LERF samples | ~8 GB | Ok | `lerf-lite` should work on a single NVIDIA 2080. -`lerf-big` is experimental, and needs further tuning. +`lerf-big` is experimental, and needs further tuning. ## Method LERF enables pixel-aligned queries of the distilled 3D CLIP embeddings without relying on region proposals, masks, or fine-tuning, supporting long-tail open-vocabulary queries hierarchically across the volume. ### Multi-scale supervision + To supervise language embeddings, we pre-compute an image pyramid of CLIP features for each training view. Then, each sampled ray during optimization is supervised by interpolating the CLIP embedding within this pyramid. ### LERF Optimization -LERF optimizes a dense, multi-scale language 3D field by volume rendering CLIP embeddings along training rays, supervising these embeddings with multi-scale CLIP features across multi-view training images. + +LERF optimizes a dense, multi-scale language 3D field by volume rendering CLIP embeddings along training rays, supervising these embeddings with multi-scale CLIP features across multi-view training images. Inspired by Distilled Feature Fields (DFF), we use DINO features to regularize CLIP features. This leads to qualitative improvements in object boundaries, as CLIP embeddings in 3D can be sensitive to floaters and regions with few views. -After optimization, LERF can extract 3D relevancy maps for language queries interactively in real-time. +After optimization, LERF can extract 3D relevancy maps for language queries interactively in real-time. ### Visualizing relevancy for text queries + Set the "Output Render" type to `relevancy_0`, and enter the text query in the "LERF Positives" textbox (see image). The output render will show the 3D relevancy map for the query. View the [project page](https://lerf.io) for more examples and details about the relevancy map normalization. +
+
## Results + For results, view the [project page](https://lerf.io)! Datasets used in the original paper can be found [here](https://drive.google.com/drive/folders/1LUzwEvBCE19PNYcwfmrG-9FLpZLbi4on?usp=sharing). @@ -85,5 +87,5 @@ Datasets used in the original paper can be found [here](https://drive.google.com title = {LERF: Language Embedded Radiance Fields}, journal = {arXiv preprint arXiv:2303.09553}, year = {2023}, -} +} ```