Update lerf docs (#1782)

This commit is contained in:
Matthew Tancik authored and GitHub committed 2023-04-20 10:06:13 -07:00
1 parent bf3ae6ac5c
commit 9a61851bcf
2 files changed
+21 -15

No files matched your search

+4
View File
@@ -10,6 +10,7 @@ We recommend the following file structure:
```
├── my_method
│ ├── __init__.py
│ ├── my_config.py
│ ├── custom_pipeline.py [optional]
│ ├── custom_model.py [optional]
@@ -56,6 +57,9 @@ dependencies = [
"nerfstudio" # you may want to consider pinning the version, ie "nerfstudio==0.1.19"
]
[tool.setuptools.packages.find]
include = ["my_method*"]
[project.entry-points.'nerfstudio.method_configs']
my-method = 'my_method.my_config:MyMethod'
```
+17 -15
View File
@@ -22,15 +22,11 @@ Code
## Installation
First install nerfstudio dependencies.
Then download LERF and install it:
First install nerfstudio dependencies. Then run:
```bash
git clone https://github.com/kerrj/lerf.git
cd lerf/ && pip install -e .
pip install git+https://github.com/kerrj/lerf
```
<!-- # pip install git+https://github.com/kerrj/lerf bugs out for the time being, WIP -->
## Running LERF
@@ -42,39 +38,45 @@ ns-train lerf --help
Three variants of LERF are provided:
| Method | Description | Memory | Quality |
| ------------ | ----------------------------------------------- | --------- | ------- |
| `lerf-big` | LERF with OpenCLIP ViT-L/14 | ~22 GB | Best |
| `lerf` | Model with OpenCLIP ViT-B/16, used in paper | ~15 GB | Good |
| `lerf-lite` | LERF with smaller network and less LERF samples | ~8 GB | Ok |
| Method | Description | Memory | Quality |
| ----------- | ----------------------------------------------- | ------ | ------- |
| `lerf-big` | LERF with OpenCLIP ViT-L/14 | ~22 GB | Best |
| `lerf` | Model with OpenCLIP ViT-B/16, used in paper | ~15 GB | Good |
| `lerf-lite` | LERF with smaller network and less LERF samples | ~8 GB | Ok |
`lerf-lite` should work on a single NVIDIA 2080.
`lerf-big` is experimental, and needs further tuning.
`lerf-big` is experimental, and needs further tuning.
## Method
LERF enables pixel-aligned queries of the distilled 3D CLIP embeddings without relying on region proposals, masks, or fine-tuning, supporting long-tail open-vocabulary queries hierarchically across the volume.
### Multi-scale supervision
To supervise language embeddings, we pre-compute an image pyramid of CLIP features for each training view. Then, each sampled ray during optimization is supervised by interpolating the CLIP embedding within this pyramid.
<img id="lerf_multiscale" src="https://www.lerf.io/data/clip_features.png" style="background-color:white;" width="100%">
### LERF Optimization
LERF optimizes a dense, multi-scale language 3D field by volume rendering CLIP embeddings along training rays, supervising these embeddings with multi-scale CLIP features across multi-view training images.
LERF optimizes a dense, multi-scale language 3D field by volume rendering CLIP embeddings along training rays, supervising these embeddings with multi-scale CLIP features across multi-view training images.
Inspired by Distilled Feature Fields (DFF), we use DINO features to regularize CLIP features. This leads to qualitative improvements in object boundaries, as CLIP embeddings in 3D can be sensitive to floaters and regions with few views.
After optimization, LERF can extract 3D relevancy maps for language queries interactively in real-time.
After optimization, LERF can extract 3D relevancy maps for language queries interactively in real-time.
<img id="lerf_render" src="https://www.lerf.io/data/nerf_render.png" style="background-color:white;" width="100%">
### Visualizing relevancy for text queries
Set the "Output Render" type to `relevancy_0`, and enter the text query in the "LERF Positives" textbox (see image). The output render will show the 3D relevancy map for the query. View the [project page](https://lerf.io) for more examples and details about the relevancy map normalization.
<center>
<img id="lerf_viewer" src="https://www.lerf.io/data/lerf_screen.png" width="40%">
</center>
## Results
For results, view the [project page](https://lerf.io)!
Datasets used in the original paper can be found [here](https://drive.google.com/drive/folders/1LUzwEvBCE19PNYcwfmrG-9FLpZLbi4on?usp=sharing).
@@ -85,5 +87,5 @@ Datasets used in the original paper can be found [here](https://drive.google.com
title = {LERF: Language Embedded Radiance Fields},
journal = {arXiv preprint arXiv:2303.09553},
year = {2023},
}
}
```