mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-14 20:03:49 +08:00
Add HDR metric/alignment helper scripts and combine READMEs
- align_two_folders.py: model-free contour+homography alignment of two image folders, with diagnostics (checkerboards, thresholds, matched contours, SIFT match panels). - align_fast_to_well.py: expose the pre-RANSAC SIFT correspondences so the new visualizations can consume them. - ldr_res_jpg.py: LDR (jpg) variant of ldr_res.py metric computation. - run_metrics_all.sh: run calculate_metrics_fast across all scenes. - collect_metrics_sheet.py: gather per-scene result JSONs into a CSV. - README.md: merge README-pandora.md and README-hdrnerfacto.md into one guide (PanDORA + HDR-Nerfacto paths). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
MrKarimiDandClaude Opus 4.8 committed
1 parent
1748a9256c
commit
287f56176b
6 files changed
+809
-331
No files matched your search
@@ -1,372 +1,323 @@
|
||||
<p align="center">
|
||||
<!-- community badges -->
|
||||
<a href="https://discord.gg/uMbNqcraFc"><img src="https://img.shields.io/badge/Join-Discord-blue.svg"/></a>
|
||||
<!-- doc badges -->
|
||||
<a href='https://plenoptix-nerfstudio.readthedocs-hosted.com/en/latest/?badge=latest'>
|
||||
<img src='https://readthedocs.com/projects/plenoptix-nerfstudio/badge/?version=latest' alt='Documentation Status' />
|
||||
</a>
|
||||
<!-- pi package badge -->
|
||||
<a href="https://badge.fury.io/py/nerfstudio"><img src="https://badge.fury.io/py/nerfstudio.svg" alt="PyPI version"></a>
|
||||
<!-- code check badges -->
|
||||
<a href='https://github.com/nerfstudio-project/nerfstudio/actions/workflows/core_code_checks.yml'>
|
||||
<img src='https://github.com/nerfstudio-project/nerfstudio/actions/workflows/core_code_checks.yml/badge.svg' alt='Test Status' />
|
||||
</a>
|
||||
<a href='https://github.com/nerfstudio-project/nerfstudio/actions/workflows/viewer_build_deploy.yml'>
|
||||
<img src='https://github.com/nerfstudio-project/nerfstudio/actions/workflows/viewer_build_deploy.yml/badge.svg' alt='Viewer build Status' />
|
||||
</a>
|
||||
<!-- license badge -->
|
||||
<a href="https://github.com/nerfstudio-project/nerfstudio/blob/master/LICENSE">
|
||||
<img alt="License" src="https://img.shields.io/badge/License-Apache_2.0-blue.svg">
|
||||
</a>
|
||||
</p>
|
||||
# nerfstudio-HDR — How it works?
|
||||
|
||||
<p align="center">
|
||||
<!-- pypi-strip -->
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://docs.nerf.studio/en/latest/_images/logo-dark.png">
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://docs.nerf.studio/en/latest/_images/logo.png">
|
||||
<!-- /pypi-strip -->
|
||||
<img alt="nerfstudio" src="https://docs.nerf.studio/en/latest/_images/logo.png" width="400">
|
||||
<!-- pypi-strip -->
|
||||
</picture>
|
||||
<!-- /pypi-strip -->
|
||||
</p>
|
||||
This repository supports two HDR NeRF pipelines that share the same capture,
|
||||
pre-processing, and evaluation workflow, and differ only in the training and
|
||||
metrics steps:
|
||||
|
||||
<!-- Use this for pypi package (and disable above). Hacky workaround -->
|
||||
<!-- <p align="center">
|
||||
<img alt="nerfstudio" src="https://docs.nerf.studio/en/latest/_images/logo.png" width="400">
|
||||
</p> -->
|
||||
- **PanDORA** — two-stage training with fast/well exposure alignment.
|
||||
- **HDR-Nerfacto (`hdr-nerfacto`)** — single-stage training.
|
||||
|
||||
<p align="center"> A collaboration friendly studio for NeRFs </p>
|
||||
Steps that are identical for both methods are written once below. Where the two
|
||||
methods diverge (setup branch, training, metrics) both variants are shown.
|
||||
|
||||
<p align="center">
|
||||
<a href="https://docs.nerf.studio">
|
||||
<img alt="documentation" src="https://user-images.githubusercontent.com/3310961/194022638-b591ce16-76e3-4ba6-9d70-3be252b36084.png" width="150">
|
||||
</a>
|
||||
<a href="https://viewer.nerf.studio/">
|
||||
<img alt="viewer" src="https://user-images.githubusercontent.com/3310961/194022636-a9efb85a-14fd-4002-8ed4-4ca434898b5a.png" width="150">
|
||||
</a>
|
||||
<a href="https://colab.research.google.com/github/nerfstudio-project/nerfstudio/blob/main/colab/demo.ipynb">
|
||||
<img alt="colab" src="https://raw.githubusercontent.com/nerfstudio-project/nerfstudio/main/docs/_static/imgs/readme_colab.png" width="150">
|
||||
</a>
|
||||
</p>
|
||||
## 0. Setup
|
||||
|
||||
<img src="https://user-images.githubusercontent.com/3310961/194017985-ade69503-9d68-46a2-b518-2db1a012f090.gif" width="52%"/> <img src="https://user-images.githubusercontent.com/3310961/194020648-7e5f380c-15ca-461d-8c1c-20beb586defe.gif" width="46%"/>
|
||||
> Tip: Open this folder in the VS Code **Dev Container** extension.
|
||||
|
||||
- [Quickstart](#quickstart)
|
||||
- [Learn more](#learn-more)
|
||||
- [Supported Features](#supported-features)
|
||||
```
|
||||
tmux new -s nerf_studio
|
||||
|
||||
# About
|
||||
docker run -it --rm -v /gel/usr/{USER_NAME}:/mnt/workspace/ -v /home-local2/{USER_NAME}.extra.nobkp/:/mnt/data/ --gpus '"device=0"' -p 8008:8008 -p 6006:6006 lantern_docker:latest /bin/bash
|
||||
|
||||
_It’s as simple as plug and play with nerfstudio!_
|
||||
cd /mnt/workspace/lantern/nerfstudio-HDR
|
||||
|
||||
Nerfstudio provides a simple API that allows for a simplified end-to-end process of creating, training, and testing NeRFs.
|
||||
The library supports a **more interpretable implementation of NeRFs by modularizing each component.**
|
||||
With more modular NeRFs, we hope to create a more user-friendly experience in exploring the technology.
|
||||
|
||||
This is a contributor-friendly repo with the goal of building a community where users can more easily build upon each other's contributions.
|
||||
Nerfstudio initially launched as an opensource project by Berkeley students in [KAIR lab](https://people.eecs.berkeley.edu/~kanazawa/index.html#kair) at [Berkeley AI Research (BAIR)](https://bair.berkeley.edu/) in October 2022 as a part of a research project ([paper](https://arxiv.org/abs/2302.04264)). It is currently developed by Berkeley students and community contributors.
|
||||
|
||||
We are committed to providing learning resources to help you understand the basics of (if you're just getting started), and keep up-to-date with (if you're a seasoned veteran) all things NeRF. As researchers, we know just how hard it is to get onboarded with this next-gen technology. So we're here to help with tutorials, documentation, and more!
|
||||
|
||||
Have feature requests? Want to add your brand-spankin'-new NeRF model? Have a new dataset? **We welcome [contributions](https://docs.nerf.studio/en/latest/reference/contributing.html)!** Please do not hesitate to reach out to the nerfstudio team with any questions via [Discord](https://discord.gg/uMbNqcraFc).
|
||||
|
||||
We hope nerfstudio enables you to build faster :hammer: learn together :books: and contribute to our NeRF community :sparkling_heart:.
|
||||
|
||||
## Sponsors
|
||||
Sponsors of this work includes [Luma AI](https://lumalabs.ai/) and the [BAIR commons](https://bcommons.berkeley.edu/home).
|
||||
|
||||
<p align="left">
|
||||
<a href="https://lumalabs.ai/">
|
||||
<!-- pypi-strip -->
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="docs/_static/imgs/luma_dark.png">
|
||||
<source media="(prefers-color-scheme: light)" srcset="docs/_static/imgs/luma_light.png">
|
||||
<!-- /pypi-strip -->
|
||||
<img alt="Luma AI" src="docs/_static/imgs/luma_light.png" width="300">
|
||||
<!-- pypi-strip -->
|
||||
</picture>
|
||||
<!-- /pypi-strip -->
|
||||
</a>
|
||||
<a href="https://bcommons.berkeley.edu/home">
|
||||
<!-- pypi-strip -->
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="docs/_static/imgs/bair_dark.png">
|
||||
<source media="(prefers-color-scheme: light)" srcset="docs/_static/imgs/bair_light.png">
|
||||
<!-- /pypi-strip -->
|
||||
<img alt="BAIR" src="docs/_static/imgs/bair_light.png" width="300">
|
||||
<!-- pypi-strip -->
|
||||
</picture>
|
||||
<!-- /pypi-strip -->
|
||||
</a>
|
||||
</p>
|
||||
|
||||
|
||||
# Quickstart
|
||||
|
||||
The quickstart will help you get started with the default vanilla NeRF trained on the classic Blender Lego scene.
|
||||
For more complex changes (e.g., running with your own data/setting up a new NeRF graph), please refer to our [references](#learn-more).
|
||||
|
||||
## 1. Installation: Setup the environment
|
||||
|
||||
### Prerequisites
|
||||
|
||||
You must have an NVIDIA video card with CUDA installed on the system. This library has been tested with version 11.8 of CUDA. You can find more information about installing CUDA [here](https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html)
|
||||
|
||||
### Create environment
|
||||
|
||||
Nerfstudio requires `python >= 3.8`. We recommend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html) before proceeding.
|
||||
|
||||
```bash
|
||||
conda create --name nerfstudio -y python=3.8
|
||||
conda activate nerfstudio
|
||||
pip install --upgrade pip
|
||||
git clone https://github.com/MrKarimiD/nerfstudio-HDR.git
|
||||
conda activate nerfstudio-HDR
|
||||
export PYTHONPATH=$pwd:$PYTHONPATH
|
||||
pip install scikit-surgerycore pydub skylibs piq OpenEXR Imath equilib
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
**Select the branch for your method:**
|
||||
|
||||
Install PyTorch with CUDA (this repo has been tested with CUDA 11.7 and CUDA 11.8) and [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn).
|
||||
`cuda-toolkit` is required for building `tiny-cuda-nn`.
|
||||
- **HDR-Nerfacto** — check out the `hdr-nerfacto` branch:
|
||||
```
|
||||
git checkout -t remotes/origin/hdr-nerfacto
|
||||
```
|
||||
- **Pandora** — use the default branch (no checkout needed).
|
||||
|
||||
For CUDA 11.7:
|
||||
Then finish the install:
|
||||
|
||||
```bash
|
||||
pip install torch==2.0.1+cu117 torchvision==0.15.2+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
|
||||
|
||||
conda install -c "nvidia/label/cuda-11.7.1" cuda-toolkit
|
||||
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
|
||||
```
|
||||
|
||||
For CUDA 11.8:
|
||||
|
||||
```bash
|
||||
pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
|
||||
|
||||
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
|
||||
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
|
||||
```
|
||||
|
||||
See [Dependencies](https://github.com/nerfstudio-project/nerfstudio/blob/main/docs/quickstart/installation.md#dependencies)
|
||||
in the Installation documentation for more.
|
||||
|
||||
### Installing nerfstudio
|
||||
|
||||
Easy option:
|
||||
|
||||
```bash
|
||||
pip install nerfstudio
|
||||
```
|
||||
|
||||
**OR** if you want the latest and greatest:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/nerfstudio-project/nerfstudio.git
|
||||
cd nerfstudio
|
||||
pip install --upgrade pip setuptools
|
||||
pip install -e .
|
||||
ns-install-cli
|
||||
|
||||
pip install -U git+https://github.com/luca-medeiros/lang-segment-anything.git
|
||||
```
|
||||
|
||||
**OR** if you want to skip all installation steps and directly start using nerfstudio, use the docker image:
|
||||
|
||||
See [Installation](https://github.com/nerfstudio-project/nerfstudio/blob/main/docs/quickstart/installation.md) - **Use docker image**.
|
||||
|
||||
## 2. Training your first model!
|
||||
|
||||
The following will train a _nerfacto_ model, our recommended model for real world scenes.
|
||||
|
||||
```bash
|
||||
# Download some test data:
|
||||
ns-download-data nerfstudio --capture-name=poster
|
||||
# Train model
|
||||
ns-train nerfacto --data data/nerfstudio/poster
|
||||
If `lang-sam` is not installed, do the following:
|
||||
```
|
||||
cd ../lang-segment-anything
|
||||
pip install .
|
||||
cd ../nerfstudio-HDR
|
||||
```
|
||||
|
||||
If everything works, you should see training progress like the following:
|
||||
|
||||
<p align="center">
|
||||
<img width="800" alt="image" src="https://user-images.githubusercontent.com/3310961/202766069-cadfd34f-8833-4156-88b7-ad406d688fc0.png">
|
||||
</p>
|
||||
|
||||
Navigating to the link at the end of the terminal will load the webviewer. If you are running on a remote machine, you will need to port forward the websocket port (defaults to 7007).
|
||||
|
||||
<p align="center">
|
||||
<img width="800" alt="image" src="https://user-images.githubusercontent.com/3310961/202766653-586a0daa-466b-4140-a136-6b02f2ce2c54.png">
|
||||
</p>
|
||||
|
||||
### Resume from checkpoint / visualize existing run
|
||||
|
||||
It is possible to load a pretrained model by running
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --data data/nerfstudio/poster --load-dir {outputs/.../nerfstudio_models}
|
||||
If there is an issue with `equilib`, do the following:
|
||||
```
|
||||
pip uninstall equilib
|
||||
cd ../equilib
|
||||
python setup.py develop
|
||||
cd ../nerfstudio-HDR
|
||||
```
|
||||
|
||||
## Visualize existing run
|
||||
## 1. Data acquisition & pre-processing
|
||||
|
||||
Given a pretrained model checkpoint, you can start the viewer by running
|
||||
1. Setup the apparatus (stick with two Ricoh cameras)
|
||||
> Cameras:
|
||||
>
|
||||
> **Camera 1**: left, well-exposed, serial: YN14100695 \
|
||||
> **Camera 3**: right, fast-exposed (under-exposed), serial: YN14111000
|
||||
|
||||
```bash
|
||||
ns-viewer --load-config {outputs/.../config.yml}
|
||||
```
|
||||
- To connect the other camera:
|
||||
- Turn wifi off
|
||||
- Close the app
|
||||
- Turn wifi on
|
||||
- Open the app
|
||||
- Connect the other camera
|
||||
- Camera setup:
|
||||
- Install Ricoh Theta app
|
||||
- Connect the camera to the app
|
||||
- Change camera parameters: select manual (bottom right)
|
||||
- First time: turn on "CT Settings" in Shooting settings
|
||||
- Apreture: 2.1
|
||||
- Shutter speed: determined below
|
||||
- ISO: 800
|
||||
- WB: 3500
|
||||
- Make sure the cameras point up to avoid having lights in the seam.
|
||||
- Make sure to film up and down to get good coverage
|
||||
|
||||
## 3. Exporting Results
|
||||
2. First capture with well-exposed cameras.
|
||||
- Change shutter speed to get a well-exposed image. **Note the shutter speed.**
|
||||
- Start the video on one camera. Connect to the other camera following the steps above and start the video on the second camera.
|
||||
- Clap at the beginning and at the end to use for synchronization.
|
||||
- Go around the scene with the two cameras well-exposed.
|
||||
|
||||
Once you have a NeRF model you can either render out a video or export a point cloud.
|
||||
3. Second capture with Camera 1 (left) well-exposed and Camera 2 (right) fast-exposed. Change shutter speed of Camera 2 to 1/25000 (fastest exposure) and repeat the steps above
|
||||
|
||||
### Render Video
|
||||
If calculating **METRICS**: Capture GT HDR brakets
|
||||
- Go to settings and change the camera mode shooting method to multiple brackets
|
||||
- Set the camera parameters:
|
||||
- Apreture: 2.1
|
||||
- Shutter speed: Calculate the 11 shutter speeds and select the closest ones on the app in order from smallest to hightes
|
||||
```
|
||||
python lantern_scripts/calculate_exposures.py --min 0.00004 --max 0.004
|
||||
```
|
||||
- ISO: 800
|
||||
- WB: 3500
|
||||
- Take 10 different locations
|
||||
|
||||
First we must create a path for the camera to follow. This can be done in the viewer under the "RENDER" tab. Orient your 3D view to the location where you wish the video to start, then press "ADD CAMERA". This will set the first camera key frame. Continue to new viewpoints adding additional cameras to create the camera path. We provide other parameters to further refine your camera path. Once satisfied, press "RENDER" which will display a modal that contains the command needed to render the video. Kill the training job (or create a new terminal if you have lots of compute) and run the command to generate the video.
|
||||
4. Import the files on computer. If on a mac, use Ricoh Theta File Transfer for Mac
|
||||
|
||||
Other video export options are available, learn more by running
|
||||
5. Get the equirectangular videos by processing them with the Ricoh Theta computer app.
|
||||
- Drag and drop the video file into the app window.
|
||||
- Make sure both boxes are unchecked.
|
||||
- Name files as fallows
|
||||
- Camera 1:
|
||||
- First capture: left_sfm
|
||||
- Second capture: left_e1
|
||||
- Camera 2 (fast exposed):
|
||||
- First capture: right_sfm
|
||||
- Second capture: right_e2
|
||||
|
||||
```bash
|
||||
ns-render --help
|
||||
```
|
||||
If calculating **METRICS**: Group GT brackets under GT folder like so
|
||||
GT/
|
||||
GT1/
|
||||
001.jpg
|
||||
002.jpg
|
||||
...
|
||||
GT2/
|
||||
...
|
||||
...
|
||||
|
||||
### Generate Point Cloud
|
||||
6. Transfer files on lab machine.
|
||||
|
||||
While NeRF models are not designed to generate point clouds, it is still possible. Navigate to the "EXPORT" tab in the 3D viewer and select "POINT CLOUD". If the crop option is selected, everything in the yellow square will be exported into a point cloud. Modify the settings as desired then run the command at the bottom of the panel in your command line.
|
||||
```
|
||||
scp -r /path/to/source/file /path/to/destination/
|
||||
```
|
||||
|
||||
Alternatively you can use the CLI without the viewer. Learn about the export options by running
|
||||
7. Process the videos for OpenSFM. Input the right shutter speed noted above.
|
||||
|
||||
```bash
|
||||
ns-export pointcloud --help
|
||||
```
|
||||
If not calculating metrics:
|
||||
```
|
||||
python lantern_scripts/process_videos_for_sfm.py --input_dir /mnt/data/scene/ --shutter_speed 0.004
|
||||
```
|
||||
|
||||
## 4. Using Custom Data
|
||||
If calculating **METRICS**:
|
||||
```
|
||||
python lantern_scripts/process_videos_for_sfm.py --input_dir /mnt/data/scene/ --shutter_speed 0.004 --gt
|
||||
```
|
||||
- Resize GT_exr and GT_jpg images to 3840x1920 with GNU Image Manipulation Program
|
||||
- Go to Image > Scale Image
|
||||
- Transfer those back on lab machine in th GT folder of the scene naming them GT_jpg_small and GT_exr_small
|
||||
- Add GT_jpg images into sfm/images/ with their appropriate mask (checkout stick_masks folder)
|
||||
|
||||
Using an existing dataset is great, but likely you want to use your own data! We support various methods for using your own data. Before it can be used in nerfstudio, the camera location and orientations must be determined and then converted into our format using `ns-process-data`. We rely on external tools for this, instructions and information can be found in the documentation.
|
||||
|
||||
| Data | Capture Device | Requirements | `ns-process-data` Speed |
|
||||
| ---------------------------------------------------------------------------------------------------- | -------------- | ----------------------------------------------------------------- | ----------------------- |
|
||||
| 📷 [Images](https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html#images-and-video) | Any | [COLMAP](https://colmap.github.io/install.html) | 🐢 |
|
||||
| 📹 [Video](https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html#images-and-video) | Any | [COLMAP](https://colmap.github.io/install.html) | 🐢 |
|
||||
| 🌎 [360 Data](https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html#360_data) | Any | [COLMAP](https://colmap.github.io/install.html) | 🐢 |
|
||||
| 📱 [Polycam](https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html#polycam-capture) | IOS with LiDAR | [Polycam App](https://poly.cam/) | 🐇 |
|
||||
| 📱 [KIRI Engine](https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html#kiri-capture) | IOS or Android | [KIRI Engine App](https://www.kiriengine.com/) | 🐇 |
|
||||
| 📱 [Record3D](https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html#record3d-capture) | IOS with LiDAR | [Record3D app](https://record3d.app/) | 🐇 |
|
||||
| 🖥 [Metashape](https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html#metashape) | Any | [Metashape](https://www.agisoft.com/) | 🐇 |
|
||||
| 🖥 [RealityCapture](https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html#realitycapture) | Any | [RealityCapture](https://www.capturingreality.com/realitycapture) | 🐇 |
|
||||
| 🛠 [Custom](https://docs.nerf.studio/en/latest/quickstart/data_conventions.html) | Any | Camera Poses | 🐇 |
|
||||
|
||||
## 5. Advanced Options
|
||||
|
||||
### Training models other than nerfacto
|
||||
|
||||
We provide other models than nerfacto, for example if you want to train the original nerf model, use the following command
|
||||
|
||||
```bash
|
||||
ns-train vanilla-nerf --data DATA_PATH
|
||||
```
|
||||
|
||||
For a full list of included models run `ns-train --help`.
|
||||
|
||||
### Modify Configuration
|
||||
|
||||
Each model contains many parameters that can be changed, too many to list here. Use the `--help` command to see the full list of configuration options.
|
||||
|
||||
```bash
|
||||
ns-train nerfacto --help
|
||||
```
|
||||
|
||||
### Tensorboard / WandB / Viewer
|
||||
|
||||
We support three different methods to track training progress, using the viewer, [tensorboard](https://www.tensorflow.org/tensorboard), and [Weights and Biases](https://wandb.ai/site). You can specify which visualizer to use by appending `--vis {viewer, tensorboard, wandb, viewer+wandb, viewer+tensorboard}` to the training command. Simultaneously utilizing the viewer alongside wandb or tensorboard may cause stuttering issues during evaluation steps. The viewer only works for methods that are fast (ie. nerfacto, instant-ngp), for slower methods like NeRF, use the other loggers.
|
||||
|
||||
# Learn More
|
||||
|
||||
And that's it for getting started with the basics of nerfstudio.
|
||||
|
||||
If you're interested in learning more on how to create your own pipelines, develop with the viewer, run benchmarks, and more, please check out some of the quicklinks below or visit our [documentation](https://docs.nerf.studio/en/latest/) directly.
|
||||
|
||||
| Section | Description |
|
||||
| -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
|
||||
| [Documentation](https://docs.nerf.studio/en/latest/) | Full API documentation and tutorials |
|
||||
| [Viewer](https://viewer.nerf.studio/) | Home page for our web viewer |
|
||||
| 🎒 **Educational** |
|
||||
| [Model Descriptions](https://docs.nerf.studio/en/latest/nerfology/methods/index.html) | Description of all the models supported by nerfstudio and explanations of component parts. |
|
||||
| [Component Descriptions](https://docs.nerf.studio/en/latest/nerfology/model_components/index.html) | Interactive notebooks that explain notable/commonly used modules in various models. |
|
||||
| 🏃 **Tutorials** |
|
||||
| [Getting Started](https://docs.nerf.studio/en/latest/quickstart/installation.html) | A more in-depth guide on how to get started with nerfstudio from installation to contributing. |
|
||||
| [Using the Viewer](https://docs.nerf.studio/en/latest/quickstart/viewer_quickstart.html) | A quick demo video on how to navigate the viewer. |
|
||||
| [Using Record3D](https://www.youtube.com/watch?v=XwKq7qDQCQk) | Demo video on how to run nerfstudio without using COLMAP. |
|
||||
| 💻 **For Developers** |
|
||||
| [Creating pipelines](https://docs.nerf.studio/en/latest/developer_guides/pipelines/index.html) | Learn how to easily build new neural rendering pipelines by using and/or implementing new modules. |
|
||||
| [Creating datasets](https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html) | Have a new dataset? Learn how to run it with nerfstudio. |
|
||||
| [Contributing](https://docs.nerf.studio/en/latest/reference/contributing.html) | Walk-through for how you can start contributing now. |
|
||||
| 💖 **Community** |
|
||||
| [Discord](https://discord.gg/uMbNqcraFc) | Join our community to discuss more. We would love to hear from you! |
|
||||
| [Twitter](https://twitter.com/nerfstudioteam) | Follow us on Twitter @nerfstudioteam to see cool updates and announcements |
|
||||
|
||||
# Supported Features
|
||||
|
||||
We provide the following support structures to make life easier for getting started with NeRFs.
|
||||
|
||||
**If you are looking for a feature that is not currently supported, please do not hesitate to contact the Nerfstudio Team on [Discord](https://discord.gg/uMbNqcraFc)!**
|
||||
|
||||
- :mag_right: Web-based visualizer that allows you to:
|
||||
- Visualize training in real-time + interact with the scene
|
||||
- Create and render out scenes with custom camera trajectories
|
||||
- View different output types
|
||||
- And more!
|
||||
- :pencil2: Support for multiple logging interfaces (Tensorboard, Wandb), code profiling, and other built-in debugging tools
|
||||
- :chart_with_upwards_trend: Easy-to-use benchmarking scripts on the Blender dataset
|
||||
- :iphone: Full pipeline support (w/ Colmap, Polycam, or Record3D) for going from a video on your phone to a full 3D render.
|
||||
|
||||
# Built On
|
||||
|
||||
<a href="https://github.com/brentyi/tyro">
|
||||
<!-- pypi-strip -->
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://brentyi.github.io/tyro/_static/logo-dark.svg" />
|
||||
<!-- /pypi-strip -->
|
||||
<img alt="tyro logo" src="https://brentyi.github.io/tyro/_static/logo-light.svg" width="150px" />
|
||||
<!-- pypi-strip -->
|
||||
</picture>
|
||||
<!-- /pypi-strip -->
|
||||
</a>
|
||||
|
||||
- Easy-to-use config system
|
||||
- Developed by [Brent Yi](https://brentyi.com/)
|
||||
|
||||
<a href="https://github.com/KAIR-BAIR/nerfacc">
|
||||
<!-- pypi-strip -->
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://user-images.githubusercontent.com/3310961/199083722-881a2372-62c1-4255-8521-31a95a721851.png" />
|
||||
<!-- /pypi-strip -->
|
||||
<img alt="tyro logo" src="https://user-images.githubusercontent.com/3310961/199084143-0d63eb40-3f35-48d2-a9d5-78d1d60b7d66.png" width="250px" />
|
||||
<!-- pypi-strip -->
|
||||
</picture>
|
||||
<!-- /pypi-strip -->
|
||||
</a>
|
||||
|
||||
- Library for accelerating NeRF renders
|
||||
- Developed by [Ruilong Li](https://www.liruilong.cn/)
|
||||
|
||||
# Citation
|
||||
|
||||
You can find a paper writeup of the framework on [arXiv](https://arxiv.org/abs/2302.04264).
|
||||
|
||||
If you use this library or find the documentation useful for your research, please consider citing:
|
||||
## 2. Process data using OpenSFM
|
||||
|
||||
```
|
||||
@inproceedings{nerfstudio,
|
||||
title = {Nerfstudio: A Modular Framework for Neural Radiance Field Development},
|
||||
author = {
|
||||
Tancik, Matthew and Weber, Ethan and Ng, Evonne and Li, Ruilong and Yi, Brent
|
||||
and Kerr, Justin and Wang, Terrance and Kristoffersen, Alexander and Austin,
|
||||
Jake and Salahi, Kamyar and Ahuja, Abhik and McAllister, David and Kanazawa,
|
||||
Angjoo
|
||||
},
|
||||
year = 2023,
|
||||
booktitle = {ACM SIGGRAPH 2023 Conference Proceedings},
|
||||
series = {SIGGRAPH '23}
|
||||
}
|
||||
bin/opensfm_run_all /mnt/data/scene/sfm/
|
||||
```
|
||||
|
||||
# Contributors
|
||||
To view results:
|
||||
```
|
||||
python3 viewer/server.py -d /mnt/data/scene/sfm/
|
||||
```
|
||||
Check that the camera positions make sense and that the number of clusters is low.
|
||||
|
||||
<a href="https://github.com/nerfstudio-project/nerfstudio/graphs/contributors">
|
||||
<img src="https://contrib.rocks/image?repo=nerfstudio-project/nerfstudio" />
|
||||
</a>
|
||||
## 3. Process data for NeRF
|
||||
|
||||
```
|
||||
ns-process-data lantern-openSFM --data /mnt/data/scene/data/ --output-dir /mnt/data/scene/scene_ns/ --metadata /mnt/data/scene/sfm/reconstruction.json
|
||||
```
|
||||
|
||||
For processing existing dataset that used colmap for camera positions:
|
||||
```
|
||||
ns-process-data images --data /mnt/data/garden/images --output-dir /mnt/data/garden_processed --skip-colmap --skip-image-processing --colmap-model-path /mnt/data/garden/sparse/0
|
||||
```
|
||||
|
||||
## 4. Train NeRF
|
||||
|
||||
Pick the method you set up in Step 0.
|
||||
|
||||
### Option A — PanDORA
|
||||
|
||||
1. Run step 1 of PanDORA.
|
||||
|
||||
```
|
||||
ns-train PanDORA --data /mnt/data/scene/scene_ns/ --viewer.websocket-port 8008 --pipeline.datamanager.train-num-images-to-sample-from 1800 --pipeline.model.lantern_steps 1 --pipeline.datamanager.pixel-sampler.lantern_steps 1 --max-num-iterations 60000
|
||||
```
|
||||
|
||||
1.1. For better alignment between the well and the fast exposed images, run the following command.
|
||||
```
|
||||
python lantern_scripts/align_fast_to_well.py --input_dir /mnt/data/scene/scene_ns/ --config outputs/scene_ns/PanDORA/2024-08-07_143013/config.yml
|
||||
```
|
||||
|
||||
2. Run step 2 of PanDORA.
|
||||
|
||||
```
|
||||
ns-train PanDORA --data /mnt/data/scene/scene_ns/ --viewer.websocket-port 8008 --pipeline.datamanager.train-num-images-to-sample-from 1800 --pipeline.model.lantern_steps 2 --pipeline.datamanager.pixel-sampler.lantern_steps 2 --load-dir /mnt/workspace/lantern/nerfstudio-HDR/outputs/scene_ns/PanDORA/2024-06-10_155559/nerfstudio_models --pipeline.model.apply_mu_law False --max-num-iterations 120000
|
||||
```
|
||||
|
||||
### Option B — HDR-Nerfacto (`hdr-nerfacto`)
|
||||
|
||||
1. Run hdr-nerfacto.
|
||||
|
||||
```
|
||||
ns-train hdr-nerfacto --data /mnt/data/nerfstudio_ds/real_data/amphitheatre_1112_2/amphitheatre_ns/ --viewer.websocket-port 8008 --pipeline.datamanager.train-num-images-to-sample-from 1800 --max-num-iterations 120000 --viewer.websocket-port 8008
|
||||
```
|
||||
|
||||
## 5. View results
|
||||
|
||||
To use the interface:
|
||||
```
|
||||
ns-viewer --load-config outputs/scene_ns/PanDORA/2024-06-06_191338/config.yml --viewer.websocket-port 8008
|
||||
```
|
||||
- For the right side up:
|
||||
- Go to SCENE sub-menu.
|
||||
- Clic on RESET UP DIRECTION.
|
||||
- To render a video with the interface command:
|
||||
- Go to RENDER sub-menu.
|
||||
- Clic on ADD CAMERA to manualy set camera view points.
|
||||
- Clic on RENDER to get the command.
|
||||
```
|
||||
ns-render camera-path --load-config outputs/scene_ns/PanDORA/2024-06-06_191338/config.yml --camera-path-filename /mnt/data/scene/scene_ns/camera_paths/2024-06-06_191338.json --output-path /mnt/data/scene/renders/2024-06-06_191338.mp4
|
||||
```
|
||||
- To render a video fast-exposed: add --rendered-output-names rgb_fast
|
||||
```
|
||||
ns-render camera-path --load-config outputs/scene_ns/PanDORA/2024-06-06_191338/config.yml --camera-path-filename /mnt/data/scene/scene_ns/camera_paths/2024-06-06_191338.json --output-path /mnt/data/scene/renders/2024-06-06_191338.mp4 --rendered-output-names rgb_fast
|
||||
```
|
||||
|
||||
To get evaluation images (well-exposed):
|
||||
```
|
||||
ns-eval --load-config=outputs/scene_ns/PanDORA/2024-06-17_152246/config.yml --output-path=output.json --render_output_path=/mnt/data/scene/eval
|
||||
```
|
||||
|
||||
- To get fast-exposed evaluation images, change these two lines in nerfstudio/lantern/model.py:
|
||||
```
|
||||
line 238: if batch["exposure"] != 1.0: # change to == for testing fast-exposed
|
||||
line 242: return None, None # comment for testing fast-exposed
|
||||
```
|
||||
|
||||
- To get evaluation images and tensorboard:
|
||||
```
|
||||
ns-eval --load-config=outputs/scene_ns/PanDORA/2024-06-20_142413/config.yml --output-path=output.json --render_output_path=/mnt/data/scene/eval --vis viewer+tensorboard
|
||||
```
|
||||
- To load tensorboard results:
|
||||
```
|
||||
tensorboard --logdir=/path/to/file
|
||||
```
|
||||
- Connect to http://localhost:6006/
|
||||
|
||||
## 6. Calculate metrics
|
||||
|
||||
The single-command convenience script differs by method:
|
||||
|
||||
- **PanDORA**:
|
||||
```
|
||||
python lantern_scripts/calculate_metrics.py --input_dir /mnt/data/coffee_room2/ --checkpoint /mnt/workspace/lantern/nerfstudio-HDR/outputs/coffee_room2_ns/PanDORA/2025-01-22_194346/ --metric_name unaligned
|
||||
```
|
||||
- **HDR-Nerfacto (`hdr-nerfacto`)**:
|
||||
```
|
||||
python lantern_scripts/calculate_metrics_hdrnerfacto.py --input_dir /mnt/data/coffee_room2/ --checkpoint /mnt/workspace/lantern/nerfstudio-HDR/outputs/coffee_room2_ns/PanDORA/2025-01-22_194346/ --metric_name hdr_nerfacto
|
||||
```
|
||||
|
||||
then point #8
|
||||
|
||||
**or**, run the steps manually (identical for both methods):
|
||||
|
||||
1. Get ground truth transformations:
|
||||
```
|
||||
ns-process-data lantern-GT-HDR --data /mnt/data/coffee_room2/coffee_room2_ns/ --output-dir /mnt/data/coffee_room2/metrics --metadata /mnt/data/coffee_room2/sfm/reconstruction.json --checkpoint /mnt/workspace/lantern/nerfstudio-HDR/outputs/coffee_room2_ns/PanDORA/2025-01-22_194346/
|
||||
```
|
||||
|
||||
2. Render panos at ground truth positions for well and fast exposed:
|
||||
```
|
||||
ns-render camera-path --load-config /mnt/workspace/lantern/nerfstudio-HDR/outputs/coffee_room2_ns/PanDORA/2025-01-22_194346/config.yml --camera-path-filename /mnt/data/coffee_room2/metrics/GT_transforms.json --output-path /mnt/data/coffee_room2/metrics/renders/unaligned --output-format images
|
||||
```
|
||||
```
|
||||
ns-render camera-path --load-config /mnt/workspace/lantern/nerfstudio-HDR/outputs/coffee_room2_ns/PanDORA/2025-01-22_194346/config.yml --camera-path-filename /mnt/data/coffee_room2/metrics/GT_transforms.json --output-path /mnt/data/coffee_room2/metrics/renders/unaligned_fast --output-format images --rendered_output_names rgb_fast
|
||||
```
|
||||
|
||||
3. Combine both well and fast exposed panos:
|
||||
```
|
||||
python lantern_scripts/combine.py --well_dir /mnt/data/coffee_room2/metrics/renders/unaligned/ --fast_dir /mnt/data/coffee_room2/metrics/renders/unaligned_fast/ --out_dir /mnt/data/coffee_room2/metrics/renders/unaligned_lin/ --experiment_location /mnt/data/coffee_room2/data/ --do_linearization
|
||||
```
|
||||
|
||||
4. Make sure the panos from renders have the same names as the GT panos.
|
||||
|
||||
5. Calculate PSNR, SSIM and LPIPS:
|
||||
```
|
||||
python lantern_scripts/ldr_res.py --gt_dir /mnt/data/coffee_room2/GT/GT_exr/ --data_dir /mnt/data/coffee_room2/metrics/renders/unaligned_lin/
|
||||
```
|
||||
|
||||
6. Generate GT and results renders with blender:
|
||||
```
|
||||
blender --background lantern_scripts/scene_new.blend -P lantern_scripts/hdr_blender.py -- /mnt/data/coffee_room2/GT/GT_exr/ /mnt/data/coffee_room2/GT/GT_exr_renders/
|
||||
```
|
||||
```
|
||||
blender --background lantern_scripts/scene_new.blend -P lantern_scripts/hdr_blender.py -- /mnt/data/coffee_room2/metrics/renders/unaligned_lin/ /mnt/data/coffee_room2/metrics/renders/unaligned_lin_renders/
|
||||
```
|
||||
|
||||
7. Calculate si-RMSE, RMSE, RGB ang. and PSNR (LDR r.). Need to change values in code:
|
||||
```
|
||||
python lantern_scripts/res_table_1.py
|
||||
```
|
||||
|
||||
8. Convert exr panos to hdr:
|
||||
```
|
||||
convert GT1.exr GT1.hdr
|
||||
```
|
||||
|
||||
9. Calulate PU-PSNR, HDR-VDP and PU-SSIM following the steps in the following link:
|
||||
https://github.com/darthgera123/PanoHDR-NeRF/tree/main/LANet/metrics
|
||||
```
|
||||
cd hdrvdp-3.0.6
|
||||
```
|
||||
```
|
||||
matlab -batch "metric('../../data/GT_meeting_room/', '../../data/results_meeting_room/')"
|
||||
```
|
||||
|
||||
```
|
||||
cd ../pu21/matlab/examples
|
||||
```
|
||||
```
|
||||
matlab -batch "pupsnr('../../../../data/GT_meeting_room/', '../../../../data/results_meeting_room/')"
|
||||
```
|
||||
@@ -68,6 +68,8 @@ class AlignImages:
|
||||
except ContourTooSmallException as e:
|
||||
continue
|
||||
self.matched_well_contours = [None] * len(self.fast_contours)
|
||||
self.well_points_combined = []
|
||||
self.fast_points_combined = []
|
||||
self.well_image = Image(well_image)
|
||||
self.well_images = []
|
||||
self.well_thresholds = (230, 235, 240, 245, 250)
|
||||
@@ -165,6 +167,9 @@ class AlignImages:
|
||||
well_matched_points, fast_matched_points = self.match_points(well_contour, fast_contour)
|
||||
well_points_combined.extend(well_matched_points)
|
||||
fast_points_combined.extend(fast_matched_points)
|
||||
# Expose the SIFT/BFMatcher point correspondences (pre-RANSAC) for visualization.
|
||||
self.well_points_combined = well_points_combined
|
||||
self.fast_points_combined = fast_points_combined
|
||||
if len(well_points_combined) >= 4:
|
||||
well_points = np.float32(well_points_combined)
|
||||
fast_points = np.float32(fast_points_combined)
|
||||
|
||||
@@ -0,0 +1,332 @@
|
||||
"""Align images from one folder to a reference folder using the same
|
||||
contour-matching + homography computation as align_fast_to_well.py, but
|
||||
WITHOUT needing a model to render the reference images.
|
||||
|
||||
It loads two folders of already-existing images:
|
||||
--well_dir : reference images to align *to* (previously the rendered ones)
|
||||
--fast_dir : images to be warped/aligned
|
||||
and, for each matched pair, computes the homography and saves the warped image.
|
||||
"""
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
import os
|
||||
import sys
|
||||
import argparse
|
||||
|
||||
# Reuse the alignment logic from the original script.
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from align_fast_to_well import AlignImages, save_transformation
|
||||
|
||||
|
||||
IMAGE_EXTS = ('.png', '.jpg', '.jpeg')
|
||||
|
||||
|
||||
def list_images(directory, prefix=""):
|
||||
files = [f for f in os.listdir(directory)
|
||||
if f.lower().endswith(IMAGE_EXTS) and f.startswith(prefix)]
|
||||
return sorted(files)
|
||||
|
||||
|
||||
def pair_images(well_files, fast_files, match_by):
|
||||
"""Return a list of (well_filename, fast_filename) pairs."""
|
||||
if match_by == 'name':
|
||||
well_by_stem = {os.path.splitext(f)[0]: f for f in well_files}
|
||||
pairs = []
|
||||
for fast_f in fast_files:
|
||||
stem = os.path.splitext(fast_f)[0]
|
||||
if stem in well_by_stem:
|
||||
pairs.append((well_by_stem[stem], fast_f))
|
||||
else:
|
||||
print(f"No matching well image for: {fast_f}")
|
||||
return pairs
|
||||
else: # 'order'
|
||||
if len(well_files) != len(fast_files):
|
||||
print(f"Warning: folder counts differ "
|
||||
f"(well={len(well_files)}, fast={len(fast_files)}); "
|
||||
f"pairing the first {min(len(well_files), len(fast_files))} by sorted order.")
|
||||
return list(zip(well_files, fast_files))
|
||||
|
||||
|
||||
def save_checkerboard(a, b, path, tiles=8):
|
||||
"""Alternating tiles of two RGB images to expose misaligned edges."""
|
||||
h, w = a.shape[:2]
|
||||
th, tw = h // tiles, w // tiles
|
||||
out = a.copy()
|
||||
for i in range(tiles):
|
||||
for j in range(tiles):
|
||||
if (i + j) % 2 == 1:
|
||||
y0, y1 = i * th, (h if i == tiles - 1 else (i + 1) * th)
|
||||
x0, x1 = j * tw, (w if j == tiles - 1 else (j + 1) * tw)
|
||||
out[y0:y1, x0:x1] = b[y0:y1, x0:x1]
|
||||
cv2.imwrite(path, cv2.cvtColor(out, cv2.COLOR_RGB2BGR))
|
||||
|
||||
|
||||
def save_matched_contours(align_images, well_path, fast_path):
|
||||
"""Save the matched contours on the well and fast images as two separate
|
||||
images. Each matched pair uses the same color and index in both, so they
|
||||
can be cross-referenced."""
|
||||
well = align_images.well_image.image.copy()
|
||||
fast = align_images.fast_image.image.copy()
|
||||
|
||||
pairs = list(zip(align_images.matched_well_contours, align_images.fast_contours))
|
||||
|
||||
def draw(canvas, contour, idx, color):
|
||||
cv2.drawContours(canvas, [contour.contour], -1, color, 2)
|
||||
cx, cy = int(contour.centroid[0]), int(contour.centroid[1])
|
||||
cv2.putText(canvas, str(idx), (cx + 5, cy - 5),
|
||||
cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 1)
|
||||
|
||||
for idx, (well_c, fast_c) in enumerate(pairs):
|
||||
if well_c is None or fast_c is None:
|
||||
continue
|
||||
# Deterministic distinct color per matched pair (same in both images).
|
||||
color_map = cv2.applyColorMap(
|
||||
np.array([[int(255 * idx / max(len(pairs), 1))]], dtype=np.uint8),
|
||||
cv2.COLORMAP_HSV)[0, 0]
|
||||
color = (int(color_map[0]), int(color_map[1]), int(color_map[2]))
|
||||
draw(well, well_c, idx, color)
|
||||
draw(fast, fast_c, idx, color)
|
||||
|
||||
cv2.imwrite(well_path, cv2.cvtColor(well, cv2.COLOR_RGB2BGR))
|
||||
cv2.imwrite(fast_path, cv2.cvtColor(fast, cv2.COLOR_RGB2BGR))
|
||||
|
||||
|
||||
def _rotate_ccw_point(x, y, width):
|
||||
"""Map a point (x, y) under a 90 deg counter-clockwise image rotation,
|
||||
where `width` is the ORIGINAL image width."""
|
||||
return y, (width - 1 - x)
|
||||
|
||||
|
||||
def _rotate_ccw_contour(contour, width):
|
||||
"""Apply the 90 deg CCW point mapping to an (N,1,2) contour array."""
|
||||
pts = contour.reshape(-1, 2)
|
||||
rotated = np.empty_like(pts)
|
||||
rotated[:, 0] = pts[:, 1]
|
||||
rotated[:, 1] = width - 1 - pts[:, 0]
|
||||
return rotated.reshape(-1, 1, 2)
|
||||
|
||||
|
||||
def save_matched_contours_panel(align_images, path):
|
||||
"""Side-by-side panel with each image rotated 90 deg counter-clockwise,
|
||||
each matched contour pair drawn in the same color and connected by a line
|
||||
across the panel."""
|
||||
well = cv2.rotate(align_images.well_image.image.copy(), cv2.ROTATE_90_COUNTERCLOCKWISE)
|
||||
fast = cv2.rotate(align_images.fast_image.image.copy(), cv2.ROTATE_90_COUNTERCLOCKWISE)
|
||||
# Original widths (needed for the coordinate rotation).
|
||||
w_w0 = align_images.well_image.image.shape[1]
|
||||
w_f0 = align_images.fast_image.image.shape[1]
|
||||
# Rotated image dimensions.
|
||||
h_w, w_w = well.shape[:2]
|
||||
h_f, w_f = fast.shape[:2]
|
||||
h = max(h_w, h_f)
|
||||
canvas = np.zeros((h, w_w + w_f, 3), dtype=np.uint8)
|
||||
canvas[:h_w, :w_w] = well
|
||||
canvas[:h_f, w_w:w_w + w_f] = fast
|
||||
|
||||
pairs = list(zip(align_images.matched_well_contours, align_images.fast_contours))
|
||||
for idx, (well_c, fast_c) in enumerate(pairs):
|
||||
if well_c is None or fast_c is None:
|
||||
continue
|
||||
# Deterministic distinct color per matched pair.
|
||||
color_map = cv2.applyColorMap(
|
||||
np.array([[int(255 * idx / max(len(pairs), 1))]], dtype=np.uint8),
|
||||
cv2.COLORMAP_HSV)[0, 0]
|
||||
color = (int(color_map[0]), int(color_map[1]), int(color_map[2]))
|
||||
|
||||
well_contour_rot = _rotate_ccw_contour(well_c.contour, w_w0)
|
||||
fast_contour_rot = _rotate_ccw_contour(fast_c.contour, w_f0)
|
||||
cv2.drawContours(canvas, [well_contour_rot], -1, color, 2)
|
||||
cv2.drawContours(canvas, [fast_contour_rot + np.array([[w_w, 0]])], -1, color, 2)
|
||||
|
||||
wx, wy = _rotate_ccw_point(int(well_c.centroid[0]), int(well_c.centroid[1]), w_w0)
|
||||
fx, fy = _rotate_ccw_point(int(fast_c.centroid[0]), int(fast_c.centroid[1]), w_f0)
|
||||
fx += w_w
|
||||
cv2.line(canvas, (wx, wy), (fx, fy), color, 3)
|
||||
|
||||
cv2.imwrite(path, cv2.cvtColor(canvas, cv2.COLOR_RGB2BGR))
|
||||
|
||||
|
||||
def save_sift_matches_panel(align_images, path, layout="vertical", max_matches=40):
|
||||
"""Panel drawing SIFT/BFMatcher point correspondences (post-BFMatcher,
|
||||
pre-RANSAC) as connecting lines.
|
||||
layout='vertical' -> fast exposure on top, well exposure on bottom.
|
||||
layout='horizontal' -> fast exposure on left, well exposure on right.
|
||||
max_matches limits how many correspondences are drawn (evenly subsampled)
|
||||
so the visualization stays readable."""
|
||||
well = align_images.well_image.image.copy()
|
||||
fast = align_images.fast_image.image.copy()
|
||||
h_w, w_w = well.shape[:2]
|
||||
h_f, w_f = fast.shape[:2]
|
||||
|
||||
if layout == "horizontal":
|
||||
canvas = np.zeros((max(h_w, h_f), w_f + w_w, 3), dtype=np.uint8)
|
||||
canvas[:h_f, :w_f] = fast # fast exposure on left
|
||||
canvas[:h_w, w_f:w_f + w_w] = well # well exposure on right
|
||||
fast_offset = (0, 0)
|
||||
well_offset = (w_f, 0)
|
||||
else: # vertical
|
||||
canvas = np.zeros((h_f + h_w, max(w_w, w_f), 3), dtype=np.uint8)
|
||||
canvas[:h_f, :w_f] = fast # fast exposure on top
|
||||
canvas[h_f:h_f + h_w, :w_w] = well # well exposure on bottom
|
||||
fast_offset = (0, 0)
|
||||
well_offset = (0, h_f)
|
||||
|
||||
well_pts = align_images.well_points_combined
|
||||
fast_pts = align_images.fast_points_combined
|
||||
n = min(len(well_pts), len(fast_pts))
|
||||
|
||||
# Evenly subsample down to max_matches for readability.
|
||||
if max_matches and n > max_matches:
|
||||
indices = np.linspace(0, n - 1, max_matches).astype(int)
|
||||
else:
|
||||
indices = range(n)
|
||||
|
||||
for k, idx in enumerate(indices):
|
||||
wp = well_pts[idx]
|
||||
fp = fast_pts[idx]
|
||||
# Distinct color per drawn correspondence.
|
||||
color_map = cv2.applyColorMap(
|
||||
np.array([[int(255 * k / max(len(indices), 1))]], dtype=np.uint8),
|
||||
cv2.COLORMAP_HSV)[0, 0]
|
||||
color = (int(color_map[0]), int(color_map[1]), int(color_map[2]))
|
||||
fx, fy = int(fp[0]) + fast_offset[0], int(fp[1]) + fast_offset[1]
|
||||
wx, wy = int(wp[0]) + well_offset[0], int(wp[1]) + well_offset[1]
|
||||
cv2.circle(canvas, (fx, fy), 5, color, -1)
|
||||
cv2.circle(canvas, (wx, wy), 5, color, -1)
|
||||
cv2.line(canvas, (fx, fy), (wx, wy), color, 3)
|
||||
|
||||
cv2.imwrite(path, cv2.cvtColor(canvas, cv2.COLOR_RGB2BGR))
|
||||
|
||||
|
||||
def save_thresholded(align_images, output_dir, base):
|
||||
"""Save the binary threshold maps that contour detection operates on."""
|
||||
cv2.imwrite(os.path.join(output_dir, base + "_fast_threshold.png"),
|
||||
align_images.fast_image.thresholded)
|
||||
for img, thr in zip(align_images.well_images, align_images.well_thresholds):
|
||||
cv2.imwrite(os.path.join(output_dir, base + f"_well_threshold_{thr}.png"),
|
||||
img.thresholded)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
argparser = argparse.ArgumentParser()
|
||||
argparser.add_argument("--well_dir", type=str, required=True,
|
||||
help="Folder of reference images to align to.")
|
||||
argparser.add_argument("--fast_dir", type=str, required=True,
|
||||
help="Folder of images to be aligned (warped).")
|
||||
argparser.add_argument("--output_dir", type=str, required=True,
|
||||
help="Folder to write the aligned images into.")
|
||||
argparser.add_argument("--match_by", type=str, default="order",
|
||||
choices=["order", "name"],
|
||||
help="Pair images by sorted order (default) or by filename stem.")
|
||||
argparser.add_argument("--prefix", type=str, default="right",
|
||||
help="Only align fast_dir images whose filename starts with this prefix "
|
||||
"(use '' to process all images).")
|
||||
argparser.add_argument("--stride", type=int, default=1,
|
||||
help="Process only every Nth image pair (e.g. 5 = every 5th).")
|
||||
argparser.add_argument("--only", type=str, default=None,
|
||||
help="Process only this single fast_dir filename (its correct "
|
||||
"well partner is still resolved via the full pairing).")
|
||||
argparser.add_argument("--save_merged", action='store_true', default=False,
|
||||
help="Also save before/after overlay visualizations.")
|
||||
argparser.add_argument("--visualize_contours", action='store_true', default=False,
|
||||
help="Save detected/matched contour visualizations for debugging.")
|
||||
argparser.add_argument("--save_diagnostics", action='store_true', default=False,
|
||||
help="Save diagnostics: before/after checkerboard composites, "
|
||||
"thresholded binary maps, and contour visualizations.")
|
||||
argparser.add_argument("--sift_layout", type=str, default="vertical",
|
||||
choices=["vertical", "horizontal"],
|
||||
help="Layout for *_sift_matches.png: vertical (fast top / well bottom) "
|
||||
"or horizontal (fast left / well right).")
|
||||
argparser.add_argument("--sift_max_matches", type=int, default=40,
|
||||
help="Max number of SIFT correspondences drawn in *_sift_matches.png "
|
||||
"(evenly subsampled for readability; 0 = draw all).")
|
||||
args = argparser.parse_args()
|
||||
|
||||
os.makedirs(args.output_dir, exist_ok=True)
|
||||
json_path = os.path.join(args.output_dir, 'alignment_matrices.json')
|
||||
merged_dir = os.path.join(args.output_dir, 'merged')
|
||||
contour_visualizations_path = os.path.join(args.output_dir, 'contour_visualizations')
|
||||
diagnostics_dir = os.path.join(args.output_dir, 'diagnostics')
|
||||
if args.save_merged:
|
||||
os.makedirs(merged_dir, exist_ok=True)
|
||||
if args.save_diagnostics:
|
||||
os.makedirs(diagnostics_dir, exist_ok=True)
|
||||
|
||||
well_files = list_images(args.well_dir)
|
||||
fast_files = list_images(args.fast_dir, prefix=args.prefix)
|
||||
pairs = pair_images(well_files, fast_files, args.match_by)
|
||||
|
||||
if args.only is not None:
|
||||
pairs = [(w, f) for (w, f) in pairs if f == args.only]
|
||||
if not pairs:
|
||||
print(f"'{args.only}' not found among the paired fast images; nothing to do.")
|
||||
sys.exit(0)
|
||||
|
||||
if args.stride > 1:
|
||||
pairs = pairs[::args.stride]
|
||||
|
||||
print(f"Aligning {len(pairs)} image pairs...")
|
||||
for well_name, fast_name in pairs:
|
||||
well_path = os.path.join(args.well_dir, well_name)
|
||||
fast_path = os.path.join(args.fast_dir, fast_name)
|
||||
well_image = cv2.cvtColor(cv2.imread(well_path), cv2.COLOR_BGR2RGB)
|
||||
fast_image = cv2.cvtColor(cv2.imread(fast_path), cv2.COLOR_BGR2RGB)
|
||||
|
||||
align_images = None
|
||||
try:
|
||||
align_images = AlignImages(well_image, fast_image)
|
||||
warped_image, matrix = align_images.alignImages()
|
||||
print(f"Aligned: {fast_name} -> {well_name}")
|
||||
except Exception as e:
|
||||
warped_image = fast_image
|
||||
matrix = np.zeros((3, 3))
|
||||
print(f"Failed to align: {fast_name} ({e})")
|
||||
|
||||
base = os.path.splitext(fast_name)[0]
|
||||
|
||||
if args.visualize_contours and align_images is not None:
|
||||
try:
|
||||
align_images.save_contour_visualizations(contour_visualizations_path, base + ".png")
|
||||
except Exception:
|
||||
print(f"Failed to visualize contours: {fast_name}")
|
||||
|
||||
out_path = os.path.join(args.output_dir, fast_name)
|
||||
cv2.imwrite(out_path, cv2.cvtColor(warped_image, cv2.COLOR_RGB2BGR))
|
||||
save_transformation(json_path, fast_name, matrix)
|
||||
|
||||
if args.save_diagnostics:
|
||||
try:
|
||||
save_checkerboard(well_image, fast_image,
|
||||
os.path.join(diagnostics_dir, base + "_checkerboard_before.png"))
|
||||
save_checkerboard(well_image, warped_image,
|
||||
os.path.join(diagnostics_dir, base + "_checkerboard_after.png"))
|
||||
if align_images is not None:
|
||||
save_thresholded(align_images, diagnostics_dir, base)
|
||||
align_images.save_contour_visualizations(diagnostics_dir, base + ".png")
|
||||
save_matched_contours(
|
||||
align_images,
|
||||
os.path.join(diagnostics_dir, base + "_matched_well.png"),
|
||||
os.path.join(diagnostics_dir, base + "_matched_fast.png"))
|
||||
save_matched_contours_panel(
|
||||
align_images,
|
||||
os.path.join(diagnostics_dir, base + "_matched_panel.png"))
|
||||
save_sift_matches_panel(
|
||||
align_images,
|
||||
os.path.join(diagnostics_dir, base + "_sift_matches.png"),
|
||||
layout=args.sift_layout,
|
||||
max_matches=args.sift_max_matches)
|
||||
except Exception as e:
|
||||
print(f"Failed to save diagnostics: {fast_name} ({e})")
|
||||
|
||||
if args.save_merged:
|
||||
alpha = 50.0 / 100.0
|
||||
merged_before = cv2.addWeighted(well_image, 1.0 - alpha, fast_image, alpha, 0)
|
||||
merged_before_path = os.path.join(merged_dir, fast_name).rsplit('.', 1)[0] + "_before.png"
|
||||
cv2.imwrite(merged_before_path, cv2.cvtColor(merged_before, cv2.COLOR_RGB2BGR))
|
||||
|
||||
merged_after = cv2.addWeighted(well_image, 1.0 - alpha, warped_image, alpha, 0)
|
||||
merged_after_path = os.path.join(merged_dir, fast_name).rsplit('.', 1)[0] + "_after.png"
|
||||
cv2.imwrite(merged_after_path, cv2.cvtColor(merged_after, cv2.COLOR_RGB2BGR))
|
||||
|
||||
print("All done :D")
|
||||
@@ -0,0 +1,72 @@
|
||||
import os
|
||||
import re
|
||||
import json
|
||||
import csv
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_number(value):
|
||||
"""Extract a float from values like '0.83', 'tensor(0.83)',
|
||||
'tensor(0.8312, device=\"cuda:0\")', etc. Returns None if not parseable."""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
match = re.search(r"[-+]?\d*\.?\d+(?:[eE][-+]?\d+)?", str(value))
|
||||
return float(match.group()) if match else None
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--data_root", type=str,
|
||||
default="/mnt/data/nerfstudio_ds/real_data",
|
||||
help="Root directory containing one folder per scene.")
|
||||
parser.add_argument("--pattern", type=str, default="*results*.json",
|
||||
help="Glob pattern for the metric JSON files to collect.")
|
||||
parser.add_argument("--output", type=str, default="metrics_sheet.csv",
|
||||
help="Path to the output CSV sheet.")
|
||||
args = parser.parse_args()
|
||||
|
||||
data_root = Path(args.data_root)
|
||||
scenes = sorted(p for p in data_root.iterdir() if p.is_dir())
|
||||
|
||||
rows = []
|
||||
metric_keys = [] # preserve discovery order of metric columns
|
||||
|
||||
for scene in scenes:
|
||||
json_files = sorted(scene.rglob(args.pattern))
|
||||
if not json_files:
|
||||
print(f"No metric JSON found for scene: {scene.name}")
|
||||
continue
|
||||
for json_path in json_files:
|
||||
try:
|
||||
with open(json_path) as f:
|
||||
data = json.load(f)
|
||||
except Exception as e:
|
||||
print(f"Could not read {json_path}: {e}")
|
||||
continue
|
||||
|
||||
row = {
|
||||
"scene": scene.name,
|
||||
"result_file": json_path.name,
|
||||
"result_path": str(json_path.relative_to(data_root)),
|
||||
}
|
||||
for key, value in data.items():
|
||||
if key not in metric_keys:
|
||||
metric_keys.append(key)
|
||||
row[key] = parse_number(value)
|
||||
rows.append(row)
|
||||
|
||||
if not rows:
|
||||
print("No metric JSON files found; nothing written.")
|
||||
raise SystemExit(0)
|
||||
|
||||
fieldnames = ["scene", "result_file", "result_path"] + metric_keys
|
||||
with open(args.output, "w", newline="") as f:
|
||||
writer = csv.DictWriter(f, fieldnames=fieldnames)
|
||||
writer.writeheader()
|
||||
for row in rows:
|
||||
writer.writerow(row)
|
||||
|
||||
print(f"Wrote {len(rows)} rows ({len(scenes)} scenes scanned) to: {os.path.abspath(args.output)}")
|
||||
@@ -0,0 +1,78 @@
|
||||
import os
|
||||
import numpy as np
|
||||
import cv2
|
||||
from pathlib import Path
|
||||
from tqdm import tqdm
|
||||
import argparse
|
||||
import json
|
||||
import torch
|
||||
import piq
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--gt_dir', type=str, default='/Users/momo/Desktop/Pandora-HDR_NeRF/final_results/lab_downstairs/GT/pano/')
|
||||
parser.add_argument('--data_dir', type=str, default='/Users/momo/Desktop/Pandora-HDR_NeRF/final_results/lab_downstairs/HDR-Nerfacto/pano/')
|
||||
parser.add_argument('--method_name', type=str)
|
||||
args = parser.parse_args()
|
||||
|
||||
# Collect GT and prediction jpg images (case-insensitive extension).
|
||||
def list_jpgs(directory):
|
||||
images = {}
|
||||
for path in Path(directory).rglob('*'):
|
||||
if path.suffix.lower() in ('.jpg', '.jpeg'):
|
||||
images[path.stem] = path
|
||||
return images
|
||||
|
||||
gt_images = list_jpgs(args.gt_dir)
|
||||
pred_images = list_jpgs(args.data_dir)
|
||||
|
||||
assert len(pred_images) == len(gt_images), "The size of images don't match GT size"
|
||||
|
||||
gts = []
|
||||
preds = []
|
||||
|
||||
for stem, gt_addr in tqdm(sorted(gt_images.items())):
|
||||
assert stem in pred_images, f"No matching prediction for GT image: {stem}"
|
||||
gt_addr = str(gt_addr)
|
||||
pred_addr = str(pred_images[stem])
|
||||
|
||||
# LDR jpg images are already gamma-encoded 8-bit; just normalize to [0, 1].
|
||||
gt_image = cv2.imread(gt_addr, cv2.IMREAD_UNCHANGED)
|
||||
gt_image = cv2.resize(gt_image, (512, 256), interpolation=cv2.INTER_LINEAR)
|
||||
gt_image = np.clip(gt_image.astype(np.float32) / 255.0, 0.0, 1.0)
|
||||
|
||||
pred_image = cv2.imread(pred_addr, cv2.IMREAD_UNCHANGED)
|
||||
pred_image = cv2.resize(pred_image, (512, 256), interpolation=cv2.INTER_LINEAR)
|
||||
pred_image = np.clip(pred_image.astype(np.float32) / 255.0, 0.0, 1.0)
|
||||
|
||||
cv2.imwrite('./test.png', (pred_image * 255).astype('uint8'))
|
||||
cv2.imwrite('./gt.png', (gt_image * 255).astype('uint8'))
|
||||
|
||||
if not np.isnan(gt_image).any():
|
||||
gt_image_torch = torch.from_numpy(gt_image).permute(2, 0, 1)
|
||||
gts.append(gt_image_torch)
|
||||
pred_image_torch = torch.from_numpy(pred_image).permute(2, 0, 1)
|
||||
preds.append(pred_image_torch)
|
||||
|
||||
ssim_index = piq.ssim(torch.stack(gts), torch.stack(preds), data_range=1.)
|
||||
psnr_index = piq.psnr(torch.stack(gts), torch.stack(preds), data_range=1.)
|
||||
|
||||
print("SSIM: ", ssim_index)
|
||||
print("PSNR: ", psnr_index)
|
||||
lpips_index = piq.LPIPS(reduction='none')(torch.stack(gts), torch.stack(preds))
|
||||
print("LPIPS: ", torch.mean(lpips_index))
|
||||
|
||||
results = {
|
||||
"SSIM": str(ssim_index),
|
||||
"PSNR": str(psnr_index),
|
||||
"LPIPS": str(torch.mean(lpips_index))
|
||||
}
|
||||
|
||||
# output_path = os.path.join(args.data_dir, "ldr_fast_results.json")
|
||||
output_path = os.path.join(args.data_dir, "hdrnerf_fast_results.json")
|
||||
|
||||
with open(output_path, 'w') as f:
|
||||
json.dump(results, f, indent=4)
|
||||
|
||||
print(f"Saved metrics to: {output_path}")
|
||||
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run calculate_metrics_fast.py for every scene folder in a data root.
|
||||
set -u
|
||||
|
||||
DATA_ROOT="/mnt/data/nerfstudio_ds/real_data"
|
||||
SCRIPT="lantern_scripts/calculate_metrics_fast.py"
|
||||
|
||||
if [ $# -lt 1 ]; then
|
||||
echo "Usage: $0 <metric_name> [data_root]"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
METRIC_NAME="$1"
|
||||
DATA_ROOT="${2:-$DATA_ROOT}"
|
||||
|
||||
failed=()
|
||||
|
||||
for scene in "$DATA_ROOT"/*/; do
|
||||
[ -d "$scene" ] || continue
|
||||
echo "=================================================================="
|
||||
echo "Processing: $scene"
|
||||
echo "=================================================================="
|
||||
if python "$SCRIPT" --input_dir "$scene" --metric_name "$METRIC_NAME"; then
|
||||
echo "OK: $scene"
|
||||
else
|
||||
echo "FAILED: $scene"
|
||||
failed+=("$scene")
|
||||
fi
|
||||
done
|
||||
|
||||
echo "=================================================================="
|
||||
if [ ${#failed[@]} -eq 0 ]; then
|
||||
echo "All scenes processed successfully."
|
||||
else
|
||||
echo "The following scenes failed:"
|
||||
for f in "${failed[@]}"; do
|
||||
echo " - $f"
|
||||
done
|
||||
exit 1
|
||||
fi
|
||||
Reference in new issue
Block a user