mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-19 22:25:44 +08:00
156 lines
5.4 KiB
Markdown
156 lines
5.4 KiB
Markdown
|
|
|
|
# How it works?
|
|
|
|
## 0. Setup
|
|
|
|
|
|
Open this folder in VS Code Dev Container extension!
|
|
|
|
```
|
|
export PYTHONPATH=$pwd:$PYTHONPATH
|
|
pip install skylibs equilib
|
|
```
|
|
|
|
## 1. Data acquisition & pre-processing
|
|
|
|
### OPTION 1: Real data
|
|
|
|
1. Acquire data using the apparatus (stick with two Ricoh cameras)
|
|
|
|
Make sure the cameras point up to avoid having lights in the seam.
|
|
|
|
Do two recordings with both Ricoh Theta cameras. Use the flash of a flashlight both times, the falling edge will later be detected by a software.
|
|
|
|
> Cameras:
|
|
>
|
|
> **camera 1**: left, well-exposed, serial: YN14100695 \
|
|
> **camera 3**: right, under-exposed, serial: YN14111000
|
|
|
|
2. Take one video capture with the two cameras well-exposed.
|
|
|
|
Use flash at the beginning (and at the end?) to use for synchronization.
|
|
|
|
3. Take one video capture with Camera 1 (left) well-exposed and Camera 2 (right) fast-exposed.
|
|
|
|
Use flash at the beginning (and at the end?) to use for synchronization.
|
|
|
|
4. Run the script for automatic flash detection / use Shotcut to manually align and trim the videos manually.
|
|
|
|
It will temporally align the frames from the left and right camera and remove the initial frames with the flash.
|
|
|
|
5. Extract frames using ffmpeg
|
|
|
|
```
|
|
python lantern_scripts/mp4_videos_to_png_frames.py --help
|
|
python lantern_scripts/mp4_videos_to_png_frames.py --input_dir data/lab_downstairs/trimmed_videos
|
|
```
|
|
|
|
It should output the following file structure
|
|
```
|
|
left_colmap_baseline/
|
|
....png
|
|
....png
|
|
....png
|
|
right_colmap_baseline/
|
|
....png
|
|
....png
|
|
....png
|
|
left_e1/
|
|
....png
|
|
....png
|
|
....png
|
|
right_e2/
|
|
....png
|
|
....png
|
|
....png
|
|
```
|
|
|
|
6. Run the code to mask out people and the stick.
|
|
|
|
Make sure the masks in "./stick_masks" are valid
|
|
```
|
|
python lantern_scripts/mask_humans2.py --help
|
|
python lantern_scripts/mask_humans2.py --input_dir data/lab_downstairs/trimmed_videos
|
|
```
|
|
|
|
|
|
### OPTION 2: Synthetic data
|
|
|
|
``
|
|
export PYTHONPATH=$pwd:$PYTHONPATH
|
|
python lantern_scripts/split_synthetic_colmap.py --has_exposure --left_dir ./Dual_Cameras/left --right_dir ./Dual_Cameras/right --out_dir ./Dual_Cameras/split_colmap_out
|
|
``
|
|
|
|
## 2. Process data using COLMAP
|
|
|
|
Example with synthetic data:
|
|
|
|
```
|
|
export PYTHONPATH=$pwd:$PYTHONPATH
|
|
ns-process-data lantern --data ./Dual_Cameras/split_colmap_subset --output-dir ./Dual_Cameras/split_colmap_subset/split_colmap_out5 --skip-image-processing --camera-type equirectangular --images-per-equirect 8
|
|
```
|
|
|
|
Example with real data (note that `--mask-dir` was added):
|
|
|
|
```
|
|
export PYTHONPATH=$pwd:$PYTHONPATH
|
|
ns-process-data lantern --data ./data/lab_downstairs/captured_data --mask-dir ./data/lab_downstairs/masks --output-dir ./data/lab_downstairs/colmap_out --skip-image-processing --camera-type equirectangular --images-per-equirect 8
|
|
```
|
|
|
|
After that, you'll need to copy manually (for now), the folder to `images`, for nerf training.
|
|
|
|
Examples
|
|
```
|
|
ns-process-data lantern --data ./data/lab_ground_floor/trimmed_videos_smart_subset --mask-dir ./data/lab_ground_floor/trimmed_videos_masks --output-dir ./data/lab_ground_floor/colmap_out_smart --skip-image-processing --camera-type equirectangular --images-per-equirect 8 --skip-colmap --skip-perspective-transform
|
|
```
|
|
|
|
## 2. Pre-training NeRF
|
|
|
|
Using both the well-exposed and fast-exposed frames, as well as their associated camera poses, a NeRF will be trained to predict both the RGB and mask to do volumetric rendering.
|
|
|
|
After convergence, go to the next step.
|
|
|
|
```
|
|
ns-train lantern-nerfacto --help
|
|
```
|
|
|
|
If you need to use less frames, add this argument: `--pipeline.datamanager.train-num-images-to-sample-from 400 --pipeline.datamanager.eval-num-images-to-sample-from 30`
|
|
|
|
> Thing to keep in mind: the way Nerfstudio splits train and test images may not be optimal: left might be in one dataset and right in another... that's a bit of a data leak.
|
|
|
|
Examples
|
|
|
|
```
|
|
ns-train lantern-nerfacto --data data/lab_ground_floor/colmap_out_smart --pipeline.datamanager.eval-num-images-to-sample-from 0
|
|
```
|
|
|
|
## 3. Running MomoNet
|
|
|
|
(Using which frames?) A pretrained UNet image-to-image network called MomoNet will infill the missing dynamic range and store these predictions in new images.
|
|
It will be trained on the Laval indoor HDR dataset to do this task.
|
|
|
|
TODO: validate this part
|
|
|
|
## 4. Evaluation
|
|
|
|
```
|
|
ns-train hdr-nerfacto --data data/lab_downstairs_ns --pipeline.datamanager.train-num-images-to-sample-from 400 --pipeline.datamanager.eval-num-images-to-sample-from 30
|
|
```
|
|
|
|
```
|
|
ns-eval --load-config outputs/LVSN_LAB_OpenSFM/hdr-nerfacto/2024-05-10_032723/config.yml --output-path ablations_out/hdr_nerfacto/output.json --render-output-path ablations_out/hdr-nerfacto
|
|
```
|
|
|
|
```
|
|
ns-render camera-path --load-config outputs/lab_downstairs_ns/hdr-nerfacto/2024-05-10_211317/config.yml --camera-path-filename data/lab_downstairs_ns/GT_transforms.json --output-path renders/lab_downstairs_ns/2024-05-10_211317/hdr-nerfacto --output-format images --image-format exr
|
|
```
|
|
|
|
```
|
|
docker run -it --rm -v /gel/usr/$USER:/mnt/workspace/ -v /home-local2/$USER.extra.nobkp/:/mnt/data/ --gpus '"device=1"' -p 7007:7007 lantern_docker:latest /bin/bash
|
|
export PYTHONPATH=$PYTHONPATH:$(pwd)
|
|
```
|
|
|
|
docker exec -it 57d2bf549c9a /bin/bash
|
|
ns-train hdr-nerfacto --data data/lab_downstairs_ns/ --pipeline.datamanager.train-num-images-to-sample-from 1800 --viewer.websocket-port 9009
|