mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-22 23:55:36 +08:00
162 lines
6.2 KiB
Markdown
162 lines
6.2 KiB
Markdown
# How it works?
|
|
|
|
## 0. Setup
|
|
|
|
```
|
|
git clone https://github.com/MrKarimiD/nerfstudio-HDR.git
|
|
conda activate nerfstudio-HDR
|
|
export PYTHONPATH=$pwd:$PYTHONPATH
|
|
pip install -e .
|
|
pip install skylibs equilib scikit-surgerycore pydub
|
|
pip install -U git+https://github.com/luca-medeiros/lang-segment-anything.git
|
|
```
|
|
|
|
If there is an issue with equilib, do the following...
|
|
```
|
|
pip uninstall equilib
|
|
cd ../equilib
|
|
python setup.py develop
|
|
cd ../nerfstudio-HDR
|
|
```
|
|
|
|
## 1. Data acquisition & pre-processing
|
|
|
|
1. Setup the apparatus (stick with two Ricoh cameras)
|
|
> Cameras:
|
|
>
|
|
> **Camera 1**: left, well-exposed, serial: YN14100695 \
|
|
> **Camera 3**: right, fast-exposed (under-exposed), serial: YN14111000
|
|
|
|
- To connect the other camera:
|
|
- Turn wifi off
|
|
- Close the app
|
|
- Turn wifi on
|
|
- Open the app
|
|
- Connect the other camera
|
|
- Camera setup:
|
|
- Install Ricoh Theta app
|
|
- Connect the camera to the app
|
|
- Change camera parameters: select manual (bottom right)
|
|
- First time: turn on "CT Settings" in Shooting settings
|
|
- Apreture: 2.1
|
|
- Shutter speed: determined below
|
|
- ISO: 800
|
|
- WB: 3500
|
|
- Make sure the cameras point up to avoid having lights in the seam.
|
|
|
|
2. First capture with well-exposed cameras.
|
|
- Change shutter speed to get a well exposed image. **Note the shutter speed.**
|
|
- Start the video on one camera. Connect to the other camera following the steps above and start the vidoe on the second camera.
|
|
- Clap at the beginning and at the end to use for synchronization.
|
|
- Go around the scene with the two cameras well-exposed.
|
|
|
|
3. Second capture with Camera 1 (left) well-exposed and Camera 2 (right) fast-exposed. Change shutter speed of Camera 2 to 1/25000 (fastest exposure) and repeat the steps above
|
|
|
|
4. Import the files on computer. If on a mac, use Ricoh Theta File Transfer for Mac
|
|
|
|
5. Get the equirectangular videos by processing them with the Ricoh Theta computer app.
|
|
- Drag and drop the video file into the app window.
|
|
- Make sure both boxes are unchecked.
|
|
- Name files as fallows
|
|
- Camera 1:
|
|
- First capture: left_sfm
|
|
- Second capture: left_e1
|
|
- Camera 2:
|
|
- First capture: right_sfm
|
|
- Second capture: right_e2
|
|
|
|
6. Transfer videos on lab machine.
|
|
|
|
```
|
|
scp -r /path/to/source/file /path/to/destination/
|
|
```
|
|
|
|
7. Process the videos for OpenSFM. Input the right shutter speed noted above.
|
|
|
|
```
|
|
python lantern_scripts/process_videos_for_sfm.py --input_dir /mnt/data/scene/ --shutter_speed 0.004
|
|
```
|
|
|
|
## 2. Process data using OpenSFM
|
|
|
|
```
|
|
bin/opensfm_run_all /mnt/data/scene/sfm/
|
|
```
|
|
|
|
To view results:
|
|
```
|
|
python3 viewer/server.py -d /mnt/data/scene/sfm/
|
|
```
|
|
Check that the camera positions make sense and that the number of clusters is low.
|
|
|
|
## 2. Process data for NeRF
|
|
|
|
```
|
|
ns-process-data lantern-openSFM --data /mnt/data/scene/data/ --output-dir /mnt/data/scene/scene_ns/ --metadata /mnt/data/scene/sfm/reconstruction.json
|
|
```
|
|
|
|
For processing existing dataset that used colmap for camera positions:
|
|
```
|
|
ns-process-data images --data /mnt/data/garden/images --output-dir /mnt/data/garden_processed --skip-colmap --skip-image-processing --colmap-model-path /mnt/data/garden/sparse/0
|
|
```
|
|
|
|
## 3. Train NeRF
|
|
|
|
1. Run step 1 of lantern-nerfacto.
|
|
|
|
```
|
|
ns-train lantern-nerfacto --data /mnt/data/scene/scene_ns/ --viewer.websocket-port 8008 --pipeline.datamanager.train-num-images-to-sample-from 1800 --pipeline.model.lantern_steps 1 --pipeline.datamanager.pixel-sampler.lantern_steps 1 --max-num-iterations 60000
|
|
```
|
|
For better alignment between the well and the fast exposed images:
|
|
|
|
1.1. Run the following command.
|
|
```
|
|
python lantern_scripts/align_fast_to_eval.py --input_dir /mnt/data/scene/scene_ns/ --config outputs/scene_ns/lantern-nerfacto/2024-08-07_143013/config.yml
|
|
```
|
|
1.2. Rerun step 1.
|
|
```
|
|
ns-train lantern-nerfacto --data /mnt/data/scene/scene_ns/ --viewer.websocket-port 8008 --pipeline.datamanager.train-num-images-to-sample-from 1800 --pipeline.model.lantern_steps 1 --pipeline.datamanager.pixel-sampler.lantern_steps 1 --max-num-iterations 60000
|
|
```
|
|
|
|
2. Run step 2 of lantern-nerfacto.
|
|
|
|
```
|
|
ns-train lantern-nerfacto --data /mnt/data/scene/scene_ns/ --viewer.websocket-port 8008 --pipeline.datamanager.train-num-images-to-sample-from 1800 --pipeline.model.lantern_steps 2 --pipeline.datamanager.pixel-sampler.lantern_steps 2 --load-dir /mnt/workspace/lantern/nerfstudio-HDR/outputs/scene_ns/lantern-nerfacto/2024-06-10_155559/nerfstudio_models --pipeline.model.apply_mu_law False
|
|
```
|
|
|
|
## 4. View results
|
|
|
|
To use the interface:
|
|
```
|
|
ns-viewer --load-config outputs/scene_ns/lantern-nerfacto/2024-06-06_191338/config.yml --viewer.websocket-port 8008
|
|
```
|
|
- For the right side up:
|
|
- Go to SCENE sub-menu.
|
|
- Clic on RESET UP DIRECTION.
|
|
- To render a video with the interface command:
|
|
- Go to RENDER sub-menu.
|
|
- Clic on ADD CAMERA to manualy set camera view points.
|
|
- Clic on RENDER to get the command.
|
|
```
|
|
ns-render camera-path --load-config outputs/scene_ns/lantern-nerfacto/2024-06-06_191338/config.yml --camera-path-filename /mnt/data/scene/scene_ns/camera_paths/2024-06-06_191338.json --output-path /mnt/data/scene/renders/2024-06-06_191338.mp4
|
|
```
|
|
- To render a video fast-exposed: add --rendered-output-names rgb_fast
|
|
```
|
|
ns-render camera-path --load-config outputs/scene_ns/lantern-nerfacto/2024-06-06_191338/config.yml --camera-path-filename /mnt/data/scene/scene_ns/camera_paths/2024-06-06_191338.json --output-path /mnt/data/scene/renders/2024-06-06_191338.mp4 --rendered-output-names rgb_fast
|
|
```
|
|
|
|
To get evaluation images:
|
|
```
|
|
ns-eval --load-config=outputs/scene_ns/lantern-nerfacto/2024-06-17_152246/config.yml --output-path=output.json --render_output_path=/mnt/data/scene/eval
|
|
```
|
|
|
|
- To get evaluation images and tensorboard:
|
|
```
|
|
ns-eval --load-config=outputs/scene_ns/lantern-nerfacto/2024-06-20_142413/config.yml --output-path=output.json --render_output_path=/mnt/data/scene/eval --vis viewer+tensorboard
|
|
```
|
|
- To load tensorboard results:
|
|
```
|
|
tensorboard --logdir=/path/to/file
|
|
```
|
|
- Connect to http://localhost:6006/
|