6.2 KiB
How it works?
0. Setup
git clone https://github.com/MrKarimiD/nerfstudio-HDR.git
conda activate nerfstudio-HDR
export PYTHONPATH=$pwd:$PYTHONPATH
pip install -e .
pip install skylibs equilib scikit-surgerycore pydub
pip install -U git+https://github.com/luca-medeiros/lang-segment-anything.git
If there is an issue with equilib, do the following...
pip uninstall equilib
cd ../equilib
python setup.py develop
cd ../nerfstudio-HDR
1. Data acquisition & pre-processing
-
Setup the apparatus (stick with two Ricoh cameras)
Cameras:
Camera 1: left, well-exposed, serial: YN14100695
Camera 3: right, fast-exposed (under-exposed), serial: YN14111000- To connect the other camera:
- Turn wifi off
- Close the app
- Turn wifi on
- Open the app
- Connect the other camera
- Camera setup:
- Install Ricoh Theta app
- Connect the camera to the app
- Change camera parameters: select manual (bottom right)
- First time: turn on "CT Settings" in Shooting settings
- Apreture: 2.1
- Shutter speed: determined below
- ISO: 800
- WB: 3500
- Make sure the cameras point up to avoid having lights in the seam.
- To connect the other camera:
-
First capture with well-exposed cameras.
- Change shutter speed to get a well exposed image. Note the shutter speed.
- Start the video on one camera. Connect to the other camera following the steps above and start the vidoe on the second camera.
- Clap at the beginning and at the end to use for synchronization.
- Go around the scene with the two cameras well-exposed.
-
Second capture with Camera 1 (left) well-exposed and Camera 2 (right) fast-exposed. Change shutter speed of Camera 2 to 1/25000 (fastest exposure) and repeat the steps above
-
Import the files on computer. If on a mac, use Ricoh Theta File Transfer for Mac
-
Get the equirectangular videos by processing them with the Ricoh Theta computer app.
- Drag and drop the video file into the app window.
- Make sure both boxes are unchecked.
- Name files as fallows
- Camera 1:
- First capture: left_sfm
- Second capture: left_e1
- Camera 2:
- First capture: right_sfm
- Second capture: right_e2
- Camera 1:
-
Transfer videos on lab machine.
scp -r /path/to/source/file /path/to/destination/ -
Process the videos for OpenSFM. Input the right shutter speed noted above.
python lantern_scripts/process_videos_for_sfm.py --input_dir /mnt/data/scene/ --shutter_speed 0.004
2. Process data using OpenSFM
bin/opensfm_run_all /mnt/data/scene/sfm/
To view results:
python3 viewer/server.py -d /mnt/data/scene/sfm/
Check that the camera positions make sense and that the number of clusters is low.
2. Process data for NeRF
ns-process-data lantern-openSFM --data /mnt/data/scene/data/ --output-dir /mnt/data/scene/scene_ns/ --metadata /mnt/data/scene/sfm/reconstruction.json
For processing existing dataset that used colmap for camera positions:
ns-process-data images --data /mnt/data/garden/images --output-dir /mnt/data/garden_processed --skip-colmap --skip-image-processing --colmap-model-path /mnt/data/garden/sparse/0
3. Train NeRF
-
Run step 1 of lantern-nerfacto.
ns-train lantern-nerfacto --data /mnt/data/scene/scene_ns/ --viewer.websocket-port 8008 --pipeline.datamanager.train-num-images-to-sample-from 1800 --pipeline.model.lantern_steps 1 --pipeline.datamanager.pixel-sampler.lantern_steps 1 --max-num-iterations 60000For better alignment between the well and the fast exposed images:
1.1. Run the following command.
python lantern_scripts/align_fast_to_eval.py --input_dir /mnt/data/scene/scene_ns/ --config outputs/scene_ns/lantern-nerfacto/2024-08-07_143013/config.yml1.2. Rerun step 1.
ns-train lantern-nerfacto --data /mnt/data/scene/scene_ns/ --viewer.websocket-port 8008 --pipeline.datamanager.train-num-images-to-sample-from 1800 --pipeline.model.lantern_steps 1 --pipeline.datamanager.pixel-sampler.lantern_steps 1 --max-num-iterations 60000 -
Run step 2 of lantern-nerfacto.
ns-train lantern-nerfacto --data /mnt/data/scene/scene_ns/ --viewer.websocket-port 8008 --pipeline.datamanager.train-num-images-to-sample-from 1800 --pipeline.model.lantern_steps 2 --pipeline.datamanager.pixel-sampler.lantern_steps 2 --load-dir /mnt/workspace/lantern/nerfstudio-HDR/outputs/scene_ns/lantern-nerfacto/2024-06-10_155559/nerfstudio_models --pipeline.model.apply_mu_law False
4. View results
To use the interface:
ns-viewer --load-config outputs/scene_ns/lantern-nerfacto/2024-06-06_191338/config.yml --viewer.websocket-port 8008
- For the right side up:
- Go to SCENE sub-menu.
- Clic on RESET UP DIRECTION.
- To render a video with the interface command:
- Go to RENDER sub-menu.
- Clic on ADD CAMERA to manualy set camera view points.
- Clic on RENDER to get the command.
ns-render camera-path --load-config outputs/scene_ns/lantern-nerfacto/2024-06-06_191338/config.yml --camera-path-filename /mnt/data/scene/scene_ns/camera_paths/2024-06-06_191338.json --output-path /mnt/data/scene/renders/2024-06-06_191338.mp4- To render a video fast-exposed: add --rendered-output-names rgb_fast
ns-render camera-path --load-config outputs/scene_ns/lantern-nerfacto/2024-06-06_191338/config.yml --camera-path-filename /mnt/data/scene/scene_ns/camera_paths/2024-06-06_191338.json --output-path /mnt/data/scene/renders/2024-06-06_191338.mp4 --rendered-output-names rgb_fast
To get evaluation images:
ns-eval --load-config=outputs/scene_ns/lantern-nerfacto/2024-06-17_152246/config.yml --output-path=output.json --render_output_path=/mnt/data/scene/eval
- To get evaluation images and tensorboard:
ns-eval --load-config=outputs/scene_ns/lantern-nerfacto/2024-06-20_142413/config.yml --output-path=output.json --render_output_path=/mnt/data/scene/eval --vis viewer+tensorboard - To load tensorboard results:
tensorboard --logdir=/path/to/file- Connect to http://localhost:6006/