Files
nerfstudio/README-pandora.md
T

6.2 KiB

How it works?

0. Setup

git clone https://github.com/MrKarimiD/nerfstudio-HDR.git
conda activate nerfstudio-HDR
export PYTHONPATH=$pwd:$PYTHONPATH
pip install -e .
pip install skylibs equilib scikit-surgerycore pydub
pip install -U git+https://github.com/luca-medeiros/lang-segment-anything.git

If there is an issue with equilib, do the following...

pip uninstall equilib
cd ../equilib
python setup.py develop
cd ../nerfstudio-HDR

1. Data acquisition & pre-processing

  1. Setup the apparatus (stick with two Ricoh cameras)

    Cameras:

    Camera 1: left, well-exposed, serial: YN14100695
    Camera 3: right, fast-exposed (under-exposed), serial: YN14111000

    • To connect the other camera:
      • Turn wifi off
      • Close the app
      • Turn wifi on
      • Open the app
      • Connect the other camera
    • Camera setup:
      • Install Ricoh Theta app
      • Connect the camera to the app
      • Change camera parameters: select manual (bottom right)
        • First time: turn on "CT Settings" in Shooting settings
        • Apreture: 2.1
        • Shutter speed: determined below
        • ISO: 800
        • WB: 3500
    • Make sure the cameras point up to avoid having lights in the seam.
  2. First capture with well-exposed cameras.

    • Change shutter speed to get a well exposed image. Note the shutter speed.
    • Start the video on one camera. Connect to the other camera following the steps above and start the vidoe on the second camera.
    • Clap at the beginning and at the end to use for synchronization.
    • Go around the scene with the two cameras well-exposed.
  3. Second capture with Camera 1 (left) well-exposed and Camera 2 (right) fast-exposed. Change shutter speed of Camera 2 to 1/25000 (fastest exposure) and repeat the steps above

  4. Import the files on computer. If on a mac, use Ricoh Theta File Transfer for Mac

  5. Get the equirectangular videos by processing them with the Ricoh Theta computer app.

    • Drag and drop the video file into the app window.
    • Make sure both boxes are unchecked.
    • Name files as fallows
      • Camera 1:
        • First capture: left_sfm
        • Second capture: left_e1
      • Camera 2:
        • First capture: right_sfm
        • Second capture: right_e2
  6. Transfer videos on lab machine.

    scp -r /path/to/source/file /path/to/destination/
    
  7. Process the videos for OpenSFM. Input the right shutter speed noted above.

    python lantern_scripts/process_videos_for_sfm.py --input_dir /mnt/data/scene/ --shutter_speed 0.004
    

2. Process data using OpenSFM

bin/opensfm_run_all /mnt/data/scene/sfm/

To view results:

python3 viewer/server.py -d /mnt/data/scene/sfm/

Check that the camera positions make sense and that the number of clusters is low.

2. Process data for NeRF

ns-process-data lantern-openSFM  --data /mnt/data/scene/data/  --output-dir /mnt/data/scene/scene_ns/ --metadata  /mnt/data/scene/sfm/reconstruction.json

For processing existing dataset that used colmap for camera positions:

ns-process-data images --data /mnt/data/garden/images --output-dir /mnt/data/garden_processed --skip-colmap --skip-image-processing --colmap-model-path /mnt/data/garden/sparse/0

3. Train NeRF

  1. Run step 1 of lantern-nerfacto.

    ns-train lantern-nerfacto --data /mnt/data/scene/scene_ns/ --viewer.websocket-port 8008 --pipeline.datamanager.train-num-images-to-sample-from 1800 --pipeline.model.lantern_steps 1 --pipeline.datamanager.pixel-sampler.lantern_steps 1 --max-num-iterations 60000
    

    For better alignment between the well and the fast exposed images:

    1.1. Run the following command.

    python lantern_scripts/align_fast_to_eval.py --input_dir /mnt/data/scene/scene_ns/ --config outputs/scene_ns/lantern-nerfacto/2024-08-07_143013/config.yml
    

    1.2. Rerun step 1.

    ns-train lantern-nerfacto --data /mnt/data/scene/scene_ns/ --viewer.websocket-port 8008 --pipeline.datamanager.train-num-images-to-sample-from 1800 --pipeline.model.lantern_steps 1 --pipeline.datamanager.pixel-sampler.lantern_steps 1 --max-num-iterations 60000
    
  2. Run step 2 of lantern-nerfacto.

    ns-train lantern-nerfacto --data /mnt/data/scene/scene_ns/ --viewer.websocket-port 8008 --pipeline.datamanager.train-num-images-to-sample-from 1800 --pipeline.model.lantern_steps 2 --pipeline.datamanager.pixel-sampler.lantern_steps 2 --load-dir /mnt/workspace/lantern/nerfstudio-HDR/outputs/scene_ns/lantern-nerfacto/2024-06-10_155559/nerfstudio_models --pipeline.model.apply_mu_law False
    

4. View results

To use the interface:

ns-viewer --load-config outputs/scene_ns/lantern-nerfacto/2024-06-06_191338/config.yml --viewer.websocket-port 8008
  • For the right side up:
    • Go to SCENE sub-menu.
    • Clic on RESET UP DIRECTION.
  • To render a video with the interface command:
    • Go to RENDER sub-menu.
    • Clic on ADD CAMERA to manualy set camera view points.
    • Clic on RENDER to get the command.
    ns-render camera-path --load-config outputs/scene_ns/lantern-nerfacto/2024-06-06_191338/config.yml --camera-path-filename /mnt/data/scene/scene_ns/camera_paths/2024-06-06_191338.json --output-path /mnt/data/scene/renders/2024-06-06_191338.mp4
    
    • To render a video fast-exposed: add --rendered-output-names rgb_fast
      ns-render camera-path --load-config outputs/scene_ns/lantern-nerfacto/2024-06-06_191338/config.yml --camera-path-filename /mnt/data/scene/scene_ns/camera_paths/2024-06-06_191338.json --output-path /mnt/data/scene/renders/2024-06-06_191338.mp4 --rendered-output-names rgb_fast
      

To get evaluation images:

ns-eval --load-config=outputs/scene_ns/lantern-nerfacto/2024-06-17_152246/config.yml --output-path=output.json --render_output_path=/mnt/data/scene/eval
  • To get evaluation images and tensorboard:
    ns-eval --load-config=outputs/scene_ns/lantern-nerfacto/2024-06-20_142413/config.yml --output-path=output.json --render_output_path=/mnt/data/scene/eval  --vis viewer+tensorboard
    
  • To load tensorboard results:
    tensorboard --logdir=/path/to/file