Omnidirectional Stereo Update - Alignment and Compositing (#2217)

Update to omnidirectional stereo rendering (VR 360) to fix slight positional alignment to accurately composite with Blender objects. Modified documentation for ODS and the Blender add-on to include details on Blender camera settings to composite with the Nerfstudio ODS camera model.
This commit is contained in:
Cyrus Vachha authored and GitHub committed 2023-07-13 07:59:10 -07:00
1 parent cebe787bed
commit 554dec77f2
3 files changed
+13 -8

No files matched your search

+6 -4
View File
@@ -355,7 +355,7 @@ Stereo equirectangular rendering for VR video is supported as omni-directional s
<img img width="300" src="https://github-production-user-asset-6210df.s3.amazonaws.com/9502341/240530353-5c05d12b-6e42-42bb-8ff5-f54aa30ca262.jpg">
</center>
To render in ODS it is essential to adjust the NeRF to have an approximately true-to-life real world scale (adjustable in the camera path) to ensure that the scene depth and IPD (distance between the eyes) is appropriate for the render to be viewable in VR. You can adjust the scene scale with the [Blender Add-on](https://docs.nerf.studio/en/latest/extensions/blender_addon.html) by appropriately scaling a point cloud representation of the NeRF.
To render in ODS it is essential to adjust the NeRF to have an approximately true-to-life real world scale (adjustable in the camera path) to ensure that the scene depth and IPD (distance between the eyes) is appropriate for the render to be viewable in VR. You can adjust the scene scale with the [Nerfstudio Blender Add-on](https://docs.nerf.studio/en/latest/extensions/blender_addon.html) by appropriately scaling a point cloud representation of the NeRF.
Results may be unviewable if the scale is not set appropriately. The IPD is set at 64mm by default but only is accurate when the NeRF scene is true to scale.
For good quality renders, it is recommended to render at 4096x2048 per eye, or 2048x1024 per eye. Render resolutions must be specified in the camera path in a 2:1 aspect ratio. The final stacked render output will automatically be constructed as 1:1.
@@ -366,12 +366,12 @@ If you are rendering an image sequence, it is recommended to render as png inste
:::
To render with the ODS camera:
1. Use the Nerfstudio Blender Add-on to set the scale of the NeRF scene and create the camera path
1. Use the [Nerfstudio Blender Add-on](https://docs.nerf.studio/en/latest/extensions/blender_addon.html) to set the scale of the NeRF scene and create the camera path
- Export a point cloud representation of the NeRF
- Import the point cloud representation in Blender and enable the Blender Add-on
- Import the point cloud representation in Blender and enable the Nerfstudio Blender Add-on
- Create a reference object such as a cube which may be 1x1x1 meter. You could also create a cylinder and scale it to an appropriate height of a viewer.
- Now scale the point cloud representation accordingly to match the reference object. This is to ensure that the NeRF scene is scaled as close to real life.
- To place the camera at the correct height from the ground in the scene, you create a cylinder representing the viewer vertically scaled to the viewer’s height, and place the camera at eye level.
- To place the camera at the correct height from the ground in the scene, you can create a cylinder representing the viewer vertically scaled to the viewer’s height, and place the camera at eye level.
- Animate the camera movement as needed
- Create the camera path JSON file with the Nerfstudio Blender Add-on
2. Edit the JSON camera path file
@@ -392,3 +392,5 @@ If the depth of the scene is unviewable and looks too close or expanded when vie
- When rendering ODS content, Nerfstudio will first render the left eye, then the right eye, and finally vertically stack the renders. During this process, Nerfstudio will create a temporary folder to store the left and right eye renders and delete this folder once the final renders are stacked.
- If rendering content where the camera is stationary for many frames, it is recommended to only render once at that position and extend the time in a video editor since ODS renders can take a lot of time to render.
- It is recommended to render a preliminary render at a much lower resolution or frame rate to test and ensure that the depth and camera position look accurate in VR.
- The IPD can be modified in the `cameras.py` script as the variable `vr_ipd` (default is 64 mm).
- If using the Nerfstudio Blender Add-on to composite Blender objects or NeRFs with the omnidirectional render, change the Stereoscopy mode to "Parallel" set the Interocular Distance to 0.064 m.