Add support for Project Aria datasets. (#2617)

* Add support for Project Aria datasets.

1) An export script for processing Project Aria datasets
2) Add support for Fisheye624 cameras
3) Add support for center-crop masking and sampling

Example commands for running nerfstudio on Project Aria data:

python scripts/datasets/process_project_aria.py --vrs_file=<path_to_vrs> --mps_data_dir=<path to MPS directory> --output_dir=<nerfstudio_data_dir>

ns-train nerfacto --data <nerfstudio_data_dir>  nerfstudio-data --orientation-method none

* Clean up type errors

* Fix formatting

* Fix linter errors, import, and use absolute file paths

* Add aria subcommand to ns-process-data

* Appease ruff

* Add Aria to docs

* Fix docs, add back import try/except

* nit: path.as_posix() -> str(path)

---------

Co-authored-by: Brent Yi <yibrenth@gmail.com>
This commit is contained in:
Chris SweeneyandBrent Yi authored and GitHub committed 2023-12-12 22:18:13 +00:00
1 parent 01d57b90dc
commit 6ccbc2de45
10 files changed
+634 -40

No files matched your search

+28 -2
View File
@@ -10,7 +10,7 @@ ns-process-data {video,images,polycam,record3d} --data {DATA_PATH} --output-dir
A full set of arguments can be found {doc}`here</reference/cli/ns_process_data>`.
We Currently support the following custom data types:
We currently support the following custom data types:
| Data | Capture Device | Requirements | `ns-process-data` Speed |
| ----------------------------- | -------------- | ----------------------------------------------- | ----------------------- |
| 📷 [Images](images_and_video) | Any | [COLMAP](https://colmap.github.io/install.html) | 🐢 |
@@ -22,6 +22,7 @@ We Currently support the following custom data types:
| 🖥 [Metashape](metashape) | Any | [Metashape](https://www.agisoft.com/) | 🐇 |
| 🖥 [RealityCapture](realitycapture) | Any | [RealityCapture](https://www.capturingreality.com/realitycapture) | 🐇 |
| 🖥 [ODM](odm) | Any | [ODM](https://github.com/OpenDroneMap/ODM) | 🐇 |
| 👓 [Aria](aria) | Aria glasses | [Project Aria](https://projectaria.com/) | 🐇 |
(images_and_video)=
@@ -348,6 +349,31 @@ ns-process-data odm --data /path/to/dataset --output-dir {output directory}
ns-train nerfacto --data {output directory}
```
(aria)=
## Aria
1. Install projectaria_tools:
```bash
conda activate nerfstudio
pip install projectaria-tools'[all]'
```
2. Download a VRS file from Project Aria glasses, and run Machine Perception Services to extract poses.
3. Convert to nerfstudio format.
```bash
ns-process-data aria --vrs-file /path/to/vrs/file --mps-data-dir /path/to/mps/data --output-dir {output directory}
```
4. Train!
```bash
ns-train nerfacto --data {output directory}
```
(360_data)=
## 360 Data (Equirectangular)
@@ -459,4 +485,4 @@ If the depth of the scene is unviewable and looks too close or expanded when vie
- The IPD can be modified in the `cameras.py` script as the variable `vr_ipd` (default is 64 mm).
- Compositing with Blender Objects and VR180 or ODS Renders
- Configure the Blender camera as panoramic and equirectangular. For the VR180 Blender camera, set the panoramic longitude min and max to -90 and 90.
- Change the Stereoscopy mode to "Parallel" set the Interocular Distance to 0.064 m.
- Change the Stereoscopy mode to "Parallel" set the Interocular Distance to 0.064 m.