From 626441e15e8e59970ba95229e40727458bbf65a3 Mon Sep 17 00:00:00 2001 From: Otto Seiskari Date: Fri, 15 Dec 2023 18:37:27 +0200 Subject: [PATCH] Add instructions for data collection and processing with Spectacular AI tools (#2668) * Add instructions for using Spectacular AI tools Adds easy support for new device types: OAK-D, RealSense and Azure Kinect, as well as an alternative pipeline for iOS. * Grammar fixes in docs/quickstart/custom_dataset.md (Spectacular AI section) Co-authored-by: Matias Turkulainen <30566358+maturk@users.noreply.github.com> * Fix section label in the Spectacular AI section of custom_dataset.md * Update custom_dataset.md: typo fix. "montions" -> "motions" --------- Co-authored-by: Matias Turkulainen <30566358+maturk@users.noreply.github.com> --- README.md | 2 ++ docs/quickstart/custom_dataset.md | 39 +++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/README.md b/README.md index b6c5bb6ff..dc4a0093e 100644 --- a/README.md +++ b/README.md @@ -241,12 +241,14 @@ Using an existing dataset is great, but likely you want to use your own data! We | 📱 [Polycam](https://docs.nerf.studio/quickstart/custom_dataset.html#polycam-capture) | IOS with LiDAR | [Polycam App](https://poly.cam/) | 🐇 | | 📱 [KIRI Engine](https://docs.nerf.studio/quickstart/custom_dataset.html#kiri-engine-capture) | IOS or Android | [KIRI Engine App](https://www.kiriengine.com/) | 🐇 | | 📱 [Record3D](https://docs.nerf.studio/quickstart/custom_dataset.html#record3d-capture) | IOS with LiDAR | [Record3D app](https://record3d.app/) | 🐇 | +| 📱 [Spectacular AI](https://docs.nerf.studio/quickstart/custom_dataset.html#spectacularai) | IOS, OAK, [others](https://www.spectacularai.com/mapping#supported-devices) | [App](https://apps.apple.com/us/app/spectacular-rec/id6473188128) / [`sai-cli`](https://www.spectacularai.com/mapping) | 🐇 | | 🖥 [Metashape](https://docs.nerf.studio/quickstart/custom_dataset.html#metashape) | Any | [Metashape](https://www.agisoft.com/) | 🐇 | | 🖥 [RealityCapture](https://docs.nerf.studio/quickstart/custom_dataset.html#realitycapture) | Any | [RealityCapture](https://www.capturingreality.com/realitycapture) | 🐇 | | 🖥 [ODM](https://docs.nerf.studio/quickstart/custom_dataset.html#odm) | Any | [ODM](https://github.com/OpenDroneMap/ODM) | 🐇 | | 👓 [Aria](https://docs.nerf.studio/quickstart/custom_dataset.html#aria) | Aria glasses | [Project Aria](https://projectaria.com/) | 🐇 | | 🛠 [Custom](https://docs.nerf.studio/quickstart/data_conventions.html) | Any | Camera Poses | 🐇 | + ## 5. Advanced Options ### Training models other than nerfacto diff --git a/docs/quickstart/custom_dataset.md b/docs/quickstart/custom_dataset.md index fddbe1d56..6444ca445 100644 --- a/docs/quickstart/custom_dataset.md +++ b/docs/quickstart/custom_dataset.md @@ -19,6 +19,7 @@ We currently support the following custom data types: | 📱 [Polycam](polycam) | IOS with LiDAR | [Polycam App](https://poly.cam/) | 🐇 | | 📱 [KIRI Engine](kiri) | IOS or Android | [KIRI Engine App](https://www.kiriengine.com/) | 🐇 | | 📱 [Record3D](record3d) | IOS with LiDAR | [Record3D app](https://record3d.app/) | 🐇 | +| 📱 [Spectacular AI](spectacularai) | IOS, OAK, others| [App](https://apps.apple.com/us/app/spectacular-rec/id6473188128) / [`sai-cli`](https://www.spectacularai.com/mapping) | 🐇 | | 🖥 [Metashape](metashape) | Any | [Metashape](https://www.agisoft.com/) | 🐇 | | 🖥 [RealityCapture](realitycapture) | Any | [RealityCapture](https://www.capturingreality.com/realitycapture) | 🐇 | | 🖥 [ODM](odm) | Any | [ODM](https://github.com/OpenDroneMap/ODM) | 🐇 | @@ -267,6 +268,44 @@ ns-process-data record3d --data {data directory} --output-dir {output directory} ns-train nerfacto --data {output directory} ``` +(spectacularai)= + +## Spectacular AI + +Spectacular AI SDK and apps can be used to capture data from various devices: + + * iPhones (with LiDAR) + * OAK-D cameras + * RealSense D455/D435i + * Azure Kinect DK + +The SDK also records IMU data, which is fused with camera and (if available) LiDAR/ToF data when computing the camera poses. This approach, VISLAM, is more robust than purely image based methods (e.g., COLMAP) and can work better and faster for difficult data (monotonic environments, fast motions, narrow FoV, etc.). + +Instructions: + +1. Installation. With the Nerfstudio Conda environment active, first install the Spectacular AI Python library + +```bash +pip install spectacularAI[full] +``` + +2. Install FFmpeg. Linux: `apt install ffmpeg` (or similar, if using another package manager). Windows: [see here](https://www.editframe.com/guides/how-to-install-and-start-using-ffmpeg-in-under-10-minutes). FFmpeg must be in your `PATH` so that `ffmpeg` works on the command line. + +3. Data capture. See [here for specific instructions for each supported device](https://github.com/SpectacularAI/sdk-examples/tree/main/python/mapping#recording-data). + +4. Process and export. Once you have recorded a dataset in Spectacular AI format and have it stored in `{data directory}` it can be converted into a Nerfstudio supported format with: + +```bash +sai-cli process {data directory} --preview3d --key_frame_distance=0.05 {output directory} +``` +The optional `--preview3d` flag shows a 3D preview of the point cloud and estimated trajectory live while VISLAM is running. The `--key_frame_distance` argument can be tuned based on the recorded scene size: 0.05 (5cm) is good for small scans and 0.15 for room-sized scans. If the processing gets slow, you can also try adding a --fast flag to `sai-cli process` to trade off quality for speed. + +5. Train. No separate `ns-process-data` step is needed. The data in `{output directory}` can now be trained with Nerfstudio: + +```bash +ns-train nerfacto --data {output directory} +``` + (metashape)= ## Metashape