diff --git a/README.md b/README.md index b6c5bb6ff..dc4a0093e 100644 --- a/README.md +++ b/README.md @@ -241,12 +241,14 @@ Using an existing dataset is great, but likely you want to use your own data! We | 📱 [Polycam](https://docs.nerf.studio/quickstart/custom_dataset.html#polycam-capture) | IOS with LiDAR | [Polycam App](https://poly.cam/) | 🐇 | | 📱 [KIRI Engine](https://docs.nerf.studio/quickstart/custom_dataset.html#kiri-engine-capture) | IOS or Android | [KIRI Engine App](https://www.kiriengine.com/) | 🐇 | | 📱 [Record3D](https://docs.nerf.studio/quickstart/custom_dataset.html#record3d-capture) | IOS with LiDAR | [Record3D app](https://record3d.app/) | 🐇 | +| 📱 [Spectacular AI](https://docs.nerf.studio/quickstart/custom_dataset.html#spectacularai) | IOS, OAK, [others](https://www.spectacularai.com/mapping#supported-devices) | [App](https://apps.apple.com/us/app/spectacular-rec/id6473188128) / [`sai-cli`](https://www.spectacularai.com/mapping) | 🐇 | | 🖥 [Metashape](https://docs.nerf.studio/quickstart/custom_dataset.html#metashape) | Any | [Metashape](https://www.agisoft.com/) | 🐇 | | 🖥 [RealityCapture](https://docs.nerf.studio/quickstart/custom_dataset.html#realitycapture) | Any | [RealityCapture](https://www.capturingreality.com/realitycapture) | 🐇 | | 🖥 [ODM](https://docs.nerf.studio/quickstart/custom_dataset.html#odm) | Any | [ODM](https://github.com/OpenDroneMap/ODM) | 🐇 | | 👓 [Aria](https://docs.nerf.studio/quickstart/custom_dataset.html#aria) | Aria glasses | [Project Aria](https://projectaria.com/) | 🐇 | | 🛠 [Custom](https://docs.nerf.studio/quickstart/data_conventions.html) | Any | Camera Poses | 🐇 | + ## 5. Advanced Options ### Training models other than nerfacto diff --git a/docs/quickstart/custom_dataset.md b/docs/quickstart/custom_dataset.md index fddbe1d56..6444ca445 100644 --- a/docs/quickstart/custom_dataset.md +++ b/docs/quickstart/custom_dataset.md @@ -19,6 +19,7 @@ We currently support the following custom data types: | 📱 [Polycam](polycam) | IOS with LiDAR | [Polycam App](https://poly.cam/) | 🐇 | | 📱 [KIRI Engine](kiri) | IOS or Android | [KIRI Engine App](https://www.kiriengine.com/) | 🐇 | | 📱 [Record3D](record3d) | IOS with LiDAR | [Record3D app](https://record3d.app/) | 🐇 | +| 📱 [Spectacular AI](spectacularai) | IOS, OAK, others| [App](https://apps.apple.com/us/app/spectacular-rec/id6473188128) / [`sai-cli`](https://www.spectacularai.com/mapping) | 🐇 | | 🖥 [Metashape](metashape) | Any | [Metashape](https://www.agisoft.com/) | 🐇 | | 🖥 [RealityCapture](realitycapture) | Any | [RealityCapture](https://www.capturingreality.com/realitycapture) | 🐇 | | 🖥 [ODM](odm) | Any | [ODM](https://github.com/OpenDroneMap/ODM) | 🐇 | @@ -267,6 +268,44 @@ ns-process-data record3d --data {data directory} --output-dir {output directory} ns-train nerfacto --data {output directory} ``` +(spectacularai)= + +## Spectacular AI + +Spectacular AI SDK and apps can be used to capture data from various devices: + + * iPhones (with LiDAR) + * OAK-D cameras + * RealSense D455/D435i + * Azure Kinect DK + +The SDK also records IMU data, which is fused with camera and (if available) LiDAR/ToF data when computing the camera poses. This approach, VISLAM, is more robust than purely image based methods (e.g., COLMAP) and can work better and faster for difficult data (monotonic environments, fast motions, narrow FoV, etc.). + +Instructions: + +1. Installation. With the Nerfstudio Conda environment active, first install the Spectacular AI Python library + +```bash +pip install spectacularAI[full] +``` + +2. Install FFmpeg. Linux: `apt install ffmpeg` (or similar, if using another package manager). Windows: [see here](https://www.editframe.com/guides/how-to-install-and-start-using-ffmpeg-in-under-10-minutes). FFmpeg must be in your `PATH` so that `ffmpeg` works on the command line. + +3. Data capture. See [here for specific instructions for each supported device](https://github.com/SpectacularAI/sdk-examples/tree/main/python/mapping#recording-data). + +4. Process and export. Once you have recorded a dataset in Spectacular AI format and have it stored in `{data directory}` it can be converted into a Nerfstudio supported format with: + +```bash +sai-cli process {data directory} --preview3d --key_frame_distance=0.05 {output directory} +``` +The optional `--preview3d` flag shows a 3D preview of the point cloud and estimated trajectory live while VISLAM is running. The `--key_frame_distance` argument can be tuned based on the recorded scene size: 0.05 (5cm) is good for small scans and 0.15 for room-sized scans. If the processing gets slow, you can also try adding a --fast flag to `sai-cli process` to trade off quality for speed. + +5. Train. No separate `ns-process-data` step is needed. The data in `{output directory}` can now be trained with Nerfstudio: + +```bash +ns-train nerfacto --data {output directory} +``` + (metashape)= ## Metashape