nerfstudio/docs/tutorials/data/creating_dataset.md

Using custom data

Training model on existing datasets is only so fun. If you would like to train on self captured data you will need to process the data into an existing format. Specifically we need to know the camera poses for each image. COLMAP is a standard tool for extracting poses. It is possible to use other methods like SLAM or hardware recorded poses. We intend to add documentation for these other methods in the future.

Recover poses using COLMAP

:::{admonition} Notes :class: warning

• This method assumes the images were captured with a standard pinhole-like camera. 360° captures will not work.
• COLMAP is notoriously finicky. Try your best to capture overlapping, non-blurry images. :::

These instructions will recover poses for your own images using COLMAP and save the data in the instant_ngp format.

Please download this file and place it at scripts/colmap2nerf.py courtesy of the Instant-NGP authors. This script will use COLMAP to process images of a video file into a reconstruction with camera poses. You can see their write-up here for instructions.

For example, maybe we have a video named bear.mp4. We can use the commands described in the Instant-NGP repo to create our dataset. We first create a a data folder. Then, we navigate to this folder. Finally, we run the colmap2nerf.py script with a specified video_fps and aabb_scale. The recommended number of frames is around 50-150 and the aabb_scale is the extent of the scene. The origin is the center of the aabb and aabb_scale is the length of each side of the box.

# Navigate to the repo.
cd /path/to/nerfactory

# Setup the environment variables.
nerfactory_DIR=`pwd`
DATASET_FORMAT=instant_ngp
DATASET_NAME=bear

# Create and navigate to the data folder.
mkdir data/$DATASET_FORMAT/$DATASET_NAME
cd data/$DATASET_FORMAT/$DATASET_NAME

# Run the COLMAP script.
python $nerfactory_DIR/scripts/colmap2nerf.py \
    --video_in /path/to/bear.mp4 \
    --video_fps 2 \
    --run_colmap \
    --aabb_scale 16

In the config set:

---
data_directory: data/instant_ngp/bear
dataset_format: instant_ngp