Files
nerfstudio/colab/demo.ipynb
T
2023-10-20 15:00:03 -07:00

12 KiB

Open In Colab

nerfstudio

Nerfstudio: A collaboration friendly studio for NeRFs

GitHub stars

This colab shows how to train and view NeRFs from Nerfstudio both on pre-made datasets or from your own videos/images.

Credit to NeX for Google Colab format.

Frequently Asked Questions

  • Downloading custom data is stalling (no output):
    • This is a bug in Colab. The data is processing, but may take a while to complete. You will know processing completed if data/nerfstudio/custom_data/transforms.json exists. Terminating the cell early will result in not being able to train.
  • Processing custom data is taking a long time:
    • The time it takes to process data depends on the number of images and its resolution. If processing is taking too long, try lowering the resolution of your custom data.
  • Error: Data processing did not complete:
    • This means that the data processing script did not fully complete. This could be because there were not enough images, or that the images were of low quality. We recommend images with little to no motion blur and lots of visual overlap of the scene to increase the chances of successful processing.
  • Training is not showing progress:
    • The lack of output is a bug in Colab. You can see the training progress from the viewer.
  • Viewer Quality is bad / Low resolution:
    • This may be because more GPU is being used on training that rendering the viewer. Try pausing training or decreasing training utilization.
  • WARNING: Running pip as the 'root' user...::
    • This and other pip warnings or errors can be safely ignored.
  • Other problems?

Nerfstudio Demo

In [ ]:
#@markdown <h3>Install Nerfstudio and Dependencies (~8 min)</h3>

%cd /content/
!pip install --upgrade pip
!pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118

# Installing TinyCuda
%cd /content/
!gdown "https://drive.google.com/u/1/uc?id=1-7x7qQfB7bIw2zV4Lr6-yhvMpjXC84Q5&confirm=t"
!pip install tinycudann-1.7-cp310-cp310-linux_x86_64.whl

# Install nerfstudio
%cd /content/
!pip install git+https://github.com/nerfstudio-project/nerfstudio.git
In [ ]:
#@markdown <h3>Install COLMAP: Necessary for custom dataset (~15 min)</h3>
#@markdown <medium>Skip to the next cell if running on demo data.<br/medium>
#@markdown <medium>Run this cell only if you are uploading your own data. Installation takes a while to complete.<br/medium>

# Installing COLMAP via conda
%cd /content
!conda install -c conda-forge colmap
In [ ]:
#@markdown <h3> Downloading and Processing Data</h3>
#@markdown <medium>Pick the preset scene or upload your own images/video</medium>

import os
import glob
from google.colab import files
from IPython.core.display import display, HTML

scene = '\uD83D\uDDBC poster' #@param ['🖼 poster', '🚜 dozer', '🌄 desolation', '📤 upload your images' , '🎥 upload your own video', '🔺 upload Polycam data', '💽 upload your own Record3D data']
scene = ' '.join(scene.split(' ')[1:])

if scene == "upload Polycam data":
    %cd /content/
    !mkdir -p /content/data/nerfstudio/custom_data
    %cd /content/data/nerfstudio/custom_data/
    uploaded = files.upload()
    dir = os.getcwd()
    if len(uploaded.keys()) > 1:
        print("ERROR, upload a single .zip file when processing Polycam data")
    dataset_dir = [os.path.join(dir, f) for f in uploaded.keys()][0]
    !ns-process-data polycam --data $dataset_dir --output-dir /content/data/nerfstudio/custom_data/
    scene = "custom_data"
elif scene == 'upload your own Record3D data':
    display(HTML('<h3>Zip your Record3D folder, and upload.</h3>'))
    display(HTML('<h3>More information on Record3D can be found <a href="https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html#record3d-capture" target="_blank">here</a>.</h3>'))
    %cd /content/
    !mkdir -p /content/data/nerfstudio/custom_data
    %cd /content/data/nerfstudio/custom_data/
    uploaded = files.upload()
    dir = os.getcwd()
    preupload_datasets = [os.path.join(dir, f) for f in uploaded.keys()]
    record_3d_zipfile = preupload_datasets[0]
    !unzip $record_3d_zipfile -d /content/data/nerfstudio/custom_data
    custom_data_directory = glob.glob('/content/data/nerfstudio/custom_data/*')[0]
    !ns-process-data record3d --data $custom_data_directory --output-dir /content/data/nerfstudio/custom_data/
    scene = "custom_data"
elif scene in ['upload your images', 'upload your own video']:
    display(HTML('<h3>Select your custom data</h3>'))
    display(HTML('<p/>You can select multiple images by pressing ctrl, cmd or shift and click.<p>'))
    display(HTML('<p/>Note: This may take time, especially on higher resolution inputs, so we recommend to download dataset after creation.<p>'))
    !mkdir -p /content/data/nerfstudio/custom_data
    if scene == 'upload your images':
        !mkdir -p /content/data/nerfstudio/custom_data/raw_images
        %cd /content/data/nerfstudio/custom_data/raw_images
        uploaded = files.upload()
        dir = os.getcwd()
    else:
        %cd /content/data/nerfstudio/custom_data/
        uploaded = files.upload()
        dir = os.getcwd()
    preupload_datasets = [os.path.join(dir, f) for f in uploaded.keys()]
    del uploaded
    %cd /content/

    if scene == 'upload your images':
        !ns-process-data images --data /content/data/nerfstudio/custom_data/raw_images --output-dir /content/data/nerfstudio/custom_data/
    else:
        video_path = preupload_datasets[0]
        !ns-process-data video --data $video_path --output-dir /content/data/nerfstudio/custom_data/

    scene = "custom_data"
else:
    %cd /content/
    !ns-download-data nerfstudio --capture-name=$scene

print("Data Processing Succeeded!")
In [ ]:
#@markdown <h2>Start Training</h2>
#@markdown <big>Open the generated share url to access viewer.<br/big>

%cd /content
if os.path.exists(f"data/nerfstudio/{scene}/transforms.json"):
    !ns-train nerfacto --viewer.websocket-port 7007 --vis viewer_beta --viewer.make-share-url True nerfstudio-data --data data/nerfstudio/$scene --downscale-factor 4
else:
    from IPython.core.display import display, HTML
    display(HTML('<h3 style="color:red">Error: Data processing did not complete</h3>'))
    display(HTML('<h3>Please re-run `Downloading and Processing Data`, or view the FAQ for more info.</h3>'))
In [ ]:
#@title # Render Video { vertical-output: true }
#@markdown <medium>Export the camera path from within the viewer, then run this cell.<br/medium>
#@markdown <medium>The rendered video should be at renders/output.mp4!</mediun>

base_dir = "/content/outputs/unnamed/nerfacto/"
training_run_dir = base_dir + os.listdir(base_dir)[0]

from IPython.core.display import display, HTML
display(HTML('<h3>Upload the camera path JSON.</h3>'))
%cd $training_run_dir
uploaded = files.upload()
uploaded_camera_path_filename = list(uploaded.keys())[0]

config_filename = training_run_dir + "/config.yml"
camera_path_filename = training_run_dir + "/" + uploaded_camera_path_filename
camera_path_filename = camera_path_filename.replace(" ", "\\ ").replace("(", "\\(").replace(")", "\\)")

%cd /content/
!ns-render camera-path --load-config $config_filename --camera-path-filename $camera_path_filename --output-path renders/output.mp4