nerfstudio/docs/tutorials/pipelines/dataparsers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DataParsers\n",
    "\n",
    "```{image} imgs/dataparsers.png\n",
    ":align: center\n",
    "```\n",
    "\n",
    "## What is a DataParser?\n",
    "\n",
    "The dataparser returns `DataparserOutputs`, which puts all the various datasets into a common format.\n",
    "\n",
    "```python\n",
    "@dataclass\n",
    "class DataparserOutputs:\n",
    "    \"\"\"Dataset inputs for the image dataset and the ray generator.\"\"\"\n",
    "\n",
    "    image_filenames: List[Path]\n",
    "    \"\"\"Filenames for the images.\"\"\"\n",
    "    cameras: Cameras\n",
    "    \"\"\"Camera object storing collection of camera information in dataset\"\"\"\n",
    "    alpha_color: Optional[TensorType[3]] = None\n",
    "    \"\"\"color of dataset background\"\"\"\n",
    "    scene_box: SceneBox = SceneBox()\n",
    "    \"\"\"scene box of dataset. Could be used to bound the scene, or to provide the scene scale depending on model.\"\"\"\n",
    "    ...\n",
    "\n",
    "@dataclass\n",
    "class DataParser:\n",
    "\n",
    "    @abstractmethod\n",
    "    def _generate_dataset_inputs(self, split: str = \"train\") -> DataparserOutputs:\n",
    "        \"\"\"Abstract method that returns the dataparser outputs for the given split.\n",
    "\n",
    "        Args:\n",
    "            split: Which dataset split to generate data for (train/eval/test).\n",
    "\n",
    "        Returns:\n",
    "            DataparserOutputs for the specified dataset and split.\n",
    "        \"\"\"\n",
    "```\n",
    "\n",
    "## Example\n",
    "\n",
    "Here is an example where we implement a DataParser for our Nerfstudio data format.\n",
    "\n",
    "```python\n",
    "@dataclass\n",
    "class NerfstudioDataParserConfig(DataParserConfig):\n",
    "    \"\"\"Nerfstudio dataset config\"\"\"\n",
    "\n",
    "    _target: Type = field(default_factory=lambda: Nerfstudio)\n",
    "    \"\"\"Target class to instantiate.\"\"\"\n",
    "    data_directory: Path = Path(\"data/nerfstudio/poster\")\n",
    "    \"\"\"Directory specifying location of data.\"\"\"\n",
    "    scale_factor: float = 1.0\n",
    "    \"\"\"How much to scale the camera origins by.\"\"\"\n",
    "    downscale_factor: int = 1\n",
    "    \"\"\"How much to downscale images.\"\"\"\n",
    "    scene_scale: float = 1.0\n",
    "    \"\"\"How much to scale the region of interest by.\"\"\"\n",
    "    orientation_method: Literal[\"pca\", \"up\"] = \"up\"\n",
    "    \"\"\"The method to use for orientation.\"\"\"\n",
    "    train_split_percentage: float = 0.9\n",
    "    \"\"\"The percent of images to use for training. The remaining images are for eval.\n",
    "    \"\"\"\n",
    "\n",
    "@dataclass\n",
    "class Nerfstudio(DataParser):\n",
    "    \"\"\"Nerfstudio Dataset\"\"\"\n",
    "\n",
    "    config: NerfstudioDataParserConfig\n",
    "\n",
    "    def _generate_dataset_inputs(self, split=\"train\") -> DataParserOutputs:\n",
    "      \"\"\"Returns DataParserOutputs.\"\"\"\n",
    "```\n",
    "\n",
    "## Our Implementations\n",
    "\n",
    "###### Nerfstudio\n",
    "\n",
    "This is our custom dataparser.\n",
    "\n",
    "```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/nerfstudio_dataparser.py\n",
    ":color: primary\n",
    ":outline:\n",
    "See the code!\n",
    "```\n",
    "\n",
    "###### Blender\n",
    "\n",
    "We support the synthetic Blender dataset from the original NeRF paper.\n",
    "\n",
    "```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/blender_dataparser.py\n",
    ":color: primary\n",
    ":outline:\n",
    "See the code!\n",
    "```\n",
    "\n",
    "###### Instant NGP\n",
    "\n",
    "This supports the Instant NGP dataset.\n",
    "\n",
    "```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/instant_ngp_dataparser.py\n",
    ":color: primary\n",
    ":outline:\n",
    "See the code!\n",
    "```\n",
    "\n",
    "###### MipNeRF 360\n",
    "\n",
    "```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/mipnerf_dataparser.py\n",
    ":color: primary\n",
    ":outline:\n",
    "See the code!\n",
    "```\n",
    "\n",
    "###### Record3D\n",
    "\n",
    "This dataparser can use recorded data from a >= iPhone 12 Pro using the Record3D app. Record a video and export with the `EXR + JPG sequence` format. Unzip export and `rgb` folder before training.\n",
    "\n",
    "```{button-link} https://github.com/nerfstudio-project/nerfstudio/blob/master/nerfstudio/data/dataparsers/record3d_dataparser.py\n",
    ":color: primary\n",
    ":outline:\n",
    "See the code!\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.13 ('nerfactory')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.8.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "b7b86c17c0dfd6b313529972d424f2e075cc198a46cda005757126afe8133c7b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}