initial dataset documentation (#36)

* initial dataset documentation

* data docs

* fixed repo name

* removed file

* fixed instruction

* adding bash

README.md

@@ -13,14 +13,13 @@
     <!-- TODO: add version number badge -->
 </p>
 
-
-* [Quickstart](#quickstart)
-* [Supported Features](#supported-features)
-* [Benchmarked Model Architectures](#benchmarked-model-architectures)
+- [Quickstart](#quickstart)
+- [Supported Features](#supported-features)
+- [Benchmarked Model Architectures](#benchmarked-model-architectures)
 
 # Quickstart
 
-The quickstart will help you get started with the default vanilla nerf trained on the classic blender lego scene. 
+The quickstart will help you get started with the default vanilla nerf trained on the classic blender lego scene.
 For more complex changes (e.g. running with your own data/ setting up a new NeRF graph, please see our [docs](https://plenoptix-pyrad.readthedocs-hosted.com/en/latest/quickstart/quick_tour.html).
 
 #### 1. Installation: Setup the environment
@@ -29,7 +28,7 @@ This repository is tested with cuda 11.3
 
 ```
 # Clone the repo
-git clone --recurse-submodules git@github.com:ethanweber/pyrad.git
+git clone --recurse-submodules git@github.com:plenoptix/pyrad.git
 
 # Create the python environment
 conda create --name pyrad python=3.8.13
@@ -61,12 +60,14 @@ Download the original [NeRF dataset](https://drive.google.com/drive/folders/128y
 #### 3. Training a model
 
 To run with all the defaults, e.g. vanilla nerf method with the blender lego image:
+
 ```
 # Run with default config
 python scripts/run_train.py
 ```
 
 With support for [Hydra](https://hydra.cc/), you can run with other configurations by changing appropriate configs defined in `configs/` or by setting flags via command-line arguments:
+
 ```
 # Run with config changes
 python scripts/run_train.py machine_config.num_gpus=1
@@ -86,6 +87,7 @@ python scripts/run_train.py ++data.dataset.use_cache=true
 ```
 
 #### 4. Visualizing training runs
+
 If you run everything with the default configuration, by default, we use [TensorBoard](https://www.tensorflow.org/tensorboard) to log all training curves, test images, and other stats. Once the job is launched, you will be able to track training by launching the tensorboard in `outputs/blender_lego/vanilla_nerf/<timestamp>/<events.tfevents>`.
 
 ```
@@ -93,12 +95,12 @@ tensorboard --logdir outputs/blender_lego/vanilla_nerf/
 ```
 
 #### 5. Rendering a trajectories during inference
+
 TODO(ethan)
 
-
 #### 6. In-depth guide
-For a more in-depth tutorial on how to modify/implement your own NeRF Graph, please see our [walk-through](#).
 
+For a more in-depth tutorial on how to modify/implement your own NeRF Graph, please see our [walk-through](#).
 
 # Supported Features
 
@@ -120,7 +122,6 @@ If you are looking for a feature that is not currently supported, please do not
 
 #### :metal: Support for Jupyter
 
-
 # Benchmarked Model Architectures
 
 | Method                                                                            | PSNR                     |

docs/reference/data.rst

@@ -1,15 +1,8 @@
 .. _dataset:
 
-Dataset
+Data
 ============
 
-Colmap Utils
-----------------
-
-.. automodule:: pyrad.data.colmap_utils
-   :members:
-   :show-inheritance:
-
 Image Dataset
 ----------------
 
@@ -17,6 +10,20 @@ Image Dataset
    :members:
    :show-inheritance:
 
+Image Sampler
+----------------
+
+.. automodule:: pyrad.data.image_sampler
+   :members:
+   :show-inheritance:
+
+Pixel Sampler
+----------------
+
+.. automodule:: pyrad.data.pixel_sampler
+   :members:
+   :show-inheritance:
+
 Structures
 ----------------
 

docs/reference/index.rst

@@ -7,7 +7,7 @@ TODO: Explanation of each component
 
 .. toctree::
 
-    dataset
+    data
     fields
     fields/modules/index
     graphs

docs/tutorials/creating_dataset.md

@@ -1,2 +1,96 @@
-# Setting up new data 
-TODO(ethan)
+# Using custom data
+
+#### Running COLMAP on your own images
+
+Please download [this file](https://github.com/NVlabs/instant-ngp/blob/07d8e2ca7232f97397ab73af9b56c7db639d3445/scripts/colmap2nerf.py) and place it at `scripts/colmap2nerf.py` courtesy of the [Instant-NGP](https://github.com/NVlabs/instant-ngp) authors. This script will use [COLMAP](https://github.com/colmap/colmap) to process images of a video file into a reconstruction with camera poses. You can see their write-up [here](https://github.com/NVlabs/instant-ngp/blob/master/docs/nerf_dataset_tips.md) for instructions.
+
+For example, maybe we have a video named `bear.mp4`. We can use the commands described in the Instant-NGP repo to create our dataset. We first create a a data folder. Then, we navigate to this folder. Finally, we run the colmap2nerf.py script with a specified `video_fps` and `aabb_scale`. The recommended number of frames is around 50-150 and the `aabb_scale` is the extent of the scene. The origin is the center of the aabb and `aabb_scale` is the length of each side of the box.
+
+```bash
+# Navigate to the repo.
+cd /path/to/pyrad
+
+# Setup the environment variables.
+PYRAD_DIR=`pwd`
+DATASET_FORMAT=instant_ngp
+DATASET_NAME=bear
+
+# Create and navigate to the data folder.
+mkdir data/$DATASET_FORMAT/$DATASET_NAME
+cd data/$DATASET_FORMAT/$DATASET_NAME
+
+# Run the COLMAP script.
+python $PYRAD_DIR/scripts/colmap2nerf.py \
+    --video_in /path/to/bear.mp4 \
+    --video_fps 2 \
+    --run_colmap \
+    --aabb_scale 16
+```
+
+###### Referring to a specific dataset
+
+Here is how we refer to a dataset. We have a `dataset_inputs_train` and a `dataset_inputs_eval` config. These live under the `data` directory. You can see `configs/graph_default.yaml` for more details on what a full config would look like.
+
+```yaml
+data:
+  dataset_inputs_train:
+    data_directory: data/instant_ngp/fox
+    dataset_format: instant_ngp
+    downscale_factor: 1
+    alpha_color: white
+  dataset_inputs_eval:
+    data_directory: data/instant_ngp/fox
+    dataset_format: instant_ngp
+    downscale_factor: 1
+    alpha_color: white
+```
+
+#### Sanity check with a notebook
+
+You can use `notebooks/visualize_dataset.ipynb` to see how your formatted dataset looks. This code will load a dataset from a yaml config, show the camera poses, and visualize rays with the specified collider in the config. TODO: add a better visualization with images too
+
+![visualize_dataset](media/creating_dataset_visualize_dataset.png)
+
+#### Handling a new dataset type
+
+In this page, we explain how to use pyRad with your own data. We've implemented many common datasets inside the `pyrad/format/<dataset_format>.py` files. Each of these files implements a function called `load_<dataset_format>_data`. Each of these functions returns a `DatasetInputs` object. Furthermore, `get_dataset_inputs` in `pyrad/data/utils.py` will have to be updated to handle the new function.
+
+```python
+# The DatasetInputs dataclass that is passed around to create datasets and initialize the graphs.
+# See `pyrad/data/structs.py` for the code.
+@dataclass
+class DatasetInputs:
+    """Dataset inputs are used to initialize datasets and the NeRF graph."""
+
+    image_filenames: List[str]
+    downscale_factor: int = 1
+    intrinsics: torch.tensor = None
+    camera_to_world: torch.tensor = None
+    mask_filenames: List[str] = None
+    depth_filenames: List[str] = None
+    scene_bounds: SceneBounds = SceneBounds()
+    semantics: Semantics = Semantics()
+    point_cloud: PointCloud = PointCloud()
+    alpha_color: Optional[TensorType[3]] = None
+
+# The method signature for the `instant_ngp` dataset format.
+# See `pyrad/data/format/instant_ngp.py` for the code.
+def load_instant_ngp_data(
+    basedir: str,
+    downscale_factor: int = 1,
+    split: str = "train", camera_translation_scalar=0.33
+) -> DatasetInputs:
+    """Returns a DatasetInputs struct."""
+
+# The method signature for get_dataset_inputs.
+# Notice that the parameters match the yaml config parameters above.
+# See `pyrad/data/utils.py` for the code.
+def get_dataset_inputs(
+    data_directory: str,
+    dataset_format: str,
+    split: str,
+    downscale_factor: int = 1,
+    alpha_color: Optional[Union[str, list, ListConfig]] = None,
+) -> DatasetInputs:
+    """Makes a call to `load_<dataset_format>_data` and returns a DatasetInputs struct."""
+```

setup.cfg

@@ -1,7 +1,7 @@
 [metadata]
 name = pyrad
 version = 0.0.0
-url = https://github.com/ethanweber/pyrad.git
+url = https://github.com/plenoptix/pyrad.git
 description = differentiable volume rendering package
 long_description = file: README.md