diff --git a/README.md b/README.md index 4d8a68954..33574e712 100644 --- a/README.md +++ b/README.md @@ -13,14 +13,13 @@

- -* [Quickstart](#quickstart) -* [Supported Features](#supported-features) -* [Benchmarked Model Architectures](#benchmarked-model-architectures) +- [Quickstart](#quickstart) +- [Supported Features](#supported-features) +- [Benchmarked Model Architectures](#benchmarked-model-architectures) # Quickstart -The quickstart will help you get started with the default vanilla nerf trained on the classic blender lego scene. +The quickstart will help you get started with the default vanilla nerf trained on the classic blender lego scene. For more complex changes (e.g. running with your own data/ setting up a new NeRF graph, please see our [docs](https://plenoptix-pyrad.readthedocs-hosted.com/en/latest/quickstart/quick_tour.html). #### 1. Installation: Setup the environment @@ -29,7 +28,7 @@ This repository is tested with cuda 11.3 ``` # Clone the repo -git clone --recurse-submodules git@github.com:ethanweber/pyrad.git +git clone --recurse-submodules git@github.com:plenoptix/pyrad.git # Create the python environment conda create --name pyrad python=3.8.13 @@ -61,12 +60,14 @@ Download the original [NeRF dataset](https://drive.google.com/drive/folders/128y #### 3. Training a model To run with all the defaults, e.g. vanilla nerf method with the blender lego image: + ``` # Run with default config python scripts/run_train.py ``` With support for [Hydra](https://hydra.cc/), you can run with other configurations by changing appropriate configs defined in `configs/` or by setting flags via command-line arguments: + ``` # Run with config changes python scripts/run_train.py machine_config.num_gpus=1 @@ -86,6 +87,7 @@ python scripts/run_train.py ++data.dataset.use_cache=true ``` #### 4. Visualizing training runs + If you run everything with the default configuration, by default, we use [TensorBoard](https://www.tensorflow.org/tensorboard) to log all training curves, test images, and other stats. Once the job is launched, you will be able to track training by launching the tensorboard in `outputs/blender_lego/vanilla_nerf//`. ``` @@ -93,12 +95,12 @@ tensorboard --logdir outputs/blender_lego/vanilla_nerf/ ``` #### 5. Rendering a trajectories during inference + TODO(ethan) - #### 6. In-depth guide -For a more in-depth tutorial on how to modify/implement your own NeRF Graph, please see our [walk-through](#). +For a more in-depth tutorial on how to modify/implement your own NeRF Graph, please see our [walk-through](#). # Supported Features @@ -120,7 +122,6 @@ If you are looking for a feature that is not currently supported, please do not #### :metal: Support for Jupyter - # Benchmarked Model Architectures | Method | PSNR | diff --git a/docs/reference/dataset.rst b/docs/reference/data.rst similarity index 69% rename from docs/reference/dataset.rst rename to docs/reference/data.rst index 62049e341..a8f50428f 100644 --- a/docs/reference/dataset.rst +++ b/docs/reference/data.rst @@ -1,15 +1,8 @@ .. _dataset: -Dataset +Data ============ -Colmap Utils ----------------- - -.. automodule:: pyrad.data.colmap_utils - :members: - :show-inheritance: - Image Dataset ---------------- @@ -17,6 +10,20 @@ Image Dataset :members: :show-inheritance: +Image Sampler +---------------- + +.. automodule:: pyrad.data.image_sampler + :members: + :show-inheritance: + +Pixel Sampler +---------------- + +.. automodule:: pyrad.data.pixel_sampler + :members: + :show-inheritance: + Structures ---------------- diff --git a/docs/reference/index.rst b/docs/reference/index.rst index 395ccb5dc..431430a5e 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -7,7 +7,7 @@ TODO: Explanation of each component .. toctree:: - dataset + data fields fields/modules/index graphs diff --git a/docs/tutorials/creating_dataset.md b/docs/tutorials/creating_dataset.md index 879992f09..6d60c0d0b 100644 --- a/docs/tutorials/creating_dataset.md +++ b/docs/tutorials/creating_dataset.md @@ -1,2 +1,96 @@ -# Setting up new data -TODO(ethan) \ No newline at end of file +# Using custom data + +#### Running COLMAP on your own images + +Please download [this file](https://github.com/NVlabs/instant-ngp/blob/07d8e2ca7232f97397ab73af9b56c7db639d3445/scripts/colmap2nerf.py) and place it at `scripts/colmap2nerf.py` courtesy of the [Instant-NGP](https://github.com/NVlabs/instant-ngp) authors. This script will use [COLMAP](https://github.com/colmap/colmap) to process images of a video file into a reconstruction with camera poses. You can see their write-up [here](https://github.com/NVlabs/instant-ngp/blob/master/docs/nerf_dataset_tips.md) for instructions. + +For example, maybe we have a video named `bear.mp4`. We can use the commands described in the Instant-NGP repo to create our dataset. We first create a a data folder. Then, we navigate to this folder. Finally, we run the colmap2nerf.py script with a specified `video_fps` and `aabb_scale`. The recommended number of frames is around 50-150 and the `aabb_scale` is the extent of the scene. The origin is the center of the aabb and `aabb_scale` is the length of each side of the box. + +```bash +# Navigate to the repo. +cd /path/to/pyrad + +# Setup the environment variables. +PYRAD_DIR=`pwd` +DATASET_FORMAT=instant_ngp +DATASET_NAME=bear + +# Create and navigate to the data folder. +mkdir data/$DATASET_FORMAT/$DATASET_NAME +cd data/$DATASET_FORMAT/$DATASET_NAME + +# Run the COLMAP script. +python $PYRAD_DIR/scripts/colmap2nerf.py \ + --video_in /path/to/bear.mp4 \ + --video_fps 2 \ + --run_colmap \ + --aabb_scale 16 +``` + +###### Referring to a specific dataset + +Here is how we refer to a dataset. We have a `dataset_inputs_train` and a `dataset_inputs_eval` config. These live under the `data` directory. You can see `configs/graph_default.yaml` for more details on what a full config would look like. + +```yaml +data: + dataset_inputs_train: + data_directory: data/instant_ngp/fox + dataset_format: instant_ngp + downscale_factor: 1 + alpha_color: white + dataset_inputs_eval: + data_directory: data/instant_ngp/fox + dataset_format: instant_ngp + downscale_factor: 1 + alpha_color: white +``` + +#### Sanity check with a notebook + +You can use `notebooks/visualize_dataset.ipynb` to see how your formatted dataset looks. This code will load a dataset from a yaml config, show the camera poses, and visualize rays with the specified collider in the config. TODO: add a better visualization with images too + +![visualize_dataset](media/creating_dataset_visualize_dataset.png) + +#### Handling a new dataset type + +In this page, we explain how to use pyRad with your own data. We've implemented many common datasets inside the `pyrad/format/.py` files. Each of these files implements a function called `load__data`. Each of these functions returns a `DatasetInputs` object. Furthermore, `get_dataset_inputs` in `pyrad/data/utils.py` will have to be updated to handle the new function. + +```python +# The DatasetInputs dataclass that is passed around to create datasets and initialize the graphs. +# See `pyrad/data/structs.py` for the code. +@dataclass +class DatasetInputs: + """Dataset inputs are used to initialize datasets and the NeRF graph.""" + + image_filenames: List[str] + downscale_factor: int = 1 + intrinsics: torch.tensor = None + camera_to_world: torch.tensor = None + mask_filenames: List[str] = None + depth_filenames: List[str] = None + scene_bounds: SceneBounds = SceneBounds() + semantics: Semantics = Semantics() + point_cloud: PointCloud = PointCloud() + alpha_color: Optional[TensorType[3]] = None + +# The method signature for the `instant_ngp` dataset format. +# See `pyrad/data/format/instant_ngp.py` for the code. +def load_instant_ngp_data( + basedir: str, + downscale_factor: int = 1, + split: str = "train", camera_translation_scalar=0.33 +) -> DatasetInputs: + """Returns a DatasetInputs struct.""" + +# The method signature for get_dataset_inputs. +# Notice that the parameters match the yaml config parameters above. +# See `pyrad/data/utils.py` for the code. +def get_dataset_inputs( + data_directory: str, + dataset_format: str, + split: str, + downscale_factor: int = 1, + alpha_color: Optional[Union[str, list, ListConfig]] = None, +) -> DatasetInputs: + """Makes a call to `load__data` and returns a DatasetInputs struct.""" +``` diff --git a/docs/tutorials/media/creating_dataset_visualize_dataset.png b/docs/tutorials/media/creating_dataset_visualize_dataset.png new file mode 100644 index 000000000..bc4338d47 Binary files /dev/null and b/docs/tutorials/media/creating_dataset_visualize_dataset.png differ diff --git a/setup.cfg b/setup.cfg index 61aec33b9..c7b36db34 100644 --- a/setup.cfg +++ b/setup.cfg @@ -1,7 +1,7 @@ [metadata] name = pyrad version = 0.0.0 -url = https://github.com/ethanweber/pyrad.git +url = https://github.com/plenoptix/pyrad.git description = differentiable volume rendering package long_description = file: README.md