From ca09acece2a49ec2d8251b41b85bd0dbf656832a Mon Sep 17 00:00:00 2001 From: evonneng Date: Mon, 12 Sep 2022 11:58:13 -0700 Subject: [PATCH] updated readme and docs with promo language (#401) * checking emojis * restructured docs * updated links * updated links * small nit --- README.md | 107 ++++++++++++------ .../incrementally_add_cameras.ipynb | 0 .../dataloader_components/index.md | 0 .../pixel_sampling_strategies.ipynb | 0 docs/guides/index.rst | 15 +++ .../model_components/imgs/frustums-dark.png | Bin .../model_components/imgs/frustums.png | Bin .../imgs/samplers_stratified-dark.png | Bin .../imgs/samplers_stratified-light.png | Bin .../imgs/samplers_type-dark.png | Bin .../imgs/samplers_type-light.png | Bin docs/{ => guides}/model_components/index.md | 0 .../model_components/visualize_cameras.ipynb | 0 .../model_components/visualize_encoders.ipynb | 0 .../model_components/visualize_samplers.ipynb | 10 +- .../model_components/visualize_samples.ipynb | 0 .../visualize_spatial_distortions.ipynb | 0 .../mipnerf/models_mipnerf_field-dark.png | Bin .../mipnerf/models_mipnerf_field-light.png | Bin .../mipnerf/models_mipnerf_pipeline-dark.png | Bin .../mipnerf/models_mipnerf_pipeline-light.png | Bin .../models/imgs/models_nerf-field-dark.png | Bin .../models/imgs/models_nerf-field-light.png | Bin .../models/imgs/models_nerf-pipeline-dark.png | Bin .../imgs/models_nerf-pipeline-field-dark.png | Bin .../imgs/models_nerf-pipeline-field-light.png | Bin .../imgs/models_nerf-pipeline-light.png | Bin .../models_nerf-pipeline-renderer-dark.png | Bin .../models_nerf-pipeline-renderer-light.png | Bin .../models_nerf-pipeline-sampler-dark.png | Bin .../models_nerf-pipeline-sampler-light.png | Bin docs/{ => guides}/models/index.md | 0 docs/{ => guides}/models/instant_ngp.md | 0 docs/{ => guides}/models/mipnerf.md | 0 docs/{ => guides}/models/mipnerf_360.md | 0 docs/{ => guides}/models/nerf.md | 2 +- docs/{ => guides}/models/nerf_w.md | 0 docs/{ => guides}/models/semantic_nerf.md | 0 docs/index.rst | 4 +- docs/tooling/benchmarking.md | 3 +- docs/tooling/config.md | 1 + docs/tooling/hydra.md | 1 - docs/tooling/index.rst | 2 +- docs/tooling/logging_profiling.md | 7 -- docs/tutorials/viewer/for_developers.md | 14 +-- 45 files changed, 105 insertions(+), 61 deletions(-) rename docs/{ => guides}/dataloader_components/incrementally_add_cameras.ipynb (100%) rename docs/{ => guides}/dataloader_components/index.md (100%) rename docs/{ => guides}/dataloader_components/pixel_sampling_strategies.ipynb (100%) create mode 100644 docs/guides/index.rst rename docs/{ => guides}/model_components/imgs/frustums-dark.png (100%) rename docs/{ => guides}/model_components/imgs/frustums.png (100%) rename docs/{ => guides}/model_components/imgs/samplers_stratified-dark.png (100%) rename docs/{ => guides}/model_components/imgs/samplers_stratified-light.png (100%) rename docs/{ => guides}/model_components/imgs/samplers_type-dark.png (100%) rename docs/{ => guides}/model_components/imgs/samplers_type-light.png (100%) rename docs/{ => guides}/model_components/index.md (100%) rename docs/{ => guides}/model_components/visualize_cameras.ipynb (100%) rename docs/{ => guides}/model_components/visualize_encoders.ipynb (100%) rename docs/{ => guides}/model_components/visualize_samplers.ipynb (99%) rename docs/{ => guides}/model_components/visualize_samples.ipynb (100%) rename docs/{ => guides}/model_components/visualize_spatial_distortions.ipynb (100%) rename docs/{ => guides}/models/imgs/mipnerf/models_mipnerf_field-dark.png (100%) rename docs/{ => guides}/models/imgs/mipnerf/models_mipnerf_field-light.png (100%) rename docs/{ => guides}/models/imgs/mipnerf/models_mipnerf_pipeline-dark.png (100%) rename docs/{ => guides}/models/imgs/mipnerf/models_mipnerf_pipeline-light.png (100%) rename docs/{ => guides}/models/imgs/models_nerf-field-dark.png (100%) rename docs/{ => guides}/models/imgs/models_nerf-field-light.png (100%) rename docs/{ => guides}/models/imgs/models_nerf-pipeline-dark.png (100%) rename docs/{ => guides}/models/imgs/models_nerf-pipeline-field-dark.png (100%) rename docs/{ => guides}/models/imgs/models_nerf-pipeline-field-light.png (100%) rename docs/{ => guides}/models/imgs/models_nerf-pipeline-light.png (100%) rename docs/{ => guides}/models/imgs/models_nerf-pipeline-renderer-dark.png (100%) rename docs/{ => guides}/models/imgs/models_nerf-pipeline-renderer-light.png (100%) rename docs/{ => guides}/models/imgs/models_nerf-pipeline-sampler-dark.png (100%) rename docs/{ => guides}/models/imgs/models_nerf-pipeline-sampler-light.png (100%) rename docs/{ => guides}/models/index.md (100%) rename docs/{ => guides}/models/instant_ngp.md (100%) rename docs/{ => guides}/models/mipnerf.md (100%) rename docs/{ => guides}/models/mipnerf_360.md (100%) rename docs/{ => guides}/models/nerf.md (98%) rename docs/{ => guides}/models/nerf_w.md (100%) rename docs/{ => guides}/models/semantic_nerf.md (100%) create mode 100644 docs/tooling/config.md delete mode 100644 docs/tooling/hydra.md diff --git a/README.md b/README.md index f43611e4c..5b37038bb 100644 --- a/README.md +++ b/README.md @@ -36,12 +36,28 @@

- [Quickstart](#quickstart) +- [Learn more](#learn-more) - [Supported Features](#supported-features) +- [See what's possible](#see-whats-possible) + + +# Philosophy +All-in-one repository for state-of-the-art NeRFs. + +Nerfactory provides a simple API that allows for a seamless and simplified end-to-end process of creating, training, and visualizing NeRFs. The library supports a **more interpretable implementation of NeRFs by modularizing each component**. With more modular NeRFs, not only does your code become far more user-friendly, but using this framework also makes it easier for the community to build upon your implementation. + +It’s as simple as plug and play with Nerfactory! + +Ontop of our API, we are commited to providing learning resources to help you understand the basics of (if you're just getting start), and keep up-to-date with (if you're a seasoned veteran) all things NeRF. As researchers, we know just how hard it is to get onboarded with this next-gen technology. So we're here to help with tutorials, documentation, and more! + +Finally, have feature requests? Want to add your brand-spankin'-new NeRF model? Have a new dataset? We welcome any and all contributions! + +We hope Nerfactory enables you to build faster :hammer: learn together :books: and contribute to our NeRF community :sparkling_heart:. # Quickstart The quickstart will help you get started with the default vanilla nerf trained on the classic blender lego scene. -For more complex changes (e.g. running with your own data/ setting up a new NeRF graph, please see our [docs](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/quickstart/quick_tour.html). +For more complex changes (e.g. running with your own data/ setting up a new NeRF graph, please refer to our [references](#learn-more). #### 1. Installation: Setup the environment @@ -50,42 +66,51 @@ This repository is tested with CUDA 11.3. Make sure to install [Conda](https://d
Installing Conda - This step is fairly self-explanatory, but here are the basic steps. You can also find countless tutorials online. +This step is fairly self-explanatory, but here are the basic steps. You can also find countless tutorials online. - ``` - cd /path/to/install/miniconda +``` +cd /path/to/install/miniconda - mkdir -p miniconda3 - wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda3/miniconda.sh - bash miniconda3/miniconda.sh -b -u -p miniconda3 - rm -rf miniconda/miniconda.sh - ``` +mkdir -p miniconda3 +wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda3/miniconda.sh +bash miniconda3/miniconda.sh -b -u -p miniconda3 +rm -rf miniconda/miniconda.sh +```
+Create the python environment ``` -# Create the python environment conda create --name nerfactory python=3.8.13 conda activate nerfactory +``` -# Clone the repo +Clone the repo +``` git clone git@github.com:plenoptix/nerfactory.git - -# Install dependencies and nerfactory as a library cd nerfactory +``` + +Install dependencies and nerfactory as a library +``` python -m pip install --upgrade pip pip install -e . +``` -# Install tiny-cuda-nn (tcnn) and apex to use with the graph_instant_ngp.yaml config +Install tiny-cuda-nn (tcnn) to run instant_ngp +``` +pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch +``` -# Run the test cases +Run the test cases +``` pytest tests ``` #### 2. Getting the data -Download the original [NeRF dataset](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) and unfold it in the following format. This is for the blender dataset type. We support the major datasets and allow users to create their own dataset, described in detail [here](docs/tutorials/data_setup.rst). +Download the original [NeRF dataset](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) and unfold it in the following format. This is for the blender dataset type. We support the major datasets and allow users to create their own dataset, described in detail [here](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/tutorials/data/index.html). ``` |─ nerfactory/ @@ -134,22 +159,24 @@ python scripts/run_train.py instant_ngp --viewer.enable --viewer.no-launch-bridg #### 4. Visualizing training runs -We support multiple methods to visualize training, the default configuration uses Tensorboard. More information on logging can be found [here](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/tooling/logging.html). +We support multiple methods to visualize training, the default configuration uses Tensorboard. More information on logging can be found [here](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/tooling/logging_profiling.html).
Real-time Viewer -We have developed our own Real-time web viewer, more information can be found [here](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/tooling/viewer.html). This viewer runs during training and is designed to work with models that have fast rendering pipelines. +We have developed our own Real-time web viewer, more information can be found [here](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/tutorials/viewer/index.html). This viewer runs during training and is designed to work with models that have fast rendering pipelines. + +To turn on the viewer, simply add the flag `--viewer.enable`.
Tensorboard -If you run everything with the default configuration we log all training curves, test images, and other stats. Once the job is launched, you will be able to track training by launching the tensorboard in `outputs/blender_lego/vanilla_nerf//`. +If you run everything with the default configuration we log all training curves, test images, and other stats. Once the job is launched, you will be able to track training by launching the tensorboard in your base experiment directory (Default: `outputs/`). ```bash -tensorboard --logdir outputs +tensorboard --logdir outputs/ ```
@@ -157,13 +184,7 @@ tensorboard --logdir outputs
Weights & Biases -We support logging to weights and biases, to enable add the following to the config: - -``` -logging: - writer: - WandbWriter -``` +We support logging to weights and biases. To enable wandb logging, add the flag `--logging.writer.1.enable`.
@@ -175,18 +196,36 @@ python scripts/run_eval.py render-trajectory --load-config=outputs/blender_lego/ #### 6. In-depth guide -For a more in-depth tutorial on how to modify/implement your own NeRF Graph, please see our [walk-through](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/tutorials/creating_graphs.html). +For a more in-depth tutorial on how to modify/implement your own NeRF Graph, please see our [walk-through](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/tutorials/pipelines/index.html). + + +# Learn More +| Section | Description | +|-|-| +| [Documentation](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/) | Full API documentation and tutorials | +| [Interactive Guides](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/guides/index.html) | Go-to spot for learning how NeRFs and each of its modules work. +| [Quick tour](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/tutorials/quickstart_index.html) | Example script on how to navigate Nerfactory from install, train, to test. +| [Creating pipelines](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/tutorials/pipelines/index.html) | Learn how to easily build new neural rendering pipelines by using and/or implementing new modules. +| [Creating datsets](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/tutorials/data/index.html) | Have a new dataset? Learn how to use it with Nerfactory. +| [Mobile Capture to NerF](#) | Step-by-step tutorial on how to create beautiful renders with just your phone. +| [Contributing](https://plenoptix-nerfactory.readthedocs-hosted.com/en/latest/reference/contributing.html) | Walk-through for how you can start contributing now. # Supported Features We provide the following support strucutures to make life easier for getting started with NeRFs. For a full description, please refer to our [features page](#). -If you are looking for a feature that is not currently supported, please do not hesitate to contact the Plenoptix team! +**If you are looking for a feature that is not currently supported, please do not hesitate to contact the Plenoptix team!** -#### :metal: Support for multiple logging interfaces +* :mag_right: Web-based visualizer that allows you to: + * Visualize training in real-time + interact with the scene + * Create and render out scenes with custom camera trajectories + * View different output types + * And more! +* :pencil2: Support for multiple logging interfaces (Tensorboard, Wandb), code profiling, and other built-in debugging tools +* :chart_with_upwards_trend: Easy-to-use benchmarking scripts on the Blender dataset +* :iphone: Full pipeline support (w/ Colmap or Record3D) for going from a video on your phone to a full 3D render. Follow our step-by-step tutorial. (TODO: walk-through page on end-to-end pipeline from capture -> render) -#### :metal: Built-in support for profiling code -#### :metal: Benchmarking scripts - -#### :metal: Speed up your code with Tiny Cuda NN +# See what's possible +TODO: insert some gallery stuff here (gifs/pretty pictures w/ visualizer) +TODO: For more see gallery \ No newline at end of file diff --git a/docs/dataloader_components/incrementally_add_cameras.ipynb b/docs/guides/dataloader_components/incrementally_add_cameras.ipynb similarity index 100% rename from docs/dataloader_components/incrementally_add_cameras.ipynb rename to docs/guides/dataloader_components/incrementally_add_cameras.ipynb diff --git a/docs/dataloader_components/index.md b/docs/guides/dataloader_components/index.md similarity index 100% rename from docs/dataloader_components/index.md rename to docs/guides/dataloader_components/index.md diff --git a/docs/dataloader_components/pixel_sampling_strategies.ipynb b/docs/guides/dataloader_components/pixel_sampling_strategies.ipynb similarity index 100% rename from docs/dataloader_components/pixel_sampling_strategies.ipynb rename to docs/guides/dataloader_components/pixel_sampling_strategies.ipynb diff --git a/docs/guides/index.rst b/docs/guides/index.rst new file mode 100644 index 000000000..7ad957e8d --- /dev/null +++ b/docs/guides/index.rst @@ -0,0 +1,15 @@ +Interactive Guides +------------------ + +Want to learn more about NeRFs and how they work? You're at the right spot! + +Here, we decompose each of the pre-implemented NeRF modules- from how the data's loaded to how the model operates-in an easy to understand way. +Interact with the notebooks to gain a visual intuition for each component. +Check back in regularly as the technology continues to develop with new modules! + +.. toctree:: + :maxdepth: 1 + + models/index + model_components/index + dataloader_components/index \ No newline at end of file diff --git a/docs/model_components/imgs/frustums-dark.png b/docs/guides/model_components/imgs/frustums-dark.png similarity index 100% rename from docs/model_components/imgs/frustums-dark.png rename to docs/guides/model_components/imgs/frustums-dark.png diff --git a/docs/model_components/imgs/frustums.png b/docs/guides/model_components/imgs/frustums.png similarity index 100% rename from docs/model_components/imgs/frustums.png rename to docs/guides/model_components/imgs/frustums.png diff --git a/docs/model_components/imgs/samplers_stratified-dark.png b/docs/guides/model_components/imgs/samplers_stratified-dark.png similarity index 100% rename from docs/model_components/imgs/samplers_stratified-dark.png rename to docs/guides/model_components/imgs/samplers_stratified-dark.png diff --git a/docs/model_components/imgs/samplers_stratified-light.png b/docs/guides/model_components/imgs/samplers_stratified-light.png similarity index 100% rename from docs/model_components/imgs/samplers_stratified-light.png rename to docs/guides/model_components/imgs/samplers_stratified-light.png diff --git a/docs/model_components/imgs/samplers_type-dark.png b/docs/guides/model_components/imgs/samplers_type-dark.png similarity index 100% rename from docs/model_components/imgs/samplers_type-dark.png rename to docs/guides/model_components/imgs/samplers_type-dark.png diff --git a/docs/model_components/imgs/samplers_type-light.png b/docs/guides/model_components/imgs/samplers_type-light.png similarity index 100% rename from docs/model_components/imgs/samplers_type-light.png rename to docs/guides/model_components/imgs/samplers_type-light.png diff --git a/docs/model_components/index.md b/docs/guides/model_components/index.md similarity index 100% rename from docs/model_components/index.md rename to docs/guides/model_components/index.md diff --git a/docs/model_components/visualize_cameras.ipynb b/docs/guides/model_components/visualize_cameras.ipynb similarity index 100% rename from docs/model_components/visualize_cameras.ipynb rename to docs/guides/model_components/visualize_cameras.ipynb diff --git a/docs/model_components/visualize_encoders.ipynb b/docs/guides/model_components/visualize_encoders.ipynb similarity index 100% rename from docs/model_components/visualize_encoders.ipynb rename to docs/guides/model_components/visualize_encoders.ipynb diff --git a/docs/model_components/visualize_samplers.ipynb b/docs/guides/model_components/visualize_samplers.ipynb similarity index 99% rename from docs/model_components/visualize_samplers.ipynb rename to docs/guides/model_components/visualize_samplers.ipynb index 08b2a7853..f4d82d203 100644 --- a/docs/model_components/visualize_samplers.ipynb +++ b/docs/guides/model_components/visualize_samplers.ipynb @@ -8,7 +8,7 @@ "\n", "## Overview\n", "\n", - "Once we have a set of cameras, we want to cast camera rays associated with each pixel. Along these ray we will sample the _field_ and aggregate the samples to predict the pixels value (ie. color). The parameterization of the samples are described [here](./visualize_samples.ipynb) however we must decide where to place these samples along a ray. For this task we will use a [`Sampler`](../reference/api/graphs/modules/ray_sampler.rst).\n", + "Once we have a set of cameras, we want to cast camera rays associated with each pixel. Along these ray we will sample the _field_ and aggregate the samples to predict the pixels value (ie. color). The parameterization of the samples are described [here](./visualize_samples.ipynb) however we must decide where to place these samples along a ray. For this task we will use a [`Sampler`](../../reference/api/graphs/modules/ray_sampler.rst).\n", "\n", "In the ideal world we would compute many dense samples along a ray. Unfortunatly each additional sample adds a computation cost to the system as it needs to be processed by the _field_ which is often a neural network. As a result it is common for NeRF methods to use on the order of 100 samples. Therefore we want to optimize where those samples are placed in the scene.\n", "\n", @@ -7289,7 +7289,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3.8.12 64-bit ('nerfactory')", + "display_name": "Python 3.8.13 ('nerfactory')", "language": "python", "name": "python3" }, @@ -7303,14 +7303,14 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.8.12" + "version": "3.8.13" }, "vscode": { "interpreter": { - "hash": "3271154905f6ca90bb0c3951bd3f2265caaa6ab68492eb831b8ba5f0e8deab4d" + "hash": "100252972d4ee5286ab455ef1c778abde05b4affca9d1be78633d3797abf7e8c" } } }, "nbformat": 4, "nbformat_minor": 2 -} \ No newline at end of file +} diff --git a/docs/model_components/visualize_samples.ipynb b/docs/guides/model_components/visualize_samples.ipynb similarity index 100% rename from docs/model_components/visualize_samples.ipynb rename to docs/guides/model_components/visualize_samples.ipynb diff --git a/docs/model_components/visualize_spatial_distortions.ipynb b/docs/guides/model_components/visualize_spatial_distortions.ipynb similarity index 100% rename from docs/model_components/visualize_spatial_distortions.ipynb rename to docs/guides/model_components/visualize_spatial_distortions.ipynb diff --git a/docs/models/imgs/mipnerf/models_mipnerf_field-dark.png b/docs/guides/models/imgs/mipnerf/models_mipnerf_field-dark.png similarity index 100% rename from docs/models/imgs/mipnerf/models_mipnerf_field-dark.png rename to docs/guides/models/imgs/mipnerf/models_mipnerf_field-dark.png diff --git a/docs/models/imgs/mipnerf/models_mipnerf_field-light.png b/docs/guides/models/imgs/mipnerf/models_mipnerf_field-light.png similarity index 100% rename from docs/models/imgs/mipnerf/models_mipnerf_field-light.png rename to docs/guides/models/imgs/mipnerf/models_mipnerf_field-light.png diff --git a/docs/models/imgs/mipnerf/models_mipnerf_pipeline-dark.png b/docs/guides/models/imgs/mipnerf/models_mipnerf_pipeline-dark.png similarity index 100% rename from docs/models/imgs/mipnerf/models_mipnerf_pipeline-dark.png rename to docs/guides/models/imgs/mipnerf/models_mipnerf_pipeline-dark.png diff --git a/docs/models/imgs/mipnerf/models_mipnerf_pipeline-light.png b/docs/guides/models/imgs/mipnerf/models_mipnerf_pipeline-light.png similarity index 100% rename from docs/models/imgs/mipnerf/models_mipnerf_pipeline-light.png rename to docs/guides/models/imgs/mipnerf/models_mipnerf_pipeline-light.png diff --git a/docs/models/imgs/models_nerf-field-dark.png b/docs/guides/models/imgs/models_nerf-field-dark.png similarity index 100% rename from docs/models/imgs/models_nerf-field-dark.png rename to docs/guides/models/imgs/models_nerf-field-dark.png diff --git a/docs/models/imgs/models_nerf-field-light.png b/docs/guides/models/imgs/models_nerf-field-light.png similarity index 100% rename from docs/models/imgs/models_nerf-field-light.png rename to docs/guides/models/imgs/models_nerf-field-light.png diff --git a/docs/models/imgs/models_nerf-pipeline-dark.png b/docs/guides/models/imgs/models_nerf-pipeline-dark.png similarity index 100% rename from docs/models/imgs/models_nerf-pipeline-dark.png rename to docs/guides/models/imgs/models_nerf-pipeline-dark.png diff --git a/docs/models/imgs/models_nerf-pipeline-field-dark.png b/docs/guides/models/imgs/models_nerf-pipeline-field-dark.png similarity index 100% rename from docs/models/imgs/models_nerf-pipeline-field-dark.png rename to docs/guides/models/imgs/models_nerf-pipeline-field-dark.png diff --git a/docs/models/imgs/models_nerf-pipeline-field-light.png b/docs/guides/models/imgs/models_nerf-pipeline-field-light.png similarity index 100% rename from docs/models/imgs/models_nerf-pipeline-field-light.png rename to docs/guides/models/imgs/models_nerf-pipeline-field-light.png diff --git a/docs/models/imgs/models_nerf-pipeline-light.png b/docs/guides/models/imgs/models_nerf-pipeline-light.png similarity index 100% rename from docs/models/imgs/models_nerf-pipeline-light.png rename to docs/guides/models/imgs/models_nerf-pipeline-light.png diff --git a/docs/models/imgs/models_nerf-pipeline-renderer-dark.png b/docs/guides/models/imgs/models_nerf-pipeline-renderer-dark.png similarity index 100% rename from docs/models/imgs/models_nerf-pipeline-renderer-dark.png rename to docs/guides/models/imgs/models_nerf-pipeline-renderer-dark.png diff --git a/docs/models/imgs/models_nerf-pipeline-renderer-light.png b/docs/guides/models/imgs/models_nerf-pipeline-renderer-light.png similarity index 100% rename from docs/models/imgs/models_nerf-pipeline-renderer-light.png rename to docs/guides/models/imgs/models_nerf-pipeline-renderer-light.png diff --git a/docs/models/imgs/models_nerf-pipeline-sampler-dark.png b/docs/guides/models/imgs/models_nerf-pipeline-sampler-dark.png similarity index 100% rename from docs/models/imgs/models_nerf-pipeline-sampler-dark.png rename to docs/guides/models/imgs/models_nerf-pipeline-sampler-dark.png diff --git a/docs/models/imgs/models_nerf-pipeline-sampler-light.png b/docs/guides/models/imgs/models_nerf-pipeline-sampler-light.png similarity index 100% rename from docs/models/imgs/models_nerf-pipeline-sampler-light.png rename to docs/guides/models/imgs/models_nerf-pipeline-sampler-light.png diff --git a/docs/models/index.md b/docs/guides/models/index.md similarity index 100% rename from docs/models/index.md rename to docs/guides/models/index.md diff --git a/docs/models/instant_ngp.md b/docs/guides/models/instant_ngp.md similarity index 100% rename from docs/models/instant_ngp.md rename to docs/guides/models/instant_ngp.md diff --git a/docs/models/mipnerf.md b/docs/guides/models/mipnerf.md similarity index 100% rename from docs/models/mipnerf.md rename to docs/guides/models/mipnerf.md diff --git a/docs/models/mipnerf_360.md b/docs/guides/models/mipnerf_360.md similarity index 100% rename from docs/models/mipnerf_360.md rename to docs/guides/models/mipnerf_360.md diff --git a/docs/models/nerf.md b/docs/guides/models/nerf.md similarity index 98% rename from docs/models/nerf.md rename to docs/guides/models/nerf.md index 3fb426f83..c24f41a3b 100644 --- a/docs/models/nerf.md +++ b/docs/guides/models/nerf.md @@ -108,7 +108,7 @@ dir_enc = NeRFEncoding( Now that we have a representation of space, we need some way to render new images of it. To accomplish this, we are going to _project_ a ray from the target pixel and evaluate points along that ray. We then rely on classic volumetric rendering techniques [[Kajiya, 1984]](https://dl.acm.org/doi/abs/10.1145/964965.808594) to composite the points into a predicted color. This compositing is similar to what happens in tools like Photoshop when you layer multiple objects of varying opacity on top of each other. The only difference is that NeRF takes into account the differences in spacing between points. -Rending RGB images is not the only type of output render supported. It is possible to render other output types such as depth and semantics. Additional renderers can be found [Here](../reference/api/renderers/index.rst). +Rending RGB images is not the only type of output render supported. It is possible to render other output types such as depth and semantics. Additional renderers can be found [Here](../../reference/api/renderers/index.rst). Associated Nerfactory code: diff --git a/docs/models/nerf_w.md b/docs/guides/models/nerf_w.md similarity index 100% rename from docs/models/nerf_w.md rename to docs/guides/models/nerf_w.md diff --git a/docs/models/semantic_nerf.md b/docs/guides/models/semantic_nerf.md similarity index 100% rename from docs/models/semantic_nerf.md rename to docs/guides/models/semantic_nerf.md diff --git a/docs/index.rst b/docs/index.rst index 3a6132fc5..c001eb917 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -45,9 +45,7 @@ Contents :hidden: :caption: Guides - models/index - model_components/index - dataloader_components/index + guides/index .. toctree:: :hidden: diff --git a/docs/tooling/benchmarking.md b/docs/tooling/benchmarking.md index c5bbf8720..d57561508 100644 --- a/docs/tooling/benchmarking.md +++ b/docs/tooling/benchmarking.md @@ -3,7 +3,6 @@ We make it easy to benchmark your new NeRF against the standard Blender dataset. -(train)= ## Launching training on Blender dataset To start, you will need to train your NeRF on each of the blender objects. @@ -29,7 +28,7 @@ A full example would be: The script will automatically launch training on all of the items and save the checkpoints in an output directory with the current date and timestamp. -(eval)= + ## Evaluating trained Blender models Once you have launched training, and training converges, you can test your method with `scripts/benchmarking/launch_eval_blender.sh`. diff --git a/docs/tooling/config.md b/docs/tooling/config.md new file mode 100644 index 000000000..1a3c6d86e --- /dev/null +++ b/docs/tooling/config.md @@ -0,0 +1 @@ +# Customizable Configs \ No newline at end of file diff --git a/docs/tooling/hydra.md b/docs/tooling/hydra.md deleted file mode 100644 index 3e8f9a44e..000000000 --- a/docs/tooling/hydra.md +++ /dev/null @@ -1 +0,0 @@ -# Hydra config structure \ No newline at end of file diff --git a/docs/tooling/index.rst b/docs/tooling/index.rst index cbb730302..fe1c92e8f 100644 --- a/docs/tooling/index.rst +++ b/docs/tooling/index.rst @@ -2,7 +2,7 @@ .. toctree:: :maxdepth: 1 - hydra.md + config.md logging_profiling.md benchmarking.md diff --git a/docs/tooling/logging_profiling.md b/docs/tooling/logging_profiling.md index d55ca7607..14556fbf7 100644 --- a/docs/tooling/logging_profiling.md +++ b/docs/tooling/logging_profiling.md @@ -1,6 +1,5 @@ # Loggers and Profilers -(logging_support)= ## Logging support We provide integration with multiple logging interfaces to log images and statistics during training. @@ -24,7 +23,6 @@ logging: Currently, we provide support for the following loggers, which can be added to the `logging.writer` section of the config. -(tensorboard)= #### Tensorboard To log training stats and images with [tensorboard](https://www.tensorflow.org/tensorboard): ``` @@ -32,7 +30,6 @@ TensorboardWriter: log_dir: "./" # the output path for which the event files are written ``` -(wnb)= #### Weights and Biases To log training stats and images with [wandb](https://wandb.ai/site): ``` @@ -40,7 +37,6 @@ WandbWriter: log_dir: "./" # the output path for which the run files are written ``` -(local)= #### Local writer ``` LocalWriter: @@ -55,11 +51,9 @@ The `LocalWriter` simply outputs to the terminal. However, you can specify addit * `max_log_size`: how much content to print onto the screen (e.g. only print 10 lines on the screen at a time). If 0, will print everything without deleting any previous lines. -(code_profiler)= ## Code profiling support We provide built-in performance profiling capabilities to make it easier for you to debug and assess the performance of your code. -(inhouse)= #### In-house profiler You can use our built-in profiler by enabling the profiler in the config: @@ -90,7 +84,6 @@ At termination of training or end of the training run, the profiler will print o Use this profiler if there are specific functions that you want to measure the times for. -(pyspy)= #### Profiling with PySpy If you want to profile the entire codebase, we provide functionality to generate flame graphs or get real-time reports using [PySpy](https://github.com/benfred/py-spy). diff --git a/docs/tutorials/viewer/for_developers.md b/docs/tutorials/viewer/for_developers.md index 2f0756177..46b85c382 100644 --- a/docs/tutorials/viewer/for_developers.md +++ b/docs/tutorials/viewer/for_developers.md @@ -2,12 +2,12 @@ The tutorial below shows you how to host the viewer yourself and how you can use it to visualize training. -(nerfactory_bridge)= + #### Connection between nerfactory and the Bridge Server In the center, we have the Bridge Server, which facilitates the connection between nerfactory code and the Client App. This server runs on the same machine that you are using nerfactory. It has a TCP Request/Reply (REQ/REP) connection that nerfactory can connect to with the Viewer object (left). We use [ZeroMQ](https://zeromq.org/), an open-sourced messaging library, to implement this lightweight TCP connection. The Viewer class can send commands to the Bridge Server and receive replies. The Bridge Server will either dispatch commands to the Client App via a websocket or it will return information stored in the Bridge Server state. -(bridge_client)= + #### Connection between the Bridge Server and the Client App The connection between the Bridge Server and the Client App works with WebSockets and WebRTC. @@ -16,10 +16,10 @@ The connection between the Bridge Server and the Client App works with WebSocket - **WebRTC connection** - We use WebRTC to stream images being rendered from nerfactory. The websocket connection if first used to establish the WebRTC connection. Then, the Client App constantly publishes camera pose information to the Bridge Server and stores the camera information (intrinsics and extrinsics). This information is then queried from the nerfactory code, used to render an image with some Graph, and then the image is send over the TCP connection and dispatched via WebRTC to render the stream of images. -(getting_started)= + ## Getting started -(bridge)= + #### Running the Bridge Server The viewer server runs on the same machine that you use for training. The training code will connect to the server with a lightweight TCP connection. @@ -35,7 +35,7 @@ It should print out something of the form: "ZMQWebSocketBridge using zmq_url=tcp://127.0.0.1:6000 and websocket_port=8051" ``` -(client)= + #### Running the Client App > We will host the viewer online in the future, but for now we have to run it locally. @@ -79,7 +79,7 @@ PORT=4000 ESLINT_NO_DEV_ERRORS=true ``` -(nerfactory)= + #### Running the nerfactory Code You can now simply run a training job and visualize progress by enabling the viewer during training: @@ -90,7 +90,7 @@ python scripts/run_train.py --config-name=graph_instant_ngp.yaml viewer.enable=t - **Notebook demo** - See [Programming the viewer](viewer_notebook.ipynb) for an overview for how to interact with the viewer with the Viewer object from nerfactory. -(ack)= + ## Acknowledgements and references We thank [Robin Deits](https://github.com/rdeits) and other contributors to the following repos, which we've started with and modified and extended for our use.