diff --git a/README.md b/README.md index b68024abe..9ee19a084 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@

-

The all-in-one repo for NeRFs

+

A collaboration friendly studio for NeRFs

@@ -55,7 +55,7 @@ It’s as simple as plug and play with nerfstudio! Ontop of our API, we are commited to providing learning resources to help you understand the basics of (if you're just getting start), and keep up-to-date with (if you're a seasoned veteran) all things NeRF. As researchers, we know just how hard it is to get onboarded with this next-gen technology. So we're here to help with tutorials, documentation, and more! -Finally, have feature requests? Want to add your brand-spankin'-new NeRF model? Have a new dataset? **We welcome any and all contributions!** Please do not hesitate to reach out to the nerfstudio team with any questions. +Finally, have feature requests? Want to add your brand-spankin'-new NeRF model? Have a new dataset? **We welcome any and all [contributions](https://docs.nerf.studio/en/latest/reference/contributing.html)!** Please do not hesitate to reach out to the nerfstudio team with any questions via [Discord](https://discord.gg/NHGtYRAW). We hope nerfstudio enables you to build faster :hammer: learn together :books: and contribute to our NeRF community :sparkling_heart:. @@ -70,7 +70,7 @@ For more complex changes (e.g. running with your own data/ setting up a new NeRF We reccomend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html) before preceding. -``` +```bash conda create --name nerfstudio -y python=3.8.13; conda activate nerfstudio python -m pip install --upgrade pip @@ -80,7 +80,7 @@ python -m pip install --upgrade pip Install pytorch with CUDA (this repo has been tested with CUDA 11.3) and [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn) -``` +```bash pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch ``` @@ -89,78 +89,61 @@ pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/to Easy option: -``` +```bash pip install nerfstudio ``` If you would want the latest and greatest: -``` +```bash git clone git@github.com:plenoptix/nerfstudio.git cd nerfstudio pip install -e . ``` -### Optional Installs - -**Tab completion (bash & zsh)** - -This needs to be rerun when the CLI changes, for example if nerfstudio is updated. - -```bash -ns-install-cli -``` - -**Development packages** - -```bash -pip install -e.[dev] -pip install -e.[docs] -``` - -#### 2. Getting the data +## 2. Setting up the data Download the original NeRF Blender dataset. We support the major datasets and allow users to create their own dataset, described in detail [here](https://plenoptix-nerfstudio.readthedocs-hosted.com/en/latest/tutorials/data/index.html). -``` +```bash ns-download-data --dataset=blender ns-download-data --dataset=nerfstudio --capture=poster ``` -Use `--help` to view all currently available datasets. The resulting script should download and unpack the dataset as follows: +### 2.x Using custom data +If you have custom data in the form of a video or folder of images, we've provided some [COLMAP](https://colmap.github.io/) and [FFmpeg](https://ffmpeg.org/download.html) scripts to help you process your data so it is compatible with nerfstudio. -``` -|─ nerfstudio/ - ├─ data/ - | ├─ blender/ - | ├─ fern/ - | ├─ lego/ - ... - |- / - |- - ... +After installing both software, you can process your data via: + +```bash +ns-process-data --data FOLDER_OR_VIDEO --output-dir PROCESSED_DATA_DIR ``` ## 3. Training a model To run with all the defaults, e.g. vanilla nerf method with the blender lego image -``` +```bash # To see what models are available. ns-train --help -# Run a vanilla nerf model. -ns-train vanilla-nerf +# To see what model-specific cli arguments are available. +ns-train nerfacto --help # Run with nerfacto model. ns-train nerfacto -# Run with nerfstudio data. You'll may have to change the ports, and be sure to forward the "websocket-port". -ns-train nerfacto --vis viewer --viewer.zmq-port 8001 --viewer.websocket-port 8002 nerfstudio-data --pipeline.datamanager.dataparser.data-directory data/nerfstudio/poster --pipeline.datamanager.dataparser.downscale-factor 4 +# We provide support for many existing models. E.g. to run mipnerf. +ns-train instant-ngp + +# To train on your custom data. +ns-train nerfacto --data PROCESSED_DATA_DIR ``` ### 3.x Training a model with the viewer +You can visualize training in real-time using our web-based viewer. + Make sure to forward a port for the websocket to localhost. The default port is 7007, which you should expose to localhost:7007. ```bash @@ -174,59 +157,39 @@ ns-train nerfacto --vis viewer --viewer.websocket-port=7008 ssh -L localhost:7008:localhost:7008 {REMOTE HOST} ``` -### 4. Visualizing training runs +For more details on how to interact with the visualizer, please visit our viewer [walk-through](). -We support multiple methods to visualize training, the default configuration uses Tensorboard. More information on logging can be found [here](https://docs.nerf.studio/en/latest/tooling/logging_profiling.html). +## 4. Rendering a trajectory during inference -

-Real-time Viewer - -We have developed our own Real-time web viewer, more information can be found [here](https://docs.nerf.studio/en/latest/tutorials/viewer/index.html). This viewer runs during training and is designed to work with models that have fast rendering pipelines. - -To turn on the viewer, simply add the flag `--vis viewer`. - -
- -
-Tensorboard - -If you run everything with the default configuration we log all training curves, test images, and other stats. Once the job is launched, you will be able to track training by launching the tensorboard in your base experiment directory (Default: `outputs/`). +After your model has trained, you can headlessly render out a video of the scene with a pre-defined trajectory. ```bash -tensorboard --logdir outputs/ +ns-eval render-trajectory --load-config=outputs/blender_lego/instant_ngp/{MOST_RECENT_RUN}/config.yml--traj=spiral --output-path=output.mp4 ``` -
- -
-Weights & Biases - -We support logging to weights and biases. To enable wandb logging, add the flag `--logging.writer.1.enable`. - -
- -## 5. Rendering a trajectory during inference - -``` -ns-render --load-config=outputs/blender_lego/instant_ngp/{MOST_RECENT_RUN}/config.yml--traj=spiral --output-path=output.mp4 -``` - -## 6. In-depth guide - -For a more in-depth tutorial on how to modify/implement your own NeRF Graph, please see our [walk-through](https://docs.nerf.studio/en/latest/tutorials/pipelines/index.html). # Learn More +And that's it for getting started with the basics of nerfstudio. If you're interested in learning more on how to create your own pipelines, develop with the viewer, run benchmarks, and more, please check out some of the qucklinks below or visit our [documentation](https://docs.nerf.studio/en/latest/) directly. + | Section | Description | | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | -| [Documentation](https://docs.nerf.studio/en/latest/) | Full API documentation and tutorials | -| [Interactive Guides](https://docs.nerf.studio/en/latest/guides/index.html) | Go-to spot for learning how NeRFs and each of its modules work. | -| [Quick tour](https://docs.nerf.studio/en/latest/tutorials/quickstart_index.html) | Example script on how to navigate nerfstudio from install, train, to test. | -| [Creating pipelines](https://docs.nerf.studio/en/latest/tutorials/pipelines/index.html) | Learn how to easily build new neural rendering pipelines by using and/or implementing new modules. | -| [Creating datasets](https://docs.nerf.studio/en/latest/tutorials/data/index.html) | Have a new dataset? Learn how to use it with nerfstudio. | -| [Mobile Capture to NerF](#) | Step-by-step tutorial on how to create beautiful renders with just your phone. | -| [Contributing](https://docs.nerf.studio/en/latest/reference/contributing.html) | Walk-through for how you can start contributing now. | -| [Discord](https://discord.gg/NHGtYRAW) | Join our community to discuss more. We would love to hear from you! | +| [Documentation](https://docs.nerf.studio/en/latest/) | Full API documentation and tutorials | +| [Viewer](https://viewer.nerf.studio/) | Home page for our web viewer | +| **Educational Guides** | +| [Model Descriptions](https://docs.nerf.studio/en/latest/guides/index.html) | Description of all the models supported by nerfstudio and explanations of component parts. | +| [Component Descriptions](https://docs.nerf.studio/en/latest/nerfology/model_components/index.html) | Interactive notebooks that explain notable/commonly used modules in various models. | +| **Tutorials** | +| [Getting Started](https://docs.nerf.studio/en/latest/quickstart/installation.html) | A more in-depth guide on how to get started with nerfstudio from installation to contributing. | +| [Using the Viewer](https://docs.nerf.studio/en/latest/quickstart/viewer_quickstart.html) | A quick demo video on how to navigate the viewer. | +| **Developer Resources** | +| [Creating pipelines](https://docs.nerf.studio/en/latest/developer_guides/pipelines/index.html) | Learn how to easily build new neural rendering pipelines by using and/or implementing new modules. | +| [Creating datasets](https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html) | Have a new dataset? Learn how to run it with nerfstudio. | | +| [Contributing](https://docs.nerf.studio/en/latest/reference/contributing.html) | Walk-through for how you can start contributing now. | +| **Community** | +| [Discord](https://discord.gg/NHGtYRAW) | Join our community to discuss more. We would love to hear from you! | +| [Twitter](#) | Follow us on Twitter to see cool updates and announcements | +| [TikTok](#) | Comming soon! Follow us on TikTok to see some of our fan favorite results | # Supported Features diff --git a/docs/index.rst b/docs/index.rst index 5470a9043..66831d446 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -27,7 +27,8 @@ It's as simple as plug and play with nerfstudio! Ontop of our API, we are commited to providing learning resources to help you understand the basics of (if you're just getting start), and keep up-to-date with (if you're a seasoned veteran) all things NeRF. As researchers, we know just how hard it is to get onboarded with this next-gen technology. So we're here to help with tutorials, documentation, and more! -Finally, have feature requests? Want to add your brand-spankin'-new NeRF model? Have a new dataset? **We welcome any and all contributions!** Please do not hesitate to reach out to the nerfstudio team with any questions. +Finally, have feature requests? Want to add your brand-spankin'-new NeRF model? Have a new dataset? **We welcome any and all [contributions](https://docs.nerf.studio/en/latest/reference/contributing.html)!** +Please do not hesitate to reach out to the nerfstudio team with any questions via [Discord](https://discord.gg/NHGtYRAW). We hope nerfstudio enables you to build faster 🔨 learn together 📚 and contribute to our NeRF community 💖. @@ -111,7 +112,7 @@ Quicklinks ============================================================ ====================== `Github `_ Official Github Repo `Discord `_ Join Discord Community - `Web-based Viewer `_ Web-based Nerf Viewer + `Viewer `_ Web-based Nerf Viewer ============================================================ ====================== diff --git a/docs/quickstart/custom_dataset.md b/docs/quickstart/custom_dataset.md index d5c6f4204..d24bf2460 100644 --- a/docs/quickstart/custom_dataset.md +++ b/docs/quickstart/custom_dataset.md @@ -1,8 +1,8 @@ -# Using Custom Data +# Using custom data Training model on existing datasets is only so fun. If you would like to train on self captured data you will need to process the data into an existing format. Specifically we need to know the camera poses for each image. [COLMAP](https://github.com/colmap/colmap) is a standard tool for extracting poses. It is possible to use other methods like [SLAM](https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping) or hardware recorded poses. We intend to add documentation for these other methods in the future. -## nerfstudio dataset +## Nerfstudio dataset To assist running on custom data we have a script that will process a video or folder of images into a format that is compatible with nerfstudio. We use [COLMAP](https://colmap.github.io) and [FFmpeg](https://ffmpeg.org/download.html) in our data processing script, please have these installed. We have provided a quickstart to installing COLMAP below, FFmpeg can be downloaded from [here](https://ffmpeg.org/download.html) diff --git a/docs/quickstart/first_nerf.md b/docs/quickstart/first_nerf.md index 4e9f645a7..e45cc2d53 100644 --- a/docs/quickstart/first_nerf.md +++ b/docs/quickstart/first_nerf.md @@ -1,4 +1,4 @@ -# Training First Model +# Training your first model ## Downloading data @@ -64,50 +64,3 @@ ns-render --load-config=outputs/blender_lego/instant_ngp/2022-07-07_230905/confi ``` Please note, this quickstart allows you to preform everything in a headless manner. We also provide a web-based viewer that allows you to easily monitor training or render out trajectories. See our [viewer docs](viewer_quickstart.md) for more. - -## FAQ - -- [TinyCUDA installation errors out with cuda mismatch](tiny-cuda-error) -- [Installation errors, File "setup.py" not found](pip-install-error) -- [Runtime errors, "len(sources) > 0".](cuda-sources-error) - -(tiny-cuda-error)= - -#### TinyCUDA installation errors out with cuda mismatch - -While installing tiny-cuda, you run into: `The detected CUDA version mismatches the version that was used to compile PyTorch (10.2). Please make sure to use the same CUDA versions.` - -**Solution**: - -``` -pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html -``` - -(pip-install-error)= - -#### Installation errors, File "setup.py" not found - -When installing dependencies and nerfstudio with `pip install -e .`, you run into: `ERROR: File "setup.py" not found. Directory cannot be installed in editable mode` - -**Solution**: -This can be fixed by upgrading pip to the latest version: - -``` -python -m pip install --upgrade pip -``` - - -(cuda-sources-error)= - -#### Runtime errors: "len(sources) > 0", "ctype = \_C.ContractionType(type.value) ; TypeError: 'NoneType' object is not callable". - -When running `train.py `, an error occurs when installing cuda files in the backend code. - -**Solution**: -This is a problem with not being able to detect the correct CUDA version, and can be fixed by updating the CUDA path environment variables: - -``` -export CUDA_HOME=/usr/local/cuda -export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64 -export PATH=$PATH:$CUDA_HOME/bin -``` diff --git a/docs/quickstart/installation.md b/docs/quickstart/installation.md index ffd78dcd1..6728a291a 100644 --- a/docs/quickstart/installation.md +++ b/docs/quickstart/installation.md @@ -1,8 +1,8 @@ # Installation -### Create environment +## Create environment -We reccomend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html) before preceding. +We recommend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/en/latest/miniconda.html) before preceding. ```bash conda create --name nerfstudio -y python=3.8.13 @@ -11,7 +11,7 @@ python -m pip install --upgrade pip ``` -### Dependencies +## Dependencies Install pytorch with CUDA (this repo has been tested with CUDA 11.3) and [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn) @@ -21,15 +21,15 @@ pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/to ``` -### Installing nerfstudio +## Installing nerfstudio -Easy option: +Pip package: ```bash pip install nerfstudio ``` -If you would want the latest and greatest: +If you want the latest and greatest: ```bash git clone git@github.com:plenoptix/nerfstudio.git @@ -38,7 +38,7 @@ pip install -e . ``` -### Optional Installs +### Optional #### Tab completion (bash & zsh) @@ -54,3 +54,50 @@ ns-install-cli pip install -e.[dev] pip install -e.[docs] ``` + +## Installation FAQ + +- [TinyCUDA installation errors out with cuda mismatch](tiny-cuda-error) +- [Installation errors, File "setup.py" not found](pip-install-error) +- [Runtime errors, "len(sources) > 0".](cuda-sources-error) + +(tiny-cuda-error)= + +#### TinyCUDA installation errors out with cuda mismatch + +While installing tiny-cuda, you run into: `The detected CUDA version mismatches the version that was used to compile PyTorch (10.2). Please make sure to use the same CUDA versions.` + +**Solution**: + +``` +pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html +``` + +(pip-install-error)= + +#### Installation errors, File "setup.py" not found + +When installing dependencies and nerfstudio with `pip install -e .`, you run into: `ERROR: File "setup.py" not found. Directory cannot be installed in editable mode` + +**Solution**: +This can be fixed by upgrading pip to the latest version: + +``` +python -m pip install --upgrade pip +``` + + +(cuda-sources-error)= + +#### Runtime errors: "len(sources) > 0", "ctype = \_C.ContractionType(type.value) ; TypeError: 'NoneType' object is not callable". + +When running `train.py `, an error occurs when installing cuda files in the backend code. + +**Solution**: +This is a problem with not being able to detect the correct CUDA version, and can be fixed by updating the CUDA path environment variables: + +``` +export CUDA_HOME=/usr/local/cuda +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64 +export PATH=$PATH:$CUDA_HOME/bin +``` \ No newline at end of file diff --git a/docs/quickstart/viewer_quickstart.md b/docs/quickstart/viewer_quickstart.md index b6a15852a..1976162cf 100644 --- a/docs/quickstart/viewer_quickstart.md +++ b/docs/quickstart/viewer_quickstart.md @@ -1 +1 @@ -# Setting up Viewer +# Setting up viewer diff --git a/docs/reference/contributing.md b/docs/reference/contributing.md index f00bd4851..5dd972336 100644 --- a/docs/reference/contributing.md +++ b/docs/reference/contributing.md @@ -1,19 +1,20 @@ -# Contributing to nerfstudio workflow +# Contributing -**We're excited to have you join the NerfStudio family!** :love_letter: +**💝 We're excited to have you join the nerfstudio family 💝** -Below, we document the contribution pipeline and good-to-knows for when you're ready to submit a PR. If you have any questions at all, please don't hesitate to reach out to us. We'd love to hear from you! +Below, we document the contribution pipeline and good-to-knows for when you're ready to submit a PR. If you have any questions at all, please don't hesitate to reach out to us on [Discord](https://discord.com/invite/NHGtYRAW). We'd love to hear from you! The project is set up for development in VSCode, we recommend using it if you plan to contribute. -## TLDR +## Overview +Below are the various tooling features our team uses to maintain this codebase. -| | | -| --------------- | ------ | -| linter | Black | -| Testing | PyTest | -| Docs | Sphinx | -| Docstring style | Google | +| Tooling | Support | +| --------------- | ------------- | +| Linter | [Black](https://black.readthedocs.io/en/stable/) | +| Testing | [PyTest](https://docs.pytest.org/en/7.1.x/) | +| Docs | [Sphinx](https://www.sphinx-doc.org/en/master/) | +| Docstring style | [Google](https://google.github.io/styleguide/pyguide.html) | ## Requirements @@ -30,10 +31,10 @@ You may also need to install [pandoc](https://pandoc.org/). If you are using `co conda install -c conda-forge pandoc ``` -## Committing +## Committing code -1. Make your modifications -2. Perform local checks +1. ✏️ Make your modifications +2. ✅ Perform local checks To ensure that you will be passing all tests and checks on github, you will need to run the following command: @@ -41,18 +42,18 @@ conda install -c conda-forge pandoc ns-dev-test ``` -This will perform the following checks and actions: + This will perform the following checks and actions: -- Black/ Linting style check: Ensure that the code is consistently and properly formatted. -- Pytests: Runs pytests locally to make sure the logic of any added code does not break existing logic. -- Documentation build: Builds the documentation locally to make sure none of the changes result in any warnings or errors in the docs. -- Licensing: automatically adds licensing headers to the correct files. + - Black/ Linting style check: Ensures code is consistently and properly formatted. + - Pytests: Runs pytests locally to make sure added code does not break existing logic. + - Documentation build: Builds docs locally. Ensures changes do not result in warnings/errors. + - Licensing: Automatically adds licensing headers to the correct files. -In order to merge changes to the code base, all of these checks must be passing. If you pass these tests locally, you will likely pass on github servers as well (results in a green checkmark next to your commit). + In order to merge changes to the code base, all of these checks must be passing. If you pass these tests locally, you will likely pass on github servers as well (results in a green checkmark next to your commit). -3. Open pull request +3. 💌 Open pull request! -## Documentation +## Maintaining documentation ### Building @@ -64,7 +65,17 @@ python scripts/docs/build_docs.py Rerun `make html` when documentation changes are made, `make clean` is necessary if the documentation structure changes. -### Notebooks +### Auto build + +As you change or add models/components, the auto-generated [Reference API](https://docs.nerf.studio/en/latest/reference/api/index.html) may change. +If you want the code to build on save you can use [sphinx autobuild](https://github.com/executablebooks/sphinx-autobuild). If changes to the structure are made, the build files may be incorrect. + +```bash +pip install sphinx-autobuild +sphinx-autobuild docs docs/_build/html +``` + +### Adding notebooks We support jupyter notbooks in our documentation. To improve the readability, the following custom tags can be added to the top of each code cell to hide or collapse the code. @@ -73,12 +84,3 @@ We support jupyter notbooks in our documentation. To improve the readability, th | # HIDDEN | Hide code block and output | | # COLLAPSED | Collapse the code in a dropdown but show the results | | # OUTPUT_ONLY | Only show the cell's output | - -### Auto build - -If you want the code to build on save you can use [sphinx autobuild](https://github.com/executablebooks/sphinx-autobuild). If changes to the structure are made, the build files may be incorrect. - -```bash -pip install sphinx-autobuild -sphinx-autobuild docs docs/_build/html -```