tink/nerfstudio
4
0 Fork 0
mirror of https://github.com/nerfstudio-project/nerfstudio.git synced 2026-11-16 04:43:52 +08:00
Code Activity

Merge pull request #13 from plenoptix/eng/docs

Browse Source 
update readme; will port over to rst later
This commit is contained in:
evonneng authored and GitHub committed 2022-06-03 12:04:53 -07:00
2 parents 230e13adbd 84fb7b12ee
commit 93ef7a729f
4 files changed
+127 -83
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+127 -24
View File
@@ -1,13 +1,29 @@
# pyRad :metal:
<h1 align="center"> pyRad🤘 </h1>
[![Documentation Status](https://readthedocs.com/projects/plenoptix-pyrad/badge/?version=latest)](https://plenoptix-pyrad.readthedocs-hosted.com/en/latest/?badge=latest)
<p align="center"> The all-in-one repo for NeRFs </p>
An all-in-one repo for NeRFs
<p align="center">
<a href="https://plenoptix-pyrad.readthedocs-hosted.com/en/latest/?badge=latest">
<img alt="Documentation Status" src="https://readthedocs.com/projects/plenoptix-pyrad/badge/?version=latest">
</a>
<!-- TODO: add license and have it point to that -->
<a href="https://opensource.org/licenses/Apache-2.0">
<img alt="Documentation Status" src="https://img.shields.io/badge/License-Apache_2.0-blue.svg">
</a>
<!-- TODO: add version number badge -->
</p>
* [Quickstart](#quickstart)
* [Walk-through tour](#walk-through-tour)
* [Feature List](#feature-list)
* [Benchmarked Model Architectures](#benchmarked-model-architectures)
# Quickstart
## Installation: Setup the environment
#### 1. Installation: Setup the environment
This repository is tested with cuda 11.3
```
# Clone the repo
git clone --recurse-submodules git@github.com:ethanweber/pyrad.git
@@ -27,18 +43,19 @@ python setup.py develop
pytest tests
```
## Getting the data
#### 2. Getting the data
Download the original NeRF dataset and put it in the following format.
Download the original [NeRF dataset](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) and unfold it in the following format:
```
- data/
- blender/
- fern/
- lego/
├── data/
| ├── blender/
| | ├── fern/
| | ├── lego/
...
```
## Training a model
#### 3. Training a model
```
# Run with default config
@@ -61,23 +78,101 @@ python scripts/run_data_preprocessor.py
python scripts/run_train.py ++data.dataset.use_cache=true
```
# Getting around the codebase
# Walk-through tour
In this quick tour, we will walk you through the core of training and building any NeRFs with pyrad.
The entry point for training starts at `scripts/run_train.py`, which spawns instances of our `Trainer()` class (in `nerf/trainer.py`). The `Trainer()` is responsible for setting up the datasets and NeRF graph depending on the config specified. If you are planning on just using our codebase to build a new NeRF method or use an existing implementation, we've abstracted away the training routine in these two files and chances are you will not need to touch them.
#### Launching a training job
The entry point for training starts at `scripts/run_train.py`, which spawns instances of our `Trainer()` class (in `nerf/trainer.py`). The `Trainer()` is responsible for setting up the datasets and NeRF graph depending on the config specified. It will then run the usual train/val routine for a config-specified number of iterations. If you are planning on using our codebase to build a new NeRF method or to use an existing implementation, we've abstracted away the training routine in these two files and chances are you will not need to think of them again.
The NeRF graph definitions can be found in `nerf/graph/`. Each implementation of NeRF is definined in its own file. For instance, `nerf/graph/instant_ngp.py` contains populates the `NGPGraph()` class with all of the appropriate fields, colliders, and misc. modules.
* Fields (`nerf/fields/`): composed of field modules (`nerf/field_modules/`) and represents the radiance field of the NeRF.
* Misc. Modules (`nerf/misc_modules`- TODO(maybe move to misc_modules? better organization)): any remaining module in the NeRF (e.g. renderers, samplers, losses, and metrics).
#### Graphs, Fields, and Modules
If you are looking to implemnet a new NeRF method or extend an existing one, you only need to edit files in `nerf/graph/`, `nerf/fields/`, `nerf/field_modules/`, `nerf/misc_modules/`. (TODO: restructuring)
To implement any pre-existing NeRF that we have not yet implemented under `nerf/graph/`, create a new graph structure by using provided modules or any new module you define. Then create an associated config making sure `__target__` points to your NeRF class (see [here](./configs/README.md) for more info on how to create the config). Then run training as described above.
# Feature Documentation
The actual NeRF graph definitions can be found in `nerf/graph/`. For instance, to implement the vanilla NeRF, we create a new class that inherits the abstract Graph class. To fully implement the any new graph class, you will need to implement the following abstract methods defined in the skeleton code below. See also `nerf/graph/vanilla_nerf.py` for the full implementation.
### 1. [Hydra config structure](./configs/README.md)
### 2. [Logging, debugging utilities](./pyrad/utils/README.md)
### 3. [Benchmarking, other tooling](./scripts/README.md)
```
class NeRFGraph(Graph):
"""Vanilla NeRF graph"""
### 4. Running other repos with our data
def __init__(self, intrinsics=None, camera_to_world=None, **kwargs) -> None:
super().__init__(intrinsics=intrinsics, camera_to_world=camera_to_world, **kwargs)
def populate_fields(self):
"""
Set all field related modules here
"""
def populate_misc_modules(self):
"""
Set all remaining modules here including: samplers, renderers, losses, and metrics
"""
def get_param_groups(self) -> Dict[str, List[Parameter]]:
"""
Create a dictionary of parameters that are grouped according to different optimizers
"""
def get_outputs(self, ray_bundle: RayBundle):
"""
Takes in a Ray Bundle and returns a dictionary of outputs.
"""
def get_loss_dict(self, outputs, batch):
"""
Computes and returns the losses.
"""
def log_test_image_outputs(self, image_idx, step, batch, outputs):
"""
Writes the test image outputs.
"""
```
Note that the graph is composed of fields and modules.
**Fields** (`nerf/fields/`) represents the actual radiance field of the NeRF and is composed of field modules (`nerf/field_modules/`). Here, we define the field as the part of the network that takes in point samples and any other conditioning, and outputs any of the `FieldHeadNames` (`nerf/field_modules/field_heads.py`). The **misc. modules** can be any module outside of the field that are needed by the NeRF (e.g. losses, samplers, renderers).
To get started on a new NeRF implementation, you simply have to define all relevant modules and populate them in the graph.
#### Dataset population TODO(ethan)
#### Config
Now that you have the graph and dataset all set up, you're ready to create the config that you pass into our run train routine. Our config system is powered by [Hydra](https://hydra.cc/). All Hydra and machine related arguments are stored in `configs/default_setup.yaml`, as well as the defaults list.
To set up the graph config, create a new yaml under `configs/`.
```
# configs/vanilla_nerf.yaml
defaults:
- default_setup # inherit the basic yaml heirarchy
- _self_
experiment_name: blender_lego
method_name: vanilla_nerf
graph:
network:
_target_: pyrad.nerf.graph.vanilla_nerf.NeRFGraph # set the target to the graph you defined
# <insert any additional graph related overrides here>
data:
# <insert any additional dataset related overrides here>
```
Once you have the config properly set up, you can begin training! Note, you can also pass in the config changes via command-line as shown above in the quick-start if you don't want to make a new config for a given job.
```
python scripts/run_train.py --config-name vanilla_nerf
```
# Feature List
#### :metal: [Hydra config structure](#)
#### :metal: [Logging, debugging utilities](#)
#### :metal: [Benchmarking, other tooling](#)
#### :metal: Running other repos with our data
```
# nerf-pytorch
@@ -90,7 +185,7 @@ conda activate jaxnerf
python -m jaxnerf.train --data_dir=/path/to/pyrad/data/blender/chair --train_dir=/path/to/pyrad/outputs/blender_chair_jaxnerf --config=/path/to/pyrad/external/jaxnerf/configs/demo --render_every 100
```
### 5. Speeding up the code
#### :metal: Speeding up the code
Documentation for running the code with CUDA.
Please see https://github.com/NVlabs/tiny-cuda-nn for how to install tiny-cuda-nn.
@@ -104,9 +199,17 @@ To run instant-ngp with tcnn, you can do the following. This is with the fox dat
python scripts/run_train.py --config-name=instant_ngp_tcnn.yaml data/dataset=instant_ngp_fox
```
### 6. Setting up Jupyter
#### :metal: Setting up Jupyter
```
python -m jupyter lab build
bash environments/run_jupyter.sh
```
# Benchmarked Model Architectures
| Method | PSNR |
| --------------------------------------------------------------------------------- | ------------------------ |
| [NeRF](https://arxiv.org/abs/2003.08934) | :hourglass_flowing_sand: |
| [instant NGP](https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.pdf) | :hourglass_flowing_sand: |
| [Mip NeRF](https://arxiv.org/abs/2103.13415) | :hourglass_flowing_sand: |
pyrad/utils/README.md
-7
View File
@@ -1,7 +0,0 @@
# Logging and profiling features
We provide many logging functionalities for timing and/or tracking losses during training. All of these loggers are configurable via `configs/logging.yml`
1. **Writer**: Logs losses and generated images during training to a specified output stream. Specify the type of writer (Tensorboard, Local Writer, Weights and Biases), and how often to log in the config.
2. **Profiler**: Computes the average total time of execution for any function with the `@profiler.time_function` decorator. Prints out the full profile at the termination of training or the program.
scripts/README.md
-52
View File
@@ -1,52 +0,0 @@
# Tooling
1. One can use [autoenv](https://github.com/hyperupcall/autoenv) to make setting up the environment and environment variables easier. This will run the `.envrc` file upon entering the `/path/to/pyrad` folder. It will also remove the environment parameters upon leaving.
```
# Install direnv.
sudo apt install direnv
# Add the following line to the bottom of your ~/.bashrc file.
eval "$(direnv hook bash)"
# Populate your .envrc with commands you want to run. Then, run the following to allow updates.
cd /path/to/pyrad
direnv allow .
```
2. To run local **github actions**, you can run:
```
python scripts/debugging/run_actions.py
```
3. To run local **profiling** to get a flame graph, make sure [pyspy](https://github.com/benfred/py-spy) is installed and you can run:
```
pip install py-spy
## for flame graph
./scripts/debugging/profile.sh -t flame -o flame.svg -p scripts/run_train.py data/dataset=blender_lego
## for live view of functions
./scripts/debugging/profile.sh -t top -p scripts/run_train.py data/dataset=blender_lego
```
4. For debugging with a debugger.
```
ipython --pdb scripts/run_train.py
```
5. **Benchmarking** For launching training jobs automatically on blender dataset
```
./scripts/benchmarking/launch_train_blender.sh <gpu0> <gpu1> ... <gpu7>
```
For testing specific methods, see scripts/benchmarking/run_benchmark.py.
Modify the `BENCH` variable to specify which jobs ("ckpt_dir") and methods ("method") you want benchmarked. Then run
```
python scripts/benchmarking/run_benchmark.py
```
tests/README.md
View File
Whitespace-only changes.