mirror of
https://github.com/nerfstudio-project/nerfstudio.git
synced 2026-11-20 22:56:03 +08:00
merging + text embedding bugfix
This commit is contained in:
1 parent
978bf1d12d
commit
a449219f91
70 files changed
+2465
-735
No files matched your search
@@ -67,10 +67,16 @@ class NerfstudioDataParserConfig(DataParserConfig):
|
||||
"""How much to downscale images. If not set, images are chosen such that the max dimension is <1600px."""
|
||||
scene_scale: float = 1.0
|
||||
"""How much to scale the region of interest by."""
|
||||
orientation_method: Literal["pca", "up"] = "up"
|
||||
orientation_method: Literal["pca", "up", "vertical", "none"] = "up"
|
||||
"""The method to use for orientation."""
|
||||
center_method: Literal["poses", "focus", "none"] = "poses"
|
||||
"""The method to use to center the poses."""
|
||||
auto_scale_poses: bool = True
|
||||
"""Whether to automatically scale the poses to fit in +/- 1 bounding box."""
|
||||
train_split_fraction: float = 0.9
|
||||
"""The fraction of images to use for training. The remaining images are for eval."""
|
||||
depth_unit_scale_factor: float = 1e-3
|
||||
"""Scales the depth values to meters. Default value is 0.001 for a millimeter to meter conversion."""
|
||||
|
||||
@dataclass
|
||||
class Nerfstudio(DataParser):
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 582 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 593 KiB |
@@ -4,7 +4,15 @@
|
||||
|
||||
The viewer is built using [ThreeJS](https://threejs.org/) and packaged into a [ReactJS](https://reactjs.org/) application. This client viewer application will connect via a websocket to a server running on your machine. The following figure helps to illustrate how our viewer framework works:
|
||||
|
||||

|
||||
```{image} imgs/viewer_light.png
|
||||
:align: center
|
||||
:class: only-light
|
||||
```
|
||||
|
||||
```{image} imgs/viewer_dark.png
|
||||
:align: center
|
||||
:class: only-dark
|
||||
```
|
||||
|
||||
## System design
|
||||
|
||||
@@ -14,12 +22,10 @@ In the center, we have the Bridge Server, which facilitates the connection betwe
|
||||
|
||||
#### Connection between the Bridge Server and the Client App
|
||||
|
||||
The connection between the Bridge Server and the Client App works with WebSockets and WebRTC.
|
||||
The connection between the Bridge Server and the Client App works with WebSockets.
|
||||
|
||||
- **WebSocket connection** - The WebSocket is used by the Bridge Server to dispatch commands coming from the nerfstudio TCP connection. Commands can be used for drawing primitives, for setting the transform of objects, for the setting various properties, and more.
|
||||
|
||||
- **WebRTC connection** - We use WebRTC to stream images being rendered from nerfstudio. The websocket connection if first used to establish the WebRTC connection. Then, the Client App constantly publishes camera pose information to the Bridge Server and stores the camera information (intrinsics and extrinsics). This information is then queried from the nerfstudio code, used to render an image with some Graph, and then the image is send over the TCP connection and dispatched via WebRTC to render the stream of images.
|
||||
|
||||
## Installing and running locally
|
||||
|
||||
#### Running the Client App
|
||||
@@ -50,7 +56,7 @@ yarn install
|
||||
yarn start
|
||||
```
|
||||
|
||||
The local webserver runs on port 4000 by default,
|
||||
The local webserver runs on port 4000 by default,
|
||||
so when `ns-train` is running, you can connect to the viewer locally at
|
||||
[http://localhost:4000/?websocket_url=ws://localhost:7007](http://localhost:4000/?websocket_url=ws://localhost:7007)
|
||||
|
||||
@@ -70,7 +76,6 @@ We thank the authors and contributors to the following repos, which we've starte
|
||||
- [meshcat](https://github.com/rdeits/meshcat) - made by [Robin Deits](https://github.com/rdeits)
|
||||
- [ThreeJS](https://threejs.org/)
|
||||
- [ReactJS](https://reactjs.org/)
|
||||
- [WebRTC](https://webrtc.org/) - WebRTC is a framework for real-time communication that allows two peers to send video, audio, or general data to and from each other with low latency. We've adopted WebRTC to stream rendered images to our viewer.
|
||||
|
||||
## FAQ
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ ns-train vanilla-nerf
|
||||
|
||||
If you have arrived to this site, it is likely that you have at least heard of NeRFs. This page will discuss the original NeRF paper, _"NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis"_ by Mildenhall, Srinivasan, Tancik et al. (2020).
|
||||
|
||||
For most tasks, using the original NeRF model is likely not a good choice and hence we provide implementations of various other NeRF related models. It is however useful to understand how NeRF's work as most follow ups follow a similar structure.
|
||||
For most tasks, using the original NeRF model is likely not a good choice and hence we provide implementations of various other NeRF related models. It is however useful to understand how NeRF's work as most follow ups follow a similar structure and it doesn't require CUDA to execute (useful for stepping through the code with a debugger if you don't have a GPU at hand).
|
||||
|
||||
The goal is to optimize a volumetric representation of a scene that can be rendered from novel viewpoints. This representation is optimized from a set of images and associated camera poses.
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
@@ -46,7 +47,7 @@
|
||||
":class: only-dark\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"### Heirarchical Sampling\n",
|
||||
"### Hierarchical Sampling\n",
|
||||
"\n",
|
||||
"It is important to sample the scene where it has content otherwise the reconstruction quality will be reduced. \n",
|
||||
"\n",
|
||||
@@ -7413,7 +7414,7 @@
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.8.12 ('nerfactory')",
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
@@ -7427,11 +7428,11 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.12"
|
||||
"version": "3.10.7"
|
||||
},
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "34c28001ff35fb390494047002768a8182dcf55b1b11415165e62ea61557ab83"
|
||||
"hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@@ -15,6 +15,7 @@ We Currently support the following custom data types:
|
||||
| ----------------------------- | -------------- | ----------------------------------------------- | ----------------------- |
|
||||
| 📷 [Images](images_and_video) | Any | [COLMAP](https://colmap.github.io/install.html) | 🐢 |
|
||||
| 📹 [Video](images_and_video) | Any | [COLMAP](https://colmap.github.io/install.html) | 🐢 |
|
||||
| 🌎 [360 Data](360_data) | Any | [COLMAP](https://colmap.github.io/install.html) | 🐢 |
|
||||
| 📱 [Polycam](polycam) | IOS with LiDAR | [Polycam App](https://poly.cam/) | 🐇 |
|
||||
| 📱 [KIRI Engine](kiri) | IOS or Android | [KIRI Engine App](https://www.kiriengine.com/) | 🐇 |
|
||||
| 📱 [Record3D](record3d) | IOS with LiDAR | [Record3D app](https://record3d.app/) | 🐇 |
|
||||
@@ -290,3 +291,39 @@ ns-process-data metashape --data {data directory} --xml {xml file} --output-dir
|
||||
```bash
|
||||
ns-train nerfacto --data {output directory}
|
||||
```
|
||||
|
||||
(360_data)=
|
||||
|
||||
## 360 Data (Equirectangular)
|
||||
|
||||
Equirectangular data is data that has been taken by a 360 camera such as Insta360. Both equirectangular image sets and videos can be processed by nerfstudio.
|
||||
|
||||
### Images
|
||||
|
||||
For a set of equirectangular images, process the data using the following command:
|
||||
|
||||
```bash
|
||||
ns-process-data images --camera-type equirectangular --images-per-equirect {8, or 14} --data {data directory} --output-dir {output directory}
|
||||
```
|
||||
|
||||
The images-per-equirect argument is the number of images that will be sampled from each equirectangular image. We have found that 8 images per equirectangular image is sufficient for most use cases so it defaults to that. However, if you find that there isn't enough detail in the nerf or that colmap is having trouble aligning the images, you can try increasing the number of images per equirectangular image to 14.
|
||||
|
||||
### Videos
|
||||
|
||||
For videos we recommend taking a video with the camera held on top of your head. This will result in any unwanted capturer to just be in the bottom of each frame image and therefore can be cropped out.
|
||||
|
||||
For a video, process the data using the following command:
|
||||
|
||||
```bash
|
||||
ns-process-data video --camera-type equirectangular --images-per-equirect {8, or 14} --num-frames-target {num equirectangular frames to sample from} --crop-factor {top bottom left right} --data {data directory} --output-dir {output directory}
|
||||
```
|
||||
|
||||
See the equirectangular images section above for a description of the `--images-per-equirect` argument.
|
||||
|
||||
The `num-frames-target` argument is optional but it is recommended to set it to 3*(seconds of video) frames. For example, if you have a 30 second video, you would use `--num-frames-target 90` (3*30=90). This number was chosen from a bit of experimentation and seems to work well for most videos. It is by no means a hard rule and you can experiment with different values.
|
||||
|
||||
The `crop-factor` argument is optional but often very helpful. This is because equirectangular videos taken by 360 cameras tend to have a portion of the bottom of the image that is the person who was holding the camera over their head.
|
||||
|
||||
<img src="imgs/equirect_crop.jpg">
|
||||
|
||||
This obscene mesh of human is obviously not helpful in training a nerf so we can remove it by cropping the bottom 20% of the image. This can be done by using the `--crop-factor 0 0.2 0 0` argument.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 388 KiB |
@@ -140,7 +140,7 @@ To restrict to only CUDA architectures that you have available locally, use the
|
||||
build arg and look up [the compute capability for your GPU](https://developer.nvidia.com/cuda-gpus).
|
||||
For example, here's how to build with support for GeForce 30xx series GPUs:
|
||||
```bash
|
||||
docker build --build-arg CUDA_ARCHITECTURES=86 -tag nerfstudio-86 .
|
||||
docker build --build-arg CUDA_ARCHITECTURES=86 --tag nerfstudio-86 -f Dockerfile .
|
||||
```
|
||||
|
||||
### Using an interactive container
|
||||
|
||||
Reference in new issue
Block a user