GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → liuyuan-pal → NeuRay

liuyuan-pal / NeuRay

Licence: GPL-3.0 license
[CVPR2022] Neural Rays for Occlusion-aware Image-based Rendering

Programming Languages

python
139335 projects - #7 most used programming language

Labels

nerfnovel-view-synthesisneural-renderingradiance-field

Projects that are alternatives of or similar to NeuRay

hypernerf
Code for "HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields".
Stars: ✭ 735 (+152.58%)
Mutual labels:  nerf, novel-view-synthesis, neural-rendering
neurecon
Multi-view 3D reconstruction using neural rendering. Unofficial implementation of UNISURF, VolSDF, NeuS and more.
Stars: ✭ 697 (+139.52%)
Mutual labels:  nerf, neural-rendering
New-View-Synthesis
Collecting papers about new view synthesis
Stars: ✭ 437 (+50.17%)
Mutual labels:  nerf, neural-rendering
3d Photo Inpainting
[CVPR 2020] 3D Photography using Context-aware Layered Depth Inpainting
Stars: ✭ 5,596 (+1823.02%)
Mutual labels:  novel-view-synthesis
cv-arxiv-daily
🎓Automatically Update CV Papers Daily using Github Actions (Update Every 12th hours)
Stars: ✭ 216 (-25.77%)
Mutual labels:  nerf
tiny-cuda-nn
Lightning fast & tiny C++/CUDA neural network framework
Stars: ✭ 908 (+212.03%)
Mutual labels:  nerf
point based clothing
Official PyTorch code for the paper: "Point-Based Modeling of Human Clothing" (ICCV 2021)
Stars: ✭ 57 (-80.41%)
Mutual labels:  neural-rendering
NovelViewSynthesis-TensorFlow
A TensorFlow implementation of a simple Novel View Synthesis model on ShapeNet (cars and chairs), KITTI, and Synthia.
Stars: ✭ 47 (-83.85%)
Mutual labels:  novel-view-synthesis
SMPL-NeRF
Embed human pose information into neural radiance fields (NeRF) to render images of humans in desired poses 🏃 from novel views
Stars: ✭ 29 (-90.03%)
Mutual labels:  nerf
DSNeRF
Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)
Stars: ✭ 343 (+17.87%)
Mutual labels:  nerf
torch-ngp
A pytorch CUDA extension implementation of instant-ngp (sdf and nerf), with a GUI.
Stars: ✭ 1,317 (+352.58%)
Mutual labels:  nerf
gnerf
[ ICCV 2021 Oral ] Our method can estimate camera poses and neural radiance fields jointly when the cameras are initialized at random poses in complex scenarios (outside-in scenes, even with less texture or intense noise )
Stars: ✭ 152 (-47.77%)
Mutual labels:  nerf
instant-ngp
Instant neural graphics primitives: lightning fast NeRF and more
Stars: ✭ 1,863 (+540.21%)
Mutual labels:  nerf
CIPS-3D
3D-aware GANs based on NeRF (arXiv).
Stars: ✭ 562 (+93.13%)
Mutual labels:  nerf
MISE
Multimodal Image Synthesis and Editing: A Survey
Stars: ✭ 214 (-26.46%)
Mutual labels:  nerf
geometry-free-view-synthesis
Is a geometric model required to synthesize novel views from a single image?
Stars: ✭ 265 (-8.93%)
Mutual labels:  novel-view-synthesis
improved-nerfmm
Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters
Stars: ✭ 172 (-40.89%)
Mutual labels:  nerf
mvsnerf
[ICCV 2021] Our work presents a novel neural rendering approach that can efficiently reconstruct geometric and neural radiance fields for view synthesis.
Stars: ✭ 533 (+83.16%)
Mutual labels:  nerf
SCNeRF
[ICCV21] Self-Calibrating Neural Radiance Fields
Stars: ✭ 383 (+31.62%)
Mutual labels:  nerf
Nerf
Code release for NeRF (Neural Radiance Fields)
Stars: ✭ 4,062 (+1295.88%)
Mutual labels:  nerf
View All Similar Projects ➔

NeuRay

Rendered video without training on the scene.

Project page | Paper

Todo List

  • Generalization models and rendering codes.
  • Training of generalization models.
  • Finetuning codes and finetuned models.

Usage

Setup

git clone [email protected]:liuyuan-pal/NeuRay.git
cd NeuRay
pip install -r requirements.txt
Dependencies
  • torch==1.7.1
  • opencv_python==4.4.0
  • tensorflow==2.4.1
  • numpy==1.19.2
  • scipy==1.5.2

Download datasets and pretrained models

  1. Download processed datasets: DTU-Test / LLFF / NeRF Synthetic.
  2. Download pretrained model NeuRay-Depth and NeuRay-CostVolume.
  3. Organize datasets and models as follows
NeuRay
|-- data
    |--model
        |-- neuray_gen_cost_volume
        |-- neuray_gen_depth
    |-- dtu_test
    |-- llff_colmap
    |-- nerf_synthetic

Render

# render on lego of the NeRF synthetic dataset
python render.py --cfg configs/gen/neuray_gen_depth.yaml \  
                 --database nerf_synthetic/lego/black_800 \ # nerf_synthetic/lego/black_400
                 --pose_type eval                 

# render on snowman of the DTU dataset
python render.py --cfg configs/gen/neuray_gen_depth.yaml \  
                 --database dtu_test/snowman/black_800 \ # dtu_test/snowman/black_400
                 --pose_type eval 
                 
# render on fern of the LLFF dataset
python render.py --cfg configs/gen/neuray_gen_depth.yaml \
                 --database llff_colmap/fern/high \ # llff_colmap/fern/low
                 --pose_type eval

The rendered images locate in data/render/<database_name>/<renderer_name>-pretrain-eval/. If the pose_type is eval, we also generate ground-truth images in data/render/<database_name>/gt.

Explanation on parameters of render.py.

  • cfg is the path to the renderer config file, which can also be configs/gen/neuray_gen_cost_volume.yaml
  • database is a database name consisting of <dataset_name>/<scene_name>/<scene_setting>.
    • nerf_synthetic/lego/black_800 means the scene "lego" from the "nerf_synthetic" dataset using "black" background and the resolution "800X800".
    • dtu_test/snowman/black_800 means the scene "snowman" from the "dtu_test" dataset using "black" background and the resolution "800X600".
    • llff_colmap/fern/high means the scene "fern" from the "llff_colmap" dataset using "high" resolution (1008X756).
    • We may also use llff_colmlap/fern/low which renders with "low" resolution (504X378)

Evaluation

# psnr/ssim/lpips will be printed on screen
python eval.py --dir_pr data/render/<database_name>/<renderer_name>-pretrain-eval \
               --dir_gt data/render/<database_name>/gt

# example of evaluation on "fern".
# note we should already render images in the "dir_pr".
python eval.py --dir_pr data/render/llff_colmap/fern/high/neuray_gen_depth-pretrain-eval \
               --dir_gt data/render/llff_colmap/fern/high/gt

Render on custom scenes

To render on custom scenes, please refer to this

Generalization model training

Download training sets

  1. Download Google Scanned Objects, RealEstate10K Space Dataset and LLFF released Scenes from IBRNet.
  2. Download colmap depth for forward-facing scenes at here.
  3. Download DTU training images at here.
  4. Download colmap depth for DTU training images at here.

Rename directories and organize datasets like

NeuRay
|-- data
    |-- google_scanned_objects
    |-- real_estate_dataset # RealEstate10k-subset  
    |-- real_iconic_noface
    |-- spaces_dataset
    |-- colmap_forward_cache
    |-- dtu_train
    |-- colmap_dtu_cache

Train generalization model

Train the model with NeuRay initialized from estimated depth of COLMAP.

python run_training.py --cfg configs/train/gen/neuray_gen_depth_train.yaml

Train the model with NeuRay initialized from constructed cost volumes.

python run_training.py --cfg configs/train/gen/neuray_gen_cost_volume_train.yaml

Models will be saved at data/model. On every 10k steps, we will validate the model and images will be saved at data/vis_val/<model_name>-<val_set_name>

Render with trained models

python render.py --cfg configs/gen/neuray_gen_depth_train.yaml \
                 --database llff_colmap/fern/high \
                 --pose_type eval

Scene-specific finetuning

Finetuning

# finetune on lego from the NeRF synthetic dataset
python run_training.py --cfg configs/train/ft/neuray_ft_depth_lego.yaml

# finetune on fern from the LLFF dataset
python run_training.py --cfg configs/train/ft/neuray_ft_depth_fern.yaml

# finetune on birds from the DTU dataset
python run_training.py --cfg configs/train/ft/neuray_ft_depth_birds.yaml

# finetune the model initialized from cost volume
python run_training.py --cfg configs/train/ft/neuray_ft_cv_lego.yaml

The finetuned models will be saved at data/model.

Finetuned models

We provide the finetuned models on the NeRF synthetic datasets at here.

Download the models and organize files like

NeuRay
|-- data
    |-- model
        |-- neuray_ft_lego_pretrain
        |-- neuray_ft_chair_pretrain
        ...

Render with finetuned models

# render on lego of the NeRF synthetic dataset
python render.py --cfg configs/ft/neuray_ft_lego_pretrain.yaml \  
                 --database nerf_synthetic/lego/black_800 \
                 --pose_type eval \
                 --render_type ft

Code explanation

We have provided explanation on variable naming convention in here to make our codes more readable.

Acknowledgements

In this repository, we have used codes or datasets from the following repositories. We thank all the authors for sharing great codes or datasets.

Citation

@inproceedings{liu2022neuray,
  title={Neural Rays for Occlusion-aware Image-based Rendering},
  author={Liu, Yuan and Peng, Sida and Liu, Lingjie and Wang, Qianqian and Wang, Peng and Theobalt, Christian and Zhou, Xiaowei and Wang, Wenping},
  booktitle={CVPR},
  year={2022}
}
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].