0% found this document useful (0 votes)

47 views94 pages

DeepLearning L9

This document discusses several neural rendering techniques: 1. RenderNet uses a deep convolutional network to perform differentiable rendering from 3D shapes to images. It encodes a 3D volume into "neural voxels" and uses 3D-2D convolutions to compute visibility and shading. 2. Neural importance sampling trains a network to predict illumination from a scene to guide Monte Carlo rendering. 3. Adversarial Monte Carlo denoising uses an encoder-decoder network conditioned on auxiliary features like albedo to denoise renderings.

Uploaded by

Ines ZBh

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

47 views94 pages

DeepLearning L9

Uploaded by

Ines ZBh

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 94

Neural Rendering

Chuan Li

Lambda Labs
Collaborators: Thu Nguyen-Phuoc, Bing Xu, Yongliang Yang, Stephen Balaban, Lucas Theis, Christian Richardt, Junfei
Zhang, Rui Wang, Kun Xu, Rui Tang
Forward (Computer Graphics)

Model Pictures
Forward (Computer Graphics)

Model Pictures

Inverse (Computer Vision)

Integral of the incident radians
BRDF
32K SPP Ray Tracing (90 mins 12 CPU Cores)
The Tungsten Renderer
P0

P1
P0

R
01 |T
01
Inverse (Computer Vision)

P1 P2

R
01 |T
T 12
01
R 12 |
Building Rome in a Day
Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz and Richard Szeliski
Sub-module
End-2-End

Model Pictures

Diﬀerentiable Rendering
1 SPP

2048 SPP
Sub-modules

Mastering the game of Go with deep neural networks and tree search
David Silver et al.
Sub-modules

Value Network

Mastering the game of Go with deep neural networks and tree search
David Silver et al.
Sub-modules

Policy Network

Value Network

Mastering the game of Go with deep neural networks and tree search
David Silver et al.
Value Networks
Denoising

4 SPP 2^15 SPP

Value Networks
Denoising

4 SPP 2^15 SPP

Policy Networks
Same SPP
Value Networks
Denoising

4 SPP 2^15 SPP

Policy Networks
Same SPP
4 SPP Denoised 32K SPP Ray Tracing
1 sec 2080 Ti 90 mins 12 cores CPU
Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation
B Xu et al. Siggraph Asia 2019
Input Encoder x Decoder Output

loss
Ref

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation

B Xu et al. Siggraph Asia 2019
L1 VGG Loss

L1 VGG Loss + GAN

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation

B Xu et al. Siggraph Asia 2019
Diffuse Diffuse
Input
Encoder x Decoder
Output
Output

loss
Specular Specular
Input
Encoder x Decoder
Output
Ref

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation

B Xu et al. Siggraph Asia 2019
Diffuse Diffuse
Input
Encoder x Decoder
Output
Output

loss
Specular Specular
Input
Encoder x Decoder
Output
Ref

Auxiliary
Albedo, normal, depth
x

Element-wise
Biasing
Conv

LeakyReLU

Conv

Auxiliary
x

Element-wise Element-wise
Scaling Biasing
Conv Conv

LeakyReLU LeakyReLU

Conv Conv

Auxiliary
x

Element-wise Element-wise
Scaling (AND) Biasing (OR)
Conv Conv

LeakyReLU LeakyReLU

Conv Conv

Auxiliary
Denoise comparison 4 SPP

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation

B Xu et al. Siggraph Asia 2019
Value Networks
Denoising

4 SPP 2^15 SPP

Policy Networks
Same SPP
Neural Importance Sampling
Thomas Müller et al. ACM Transactions on Graphics 2019
incidence radiance map

Neural Importance Sampling

Thomas Müller et al. ACM Transactions on Graphics 2019
Neural Importance Sampling
Thomas Müller et al. ACM Transactions on Graphics 2019
Neural Importance Sampling
Thomas Müller et al. ACM Transactions on Graphics 2019
Sub-module
End-2-End

Model Pictures

Diﬀerentiable Rendering
Ray Tracing Rasterization
Image Centric Object Centric
Visibility

Ray Tracing Rasterization

Image Centric Object Centric
Shading

Ray Tracing Rasterization

Image Centric Object Centric
Depth Map Voxel Point Cloud Mesh

Memory Good Very Poor Poor Very Good

NN friendly Great Yes No Enemy

Depth Map Voxel Point Cloud Mesh

Memory Good Very Poor Poor Very Good

NN friendly Great Yes No Enemy

Depth Map Voxel Point Cloud Mesh

Memory Good Very Poor Poor Very Good

NN friendly Great Yes No Enemy

Depth Map Voxel Point Cloud Mesh

Memory Good Very Poor Poor Very Good

NN friendly Great Yes No Enemy

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018
Neural Voxels
3D Neural
Encoder Voxels

32 x 32 x 32 x 16

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018
Neural Voxels Visibility
3D Neural Neural
3D-2D
Encoder Voxels Pixels

32 x 32 x 32 x 16 32 x 32 x 512

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018
Neural Voxels Visibility
3D Neural Neural
3D-2D
Encoder Voxels Pixels

32 x 32 x 32 x 16 32 x 32 x 512

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018
Neural Voxels Visibility Shading
3D Neural Neural 2D
3D-2D
Encoder Voxels Pixels Decoder

32 x 32 x 32 x 16 32 x 32 x 512

MSE pixel loss

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018
RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018
Contour

Toon

Ambient Occlusion
RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018
RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018
RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018
3D Neural Neural 2D
3D-2D
Encoder Voxels Pixels Decoder

64 x 64 x 64 x 1

Channel-wise Concatenation

Texture Neural
or Texture
Network
Voxels

64 x 64 x 64 x 4

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018
RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018
Same shape, diﬀerent textures

Same texture, diﬀerent shapes

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018
Depth Map Voxel Point Cloud Mesh

Memory Good Very Poor Poor Very Good

NN friendly Great Yes No Enemy

Rasterization a RGB point cloud

Neural Point-Based Graphics

KA Aliev et al, arxiv 2019
Rasterization a neural point cloud
(First three PCA dimensions of the neural descriptor)

Neural Point-Based Graphics

KA Aliev et al, arxiv 2019
Rasterization a neural point cloud
(First three PCA dimensions of the neural descriptor)

Neural Point-Based Graphics

KA Aliev et al, arxiv 2019
RBG rasterization
Neural rasterization

Neural Point-Based Graphics

KA Aliev et al, arxiv 2019
Deferred Neural Rendering:
Image Synthesis using Neural Textures Neural 3D Mesh Renderer
H Kato et al, CVPR 2018
J Thies et al, Siggraph 2019
Sub-module
End-2-End

Model Pictures
?
Approximation Target
Approximation Rendered Target
Approximation
Approximation Rendered Target
Approximation

Loss

Back-propagate
Updated Rendered Target
Approximation Approximation

Loss

Back-propagate
Updated Rendered Target
Approximation Approximation

Loss

Back-propagate
For Free
Updated Rendered Target
Approximation Approximation

Expensive
Loss

Back-propagate
Rendered Target
Approximation

Encoder Decoder

Reconstruction Rendering Loss

Inductive Bias: Separate Appearance from Pose

Human perception imposes coordinate frame on objects

Learning 3D representation from natural images without 3D supervision

HoloGAN: Unsupervised learning of 3D representations from natural images

Thu Nguyen-Phuoc et al, ICCV 2019
Conditional GANs

HoloGAN: Unsupervised learning of 3D representations from natural images

Thu Nguyen-Phuoc et al, ICCV 2019
Conditional GANs

Info GANs

HoloGAN: Unsupervised learning of 3D representations from natural images

Thu Nguyen-Phuoc et al, ICCV 2019
HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019
3D Generator RenderNet