-
Inverse Rendering of Fusion Plasmas: Inferring Plasma Composition from Imaging Systems
Authors:
Ekin Öztürk,
Rob Akers,
Stanislas Pamela,
The MAST Team,
Pieter Peers,
Abhijeet Ghosh
Abstract:
In this work, we develop a differentiable rendering pipeline for visualising plasma emission within tokamaks, and estimating the gradients of the emission and estimating other physical quantities. Unlike prior work, we are able to leverage arbitrary representations of plasma quantities and easily incorporate them into a non-linear optimisation framework. The efficiency of our method enables not on…
▽ More
In this work, we develop a differentiable rendering pipeline for visualising plasma emission within tokamaks, and estimating the gradients of the emission and estimating other physical quantities. Unlike prior work, we are able to leverage arbitrary representations of plasma quantities and easily incorporate them into a non-linear optimisation framework. The efficiency of our method enables not only estimation of a physically plausible image of plasma, but also recovery of the neutral Deuterium distribution from imaging and midplane measurements alone. We demonstrate our method with three different levels of complexity showing first that a poloidal neutrals density distribution can be recovered from imaging alone, second that the distributions of neutral Deuterium, electron density and electron temperature can be recovered jointly, and finally, that this can be done in the presence of realistic imaging systems that incorporate sensor cropping and quantisation.
△ Less
Submitted 14 August, 2024;
originally announced August 2024.
-
Neural Appearance Modeling From Single Images
Authors:
Jay Idema,
Pieter Peers
Abstract:
We propose a material appearance modeling neural network for visualizing plausible, spatially-varying materials under diverse view and lighting conditions, utilizing only a single photograph of a material under co-located light and view as input for appearance estimation. Our neural architecture is composed of two network stages: a network that infers learned per-pixel neural parameters of a mater…
▽ More
We propose a material appearance modeling neural network for visualizing plausible, spatially-varying materials under diverse view and lighting conditions, utilizing only a single photograph of a material under co-located light and view as input for appearance estimation. Our neural architecture is composed of two network stages: a network that infers learned per-pixel neural parameters of a material from a single input photograph, and a network that renders the material utilizing these neural parameters, similar to a BRDF. We train our model on a set of 312,165 synthetic spatially-varying exemplars. Since our method infers learned neural parameters rather than analytical BRDF parameters, our method is capable of encoding anisotropic and global illumination (inter-pixel interaction) information into individual pixel parameters. We demonstrate our model's performance compared to prior work and demonstrate the feasibility of the render network as a BRDF by implementing it into the Mitsuba3 rendering engine. Finally, we briefly discuss the capability of neural parameters to encode global illumination information.
△ Less
Submitted 8 June, 2024;
originally announced June 2024.
-
MatFusion: A Generative Diffusion Model for SVBRDF Capture
Authors:
Sam Sartor,
Pieter Peers
Abstract:
We formulate SVBRDF estimation from photographs as a diffusion task. To model the distribution of spatially varying materials, we first train a novel unconditional SVBRDF diffusion backbone model on a large set of 312,165 synthetic spatially varying material exemplars. This SVBRDF diffusion backbone model, named MatFusion, can then serve as a basis for refining a conditional diffusion model to est…
▽ More
We formulate SVBRDF estimation from photographs as a diffusion task. To model the distribution of spatially varying materials, we first train a novel unconditional SVBRDF diffusion backbone model on a large set of 312,165 synthetic spatially varying material exemplars. This SVBRDF diffusion backbone model, named MatFusion, can then serve as a basis for refining a conditional diffusion model to estimate the material properties from a photograph under controlled or uncontrolled lighting. Our backbone MatFusion model is trained using only a loss on the reflectance properties, and therefore refinement can be paired with more expensive rendering methods without the need for backpropagation during training. Because the conditional SVBRDF diffusion models are generative, we can synthesize multiple SVBRDF estimates from the same input photograph from which the user can select the one that best matches the users' expectation. We demonstrate the flexibility of our method by refining different SVBRDF diffusion models conditioned on different types of incident lighting, and show that for a single photograph under colocated flash lighting our method achieves equal or better accuracy than existing SVBRDF estimation methods.
△ Less
Submitted 23 April, 2024;
originally announced June 2024.
-
DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation
Authors:
Chong Zeng,
Yue Dong,
Pieter Peers,
Youkang Kong,
Hongzhi Wu,
Xin Tong
Abstract:
This paper presents a novel method for exerting fine-grained lighting control during text-driven diffusion-based image generation. While existing diffusion models already have the ability to generate images under any lighting condition, without additional guidance these models tend to correlate image content and lighting. Moreover, text prompts lack the necessary expressional power to describe det…
▽ More
This paper presents a novel method for exerting fine-grained lighting control during text-driven diffusion-based image generation. While existing diffusion models already have the ability to generate images under any lighting condition, without additional guidance these models tend to correlate image content and lighting. Moreover, text prompts lack the necessary expressional power to describe detailed lighting setups. To provide the content creator with fine-grained control over the lighting during image generation, we augment the text-prompt with detailed lighting information in the form of radiance hints, i.e., visualizations of the scene geometry with a homogeneous canonical material under the target lighting. However, the scene geometry needed to produce the radiance hints is unknown. Our key observation is that we only need to guide the diffusion process, hence exact radiance hints are not necessary; we only need to point the diffusion model in the right direction. Based on this observation, we introduce a three stage method for controlling the lighting during image generation. In the first stage, we leverage a standard pretrained diffusion model to generate a provisional image under uncontrolled lighting. Next, in the second stage, we resynthesize and refine the foreground object in the generated image by passing the target lighting to a refined diffusion model, named DiLightNet, using radiance hints computed on a coarse shape of the foreground object inferred from the provisional image. To retain the texture details, we multiply the radiance hints with a neural encoding of the provisional synthesized image before passing it to DiLightNet. Finally, in the third stage, we resynthesize the background to be consistent with the lighting on the foreground object. We demonstrate and validate our lighting controlled diffusion model on a variety of text prompts and lighting conditions.
△ Less
Submitted 27 May, 2024; v1 submitted 19 February, 2024;
originally announced February 2024.
-
In the Blink of an Eye: Event-based Emotion Recognition
Authors:
Haiwei Zhang,
Jiqing Zhang,
Bo Dong,
Pieter Peers,
Wenwei Wu,
Xiaopeng Wei,
Felix Heide,
Xin Yang
Abstract:
We introduce a wearable single-eye emotion recognition device and a real-time approach to recognizing emotions from partial observations of an emotion that is robust to changes in lighting conditions. At the heart of our method is a bio-inspired event-based camera setup and a newly designed lightweight Spiking Eye Emotion Network (SEEN). Compared to conventional cameras, event-based cameras offer…
▽ More
We introduce a wearable single-eye emotion recognition device and a real-time approach to recognizing emotions from partial observations of an emotion that is robust to changes in lighting conditions. At the heart of our method is a bio-inspired event-based camera setup and a newly designed lightweight Spiking Eye Emotion Network (SEEN). Compared to conventional cameras, event-based cameras offer a higher dynamic range (up to 140 dB vs. 80 dB) and a higher temporal resolution. Thus, the captured events can encode rich temporal cues under challenging lighting conditions. However, these events lack texture information, posing problems in decoding temporal information effectively. SEEN tackles this issue from two different perspectives. First, we adopt convolutional spiking layers to take advantage of the spiking neural network's ability to decode pertinent temporal information. Second, SEEN learns to extract essential spatial cues from corresponding intensity frames and leverages a novel weight-copy scheme to convey spatial attention to the convolutional spiking layers during training and inference. We extensively validate and demonstrate the effectiveness of our approach on a specially collected Single-eye Event-based Emotion (SEE) dataset. To the best of our knowledge, our method is the first eye-based emotion recognition method that leverages event-based cameras and spiking neural network.
△ Less
Submitted 6 October, 2023;
originally announced October 2023.
-
Relighting Neural Radiance Fields with Shadow and Highlight Hints
Authors:
Chong Zeng,
Guojun Chen,
Yue Dong,
Pieter Peers,
Hongzhi Wu,
Xin Tong
Abstract:
This paper presents a novel neural implicit radiance representation for free viewpoint relighting from a small set of unstructured photographs of an object lit by a moving point light source different from the view position. We express the shape as a signed distance function modeled by a multi layer perceptron. In contrast to prior relightable implicit neural representations, we do not disentangle…
▽ More
This paper presents a novel neural implicit radiance representation for free viewpoint relighting from a small set of unstructured photographs of an object lit by a moving point light source different from the view position. We express the shape as a signed distance function modeled by a multi layer perceptron. In contrast to prior relightable implicit neural representations, we do not disentangle the different reflectance components, but model both the local and global reflectance at each point by a second multi layer perceptron that, in addition, to density features, the current position, the normal (from the signed distace function), view direction, and light position, also takes shadow and highlight hints to aid the network in modeling the corresponding high frequency light transport effects. These hints are provided as a suggestion, and we leave it up to the network to decide how to incorporate these in the final relit result. We demonstrate and validate our neural implicit representation on synthetic and real scenes exhibiting a wide variety of shapes, material properties, and global illumination light transport.
△ Less
Submitted 25 August, 2023;
originally announced August 2023.
-
Intrinsic Mesh Simplification
Authors:
Randy Shoemaker,
Sam Sartor,
Pieter Peers
Abstract:
This paper presents a novel simplification method for removing vertices from an intrinsic triangulation corresponding to extrinsic vertices lying on near-developable (i.e., with limited Gaussian curvature) and general surfaces. We greedily process all intrinsic vertices with an absolute Gaussian curvature below a user selected threshold. For each vertex, we repeatedly perform local intrinsic edge…
▽ More
This paper presents a novel simplification method for removing vertices from an intrinsic triangulation corresponding to extrinsic vertices lying on near-developable (i.e., with limited Gaussian curvature) and general surfaces. We greedily process all intrinsic vertices with an absolute Gaussian curvature below a user selected threshold. For each vertex, we repeatedly perform local intrinsic edge flips until the vertex reaches the desired valence (three for internal vertices or two for boundary vertices) such that removal of the vertex and incident edges can be locally performed in the intrinsic triangulation. Each removed vertex's intrinsic location is tracked via (intrinsic) barycentric coordinates that are updated to reflect changes in the intrinsic triangulation. We demonstrate the robustness and effectiveness of our method on the Thingi10k dataset and analyze the effect of the curvature threshold on the solutions of PDEs.
△ Less
Submitted 13 July, 2023;
originally announced July 2023.
-
Estimating Homogeneous Data-driven BRDF Parameters from a Reflectance Map under Known Natural Lighting
Authors:
Victoria L. Cooper,
James C. Bieron,
Pieter Peers
Abstract:
In this paper we demonstrate robust estimation of the model parameters of a fully-linear data-driven BRDF model from a reflectance map under known natural lighting. To regularize the estimation of the model parameters, we leverage the reflectance similarities within a material class. We approximate the space of homogeneous BRDFs using a Gaussian mixture model, and assign a material class to each G…
▽ More
In this paper we demonstrate robust estimation of the model parameters of a fully-linear data-driven BRDF model from a reflectance map under known natural lighting. To regularize the estimation of the model parameters, we leverage the reflectance similarities within a material class. We approximate the space of homogeneous BRDFs using a Gaussian mixture model, and assign a material class to each Gaussian in the mixture model. We formulate the estimation of the model parameters as a non-linear maximum a-posteriori optimization, and introduce a linear approximation that estimates a solution per material class from which the best solution is selected. We demonstrate the efficacy and robustness of our method using the MERL BRDF database under a variety of natural lighting conditions, and we provide a proof-of-concept real-world experiment.
△ Less
Submitted 11 June, 2019;
originally announced June 2019.
-
Synthesizing 3D Shapes from Silhouette Image Collections using Multi-projection Generative Adversarial Networks
Authors:
Xiao Li,
Yue Dong,
Pieter Peers,
Xin Tong
Abstract:
We present a new weakly supervised learning-based method for generating novel category-specific 3D shapes from unoccluded image collections. Our method is weakly supervised and only requires silhouette annotations from unoccluded, category-specific objects. Our method does not require access to the object's 3D shape, multiple observations per object from different views, intra-image pixel-correspo…
▽ More
We present a new weakly supervised learning-based method for generating novel category-specific 3D shapes from unoccluded image collections. Our method is weakly supervised and only requires silhouette annotations from unoccluded, category-specific objects. Our method does not require access to the object's 3D shape, multiple observations per object from different views, intra-image pixel-correspondences, or any view annotations. Key to our method is a novel multi-projection generative adversarial network (MP-GAN) that trains a 3D shape generator to be consistent with multiple 2D projections of the 3D shapes, and without direct access to these 3D shapes. This is achieved through multiple discriminators that encode the distribution of 2D projections of the 3D shapes seen from a different views. Additionally, to determine the view information for each silhouette image, we also train a view prediction network on visualizations of 3D shapes synthesized by the generator. We iteratively alternate between training the generator and training the view prediction network. We validate our multi-projection GAN on both synthetic and real image datasets. Furthermore, we also show that multi-projection GANs can aid in learning other high-dimensional distributions from lower dimensional training datasets, such as material-class specific spatially varying reflectance properties from images.
△ Less
Submitted 10 June, 2019;
originally announced June 2019.
-
Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks
Authors:
Xiao Li,
Yue Dong,
Pieter Peers,
Xin Tong
Abstract:
We present a convolutional neural network (CNN) based solution for modeling physically plausible spatially varying surface reflectance functions (SVBRDF) from a single photograph of a planar material sample under unknown natural illumination. Gathering a sufficiently large set of labeled training pairs consisting of photographs of SVBRDF samples and corresponding reflectance parameters, is a diffi…
▽ More
We present a convolutional neural network (CNN) based solution for modeling physically plausible spatially varying surface reflectance functions (SVBRDF) from a single photograph of a planar material sample under unknown natural illumination. Gathering a sufficiently large set of labeled training pairs consisting of photographs of SVBRDF samples and corresponding reflectance parameters, is a difficult and arduous process. To reduce the amount of required labeled training data, we propose to leverage the appearance information embedded in unlabeled images of spatially varying materials to self-augment the training process. Starting from an initial approximative network obtained from a small set of labeled training pairs, we estimate provisional model parameters for each unlabeled training exemplar. Given this provisional reflectance estimate, we then synthesize a novel temporary labeled training pair by rendering the exact corresponding image under a new lighting condition. After refining the network using these additional training samples, we re-estimate the provisional model parameters for the unlabeled data and repeat the self-augmentation process until convergence. We demonstrate the efficacy of the proposed network structure on spatially varying wood, metals, and plastics, as well as thoroughly validate the effectiveness of the self-augmentation training process.
△ Less
Submitted 4 September, 2018;
originally announced September 2018.
