MULTIMODAL DATA VISUALIZATION AND DENOISING WITH INTEGRATED DIFFUSION

2.1. Manifold learning via data diffusion

Intuitively, while measurement strategies often produce high dimensional observations, their intrinsic dimensionality, or number of degrees of freedom, is relatively low. In [4], diffusion maps were proposed as a robust way to capture intrinsic manifold geometry in dataset by eigendecomposing diffusion operators. Using t-step random walks that aggregate local affinity, nonlinear relations are revealed in data, which allows their embedding in low dimensions. These local affinities are commonly constructed using a Gaussian kernel

where K is an N × N Gram matrix whose (i, j) entry is denoted by K(x_i, x_j) to emphasize the dependency on X, based on bandwidth parameter ε that controls local neighborhood sizes. A diffusion operator is defined as the row-stochastic

where D is diagonal matrix: $D\left(x_i,x_i\right)={\sum }_{j}K\left(x_i,x_j\right)$.

The matrix P, or diffusion operator, defines single-step transition probabilities for a time-homogeneous diffusion process, or a Markovian random walk, over the data. The eigenvectors of P, denoted Φ = ϕ₀, ϕ₁, …, ϕ_n, represent frequency harmonics over the graph based on equivalence to eigenvectors of a normalized graph Laplacian. The eigenvalues Λ = λ₁, λ₂, λ_i … λ_n directly indicate frequencies, as they are related to the eigenvectors.

The eigenvectors of the diffusion operator are equivalent to those of the normalized graph Laplacian L = I − P = D⁻¹L_u = D⁻¹(D − K), where I is the identity matrix, D is the degree matrix, K is the kernel affinity, L_u is the unnormalized graph Laplacian. Graph Laplacian eigenvectors have been shown to be equivalent to graph frequency harmonics [8]. Thus, signal loadings on to diffusion eigenvectors create a graph Fourier transform defined as Φ^T f for a graph signal f.

Signals can be filtered using the graph Fourier transform by altering their loading coefficients on to eigenvectors of the graph Laplacian. Thus, a graph filter can be defined as

using a slight abuse of notation with Λ being a diagonal matrix of eigenvalues. Here, h rescales the eigenvalues to modulate frequency components of f. In [4], powers P^t of the diffusion operator, for t > 0, not only simulate t step random walks over the data, but can also be seen as soft low-pass graph filters $h\left({\lambda }_i,t\right)={\lambda }_i^t$, which diminish higher frequency noise components more rapidly than lower frequency informative components. In [5], such filters were used on biological data to denoise single cell RNA sequencing measurements by simply applying the powered diffusion operator (or diffusion filter) to the data as

thus avoiding eigendecomposition.

2.2. Alternating diffusion

Recently, alternating diffusion has been proposed to combine diffusion operators created from multimodal data [9]. Intuitively, this generalizes the random walk to “hop” between different metric spaces by taking a matrix product of the Markov transition matrices

Finally, the resultant alternating diffusion operator P is powered to stimulate “hopping” across modalities. While this approach is able to construct a joint diffusion operator, it is sensitive to local and global noise found in each dataset creating a joint manifold that represents not only modality specific sources of information, but also noise.

3.1. Problem formulation and approach

Let $X\subseteq {ℝ}^{D_X}$ and $Y\subseteq {ℝ}^{D_Y}$ be two datasets generated by measuring the same system with two sets of sensors in different metric spaces. Each of these datasets contains nonoverlapping features with different scales, and is subject to differing degrees of noise. Here, we propose an approach for generating an integrated diffusion operator that selects information from both modalities at multiple scales. Our approach is based on the idea that frequency components of the diffusion operator can be low-pass filtered using particular powers of this operators. By using multiple scales, we perform both local and global denoising of the data modalities to a degree where highly relevant information is retained in the joint integrated operator. This integrated diffusion operator can then be used to visualize and denoise both datasets (Fig. 1).

3.2. Neighborhood reconstruction via multiscale spectral denoising

Specific areas of each modality’s data manifold can contain different amounts of noise that may obscure structure in joint embeddings. Therefore, we translate the global denoising idea from [5] to local regions and multiple scales by creating and applying hierarchical sets of diffusion operators as described in Alg. 3.2. This recursive approach calculates increasingly local diffusion operators and denoises the original modality-specific data at multiple scales. At each scale, the original input data, represented in by X (see Alg. 3.2), is averaged with the denoised data $\widehat{X}$. Each scale of the hierarchy contributes half as much correction as the previous scale, with the overall effect summing to one. We apply this modality specific local denoising approach to correct all data modalities before integrating them. It should be noted that the filter we use from Eq. 4 can be replaced with a more general filter, i.e., any filter that takes the form of Eq. 3.

3.3. Global denoising via spectral entropy

In addition to correcting for varying local noise within a single modality, it is crucial to only maintain the most globally important eigenvectors in each data modality. While [5] did globally denoise by taking the diffusion operator powers P^t, the methodology used there for tuning t essentially required manual trial and error. Such tuning is crucial here as a small t could incorporate significant modality specific noise, while a high t could improperly diminish the effect of informative eigenvectors. Therefore, we propose to select t for each data modality separately by using spectral entropy to evaluate how much information it encodes in the diffusion operator for each candidate value of t. We can then methodically tune t to be a scale where the information loss stabilizes, with the reasoning that signals are harder to remove than the noise.

Spectral entropy is defined as the Shannon entropy of normalized eigenvalues, i.e.,

which in this context quantifies the spread of information throughout the eigenspectrum of the diffusion operator. We reason that innumerable noise dimensions will quickly drop off while informative dimensions are harder to remove. Hence, we choose the elbow of this curve to find an inflection point k in the spectral entropy S(p) as can be seen in Fig. 4C.

3.4. Fusion of operators

While the spectral entropy heuristic is used to compute t for each modality independently, it cannot directly be used to tune timescales across modalities to combine their diffusion operators together. Since noise may be inherently present in the system being measured, and therefore be present in both datasets, the integrated operator must again be used by taking its powers to a given time scale. Powering directly by t₁, t₂ and t_integrated would lead to an oversmoothing effect that would eliminate information from the low frequency eigenvectors in the final computed manifold, effectively collapsing independent informative data points together. To alleviate this concern, we raise each modalities diffusion operator to the lowest possible multiple of the ideal view-specific t. This means we can write our integrated diffusion operator, J, to reflect the differing levels of global information between views as

where t₁ and t₂ are integer values obtained from the reduced ratio as described above, and P₁ and P₂ are modality specific diffusion operators. This integrated diffusion operator can be applied directly to one of the data modalities as a low pass denoising diffusion filter as done in Eq. 4 or can be powered and embedded using the methods of [4] or [7].

4.1. Visualization

To quantify the differences in visualizations produced from differing multimodal integration strategies, we compared the first 20 diffusion map components computed from diffusion operator constructions based off of multimodal feature concatenation, distance addition, affinity addition, affinity multiplication and alternating diffusion, to integrated diffusion. We also performed ablation studies, comparing these techniques to various diffusion operators: alternating diffusion with local correction and alternating diffusion with modality specific powering of diffusion operators via spectral entropy ratio. We also compared to non-diffusion based embeddings produced by cycle GANs, autoencoders and domain transfer autoencoders. For our MNIST comparisons, we train a kNN-classifier to predict MNIST digit of origin from the embedding trained from each technique. For our tree comparisons, with branches with differing amounts of local noise, we try to determine how successful our embeddings are using DeMAP (Denoised Manifold-Affinity Preservation) proposed in [7]. DeMAP computes geodesic distance between all data points in a noiseless dataset and correlates it with distance between these data points in an embedding. This method tries to determine if the embedding accurately maintains ground truth point to point distances in compressed space.

All strategies performed comparably when both modalities had a similar degree of local and global noise. As the difference in global noise increased in our MNIST embedding classification task, strategies that powered the diffusion operators to account for global noise outperformed strategies that did not (Fig. 2C). When embedding trees with varying degrees of local branch specific noise, methods that perform local correction with multiscale spectral denoising significantly outperformed methods that did not (Fig. 2D).

4.2. Denoising

Previous work in diffusion filters has shown that low pass filtering can correct many types of noise present in real world datasets, allowing for downstream analysis [5]. Here, we compare several methods for data denoising with our integrated diffusion approach. As done previously, we created multimodal MNIST data by adding differing amounts of global noise. After computing the integrated diffusion operator with each of these comparison methods, we filter the noisier MNIST modality as done previously [5] and as can be seen in Fig. 3A. To get quantitative results, we train a kNN-classifier on the denoised pixel values to determine how well each operator is able to predict the digit (Fig. 3). As shown in Fig. 3B, across all denoising comparisons, classification accuracy on increasingly noisy MNIST digits were best recovered by integrated diffusion followed by alternating diffusion with modality specific view powering, both methods account for global information within each noisy modality.