Semi-Confirmatory Factor Analysis for High-Dimensional Data with Interconnected Community Structures

Abstract

Confirmatory factor analysis (CFA) is a statistical method for identifying and confirming the presence of latent factors among observed variables through the analysis of their covariance structure. Compared to alternative factor models, CFA offers interpretable common factors with enhanced specificity and a more adaptable approach to modeling covariance structures. However, the application of CFA has been limited by the requirement for prior knowledge about “non-zero loadings” and by the lack of computational scalability (e.g., it can be computationally intractable for hundreds of observed variables). We propose a data-driven semi-confirmatory factor analysis (SCFA) model that attempts to alleviate these limitations. SCFA automatically specifies “non-zero loadings” by learning the network structure of the large covariance matrix of observed variables, and then offers closed-form estimators for factor loadings, factor scores, covariances between common factors, and variances between errors using the likelihood method. Therefore, SCFA is applicable to high-throughput datasets (e.g., hundreds of thousands of observed variables) without requiring prior knowledge about “non-zero loadings”. Through an extensive simulation analysis benchmarking against standard packages, SCFA exhibits superior performance in estimating model parameters with a much reduced computational time. We illustrate its practical application through factor analysis on a high-dimensional RNA-seq gene expression dataset.

section INTRODUCTION

Factor analysis is a commonly used statistical technique to elucidate the relationship between multivariate observations. Factor models aim to identify the underlying factors that collectively describe the interdependencies present in multivariate observed data (Anderson, 2003). As correlated high-dimensional observed variables can be effectively decomposed into a smaller number of common factors, i.e., achieving dimension reduction, factor analysis models have garnered popularity in various fields, including social science, psychology, molecular biology, and others (Schreiber et al., 2006; Fan et al., 2020, 2021).

Factor analysis is broadly classified into two categories: exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). EFA is frequently employed to explore the interactive relationships among observed variables and to identify latent common factors, without prerequisite knowledge of grouping these observed variables. In contrast, CFA is commonly utilized to validate whether the empirical evidence supports a predetermined latent structure of the shared variance in the model specification. In a CFA model, prerequisite knowledge or empirical evidence regarding the grouping of observed variables is represented by predefined “non-zero loadings” in the factor loading matrix, establishing a rule or a factor membership that exclusively and exhaustively assigns each observed variable to a certain common factor (Browne, 2001). In practical applications, a combined approach is often adopted, starting with EFA to investigate the underlying dependence pattern, followed by CFA for model verification and justification (Basilevsky, 2009; Brown, 2015; Gana & Broc, 2019).

The above distinctive model specifications between EFA and CFA confer unique strengths and limitations to each approach. EFA exhibits greater flexibility by not necessitating specified “non-zero loadings” and can be adapted to accommodate high-dimensional observations, reaching thousands of observed variables or more (Friguet et al., 2009; Bai & Li, 2012; Fan et al., 2013; Fan & Han, 2017). However, the factor loadings in classical EFA models are typically non-zero, resulting in less interpretable relationships between common factors and observed variables. Moreover, EFA models are limited to explicitly estimating the covariance matrix of common factors for high-dimensional data. In contrast, CFA naturally specifies a sparse factor loading matrix guided by prior knowledge, enhancing interpretability, as exemplified in Carlson & Mulaik (1993), and establishes arbitrary covariances between common factors, as demonstrated in Lawley (1958) and Jackson et al. (2009). As previously mentioned, classical CFA models encounter two primary limitations: (1) pre-determined “non-zero loadings” in the factor loading matrix or factor membership is typically lacking, resulting in the nonexistence of a rule that exclusively and exhaustively assigns each observed variable to a certain common factor; and (2) the computational burden of estimating a CFA model in high-dimensional scenarios becomes intractable because the existing standard computational packages struggle to handle datasets containing hundreds or thousands of observed variables (Fox, 2006; Rosseel, 2012; Oberski, 2014).

In the current research, we concentrate on the CFA approach while attempting to address the two aforementioned limitations. We propose a semi-confirmatory factor analysis (SCFA) model that addresses the specification of “non-zero loadings” through the covariance structure learned from high-dimensional data, and significantly alleviates the computational burden with theoretically guaranteed solutions in closed form. Specifically, to overcome the first limitation, we incorporate a prevalent covariance structure, namely, the interconnected community structure, into the conventional CFA model. We focus on the interconnected community structure, as it is widely prevalent in the covariance matrices of various high-dimensional datasets, as illustrated in Figure figure. Notably, interconnected community structures, which enable features between communities to exhibit correlations, encompass various well-known patterns, including all independent community structures (Newman & Girvan, 2004; Fortunato, 2010) and most hierarchical community structures (Li et al., 2022; Schaub et al., 2023). Therefore, they provide a versatile covariance structure to model various practical applications, including brain imaging, gene expression, multi-omics, metabolomics, and more (Girvan & Newman, 2002; Colizza et al., 2006; Simpson et al., 2013; Levine et al., 2015; Huttlin et al., 2017; Zitnik et al., 2018; Perrot-Dockès et al., 2022). However, we acknowledge that the proposed structures can be limited in representing certain covariance patterns (e.g., Toeplitz). In such cases, traditional factor models remain suitable. Interconnected community structures are latent in many studies, which can be accurately and robustly estimated and extracted by recently developed network structure detection approaches (Wang et al., 2020; Li et al., 2022; Yang et al., 2024). Consequently, the detected community membership can serve as a guide to specifying the previously unknown “non-zero loadings” for CFA, effectively addressing the first limitation. The SCFA model also alleviates the computational burden by deriving closed-form solutions for all CFA model parameters, including the factor loadings, factors, covariance matrix between common factors, and covariance matrix for error terms. The closed-form estimators not only improve estimation accuracy and stability but also drastically reduce computational load and ensure scalability (e.g., handling thousands of observed variables).

Figure figure: We provide examples demonstrating that the interconnected community structures are frequently observed across various real applications: A: a brain imaging study (Chiappelli et al., 2019); B: a gene expression study (Tomczak et al., 2015); C: a multi-omics study (Perrot-Dockès et al., 2022); D: a plasma metabolomics study (Ritchie et al., 2023); E and F: an environmental study involving exposome and plasma metabolomics (ISGlobal, 2021).

SCFA presents several methodological contributions. Firstly, SCFA alleviates the requirement of “non-zero loadings” in CFA models. The acquired interconnected community structure assigns observed variables to common factors, specifying “non-zero loadings” in an adaptive manner. Secondly, SCFA provides a computationally efficient approach for conducting high-dimensional CFA (e.g., with thousands or more observed variables). All estimators are obtained by the likelihood approach and in closed form, substantially mitigating the computational burden. Thirdly, SCFA yields more accurate and reliable estimates, since all matrix estimators are uniformly minimum-variance unbiased estimators (UMVUEs). The factor scores can also be conveniently estimated using the feasible generalized least-square (FGLS) method. We further show that FGLS estimators have an identical solution to those obtained through ordinary least-square (OLS) and generalized least-square (GLS) methods. Lastly, we derive explicit variance estimators that facilitate statistical inference concerning model parameters and factor scores in a SCFA model.

The remainder of the paper is structured as follows. In Section section, we introduce the SCFA model, detailing its specifications for the factor loading matrix and the covariance matrix of observations. We subsequently present the estimation and inference procedures for all unknown matrices and factor scores. Section section and Section section are dedicated to evaluating the proposed model and approach. Section section assesses the performance of our model through simulated data, while Section section demonstrates the application to a genomics dataset without prior knowledge of “non-zero loadings”. All proofs and additional tables are included in the Supplementary Material.

section METHOD

section.subsection Background

Confirmatory factor analysis model. Let $X_{p\times 1}$ represent a $p$-dimensional vector of observations, $f_{K\times 1}$ denote the $K$-dimensional vector of common factors, with $1<K<p$, and $u_{p\times 1}$ denote the $p$-dimensional vector of error terms. A factor model can be expressed as

$$X_{p\times 1}=\mu_{p\times 1}+L_{p\times K}f_{K\times 1}+u_{p\times 1},\operatorname{E}(f)=0_{K\times 1},\operatorname{E}(u)=0_{p\times 1},\operatorname{cov}\left(f,u\right)=0_{K\times p},$$ (equation)

where $0_{p\times q}$ represents the $p$ by $q$ zero matrix; without loss of generality, the $p$-dimensional mean vector is denoted by $\mu=0_{p\times 1}$; and $L$ represents the $p$ by $K$ factor loading matrix. Furthermore, let $\Sigma=\operatorname{cov}(X)$, $\Sigma_{f}=\operatorname{cov}(f)$, and $\Sigma_{u}=\operatorname{cov}(u)$ denote the $p$ by $p$, $K$ by $K$, and $p$ by $p$ covariance matrices, respectively, where $\Sigma_{u}$ is assumed to be diagonal (Fan et al., 2021).

When performing CFA, we may introduce zero entries at specified positions in the factor loading matrix $L$ (Jöreskog, 1969) and assume the common factors to be oblique, e.g., the covariance matrix of common factors $\Sigma_{f}$ can be arbitrarily positive definite. Following the model in (equation) and these two assumptions of $L$ and $\Sigma_{f}$, we can derive the following relationship between covariance matrices in the CFA model:

$$\Sigma=L\Sigma_{f}L^{\mathrm{\scriptscriptstyle T}}+\Sigma_{u},$$ (equation)

where ${\mathrm{\scriptscriptstyle T}}$ denotes the transpose, $L=\operatorname{Bdiag}(\ell_{1},\ldots,\ell_{K})$ becomes a $p$ by $K$ block-diagonal matrix with $p_{k}$-dimensional “non-zero loadings” $\ell_{k}$ for $k=1,\ldots,K$; $\Sigma_{f}=\left(\sigma_{f,kk^{\prime}}\right)$ is a $K$ by $K$ symmetric positive-definite matrix; and $\Sigma_{u}=\left(\Sigma_{u,kk^{\prime}}\right)$ is a $p$ by $p$ diagonal positive-definite matrix satisfying the submatrix $\Sigma_{u,kk^{\prime}}$ is a $p_{k}$ by $p_{k}^{\prime}$ diagonal matrix if $k^{\prime}=k$ or zero matrix if $k^{\prime}\neq k$. In particular, $p_{k}$ represents the number of observed variables within the $k$th factor, for $k=1,\ldots,K$, satisfying that each $p_{k}>1$ and $p=p_{1}+\cdots+p_{K}$.

We remark that the non-overlapping factor loading pattern in (equation) permits only a single non-zero entry within each row. This configuration aligns with the preferences of the majority of existing complexity criteria (Browne, 2001).

Covariance matrix $\Sigma$ in a block form. As defined in (equation), the covariance matrix of observed variables $\Sigma$ is structured as a block matrix. Specifically, the block structure of $\Sigma=\left(\Sigma_{kk^{\prime}}\right)$ is determined by the “non-zero loadings” $\ell_{k}$ in $L$ for $k=1,\ldots,K$:

$$\Sigma=\left(\Sigma_{kk^{\prime}}\right)=\begin{pmatrix}\Sigma_{11}&\Sigma_{12}&\dots&\Sigma_{1K}\\ \Sigma_{21}&\Sigma_{22}&\dots&\Sigma_{2K}\\ \vdots&\vdots&\ddots&\vdots\\ \Sigma_{K1}&\Sigma_{K2}&\dots&\Sigma_{KK}\end{pmatrix},\quad\Sigma_{kk^{\prime}}=\sigma_{f,kk^{\prime}}\ell_{k}\ell_{k^{\prime}}^{\mathrm{\scriptscriptstyle T}}+\Sigma_{u,kk^{\prime}}\in\mathbb{R}^{p_{k}\times p_{k^{\prime}}},$$

for all $k$ and $k^{\prime}$, where $\Sigma_{k^{\prime}k}=\Sigma_{kk^{\prime}}$ for $k^{\prime}\neq k$ and $\Sigma_{u,kk^{\prime}}$ is a diagonal matrix.

In practical applications, it is often the case that information about both the components and sizes of each “non-zero loadings” $\ell_{k}$ is unavailable, particularly in domains such as omics and imaging data. Nevertheless, covariance matrices in these applications often exhibit block patterns, albeit these block patterns may be latent and require pattern extraction algorithms. Recent advancements in statistics have provided convenient tools to accurately identify latent block structures in covariance matrices and precisely estimate covariance parameters (Lei & Rinaldo, 2015; Wu et al., 2021; Li et al., 2022). Inspired by the above block structure of $\Sigma=\left(\Sigma_{kk^{\prime}}\right)$ in the CFA model, we are motivated to extract knowledge regarding all “non-zero loadings” from structured covariance matrix estimation for high-dimensional data, thereby facilitating the estimation of the CFA model, as elaborated in the following sections.

Interconnected community structure. The block-structured covariance matrix in the CFA model is naturally linked with the community-based network structure (Wu et al., 2021). In the current research, we focus on the interconnected community structure, which is more general and prevalent in real-world applications. As demonstrated in Figure figure, the interconnected community covariance structure presents in the high-throughput datasets such as genetics, imaging, gene-expression, DNA-methylation, and metabolomics, among many (Yang et al., 2024). Therefore, we build the proposed model based on the interconnected community covariance structure for factor analysis on these datasets.

We characterize the interconnected community covariance structure as follows. In this structure, all features can be categorized into multiple mutually exclusive and exhaustive communities. In other words, there implicitly exists a community-membership function $\varphi$ that operates as a bijection. In contrast to the classical community network structure, the interconnected community structure exhibits correlations among features within and between communities (i.e., interactive communities), as demonstrated in Figure figure. Thus, the (population) covariance matrix with an interconnected community structure has a block structure: diagonal blocks represent the intra-community correlations while the off-diagonal blocks characterize the inter-community relationships. Due to the high resemblance of entries within each block, a parametric covariance model is often used by assigning two (or one) parameters for each diagonal (or off-diagonal) block (Yang et al., 2024).

A two-step procedure is commonly employed for estimating large covariance matrices in these datasets: firstly, extracting the latent structure from the sample covariance matrix, and subsequently estimating the covariance parameters under the learned covariance structure (Chen et al., 2018; Wu et al., 2021; Chen et al., 2023). Given the reliable and replicable performance of existing algorithms in extracting latent network structures (Chen et al., 2018; Wang et al., 2020; Li et al., 2022) (see almost identical results of detected interconnected structures by different algorithms in the Supplementary Material), it is plausible to consider the estimated interconnected community structure of the covariance matrix as “known” prior knowledge for SCFA models.

Figure figure: The overview of SCFA from the perspective of a covariance matrix. The left subfigure illustrates the population covariance matrix $\Sigma$ with an explicit interconnected community structure. The parameters of the SCFA model, $L$, $\Sigma_{f}$, and $\Sigma_{u}$, are linked with $\Sigma$ as demonstrated by the right-hand side of the equation. The community membership in the interconnected community structure can specify the “non-zero loadings” in $L$, while $\Sigma_{f}$ and $\Sigma_{u}$ are determined by the parametric covariance structure in $\Sigma$.

section.subsection Semi-parametric confirmatory analysis model

We propose a semi-confirmatory factor analysis (SCFA) model for multivariate $X$ with a covariance matrix $\Sigma$ having an interconnected community structure. In this section, we will show that $L$, $\Sigma_{f}$, and $\Sigma_{u}$ in the SCFA model can be represented by the parameters in the parametric $\Sigma$ having an interconnected community structure in closed forms, as demonstrated in Figure  figure. This reparameterization facilitates accurate and computationally efficient estimation of $L$, $\Sigma_{f}$, and $\Sigma_{u}$. Specifically, following (equation) and (equation), we have:

$$X=Lf+u,\quad\Sigma=L\Sigma_{f}L^{\mathrm{\scriptscriptstyle T}}+\Sigma_{u},\text{ where }$$ (equation)

$$\begin{aligned} L_{p\times K}&=\operatorname{Bdiag}\left(\ell_{1},\ldots,\ell_{K}\right)\text{ with }\ell_{k}=\begin{pmatrix}\ell_{\bar{p}_{k-1}+1,k}\\ \vdots\\ \ell_{\bar{p}_{k},k}\end{pmatrix}_{p_{k}\times 1},\ell_{\bar{p}_{k-1}+1,k}=\tau_{k}\neq 0,\\ \Sigma_{p\times p}&=\left(\Sigma_{kk^{\prime}}\right)=\begin{pmatrix}a_{11}I_{p_{1}}+b_{11}J_{p_{1}}&b_{12}1_{p_{1}\times p_{2}}&\cdots&b_{1K}1_{p_{1}\times p_{K}}\\ b_{21}1_{p_{2}\times p_{1}}&a_{22}I_{p_{2}}+b_{22}J_{p_{2}}&\cdots&b_{2K}1_{p_{2}\times p_{K}}\\ \vdots&\vdots&\ddots&\vdots\\ b_{K1}1_{p_{K}\times p_{1}}&b_{K2}1_{p_{K}\times p_{2}}&\cdots&a_{KK}I_{p_{K}}+b_{KK}J_{p_{K}}\end{pmatrix},b_{kk^{\prime}}=b_{k^{\prime}k},\end{aligned}$$ (equation)

where $k,k^{\prime}=1,\ldots,K$ and $K$, the number of common factors, is set to be the number of interconnected communities. $p_{1},\ldots,p_{K}$ are set to be the interconnected community sizes satisfying that $p_{k}>1$ for every $k$ and $p=p_{1}+\cdots+p_{K}$, and $\bar{p}_{k}=\sum_{k^{\prime}=1}^{k}p_{k^{\prime}}$ is the sum (we define $\bar{p}_{0}=0$). Without loss of generality, the first entry in $\ell_{k}$ is assumed to be a known constant $\tau_{k}\neq 0$, i.e., $\ell_{\bar{p}_{k-1}+1,k}=\tau_{k}$ is known, while the other entries in $\ell_{k}$ are unknown for every $k$. $I_{p}$ denotes the $p$ by $p$ identity matrix, $J_{p}$ and $1_{p\times q}$ denote $p$ by $p$ and $p$ by $q$ all-one matrices, respectively. $a_{kk}$ and $b_{kk^{\prime}}$ are (unknown) entries of the parametric population covariance matrix.

The parametric covariance matrix $\Sigma$ defined in (equation) is derived from covariance patterns inherent in interconnected community structures (Yang et al., 2024). When viewed through the lens of network analysis, $\Sigma$ takes a block form, where each diagonal block represents a uniform correlation relationship between features within a single community, and each off-diagonal block represents a uniform correlation relationship between features from two distinct communities.

The SCFA model is distinct from the conventional CFA model because it can leverage the covariance matrix with an interconnected community structure to derive closed-form solutions for all parameters in the factor loading matrix $L$ and in the covariance matrices $\Sigma_{f}$ and $\Sigma_{u}$.

Specifying all “non-zero loadings” in $L$. In the SCFA model, we specify the “non-zero loadings” in $L$ based on the interconnected community structure. For an interconnected community structure of $K$ communities with corresponding community sizes $\ell_{1},\ldots,\ell_{K}$, the factor loading matrix is

$$L=\begin{pmatrix}\ell_{\bar{p}_{0}+1,1}&&&\\ \ell_{\bar{p}_{0}+2,1}&&&\\ \vdots&&&\\ \ell_{\bar{p}_{1},1}&&&\\ &\ell_{\bar{p}_{1}+1,2}&&\\ &\ell_{\bar{p}_{1}+2,2}&&\\ &\vdots&&\\ &\ell_{\bar{p}_{2},2}&&\\ &&\ddots&\\ &&&\ell_{\bar{p}_{K-1}+1,K}\\ &&&\ell_{\bar{p}_{K-1}+2,K}\\ &&&\vdots\\ &&&\ell_{\bar{p}_{K},K}\\ \end{pmatrix}_{p\times K}.$$

We utilize the community-membership function $\varphi$ based on the interconnected community structure to partition $p$ observed variables into $K$ mutually exclusive and exhaustive common factors, i.e., $\varphi:\{1,2,\ldots,p\}\to\{1,2,\ldots,K\},j\mapsto\varphi(j)$. Without loss of generality, the $j$th observed variable is mapped to the $k$th common factor, which includes $p_{k}$ distinct observed variables $\{j:\varphi(j)=k\}=\{\varphi^{-1}(k)[1],\ldots,\varphi^{-1}(k)[p_{k}]\}$ and satisfies that $p=p_{1}+\cdots+p_{K}$. Next, we reorder $p$ observed variables by listing them from the same common factor as neighbors. For the input vector of observed variables $X=(X^{(1)},\ldots,X^{(p)})^{\mathrm{\scriptscriptstyle T}}$ in (equation), we reorder the elements and obtain $X^{(\varphi)}=(X^{\left(\varphi^{-1}(1)[1]\right)},\ldots,X^{\left(\varphi^{-1}(1)[p_{1}]\right)},\ldots,X^{\left(\varphi^{-1}(K)[1]\right)},\ldots,X^{\left(\varphi^{-1}(K)[p_{K}]\right)})^{\mathrm{\scriptscriptstyle T}}$ using the function $\varphi$. Consequently, $\varphi$ dictates that the first $p_{1}$ elements of $X^{(\varphi)}$ are categorized into the first common factor, and so forth, with the last $p_{K}$ elements of $X^{(\varphi)}$ being categorized into the $K$th common factor. For simplicity, we denote $X^{(\varphi)}=X$ throughout the remainder of this paper. Moreover, the adoption of a non-overlapping factor loading pattern in (equation) and (equation) stems from the fact that $\varphi$ is a bijective function.

With the exception of components $\ell_{\bar{p}_{k}+j,k+1}$ for $j=2,\ldots,p_{k+1}$ and $k=1,\ldots,K$, which will be determined in the subsequent corollary), all information about the factor loading matrix has been completely determined by the community membership.

Reparameterizing $\Sigma_{f}$ and $\Sigma_{u}$ by $\Sigma$. With a covariance matrix $\Sigma$ characterized by an interconnected community structure and corresponding “non-zero loadings” specified in $L$, the SCFA model in (equation) can represent model parameters $\Sigma_{f}$ and $\Sigma_{u}$ using the covariance parameters in $\Sigma$.

Corollary section.theorem.

Consider the blocks $\Sigma_{kk^{\prime}}=\sigma_{f,kk^{\prime}}\ell_{k}\ell_{k^{\prime}}^{\mathrm{\scriptscriptstyle T}}+\Sigma_{u,kk^{\prime}}$ of a classical CFA model as shown in (equation). All parameters in $\Sigma_{kk^{\prime}}$ can be determined by the parametric covariance matrix in (equation) in terms of $a_{kk}$ and $b_{kk^{\prime}}$ through:

$$\Sigma_{kk^{\prime}}=\sigma_{f,kk^{\prime}}\ell_{k}\ell_{k^{\prime}}^{\mathrm{\scriptscriptstyle T}}+\Sigma_{u,kk^{\prime}}=\begin{cases}a_{kk}I_{p_{k}}+b_{kk}J_{p_{k}},&k^{\prime}=k\\ b_{kk^{\prime}}1_{p_{k}\times p_{k^{\prime}}},&k^{\prime}\neq k\end{cases},b_{kk^{\prime}}=b_{k^{\prime}k},$$

with $p_{k}>2$ for every $k$. Then, we have the following equations:

(1) $\ell_{k}=\left(\tau_{k},\ldots,\tau_{k}\right)^{\mathrm{\scriptscriptstyle T}}=\tau_{k}1_{p_{k}\times 1}$ for all $k$, so $L=\operatorname{Bdiag}\left(\tau_{1}1_{p_{1}\times 1},\ldots,\tau_{K}1_{p_{K}\times 1}\right)$;

(2) $\sigma_{f,kk^{\prime}}=b_{kk^{\prime}}/(\tau_{k}\tau_{k^{\prime}})$ for all $k,k^{\prime}$, so $\Sigma_{f}=\left\{b_{kk^{\prime}}/\left(\tau_{k}\tau_{k^{\prime}}\right)\right\}$ with $b_{kk^{\prime}}=b_{k^{\prime}k}$;

(3) $\Sigma_{u,kk}=a_{kk}I_{p_{k}}$ for all $k$, so $\Sigma_{u}=\operatorname{Bdiag}\left(a_{11}I_{p_{1}},\ldots,a_{KK}I_{p_{K}}\right)$, where we assume that the symmetric matrix $\left(b_{kk^{\prime}}\right)$ is positive definite and $a_{kk}>0$ for all $k$.

The conditions $a_{kk}>0$ for all $k$ and the positive definiteness of $\left(b_{kk^{\prime}}\right)$ are required to ensure that $\Sigma$ is positive definite (see Supplementary Material). Corollary section.theorem demonstrates that the SCFA model can (1) directly specify the “non-zero loadings” in $L$ based on the function $\varphi$ and derive that all loadings belonging to factor $k$ follow $\ell_{\bar{p}_{k-1}+1,k}=\cdots=\ell_{\bar{p}_{k},k}=\tau_{k}$; and (2) parameters for all entries in the matrices $\Sigma_{f}$ and $\Sigma_{u}$ can be expressed by the parameters in the parametric covariance matrix $\Sigma$ (i.e., $a_{kk}$ and $b_{kk^{\prime}}$). This reparametrization facilitates much improved parameter estimation accuracy and computational efficiency of the SCFA model.

section.subsection Estimation and inference

In this section, we introduce a likelihood-based parameter estimation procedure for SCFA applied to sample data $\textrm{X}_{n\times p}$ with an unknown covariance matrix having an interconnected community structure. We derive closed-form estimators for SCFA parameters $L$, $\Sigma_{f}$ and $\Sigma_{u}$, and calculate closed-form factor scores using the least-square approach. Lastly, we establish the theoretical properties of the proposed estimators and delineate the inference procedure.

Suppose that the rows $X_{1},\ldots,X_{n}$ of data matrix $\textrm{X}_{n\times p}$ are independently and identically distributed as $N\left(0_{p\times 1},\Sigma\right)$, satisfying the proposed SCFA model in (equation). Let $f_{i}$ and $u_{i}$ be the $i$th factor score and error term, respectively. We define $S_{p\times p}=n^{-1}\textrm{X}^{\mathrm{\scriptscriptstyle T}}\textrm{X}$ as the unbiased sample covariance matrix. The typical estimation objectives encompass the $(p-K)$ non-zero elements of $L$, the $K(K+1)/2$ elements of symmetric $\Sigma_{f}$, and the $p$ diagonal elements of $\Sigma_{u}$ based on $S$. We assume that the interconnected community structure and $\varphi$ are known for $\Sigma$, while parameters $a_{kk}$ and $b_{kk^{\prime}}$ need to be estimated for each $k$ and $k^{\prime}$. Without loss of generality, we let the high-dimensional framework in this paper follow that $n<p$ and $K+K(K+1)/2<n$.

We maximize the likelihood function with respect to $L=\operatorname{Bdiag}\left(\ell_{1},\ldots,\ell_{K}\right)$, $\Sigma_{f}=\left\{b_{kk^{\prime}}/(\tau_{k}\tau_{k^{\prime}})\right\}$, and $\Sigma_{u}=\operatorname{Bdiag}\left(a_{11}I_{p_{1}},\ldots,a_{KK}I_{p_{K}}\right)$:

$$\begin{split}\left(\widehat{L},\widehat{\Sigma}_{f},\widehat{\Sigma}_{u}\right)&=\operatorname*{argmax}\limits_{L,\Sigma_{f},\Sigma_{u}}\mathcal{L}_{n}\left(L,\Sigma_{f},\Sigma_{u};S,\tau_{1},\ldots,\tau_{K}\right)\\ &=\operatorname*{argmax}\limits_{a_{11},\ldots,a_{KK},b_{11},b_{12},\ldots,b_{KK}}\mathcal{L}_{n}\left(a_{11},\ldots,a_{KK},b_{11},b_{12},\ldots,b_{KK},\widehat{L};S,\tau_{1},\ldots,\tau_{K}\right)\\ &=\operatorname*{argmax}\limits_{a_{11},\ldots,a_{KK},b_{11},b_{12},\ldots,b_{KK}}\bigg{[}-\frac{n}{2}\log\det\left(\Sigma\left(a_{11},\ldots,a_{KK},b_{11},b_{12},\ldots,b_{KK}\right)\right)\\ &-\frac{n}{2}\operatorname{tr}\left\{S\Sigma^{-1}\left(a_{11},\ldots,a_{KK},b_{11},b_{12},\ldots,b_{KK}\right)\right\}\bigg{]},\end{split}$$

where $\mathcal{L}_{n}$ denotes the log-likelihood function of normal data X, $\det(\cdot)$ denotes the determinant, $\operatorname{tr}(\cdot)$ denotes the trace, $\Sigma=\Sigma\left(a_{11},\ldots,a_{KK},b_{11},b_{12},\ldots,b_{KK}\right)$ is defined in (equation). By maximizing $\mathcal{L}_{n}$ and letting $\tau_{1}=\cdots=\tau_{K}=1$ throughout the rest of the paper, we obtain the following (unique) maximum likelihood estimators:

$$\widehat{L}=\operatorname{Bdiag}\left(1_{p_{1}\times 1},\ldots,1_{p_{K}\times 1}\right),\quad\widehat{\Sigma}_{u}=\operatorname{Bdiag}\left(\widehat{a}_{11}I_{p_{1}},\ldots,\widehat{a}_{KK}I_{p_{K}}\right),\quad\widehat{\Sigma}_{f}=\left(\widehat{b}_{kk^{\prime}}\right),$$

with $\widehat{b}_{kk^{\prime}}=\widehat{b}_{k^{\prime}k}$, where

$$\widehat{a}_{kk}=\frac{p_{k}\operatorname{tr}\left(S_{kk}\right)-\operatorname{sum}\left(S_{kk}\right)}{p_{k}(p_{k}-1)},\quad\widehat{b}_{kk^{\prime}}=\begin{cases}\dfrac{\operatorname{sum}\left(S_{kk^{\prime}}\right)}{p_{k}p_{k^{\prime}}},&k\neq k^{\prime}\\ \dfrac{\operatorname{sum}\left(S_{kk}\right)-\operatorname{tr}\left(S_{kk}\right)}{p_{k}(p_{k}-1)},&k=k^{\prime}\end{cases},$$ (equation)

for all $k$ and $k^{\prime}$, $S_{kk^{\prime}}$ denotes the $p_{k}$ by $p_{k’}$ submatrix of $S=\left(S_{kk^{\prime}}\right)$, $\operatorname{sum}(\cdot)$ denotes the sum of all elements of a matrix.

The estimation procedure for SCFA parameters is scalable for high-dimensional data (e.g., $p>10^{4}$) as long as $n>K+K(K+1)/2$. In contrast, the classic CFA model faces challenges in computation when (1) $p>n$ and (2) $n>p$ but $p$ is several hundred, due to the bottleneck of computing the large covariance (and its inverse) using the maximum likelihood approach. Therefore, SCFA addresses a longstanding limitation in CFA regarding dimensionality constraints.

Furthermore, the proposed factor score estimator of $f_{i}$ is

$$\widehat{f}_{i}=\left(\widehat{L}^{\mathrm{\scriptscriptstyle T}}\widehat{L}\right)^{-1}\widehat{L}^{\mathrm{\scriptscriptstyle T}}X_{i}=\left(\widehat{L}^{\mathrm{\scriptscriptstyle T}}\Sigma_{u}^{-1}\widehat{L}\right)^{-1}\widehat{L}^{\mathrm{\scriptscriptstyle T}}\Sigma_{u}^{-1}X_{i}=\left(\widehat{L}^{\mathrm{\scriptscriptstyle T}}\widehat{\Sigma}_{u}^{-1}\widehat{L}\right)^{-1}\widehat{L}^{\mathrm{\scriptscriptstyle T}}\widehat{\Sigma}_{u}^{-1}X_{i},$$ (equation)

for $i=1,\ldots,n$, where the OLS estimator is identical to the generalized least-square (GLS) estimator and the feasible generalized least-square (FGLS) estimator. The derivation is provided in the Supplementary Material. The following theorems exhibit the theoretical properties of the above estimators.

Theorem section.theorem.

If $K+K(K+1)/2<n$, then the proposed matrix estimators $\widehat{L}$, $\widehat{\Sigma}_{f}$, and $\widehat{\Sigma}_{u}$ are uniformly minimum-variance unbiased estimators (UMVUEs).

Please refer to the proof in the Supplementary Material. As $\widehat{a}_{kk}$ and $\widehat{b}_{kk^{\prime}}$ are (unique) maximum likelihood estimators, they also exhibit large-sample properties such as consistency, asymptotic efficiency, and asymptotic normality, under the conditions that $K+K(K+1)/2<n$ and $n\to\infty$ for fixed $K$ and $p$.

Theorem section.theorem.

If $K+K(K+1)/2<n$ and $\Sigma_{u}$ is positive definite (i.e., $a_{kk}>0$ for every $k$), then $\widehat{f}_{i}$ defined as (equation) is UMVUE, following a multivariate normal distribution with mean $f_{i}$ and covariance matrix presented in Theorem section.theorem.

The proofs of the equivalence (equation) and Theorem section.theorem are elaborated in the Supplementary Material. $\widehat{f}_{i}$ also exhibits large-sample properties such as consistency, asymptotic efficiency, and asymptotic normality, as $p\to\infty$. The normality result in Theorem section.theorem extends the confidence intervals for factor scores $f_{i}$. In addition to the maximum likelihood estimators $\widehat{a}_{kk}$ and $\widehat{b}_{kk^{\prime}}$, we provide their exact closed-form variance estimators in the following theorem. Moreover, the exact covariance matrix of the proposed factor score estimator $\widehat{f}_{i}$ is obtainable from (equation).

Theorem section.theorem.

(1) The exact variance estimators of $\widehat{a}_{kk}$ and $\widehat{b}_{kk^{\prime}}$ are

$$\begin{split}\operatorname{var}(\widehat{a}_{kk})&=\frac{2a_{kk}^{2}}{(n-1)(p_{k}-1)},\\ \operatorname{var}(\widehat{b}_{kk^{\prime}})&=\begin{cases}\frac{2}{(n-1)p_{k}(p_{k}-1)}\left\{(a_{kk}+p_{k}b_{kk})^{2}-(2a_{kk}+p_{k}b_{kk})b_{kk}\right\},&k=k^{\prime}\\ \frac{1}{2(n-1)p_{k}p_{k^{\prime}}}\left\{p_{k}p_{k^{\prime}}(b_{kk^{\prime}}^{2}+b_{k^{\prime}k}^{2})+2(a_{kk}+p_{k}b_{kk})(a_{k^{\prime}k^{\prime}}+p_{k^{\prime}}b_{k^{\prime}k^{\prime}})\right\},&k\neq k^{\prime}\end{cases}\end{split}$$

for every $k$ and $k^{\prime}$.

(2) The exact covariance matrix estimator of $\widehat{f}_{i}$ is

$$\begin{split}\operatorname{cov}\left(\widehat{f}_{i}\right)=\left(\widehat{L}^{\mathrm{\scriptscriptstyle T}}\widehat{L}\right)^{-1}\widehat{L}^{\mathrm{\scriptscriptstyle T}}\Sigma\widehat{L}\left(\widehat{L}^{\mathrm{\scriptscriptstyle T}}\widehat{L}\right)^{-1}=\operatorname{diag}\left(\frac{a_{11}}{p_{1}},\ldots,\frac{a_{KK}}{p_{K}}\right)+\left(b_{kk^{\prime}}\right).\end{split}$$

In particular, as $p_{k}\to\infty$ for all $k$, then $p\to\infty$ and $\operatorname{cov}\left(\widehat{f}_{i}\right)\to\left(b_{kk^{\prime}}\right)=\Sigma_{f}$ for each $i$.

Utilizing Theorem section.theorem and the estimates provided in (equation), we can perform Wald-type hypothesis tests and compute interval estimates for all model parameters $\Sigma_{f}$, $\Sigma_{u}$, and $f_{i}$.

section SIMULATION STUDIES

section.subsection Data generation

We perform Monte Carlo simulation to generate the observation vector $X_{i}=Lf_{i}+u_{i}$ for $i=1,\ldots,n$, where the factor loading matrix $L=\operatorname{Bdiag}\left(1_{p_{1}\times 1},\ldots,1_{p_{K}\times 1}\right)$, the common factor $f_{i}\sim N\left(0_{K\times 1},\left(b_{kk^{\prime}}\right)\right)$, and the error term $u_{i}\sim N\left(0_{p\times 1},\operatorname{Bdiag}(a_{11}I_{p_{1}},\ldots,a_{KK}I_{p_{K}})\right)$ for $i=1,\ldots,n$. Specifically, we explore various values for $(p_{1},\ldots,p_{K})^{\mathrm{\scriptscriptstyle T}}$ (as indicated in Table table) under different sample sizes, i.e., $n\in\{40,80,120\}$, with $K=3$ and

$$a_{11}=0.1,a_{22}=0.2,a_{33}=0.5,\quad\left(b_{kk^{\prime}}\right)=\begin{pmatrix}2.02&0.73&1.15\\ &3.13&1.63\\ &&3.69\end{pmatrix}\text{ is symmetric}.$$

We repeat the above data generation procedure $100$ times.

section.subsection Assessing the estimated factor scores and model parameters

We apply the SCFA model to each simulated dataset to estimate $L$, $f_{i}$, $\Sigma_{f}$, and $\Sigma_{u}$, respectively. Since a primary objective of CFA is dimension reduction, yielding reliable common factors, we first focus on evaluating the performance of $f_{i}$ estimation. Specifically, we employ the Euclidean loss $\sum_{i=1}^{n}\left\|\widehat{f}_{i}-f_{i}\right\|$ as the evaluation criterion. We calculate $\widehat{f}_{i}$ by (equation) and benchmark it against competing methods including the various CFA computational methods implemented by the R packages of “sem” (Fox, 2006; Fox et al., 2022), “OpenMx” (Neale et al., 2016; Boker et al., 2023), and “lavaan” (Rosseel, 2012; Rosseel et al., 2023), respectively. We constrain the scales of loadings by all methods to be identical for fair comparison.

In Table table, we summarize the mean and standard deviation of the Euclidean losses for each approach across the $100$ replicates under different settings. The results indicate that SCFA outperforms conventional numerical approaches, exhibiting the lowest average loss, while all other methods demonstrate, on average, twice the SCFA loss. Additionally, SCFA shows a much-reduced variation in loss (e.g., around one-fifth) compared to the competing methods. Lastly, SCFA drastically improves computational efficiency by at least $100$ times. As mentioned earlier, the numerical implementations of the conventional CFA method may be limited for input datasets with $p\geq n$ and may thus yield no results (i.e., “NA” entries in Table table). In contrast, SCFA provides a viable solution for datasets with $p\geq n$, which are very common in practice (e.g., omics, imaging, and financial data).

	SCFA				sem				OpenMx				lavaan
$(n,p)$	mean	standard deviation	time (secs)		mean	standard deviation	time (secs)		mean	standard deviation	time (secs)		mean	standard deviation	time (secs)
$(40,20)$	12.16	0.78	0.04		20.7	5.93	23.03		21.31	5.84	139.89		21.24	5.84	16.21
$(40,30)$	10.03	0.67	0.14		20.26	6.95	102.29		20.7	6.85	331.37		20.66	6.86	22.39
$(40,40)$	8.72	0.58	0.04		NA	NA	NA		19.53	8.07	866.16		NA	NA	NA
$(80,40)$	17.26	0.87	0.04		30.06	10.06	423.17		30.98	9.75	904.6		30.93	9.79	47.75
$(80,80)$	12.27	0.65	0.05		NA	NA	NA		31.05	14.22	262.32		NA	NA	NA
$(80,120)$	9.83	0.49	0.22		NA	NA	NA		NA	NA	NA		NA	NA	NA
$(120,40)$	25.88	1.01	0.04		39.75	10.98	409.12		40.58	10.82	866.05		40.54	10.82	47.29
$(120,120)$	14.95	0.60	0.07		NA	NA	NA		NA	NA	NA		NA	NA	NA
$(120,200)$	11.49	0.51	0.13		NA	NA	NA		NA	NA	NA		NA	NA	NA

Table table: We present the means, standard deviations, and computation times (in seconds) for the Euclidean losses $\sum_{i=1}^{n}\|\widehat{f}_{i}-f_{i}\|$ across $100$ replicates, comparing the results obtained from our proposed method and computational packages for different values of $p$ and $n$, with “NA” indicating “not available”.

In addition to the factor scores, we evaluate the performance of estimating model parameters $\Sigma_{f}$ and $\Sigma_{u}$. As parametric matrices $\Sigma_{f}$ and $\Sigma_{u}$ can be represented by parameters $a_{kk}$ and $b_{kk^{\prime}}$, we assess the accuracy of the estimators $\widehat{a}_{kk}$ and $\widehat{b}_{kk^{\prime}}$ in (equation).

The estimation results are summarized in Table table. We consider the following metrics in Table table: the average bias, Monte Carlo standard deviation, average standard error, and $95\%$ Wald-type empirical coverage probability using the proposed estimates for each $a_{kk}$ and $b_{kk^{\prime}}$. The results in Table table demonstrate that our estimation is generally accurate with small biases and approximate $95\%$ coverage probabilities. Specifically, for each parameter, the bias is relatively small when compared to the Monte Carlo standard deviation, while the average standard error is close to the Monte Carlo standard deviation. Furthermore, both the bias and average standard error decrease as the sample size increases, and the $95\%$ coverage probability approaches the nominal level as the sample size grows.

In comparison, we also utilize the aforementioned three R packages to estimate all $\ell_{k}$ in the factor loading matrix $L$, all diagonal elements of $\Sigma_{u}=\operatorname{Bdiag}\left(a_{11}I_{p_{1}},\ldots,a_{KK}I_{p_{K}}\right)$, and all elements of $\Sigma_{f}=\left(b_{kk^{\prime}}\right)$. We also calculate the average bias and asymptotic standard error using the results produced by “sem”, “OpenMx”, and “lavaan”, respectively, for all elements of $\ell_{k}$, $\Sigma_{u}=\operatorname{Bdiag}\left(a_{11}I_{p_{1}},\ldots,a_{KK}I_{p_{K}}\right)$, and $\Sigma_{f}=\left(b_{kk^{\prime}}\right)$. Due to the page limit, we present the results in the Supplementary Material. In cases where $p<n$, the proposed estimators demonstrate superior performance compared to the estimates produced by the R packages “sem”, “OpenMx”, and “lavaan” with lower average standard errors and much shorter computational time. When $p>120$, the computational times of conventional methods become long and even intractable, resulting in “NA” values for model parameters. This is primarily due to the presence of singular sample covariance matrices. In general, the performance of all methods is comparable, with computational cost being the bottleneck for the competing methods.

	bias	MCSD	ASE	$95\%$ CP		bias	MCSD	ASE	$95\%$ CP		bias	MCSD	ASE	$95\%$ CP
$n=40$	$2\times(3,3,4)^{\mathrm{\scriptscriptstyle T}}$, $p=20$					$3\times(3,3,4)^{\mathrm{\scriptscriptstyle T}}$, $p=30$					$4\times(3,3,4)^{\mathrm{\scriptscriptstyle T}}$, $p=40$
$a_{11}$	0.0	1.0	1.0	92		0.0	0.7	0.8	93		0.0	0.6	0.7	97
$a_{22}$	-0.1	2.1	2.0	92		0.2	1.6	1.6	97		-0.3	1.3	1.3	97
$a_{33}$	-0.4	3.7	4.2	96		0.1	3.5	3.4	94		-0.1	2.8	2.9	96
$b_{11}$	-2.5	45.9	45.6	89		-0.3	52.1	46.0	91		0.7	46.7	46.1	96
$b_{12}$	-3.0	41.4	40.9	92		-4.3	43.3	41.6	92		-5.4	47.1	41.8	94
$b_{13}$	2.8	45.5	48.0	94		2.6	50.4	47.5	92		-3.6	51.6	47.2	89
$b_{22}$	-14.2	76.1	68.4	90		-2.5	78.3	70.8	88		-2.5	66.9	70.7	92
$b_{23}$	-7.9	62.9	59.5	90		-2.8	58.9	59.7	96		-3.7	64.1	59.7	91
$b_{33}$	3.3	74.8	85.8	97		-7.4	80.0	82.9	94		-6.2	84.7	82.9	89
$n=80$	$4\times(3,3,4)^{\mathrm{\scriptscriptstyle T}}$, $p=40$					$8\times(3,3,4)^{\mathrm{\scriptscriptstyle T}}$, $p=80$					$12\times(3,3,4)^{\mathrm{\scriptscriptstyle T}}$, $p=120$
$a_{11}$	0.0	0.5	0.5	95		0.1	0.4	0.3	93		0.0	0.2	0.3	98
$a_{22}$	0.0	0.9	1.0	97		0.0	0.7	0.7	93		0.0	0.5	0.5	96
$a_{33}$	-0.4	1.9	2.0	95		0.0	1.5	1.4	96		-0.1	1.3	1.2	93
$b_{11}$	3.1	33.3	32.8	92		1.0	28.8	32.4	97		0.4	31.6	32.3	95
$b_{12}$	-0.1	31.3	29.6	95		0.3	28.4	29.7	97		1.9	26.6	29.6	96
$b_{13}$	-0.4	35.5	33.2	96		-0.2	33.2	33.2	96		2.0	33.4	33.8	95
$b_{22}$	-5.3	48.3	49.2	94		3.3	50.1	50.5	95		3.0	54.0	50.4	92
$b_{23}$	-3.9	41.0	41.5	93		-3.6	44.1	42.2	93		6.4	42.8	43.2	97
$b_{33}$	-13.2	55.1	57.2	93		-7.4	56.1	57.8	92		6.3	60.4	59.9	96
$n=120$	$4\times(3,3,4)^{\mathrm{\scriptscriptstyle T}}$, $p=40$					$12\times(3,3,4)^{\mathrm{\scriptscriptstyle T}}$, $p=120$					$20\times(3,3,4)^{\mathrm{\scriptscriptstyle T}}$, $p=200$
$a_{11}$	-0.1	0.3	0.4	100		0.0	0.2	0.2	95		0.0	0.2	0.2	96
$a_{22}$	-0.1	0.8	0.8	94		-0.1	0.4	0.4	92		0.0	0.3	0.3	94
$a_{33}$	0.1	1.6	1.7	97		-0.1	0.9	0.9	97		0.0	0.6	0.7	98
$b_{11}$	-0.9	23.3	26.2	97		-1.4	25.3	26.1	95		-0.5	26.1	26.2	95
$b_{12}$	0.1	23.2	23.9	96		-4.3	21.0	23.5	97		0.7	23.7	23.9	95
$b_{13}$	-2.2	26.9	26.8	94		-1.1	27.9	26.9	93		-0.3	28.5	27.1	92
$b_{22}$	-3.1	40.2	40.4	90		-9.1	36.9	39.5	93		-4.8	37.7	40.0	97
$b_{23}$	-3.5	32.0	34.0	96		-6.4	32.9	33.7	92		-0.9	33.6	34.3	94
$b_{33}$	-12.8	47.1	46.6	92		-5.6	46.2	47.3	94		-1.1	51.8	47.8	90

Table table: We present the estimation results ($\times 100$) using the proposed method for $a_{11}$, $a_{22}$, $a_{33}$, and $b_{kk^{\prime}}$ across various $p$ (i.e., $(p_{1},\ldots,p_{K})^{\mathrm{\scriptscriptstyle T}}$) and $n$, based on $100$ replicates, where “bias” denotes the average of estimation bias, “MCSD” denotes the Monte Carlo standard deviation, “ASE” denotes the average standard error, “$95\%$ CP” denotes the coverage probability based on a $95\%$ Wald-type confidence interval.

section.subsection Simulation results with model misspecification

We evaluate the performance of SCFA in scenarios of model misspecification, particularly when the covariance matrix deviates from an interconnected community structure. Under the misspecified structure, we generate $X_{i}\sim N\left(0_{p\times 1},\Sigma_{\kappa}\right)$ with $\Sigma_{\kappa}=L\Sigma_{f}L^{\mathrm{\scriptscriptstyle T}}+\Sigma_{u}+E_{\kappa}$, where $L$, $\Sigma_{f}$, and $\Sigma_{u}$ are model parameters defined in Section section.subsection. We set $n=120$, $(p_{1},\ldots,p_{K})^{\mathrm{\scriptscriptstyle T}}=\left(60,60,80\right)^{\mathrm{\scriptscriptstyle T}}$, $K=3$, and $p=200$. $E_{\kappa}$ is an additional noise term that results in the covariance structure deviating from the interconnected community structure. Specifically, $E_{\kappa}\sim\text{Wishart}\left(p,\kappa I_{p}\right)$, where the noise level $\kappa\in 10^{-2}\times\{1,3,5\}$ and the noise scale of each entry in $E_{\kappa}$ averages approximately $\{2,6,10\}$, which overwhelms $\Sigma_{u}$. We repeat this data generation procedure $100$ times.

We apply the SCFA model to each simulated dataset and calculate the average bias, Monte Carlo standard deviation, average standard error, and $95\%$ coverage probability for each model parameter. The summarized results are presented in Table table. Additional results concerning higher values of $\kappa$ are available in the Supplementary Material.

The results demonstrate that both means and standard deviations of the Euclidean losses for estimated factor scores are larger than those calculated in a correctly specified model. As the noise level increases, both the mean and standard deviation of the losses increase. When the proposed interconnected community structure is violated by moderate random noises (e.g., the diagonal elements of mean matrix of $E_{\kappa}$ are larger than those of $\Sigma_{u}$), the performance of the SCFA model remains robust, yielding reliable common factor estimates $\widehat{f}_{i}$ and covariance parameter estimates among common factors $\widehat{\Sigma}_{f}=\left(\widehat{b}_{kk^{\prime}}\right)$ (see Table table).

	noise scale $=2$					noise scale $=6$					noise scale $=10$
	bias	MCSD	ASE	$95\%$ CP		bias	MCSD	ASE	$95\%$ CP		bias	MCSD	ASE	$95\%$ CP
$a_{11}$	203.2	3.8	3.6	0		602.3	13.4	10.3	0		989.9	21.4	16.9	0
$a_{22}$	203.5	4.6	3.8	0		596.1	13.2	10.4	0		986.7	19.5	17	0
$a_{33}$	201.9	4.5	3.7	0		614.1	11.8	9.7	0		1000.4	18.9	15.3	0
$b_{11}$	-2.8	25.7	26.3	96		-1.3	27	27.3	96		-2.5	26.9	28	94
$b_{12}$	0.5	26.3	24.2	91		2.7	28.5	25.1	90		2.3	28.9	25.6	94
$b_{13}$	-2.9	26.2	27.2	98		-1.6	27.2	27.9	96		0.7	26.9	28.5	98
$b_{22}$	1.2	39.5	41.2	94		3.3	40.8	42.4	94		4.8	40.4	43.4	94
$b_{23}$	-0.9	33.4	34.9	96		0.8	33.8	35.5	95		0.6	34	36.1	96
$b_{33}$	1.1	51.3	48.4	97		0.3	51.8	49	95		2.6	51.9	49.9	96
$\sum_{i=1}^{n}\|\widehat{f}_{i}-f_{i}\|$	34.70	1.23				58.13	2.26				75.94	2.68

Table table: We present the estimation results ($\times 100$) using the proposed method for $a_{11}$, $a_{22}$, $a_{33}$, and $b_{kk^{\prime}}$ under $n=120$, $(p_{1},\ldots,p_{K})^{\mathrm{\scriptscriptstyle T}}=(60,60,80)^{\mathrm{\scriptscriptstyle T}}$, $p=200$, and various noise terms $E_{\kappa}$, where “bias” denotes the average of estimation bias, “MCSD” denotes the Monte Carlo standard deviation, “ASE” denotes the average standard error, “$95\%$ CP” denotes the coverage probability based on a $95\%$ Wald-type confidence interval. We also present means and standard deviations (not $\times 100$) of the Euclidean losses $\sum_{i=1}^{n}\|\widehat{f}_{i}-f_{i}\|$ for estimated factor scores across $100$ replicates under various noise levels.

section APPLICATION TO GENE EXPRESSION DATA

We applied the SCFA model to the dataset for microRNA regulation on gene expression using the Pan-kidney dataset from The Cancer Genome Atlas (TCGA) project, as outlined by Tomczak et al. (2015). The gene expression dataset comprised information on the expression levels in Reads Per Million mapped reads (RPM) for $13408$ genes, collected from $712$ patients with age (mean $59.66$ and standard deviation $12.59$ years) and sex ($34.27\%$ female and $65.73\%$ male).

In this analysis, we extracted an interconnected community structure from the gene co-expression matrix using the algorithm by Wu et al. (2021), while other community detection algorithms provided similar findings. The detected interconnected community structure comprised $K=6$ communities from $p=1777$ genes with varying sizes: $p_{1}=384$, $p_{2}=503$, $p_{3}=314$, $p_{4}=268$, $p_{5}=158$, and $p_{6}=150$, as illustrated in Figure figure. Genes were highly correlated within each module, while the interconnections between modules could be either positive or negative. Our goal was to identify latent factors for each module (i.e., dimension reduction of correlated genes) and estimate module-to-module interactions.

Based on the learned interconnected community structure, we applied the SCFA model for parameter estimation. The results were presented in Figure figure. The first row of Figure figure (left to right) demonstrated the extracted interconnected community structure (middle) from the original data (left), and the large covariance estimate (right) by Yang et al. (2024). In the second row of Figure figure, we showed the results of decomposing the above-estimated covariance matrix of $\Sigma$ into SCFA parameters $L\Sigma_{f}L^{\mathrm{\scriptscriptstyle T}}+\Sigma_{u}$. We exhibited the estimated $\widehat{L}$, $\widehat{\Sigma}_{f}$, and $\widehat{\Sigma}_{u}$ using heatmaps. The middle heatmap in row two illustrated the interactive relationships between $6$ common factors. The bottom subfigure of Figure figure was a classic path diagram demonstration of factor analysis with factor memberships, intra-factor, and inter-factor correlations. As demonstrated, SCFA reduced $p=1777$ gene expression variables to $K=6$ correlated common factors (with the estimated covariance matrix $\widehat{\Sigma}_{f}$), while accurately representing the complex relationships among all $1777$ variables. The computational time was less than one minute. Due to the page limit, we provided detailed numerical estimates, gene names, the corresponding community membership, and estimated factor scores in the Supplementary Material.

Figure figure: We present the workflow for analyzing the TCGA gene expression data. In the first row, the subfigures illustrate: the input correlation matrix of $1777$ genes (left), the reordered correlation matrix highlighting the detected interconnected community structure (middle), and the (estimated) population correlation matrix (right), respectively. Utilizing the community membership from the $6$ estimated interconnected communities, we applied the SCFA model to the dataset. In the second row, the subfigures depict the estimation results: $\widehat{L}$ (left), $\widehat{\Sigma}_{f}$ (middle), and $\widehat{\Sigma}_{u}$ (right). The bottom path diagram illustrates gene subsets (in rectangles) and their respective common factors (in oval circles). The lines (with double-ended arrows) among the $6$ common factors denote their estimated correlation coefficients (i.e., the off-diagonal entries of $\widehat{\Sigma}_{f}$). Specifically, the red solid lines represent significant positive correlations, the blue solid lines represent significant negative correlations, and the dashed lines represent non-significant correlations.

We conducted pathway analysis to investigate the relationships between each common factor and kidney cancer. The results revealed that each common factor characterized unique cellular and molecular functions related to kidney cancer. Specifically, the first common factor exhibited enrichment in pathways related to G protein-coupled receptors (GPCRs), that play pivotal roles in various aspects of cancer progression, including tumor growth, invasion, migration, survival, and metastasis (Arakaki et al., 2018). The second common factor was enriched with pathways related to cellular respiration in mitochondria, which are essential to cancer cells, including renal adenocarcinoma (Wallace, 2012). The third common factor was enriched with pathways associated with the body’s immune response: renal cell carcinoma (RCC) is considered to be an immunogenic tumor but is known to mediate immune dysfunction (Díaz-Montero et al., 2020). For the remaining common factors, the uniqueness of associations might be less pronounced, and detailed information was provided in the Supplementary Material.

For comparison, we also applied the conventional CFA model with the same detected community membership to the dataset using R packages “sem”, “OpenMx”, and “lavaan”, respectively. However, none of these standard computational packages could handle the gene expression dataset due to $n=712<p=1777$. Thus, the SCFA model stands out as an unprecedented toolkit for CFA in high-dimensional applications.

section DISCUSSION

We have developed a novel confirmatory factor analysis model, the SCFA model, designed for high-dimensional data. In contrast to the traditional CFA model equipped with standard computational packages, which relies on predefined model specifications and faces computational challenges with high-dimensional datasets, the SCFA model simultaneously addresses these limitations by leveraging a data-driven covariance structure. Due to its prevalence and block structure, we integrate the ubiquitous interconnected community structure observed in covariance matrices across diverse high-dimensional data types into the SCFA model. This integration yields likelihood-based UMVUEs for the factor loading matrix $L$, the covariance matrix of common factors $\Sigma_{f}$, and the covariance matrix of error terms $\Sigma_{u}$ in closed forms, as well as explicitly consistent least-square estimators for the factor scores. Additionally, explicit variance estimators further facilitate statistical inference for these parameters.

Extensive simulation studies have demonstrated the superior performance of the SCFA model in both parameter estimation and factor score accuracy. Notably, SCFA significantly reduces the computational burden compared to existing CFA computational packages. The SCFA model also exhibits robustness to moderate violations of the interconnected community structure. In an application to TCGA gene expression data, we employed the SCFA model to explore the relationships between $6$ common factors and $1777$ genes, conducting statistical inference on covariances between these common factors. These findings underscore the limitations of classical CFA models and standard computational packages in handling such high-dimensional data analyses.

In summary, SCFA models combine the benefits of conventional CFA models, including dimension reduction and flexibility in covariance matrix modeling for common factors, with the ability to adapt to data-driven covariance structures. As many high-dimensional datasets implicitly exhibit interconnected community structures, SCFA holds broad applicability. The software package is available at https://github.com/yiorfun/SCFA.