1

Improving Power System State Estimation Based on

Matrix-Level Cleaning
Haosen Yang1 , Robert C. Qiu1 , IEEE Fellow, Lei Chu1 IEEE Student Member, Tiebin Mi1 IEEE Member, Xin
Shi1 IEEE Student Member, Chaoyuan Mary Liu2

Abstract—Power system state estimation is heavily subjected degree of estimated error. Besides, WLS heavily suffers from
arXiv:1904.06479v3 [eess.SP] 6 Apr 2020

to measurement error, which comes from the noise of measuring the ill-conditional gain matrix and bad data, and it is sensitive
instruments, communication noise, and some unclear random- to the initialization of state variables.
ness. Traditional weighted least square (WLS), as the most
universal state estimation method, attempts to minimize the In recent years, many researchers have been searching novel
residual between measurements and the estimation of measured approaches to improve power system state estimation [3]–[10].
variables, but it is unable to handle the measurement error. To In [6], a least-absolute-value (LAV) guided estimator was pro-
solve this problem, based on random matrix theory, this paper posed, which is more robust and exhibits many advantages for
proposes a data-driven approach to clean measurement error in phasor measurements. In [7], an iterative l1 -l2 mixed convex
matrix-level. Our method significantly reduces the negative effect
of measurement error, and conducts a two-stage state estimation programming was used for state estimation by linearizing the
scheme combined with WLS. In this method, a Hermitian matrix nonlinear physical equations. In [8], an autoencoder based pre-
is constructed to establish an invertible relationship between filtering was proposed to clean measurement noise and remove
the eigenvalues of measurements and their covariance matrix. gross errors. But as a deep learning method, autoencoder
Random matrix tools, combined with an optimization scheme, spends too much time in off-line training. In [9] and [10],
are used to clean measurement error by shrinking the eigenvalues
of the covariance matrix. With great robustness and generality, a two-stage state estimation method was designed, in which
our approach is particularly suitable for large interconnected the measured variables are transformed into a new group of
power grids. Our method has been numerically evaluated using variables at first so that the measurement model in the second
different testing systems, multiple models of measured noise and stage is linear.
matrix size ratios. By reviewing, even though many studies investigated new
Index Terms—state estimation, two-stage, measurement error, approaches to improve state estimation, few researchers con-
random matrix, Hermitian matrix construction, eigenvalues sidered directly processing measurement error in any system-
atic way. Our work aims to fill this gap and improve the
I. I NTRODUCTION accuracy of state estimation. Because of the strong randomness
and unclear influence of multiple noise, measurement error is
P OWER system state estimation aims to estimate state
variables from measurement data corrupted with noise,
and it plays an important role in power system operations, such
difficult to be handled in vector form or single-value form in
the past. Nevertheless, along with the well-established research
line of random matrix theory (RMT), some deterministic
as optimal power flow, stability analysis, and economic dis-
properties are workable when gathering the fully stochastic
patch. With the development of energy management systems
measurement error in matrix form. For instance, the M-P law,
(EMS) and smart grids, the requirement for accurate operating
proposed in [11], reveals that the eigenvalues of a Gaussian
parameters has been increasing greatly [1].
covariance matrix asymptotically converge to a deterministic
Conventional state estimation is mainly based on WLS,
probability distribution. Inspired by this, based on RMT and
which solves the normal equation iteratively by Gauss-Newton
an optimization scheme, this paper proposes a two-stage state
method [2]. Despite long and wide applications of WLS, there
estimation method, in which we process measurements by
is increasing concern for its accuracy and robustness. The
RMT firstly and then use WLS to estimate state variables.
objective function of WLS is the residual between measure-
RMT, aiming to extract insightful information from eigenval-
ments and the estimation of measured variables. Minimizing
ues distributions of large covariance matrices, has emerged as
the residual brings the estimation of measured variables close
a particularly useful framework for many theoretical questions
to measurements. However, approaching measurements is dis-
associated with high-dimensional big data analytics [12] [13].
similar from approaching true values, since measurements are
It has been successfully applied in quantum physics [14],
corrupted by the noise of measuring instruments, communi-
wireless communication [15], and signal processing [16], for
cation noise and random fluctuations. So only minimizing the
its remarkable effect in dealing with measurement noise in
residual but disregarding measurement error results in a certain
matrix-level. And large amounts of measurement data col-
1 Department of Electrical Engineering, Center for Big Data and Artificial lected from monitoring systems provide a new opportunity
Intelligence, Shanghai Jiaotong University, Shanghai 200240, China. for proper applications of RMT in power systems, including
2 Department of Mathematics and Statistics, Eastern Kentucky University,
event detection [17] and correlation analysis [18]. In this work,
Richmond, KY 40475, USA.
Email: Robert C. Qiu: [email protected]; Haosen Yang: an optimization framework, derived from RMT, is used to
[email protected]. clean measurement error by filtering the eigenvalues of the
 2

covariance matrix, after forming a Hermitian matrix which

0.18
is used to establish an invertible relationship between the residual
eigenvalues of measurements and those of their covariance 0.16 estimated error

matrix. To my best knowledge, it is the first time for RMT to 0.14

be applied in power system state estimation. 0.12

The contributions of this paper are listed as follows:

residual(p.u.)
0.1
(1) A crucial drawback of traditional state estimation meth-
ods is clearly analyzed that WLS does not take the effect of 0.08

measurement error into account. And the association among 0.06

measurement error, residual and estimated error is quantita- 0.04

tively discussed.
0.02
(2) We propose a Hermitian matrix construction method
to extend the application scope of previous noise-cleaning 0
0 10 20 30 40 50 60
methods. Most of previous papers (e.g., [19]–[21]), aimed index

to clean measurement noise based on RMT, only involved

Fig. 1. The comparison between the residual and the estimated error of every
eliminating noise of covariance matrices, but they did not variable. The index 1 to 30 in the abscissa represents the magnitude of every
intend to clean errors in original data of power systems. So nodal voltage, while 31 to 60 means nodal voltage angles. This case is tested
the Hermitian matrix construction is designed to enable the in IEEE 30-bus system, and all the variables are standardized.
RMT based error-cleaning scheme to adapt to measurements
of power systems.
where H = ∂h(x)/∂x is the jacobian matrix of h(x) w.r.t x.
(3) A two-stage state estimation framework is proposed, in
Upon convergence, an estimated state vector x̂ is obtained, and
which a matrix-level cleaning method is used at first to obtain
some techniques on gross error detection will be operated.
more reasonable measured values, and then WLS is employed
to eventually calculate the state vector. This framework solves
what the previous state estimation methods neglect, and greatly B. Issue of Residual
improves the accuracy of state estimation. It is clear that we have three vectors of measured vari-
The rest of this paper is organized as follows, section ables: measurements z, true values h(x) and estimated values
II introduces the problem statement and current drawbacks. ẑ = h(x̂). The residual is defined as the difference vector
Section III talks about the proposed methodology. Section IV between measurements z and estimated values h(x̂):
is case studies, and section V summarizes our work.
r = z − h(x̂) (4)

II. P ROBLEM S TATEMENT The vector of measurement error is the difference between
measurements z and real values h(x):
A. State Estimation
e = z − h(x) (5)
The measurement model of a power system is:
The vector of estimated error is the difference between real
z = h(x) + e (1) values h(x) and estimated values h(x̂):
where z denotes the measurement vector. When z comes from Re = h(x̂) − h(x) (6)
SCADA, it usually contains power flows, nodal injective power
and nodal voltage magnitudes. x represents the state vector To guide the optimization process, a distance function L(·) is
including nodal voltage magnitudes and voltage angles. h(·) required to measure the scale of these distance vectors, such as
denotes the nonlinear function relating z to x. And e is the l2 norm, l1 norm, etc. The distance function L(·) must satisfy
vector of measurement error whose elements are generally three requirements: (1) Triangular inequality, i.e., L(a + b) ≤
assumed to follow Gaussian distributions. State estimation is L(a) + L(b); (2) Symmetry, L(−a) = L(a); (3) L(·) ≥ 0. So
usually considered as a typical WLS problem, in which the function (6) can be resolved:
objective function is: L(WRe ) = L(W(h(x̂) − h(x)))
T = L(W(h(x̂) − z + z − h(x)))
J = ||(z − h(x)) W(z − h(x))||2 (2) (7)
≤ L(W(z − h(x̂)) + L(W(z − h(x)))
where || · ||2 is the l2 norm. W = diag{σ12 , 1/σ22 , 1/σ32 , · · = L(W(r + e))
·1/σn2 } is a diagonal weighted matrix whose elements are
reciprocals of the variance of measurement errors. σi2 denotes where W is defined similarly as that in function (2). According
the variance of the Gaussian error for the i-th measured to function (7), it is obvious that the estimated error Re is
variable. mainly influenced by two crucial components: the residual r
Function (2) can be minimized by iteratively solving the and the measurement error e. However, WLS fails to consider
well-known normal equation: the influence of e, thus the estimation effect is significantly
limited by its nature.
HT WH∆x = HW(z − h(x)) (3) To support the above analysis, a quick test using WLS is
 3

operated in IEEE 30-bus system, which contains 254 measured may do not rigorously follow a Gaussian distribution, because
variables (including power flows, nodal injective power, and some indescribable factors may slightly change it. This minor
voltage magnitudes) and 60 state variables (nodal voltage violation of the Gaussian hypothesis will reduce the effective-
magnitudes and angles) in total. In the simulation, Gaussian ness of our approach. To empirically illustrate the performance
measurement errors with zero mean are added, whose variance in the case of non-Gaussian distributions, many experiments
is 5% of original power flows, as well as 1% for original volt- using different distributions of measurement error are tested
age magnitudes. The initial values of voltage magnitudes and in case studies.
angles are randomly sampled from the Gaussian distribution
N (1, 0.05) and N (0, 0.157).
A. Normalization and Matrix Formation
The residual and the estimated error of every state variable
are plotted in Fig. 1. Though the residuals of most of variables Our method cleans measurement noise in matrix-level, for
have been shrunk well, the estimation of partial measured it is discovered that Gaussian measurement error in matrix
variables is still unsatisfactory, showing that it is extremely form possesses some excellent and analytical properties in the
unfeasible to ignore the existence of measurement error. To distribution of its eigenvalues. Before forming a matrix, we
address this issue, we propose a data-driven method to process normalize the measurement error of each measured variable
measurement error. by:
zi − bi √
z˜i = = (zi − bi ) wi (8)
σi
III. P ROPOSED M ETHODOLOGY
where zi denotes the i-th variable in the measurement vector,
Our method to clean measurements in matrix-level involves σi represents the standard deviation of its measurement error
two main parts: Hermitian matrix construction and RMT based ei . bi is the bias of ei , and wi = 1/σi2 is the i-th weighted
error cleaning (RBEC). The Hermitian matrix construction is coefficient in WLS. Then we utilize a split sample window to
responsible for recovering the eigenvalues of the measurement form an N × T (N < T ) measurement matrix Z ∈ RN ×T :
matrix from those of its covariance matrix. And the purpose
of RBEC is to clean the covariance matrix through shrinking Z = [z̃k−N+1 , z̃k−N+2 ......z̃k−1 , z̃k ] (9)
its eigenvalues.
where z̃k , by slight abuse of notations, is the normalized
Assumptions: (1) The topology and parameters of the es-
current measurement vector. N represents the number of
timated power system are available, which is a necessary
rows (the number of sample variables), specifically each row
condition for estimating state variables.
represents one variable sampled from SCADA. The number of
(2) The variance of measurement error can be estimated.
columns T denotes the number of continuous samplings over
This assumption, as same as most of studies about power
a period of time.
system state estimation, arises from the fact that there are
Assuming the matrix of real values H ∈ RN ×T correspond-
a large number of approaches to estimate the variance of
ing to Z is:
measurement noise by pseudo-measurements (historical data),
including direct method [22], statistical inference [23], em- g
H = [h(x) g g g
k−N+1 , h(x)k−N+2 ......h(x) k−1 , h(x)k ] (10)
pirical Bayes estimation [24] and covariance matrix analysis
[25] [26], etc. For instance, the variance of the individual g is the vector of real values after normalization.
where h(x) k
model zi = h(xi ) + ei can be written as σz2 = σh2 + σe2 , Then we have the model:
where σh is the variance of normal fluctuations in operating
state, and σe denotes the variance of measurement error. So Z=H+G (11)
that the commonest method is performing a large number where G is the normalized Gaussian matrix in which every
of independent repeated samplings over a time period during entry follows an independent identical distribution (i.i.d) with
which the operating state does not change much (σh ≈ 0). As zero mean and unit variance.
for the bias of measurement error, it can also be estimated
easily by calculating the average of the non-absolute error
(i.e., bi = E(zi − ẑi ) ). Readers can refer to [22] for more B. Hermitian Matrix Construction
information about methods to estimate the variance and bias The reason why we construct a Hermitian matrix is to
of measurement error. utilize its property: The absolute values of the eigenvalues of
(3) The measurement noise is Gaussian distributed. This a Hermitian matrix are equal to their corresponding singular
assumption is based on the central limit theorem that the values. Here the Hermitian matrix DZ ∈ R(N +T )×(N +T ) is
sum of multiple independent random variables tends to be constructed by:  
a Gaussian variable even if the respective variables are not Z
DZ = (12)
Gaussian distributed. Measurement error is caused by the ZH
accumulation of uncertainties of measuring instruments, com-
munication noise and other unclear randomness, so it is where ZH is the associate matrix (conjugate transpose) of Z.
generally modeled as a Gaussian distribution. If measurement Then the covariance matrix of DZ is:
 H 
error is strictly Gaussian distributed, our method can obtain ZZ
the optimal result. However, in practice, measurement error FZ = (13)
ZH Z
 4

Flow of eigenvalues calculation where we denote the last three terms GGH + HGH + GHH
by B. Let ci , vli and vri denote the i-th eigenvalue, left and
14 15 right eigenvector of C, respectively. λi , uli and uri denote the
12
Z DZ FZ ZZH i-th eigenvalue, left and right eigenvector of E, respectively.
ξi is the i-th estimated eigenvalue corresponding to λi . The
objective function of our RBEC is:
16 17 17
H DH FH HHH min ||Γ(E) − C||2
Γ(E)
(19)
2
= T r[(Γ(E) − C) ]
Fig. 2. The flow of the invertible relationship between the eigenvalues
of different matrices. The bracket above arrows is the function this step where Γ(E) is the estimation of C from E. Since Γ(E) and C
corresponds to. are both symmetric matrices, the l2 norm of Γ(E) − C is equal
to the trace of its square, as the second line in function (19)
shows. RMT shows the asymptotically deterministic property
Besides, another important property of the matrix DZ is that in the eigenvalues distribution of Gaussian error matrices by
if λD is an eigenvalue of DZ , its contrary value −λD must using multiple transforms (see appendix. A). So the eigen-
be another eigenvalue. According to this and the property of values distributions of matrices in B can be analytically
Hermitian matrices introduced above, the relationship between connected with E and C. Thus the trace (equal to the sum
the eigenvalues of DZ and FZ is: of its eigenvalues) of the matrix C can be approximated.
p To solve this optimization problem, similar to the family of
λDZ = ± λFZ (14)
rotational invariant methods [27], we assume that Γ(E) and
where λDZ and λFZ denote the eigenvalues of DZ and FZ E share the same eigenvectors. Here we have:
separately. It is easy to find that ZZH and ZH Z have the N
X
same nonzero eigenvalues. And ZH Z has other T − N zero Γ(E) = ξi uli uT (20)
ri
eigenvalues when N < T , so that the eigenvalues of FZ and i=1
ZZH are reversibly related:
 In other words, the eigenvectors are fixed while the eigenvalues
λZZ H λFZ 6= 0 are free variables in this optimization. Therefore, it is easy to
λFZ = (15)
0 λFZ = 0 find the optimal solution:
Same as Z, for the matrix of real values H, we have: N
X
    ξi = cj (vlj · uli )2 (21)
H HHH
DH = , FH = (16) j=1
HH HH H
where (·) denotes the inner product between two vectors. Then
and: the expectation of (vlj ·uli )2 is considered as a more reasonable
p
λDH = ± λFH estimator:
 XN
6 0
λHH H λFH = (17) ξi = cj E[(vlj · uli )2 ] (22)
λFH =
0 λFH = 0 j=1

where λDH , λFH and λHH H denote the eigenvalues of DH , Although this framework only involves cleaning the eigenval-
FH and HHH , respectively. ues, the eigenvectors are also thoroughly considered in this
The overall computing flow of eigenvalues is shown in Fig. optimization problem, whose optimal solution (22) can be
2, in which we aim to connect the eigenvalues of Z and H. regarded as converting the task of cleaning the eigenvectors
For now, except for the calculation from ZZH to HHH (the into cleaning the eigenvalues. The E[(vlj · uli )2 ] can be ana-
red arrow in Fig. 2), all procedures in this eigenvalues flow lytically expressed in [28]:
are available. qα(λi )2 (λi α(λi ) + ci )
E[(vlj · uli )2 ] = (23)
β(λi )2 + γ(λi )2
C. RMT Based Error Cleaning where:
Now we introduce how to estimate the eigenvalues of HHH α(λi ) = (1 − qhE (λi ))2 + q 2 π 2 ρ2E (λi )
from ZZH by RBEC and how to estimate the real values
β(λi ) = (λi α(λi ) + ci )(1 − q)hE (λi ) − α(λi )(1 − q) (24)
of measured variables. To facilitate reading, some related
mathematical equations of RMT are presented in appendix. A. γ(λi ) = (λi α(λi ) + ci )qπρE (λi )
Now we denote the covariance matrices by E = ZZH and where hE (λi ) is the real part of the Stieltjes transform (31),
C = HHH . Then model (11) can be expressed in the form of and ρE (λi ) is obtained from (32).
covariance matrices: Combining (22) and (23), the cleaning function for the
H eigenvalues is obtained (derivations are shown in appendix.
E = (H + G)(H + G)
B):
= C + GGH + HGH + GHH (18)
=C+B ξi = (1−qhE (λi ))(λi −(1−q)−2qλi hE (λi ))+qϕ(λi ) (25)
 5

where: zk-N+1
ϕ(λi ) = 1 − hE (λi )(λi − (1 − q)) − qλi (π 2 ρ2E (λi ) − h2E (λi )) zk-N+2
(26)
By function (26), we can obtain the eigenvalues of the cleaned zk-N+3
covariance matrix Γ(E), then the cleaned eigenvalues of the ...... Hermitian
Z FZ E
matrix FZ and DZ are obtained successively by (17). As same matrix DZ
as reconstructing C from E, we reconstruct DH from DZ : zk-2
N
X +T zk-1
Γ(DZ ) = ξDi ulDi uTrDi (27) Eigenvectors Eigenvalues
i=1 zk of DZ of E
where ξDi , ulDi and urDi denote the estimated i-th eigenvalue
of DH and the corresponding left and right eigenvector of DZ . clean
Finally, the cleaned matrix Γ(Z) is a part of Γ(DZ ), following Γ(Z) Γ(DZ)
which we obtain the cleaned measurements γ(z). The whole Eigenvalues Eigenvalues
calculation flow of RBEC is shown in Fig. 3. of Γ(DZ) of Γ(E)
Remark 1: RBEC works in matrix-level, thus measurement
γ(z)
noise in every entry of the matrix Z can be cleaned jointly.
In this paper, historical data is used to form the matrix Z,
and is cleaned together with current data, so that it is an Fig. 3. The entire process of RBEC.
ensemble processing of measurement data. In this paper, for
convenience, current measurements are solely extracted for the
Algorithm 1 The Entire Process of R-WLS
next stage.
1: Normalize the measurement data by (8);
Remark 2: Even if we reconstruct the Hermitian matrix FZ ,
2: Construct the matrix Z by (9);
we clean the eigenvalues of E, because the R-transform of
3: Construct the matrix DZ by (12) and FZ by (13);
FZ − FH is not analytical.
4: Calculate the eigenvalues of E;
5: Calculate the eigenvectors of DZ ;
D. Two-stage State Estimation 6: Clean the eigenvalues of E by function (25);
After RBEC, the cleaned measurement vector γ(z) is input 7: Obtain the cleaned eigenvalues ξDi of Γ(DZ ) by (17);
to WLS to estimate state variables. So it is a two-stage state 8: Reconstruct the matrix Γ(DZ ) by (27);
estimation method, named RBEC-WLS (R-WLS), in which 9: Choose the rows 1 : N and columns N + 1 : N + T
RBEC and WLS operate successively. The entire process of in Γ(DZ ) as Γ(Z), thus the cleaned measurements are
our method is shown in Algorithm. 1. obtained;
10: Input the cleaned measurement vector into WLS to esti-

E. Boundary Condition mate state variables.

Now we list the boundary condition of our matrix-level
cleaning method. The first condition is that the size of Z is
required to follow the Kolmogorov limit: (1) N → ∞ and that the eigenvalues distribution of a Gaussian covariance
T → ∞: the number of rows N and columns T should be matrix GGH /T converges asymptotically to a deterministic
sufficiently large. (2) N ∼ O(T )(N < T ): N is required to probability distribution. The Stieltjes transform, R-transform
be comparable to T [29]. If N > T or T >> N , the effect and S-transform (see appendix. A) are ways by which the
of our method will be unsatisfactory, which will be clearly eigenvalues can be readily analyzed in a theoretical way.
discussed in case studies. Specifically, the S-transform (35) allows us to approximate
The second boundary condition is that the variance of mea- the theoretical eigenvalues distribution of matrices product,
surement error can be estimated. Since our method is based such as GGH , HGH and GHH . And the R-transform (33)
on the determinacy of the eigenvalues of large-dimensional allows us to analytically compute the eigenvalues distribution
Gaussian random matrices, measurement errors with different of the sum of these matrices. So by these transforms, the
variance need to be normalized before operation. eigenvalues of the population matrix C are connected with
Also, other common settings of static state estimation are those of E, and the overlaps of arbitrary eigenvectors of E
required, such as the information of topology and system and C are completely computable [28]. Then an optimization
parameters, Gaussian assumption of measurement error and scheme is designed to convert the task of cleaning the whole
so on. covariance matrix into cleaning its eigenvalues. The solution
to this optimization problem merely involves the eigenvalues
and the overlaps of its eigenvectors, which are completely
F. More Discussions available by these transforms, then the cleaning equation (25)
The underlying mechanism inside our method is the well- is obtained. In addition, this scheme tends to focus on cleaning
known M-P law (see appendix. A). The M-P law reveals the covariance matrix, so we propose a Hermitian matrix
 6

12000 200
12000 real values real values real values
measurements measurements measurements
cleaned values 10000 cleaned values 150 cleaned values
10000

8000 100
8000
eigenvalues

eigenvalues

eigenvalues
6000 50
6000

4000 0
4000

2000 2000 -50

0 0 -100
0 2000 4000 6000 8000 10000 0 0.5 1 1.5 2 2.5 0 0.5 1 1.5 2 2.5
index index 10 4 index 10 4

(a) E (b) FZ (c) DZ

Fig. 4. The cleaning results of the eigenvalues of E, FZ and DZ in case. 1. The abscissa is the numbering of the eigenvalues (sorted), as well as the ordinate
is the eigenvalues.

construction approach to enable the optimization scheme to eigenvalues of E by function (25), followed by obtaining the
clean the original measurements of power systems. cleaned eigenvalues of FZ and DZ .
The cleaned results of the eigenvalues of E, FZ and DZ
IV. C ASE S TUDY are shown in Fig. 4(a), Fig. 4(b) and Fig. 4(c) respectively.
In this section, the accuracy and effectiveness of the pro- References [30] and [31] have demonstrated empirically that
posed approach are verified explicitly. In the first case in the measurement matrix in power systems is low-rank, which
European 1354-bus high voltage transmission system, the means that only a few eigenvalues (usually one or two) are far
details of the calculation process are elaborated. In the second greater than zero, as the red lines reveal in Fig. 4 ∼ 6. The blue
case, we test our approach using multiple testing systems dashed lines, denoting the eigenvalues of the measurement ma-
and various magnitudes of measurement error. The third case trix in Fig. 4 ∼ 6, are smoother curves (i.e., more eigenvalues
shows the performance of our method in the case of different are much greater than zero). Therefore, measurement noise
size ratios q = T /N of the measurement matrix Z. The fourth breaks the low-rank property of the monitoring data matrix,
case is one in which the measured noise is modeled into non- yielding a lot of disturbing eigenvalues (components). And
Gaussian distributions. And the fifth case aims to demonstrate greatly reducing these disturbing eigenvalues in the absence
that our method is still valid when we need to divide the of any knowledge of true values is the main task of RBEC, as
sample variables into matrices of appropriate size, in order to the green lines reveal in Fig. 4 ∼ 6. The explicit results of the
overcome the problem of too many measured variables. In the cleaned eigenvalues are listed in Table. I. The mean absolute
sixth case, other noise-cleaning methods are compared with error (MAE) is used to measure the difference between two
our approach. The final case demonstrates that our method vectors:
N
has little relevance to the operating state of power systems. 1 X
M AE = |ξi − λi | (28)
N i=1
A. Detailed Process where ξi and λi are the i-th element in two vectors. The reason
This case is tested in European 1354-bus high voltage why the root mean square error (RMSE) is not selected is that
transmission system, which has 1354 bus, 1991 branches the RMSE puts too much emphasis on big outliers. Hence
and 260 generators. In the simulation, Gaussian measurement if there are one or two values whose estimated errors are
errors with randomly selected bias bi ∼ U(−0.03, 0.03) are very large, squaring them will cause a very large RMSE, thus
added, whose variance is 5% of original values for power it cannot properly reflect the overall difference between two
flows, as well as 1% for voltage magnitudes. The initial vectors. Furthermore, the mean absolute percent error (MAPE)
values of voltage magnitudes for the iteration of WLS are is also dropped because it is severely influenced by small
randomly sampled from the Gaussian distribution N (1, 0.05), values. If there are one or two very small real values, the
while the initial values of voltage angles are chosen from division w.r.t. them will result in a large MAPE even though
N (0, 0.157). The results are the average of ten identical exper- the difference between real values and the estimation is not
iments. The measurements from SCADA include branch active great.
and reactive power from bus (1991+1991 variables), branch As shown in Table. I, the errors of the eigenvalues of E,
active and reactive power to bus (1991+1991 variables), nodal FZ and DZ are greatly reduced. Then we construct the matrix
injective active and reactive power (1354+1354 variables) as Γ(DZ ) by function (27), followed by obtaining Γ(Z) and γ(z).
well as nodal voltage magnitudes (1354 variables). Thus the Compared with the measurement vector z with the MAE of
size of the measurement vector is 12015 × 1, then we choose 0.0395, the MAE of the cleaned vector γ(z) is only 0.0082
1.2 ∗ 12015 ≈ 14418 recent historical measurement vectors to left, proving that our RBEC is effective to clean measurement
construct a 12015 × 14418 matrix. error.
Using Algorithm. 1, the eigenvalues of DZ , FZ , E and The MAE of the state vector decreases from 0.0183 to
the eigenvectors of DZ can be calculated. Then we clean the 0.0011, demonstrating that our method significantly improves
 7

TABLE I
0.03
T HE CLEANING RESULTS OF THE EIGENVALUES OF E, FZ AND DZ . T HE
residual
ERROR IS MEASURED BY MAE
estimated error
0.025
Matrices Measurement error Estimated error
0.02
E 1623.3825 102.3944

residual(p.u.)
FZ 1448.2478 93.0455
DZ 62.4250 4.0329 0.015

0.01
TABLE II
T HE MAE OF THE ESTIMATION ERROR OF MEASURED VARIABLES
0.005
(U NITS : P. U .).

Variables Measurements WLS R-WLS 0

0 10 20 30 40 50 60
Pt 0.0440 3.5306 0.1206 index
Pf 0.0469 3.6494 0.1315
Pb 0.0390 2.9567 0.1287
Qt 0.0441 3.6455 0.1360 Fig. 5. The comparison between the residual and the estimated error after
Qf 0.0473 3.9787 0.1324 R-WLS. The settings of the simulation are the same as Fig. 1.
Qb 0.0350 2.4066 0.0978
Vm 0.0062 0.0039 0.0012 TABLE III
T HE MATRIX SIZES OF DIFFERENT SYSTEMS .

the classic WLS based state estimation. Systems Sizes of matrices

Then the estimation of measured variables is calculated by European 1354-bus 12015×14418
ẑ = h(x̂). The estimation error of the measurement vector is Polish 3120-bus 24117×28940
shown in Table. II, where Pt, Pf, Pb, Qt, Qf, Qb, Vm denote French 6468-bus 55389×66467
European 9241-bus 91904×110285
active power to bus, active power from bus, nodal injective
active power, reactive power to bus, reactive power from
bus, nodal injective reactive power and voltage magnitudes,
measurement errors, 2.5%, 5%, 7.5%, 10% error for power
respectively. We must express the point that the estimation of
flows and 0.5%, 1%, 1.5%, 2% for voltage magnitudes are
measured variables ẑ is not the cleaned measurement vector
set. The sizes of the matrices are listed in Table. III, where
γ(z) which is obtained after WLS. Although the MAE of γ(z)
the number of rows corresponds to the number of sample
is smaller than ẑ in this case, ẑ is a more reasonable estimation
variables, and the number of columns represents the length
result, for ẑ is more in line with the physical equations of the
of the split window. The explicit results are shown in Table.
testing system.
IV.
1) Problem of Residual: Our original intention of designing
According to Table. IV, two important properties of our
RBEC is to handle the problem of WLS that the residual is
method are clearly revealed. At first, with the increase of
only a part of estimated error. As shown in Fig. 1, partial
system scale, the performance of our approach improves,
estimated errors are also significant when most of residuals
while the performance of traditional WLS becomes worse by
are shrunk to small values. To prove that our R-WLS is able
contrast. This can be explained by that some equations of
to overcome this problem, we employ the same simulation
RMT on which our method based asymptotically hold with the
settings with the quick fact in IEEE 30-bus system. As shown
matrix size increasing to infinity, such as the inverse stieltjes
in Fig. 5, both residuals and estimation errors become very
transform (32) and the R-transform of Gaussian covariance
small.
matrices. RMT derives these equations by assuming that the
matrix size converges to the Kolmogorov limit. So that a larger
B. Numerous Tests matrix would lead to a better result which is closer to the
The above case clarifies the detailed process of our ap- theoretical conclusion. Therefore, our approach is particularly
proach. Now, in this case, numerous tests in different power suitable for large interconnected systems.
systems are conducted using different magnitudes of measure- Secondly, the results are getting worse as a response to the
ment error. The size ratio q is set as 1.2 in all tests. The initial increase of measurement error. Obviously, the greater error
values of voltage magnitudes are randomly sampled from the will make it more difficult for WLS to find the optimal
Gaussian distribution N (1, 0.05), while the initial values of solution, as well as for our RBEC to clean measurements.
voltage angles are chosen from N (0, 0.157). The IEEE 30- The numerous tests, using different systems and various mag-
bus, 57-bus, 118-bus, 300-bus systems, European 1354-bus nitudes of measurement error, demonstrate the effectiveness
high voltage transmission system, Polish 3120-bus system and generality of our method.
at summer 2008 morning peak, French 6468-bus very high
voltage and high voltage transmission network, and European C. Case with Different Size Ratios
9241-bus system are utilized to test our method. The explicit As introduced above, the size of the measurement matrix Z
parameters of these systems are referred to [32]. In terms of is required that the number of rows N should be comparable
 8

TABLE IV
T HE MAE OF CASES OF DIFFERENT SYSTEMS AND VARIOUS MAGNITUDES OF MEASUREMENT ERROR (U NITS : P. U .)

Error of power flows 2.5% 5% 7.5% 10%

Error of Vm 0.5% 1% 1.5% 2%
Methods WLS R-WLS WLS R-WLS WLS R-WLS WLS R-WLS
IEEE 30-bus 0.0039 0.0010 0.0082 0.0015 0.0154 0.0015 0.0174 0.0014
IEEE 57-bus 0.0049 0.0010 0.0108 0.0016 0.0166 0.0021 0.0195 0.0022
IEEE 118-bus 0.0052 0.0007 0.0146 0.0013 0.0186 0.0017 0.0174 0.0018
IEEE 300-bus 0.0082 0.0006 0.0184 0.0008 0.0233 0.0014 0.0239 0.0017
European 1354-bus 0.0063 0.0011 0.0183 0.0011 0.0245 0.0014 0.0273 0.0019
Polish 3120-bus 0.0332 0.0013 0.0482 0.0015 0.0547 0.0015 0.0674 0.0018
French 6468-bus 0.0186 0.0010 0.0315 0.0013 0.0424 0.0015 0.0470 0.0018
European 9241-bus 0.0271 0.0005 0.0494 0.0007 0.0631 0.0009 0.0780 0.0013

5.5
10-3
5.5
10-3
TABLE V
IEEE 118-bus IEEE 118-bus
5 IEEE 300-bus
European 1354-bus
5 IEEE 300-bus
European 1354-bus
T HE P. D . F AND COEFFICIENTS OF VARIOUS NOISE MODELS .
4.5 4.5

4 4
estimated error(p.u.)

estimated error(p.u.)

3.5 3.5 Distributions P.d.f Coefficients

3 3

2.5 2.5
1 |x−µ| √
2 2
Laplace f (x) = 2b exp(−
√ b
) µ = 0,b = σi / 2
2
1.5 1.5 SC f (x) = πa a2 − x2 a = 2σi
2 √
1 1
SL y = a12 (a − |x|) a = 6σi
0.5 0.5
0 5 10 15
q
20 25 30 -1 -0.5 0 0.5 1 1.5
log(q)
2 2.5 3 3.5
NIG f (x) = N IG(x) see appendix. C
(x−µ)2
Gaussian f (x) = √ 1 exp(− ) µ = 0,b = σi
2πb 2b2
(a) q (b) ln(q)

Fig. 6. The results of different q in IEEE 118-bus, 300-bus and European

1354-bus systems. To clearly reveal the results when q < 1, we modify the
abscissa as ln(q) in subfigure(b). tion that measurement error follows a Gaussian distribution,
for Gaussian noise is the commonest model of unknown
measurement error [22]. However, some unclear randomness
to the number of columns T , i.e., N ∼ O(T )(N < T ) or may violate this assumption, leading to non-Gaussian noise. So
q = T /N > 1. So it is very meaningful to test our approach it is necessary to discuss the performance of our approach in
of different q values, if or if not the requirements of size are the case of non-Gaussian distributions of measurement error.
met. In this simulation, q is set to range from 0.3 to 50, and In this case, the Laplace distribution, semi-circle distribution
other settings are the same as the first case. The results are (SC), symmetric-linear error (SL) and normal-inverse Gaus-
shown in Fig. 6. sian distribution (NIG) are simulated, for they are somewhat
According to Fig. 6, if T >> N , the effect of our RBEC similar to noise models. The probability density functions
becomes unpleasant, since many analytical equations of RMT, (p.d.f.) and coefficients are listed in Table. V. The settings
such as the M-P Law (29) and the inverse Stieltjes transform of coefficients aim to make the variance of these distributions
(32), do not hold. Refer to appendix. D. for more information equal to σi . The Monte Carlo method is used to generate sam-
about what role this condition plays. Additionally, when N < plings of these distributions. This case uses European 1354-
T , the effect is also unacceptable because the N -dimensional bus system, and the averaged results of five tests are shown in
noise space of measurement error cannot be reconstituted. Table. VI. The Inc.Rat in Table. VI means the increasing ratio
Specifically, reconsidering model (11) and (18), the noise of MAE, i.e., Inc.Rat = (M AEW − M AER )/M AEW .
space is N -dimensional since the stochastic measurement error The results agree with our analysis that in the case of
of every sensor or substation (every row) is independent. Gaussian noise, the promotion of traditional WLS is the
However, the number of non-zero eigenvalues is min{T, N }. highest. And then, though the MAE of WLS in the case of NIG
So if T < N , RBEC can only span a T -dimensional subspace distribution 0.0134 is smaller than the Gaussian case 0.0164,
instead of the entire noise space, which is not conducive to the MAE of R-WLS in the NIG case 0.0026 is not as small
our approach. as the Gaussian case 0.0013, which reveals that the cleaning
Roughly speaking, the results are satisfactory and keep effect of NIG noise is not as high as the case of Gaussian error.
steady when q ranges approximately from 1 to 8. And the Furthermore, in all cases, R-WLS successfully improves the
larger the matrix Z, the more computing resources our ap- performance of WLS, showing that our approach has certain
proach will consume. Therefore, the most reasonable value of generality and can be effective in practice.
q is little greater than 1.
E. Case with Divided Matrices
D. Case with Different Noise Models According to the boundary condition introduced in Sec-
This subsection discusses the effectiveness of our method tion III, the size of Z is constrained by N < T . In practice, the
when measurement error is not Gaussian distributed. The number of measured variables can easily be tens of thousands,
mathematical derivation of our method is based on the assump- so the measurement samples should span a time window
 9

TABLE VI TABLE VIII

T HE MAE OF THE ESTIMATION ERROR IN THE CASE OF DIFFERENT NOISE T HE C OMPARISON RESULTS OF DIFFERENT CLEANING METHODS (MAE,
DISTRIBUTIONS (MAE, U NITS : P. U .). U NITS : P. U .)

Distributions WLS R-WLS Inc.Rat Methods WLS RBEC NLS CVC iso-CVC
Laplace 0.0246 0.0102 58.5% IEEE 30-bus 0.0082 0.0015 0.0019 0.0074 0.0016
SC 0.0427 0.0292 31.6% IEEE 57-bus 0.0108 0.0016 0.0019 0.0089 0.0017
SL 0.0280 0.0141 49.7% IEEE 118-bus 0.0146 0.0013 0.0015 0.0102 0.0012
NIG 0.0195 0.0032 83.6% IEEE 300-bus 0.0184 0.0008 0.0012 0.0098 0.0007
Gaussian 0.0183 0.0011 94.0%
10-4
350 13
active power of node 6 MAE
12
TABLE VII 300
active power of node 21

11
0.0008

T HE CLEANING EFFECTS OF DIVIDED MATRICES (MAE, U NITS : P. U .). 10

250

active power(MW)
9

MAE
Variables Size Measurement Cleaned 200 8

Pt 1991 × 2389 0.0440 0.0117 150

7

6
Pf 1991 × 2389 0.0469 0.0115 5
100
Pb 1354 × 1625 0.0390 0.0106 4

Qt 1991 × 2389 0.0441 0.0102 50

0 500 1000 1500
3
0 500 1000 1500

Qf 1991 × 2389 0.0473 0.0146 time time

Qb 1354 × 1625 0.0350 0.0083

Vm 1354 × 1625 0.0062 0.0016 (a) load settings (b) MAE

Fig. 7. The changing profile of active loads and the MAE over time.

of commensurable size. This may take a long time during

which the system topology may have changed. To address this our method also obtains comparable results, demonstrating the
problem, we divide measured variables into sets of appropriate effectiveness and competitiveness of our approach.
size if there are too many measured variables, or the equipment
for running algorithms is limited. For the sake of brevity, G. Case with time-varying operating state
this case uses European 1354-bus high voltage transmission
In this case, we aim to illustrate the performance of the
network, which contains 12015 measured variables, in order
proposed method under the scenario of time-varying operating
are active power to bus Pt, active power from bus Pf, injective
state. We use IEEE 300-bus system, and the same settings of
active power Pb, reactive power to bus Qt, reactive power from
measurement error and initialization method are adopted. The
bus Qf, injective reactive power Qb, nodal voltage magnitudes
active load of node 6 and node 21 is selected to be time-
Vm. We divide them into seven groups according to their
varying, while the others remain constant. The details of the
physical meaning. After cleaning all respective matrices, we
load settings are shown in Fig. 7(a).
gather them to operate WLS.
The MAE curve over time is shown in Fig. 7(b). According
The explicit sizes of matrices and the results of cleaning
to the results, the effectiveness of our approach remains
are shown in Table. VII. After cleaning, the averaged MAE
almost independent of the time-varying operating state. This
of the measured vector decreases from 0.0395 to 0.0097. The
is because our cleaning method rebuilds the measurement
MAE of the state vector after R-WLS is 0.0014, while MAE
model in matrix-level, and it is based on the determinacy
is 0.0011 if measured variables are cleaned in one matrix.
of the eigenvalues of G in the limit of large dimension.
Therefore, by dividing the variables, we can speed up the
Therefore, whatever the matrix H is, as long as H follows the
operation without much worsening of effectiveness.
Kolmogorov limit, our method is capable of working normally.
Besides, the results fluctuate from around 0.00075 to 0.00095,
F. Comparison of Different Cleaning Methods because the measurement noise imposed in each experiment
is randomly generated.
It is necessary to compare our approach with other RMT
based cleaning methods. In this case, the nonlinear shrinkage
(NLS) [33], loo-cross-validation covariance (CVC) [34] and H. Case of Inaccurate Variance
isotonic regression based CVC (iso-CVC) [34] are compared. As shown in the second assumption, the variance of mea-
The simulation settings are the same as those in Section. IV. surement error is assumed to be known. However, in practice,
B, and the results are shown in Table. VIII. the variance estimation may fail to obtain accurate results, due
According to Table. VIII, the comparison results in power to the lack of historical data, sudden events and some system-
systems of different sizes are different. In IEEE 30-bus system, atic errors. Thus in this case, we examine the performance
our method is superior to the other methods, which shows of our approach when the estimated variance of measurement
that our method has weaker requirement for matrix size (N error is also biased from the accurate variance. We use Eu-
and T should be large enough). This requirement for matrix ropean 1354-bus, Polish 3120-bus and IEEE 300-bus system,
size, or more broadly the Kolmogorov limit, is general and and adopt the same hyper parameters and simulation settings
fundamental for all RMT based methods. So it can be seen with the above cases. The error ratios of variance estimation
that our method is more tolerant for this crucial condition. As range from 0 to 50% of the real variance. The testing results
for the case of IEEE 57-bus, 118-bus and 300-bus systems, are shown in Fig. 8.
 10

√
where a < λ < b, q = T /N , a = σ 2 (1 − q)2 and b =
0.012 √
IEEE 300-bus σ 2 (1 + q)2 .
0.01
European 1354-bus
Polish 3120-bus
Theorem 2: Stieltjes Transform One of the most general
resolvent of the covariance matrix E is:
0.008

GE (s) = (sIN − E)−1 (30)

MAE

0.006

where s ∈ C+ is a complex variable, and IN ∈ RN ×N is the

0.004
unit matrix. The Stieltjes transform is defined as [29]:
N
1 X 1
0.002
1
gE (s) = T r(GE (s)) = (31)
0 N N i=1 s − λi
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5
Variance Error Ratio
where T r(·) is the trace of E. N denotes the number of rows
Fig. 8. The results of different error ratios of variance estimation. of E, while λi is the i-th eigenvalue of E.
Theorem 3: Inverse Stieltjes Transform The inverse
transform of the Stieltjes transform is;
From the results, the influence of inaccurate variance esti-
mation decreases with the system scale increasing, especially 1
ρE (λi ) = Im[ lim gE (λi − iη)] (32)
when the variance error ratio is great. Besides, even though π η→0
the inaccurate variance estimation generally reduces the effec-
tiveness of our method, the MAE of final results is still better where η is a small integer closed to zero, Im[·] denotes the
than those without cleaning. image part.
Theorem 4: R-Transform The R-transform is defined as:
V. C ONCLUSION 1
RE (s) =< gE (s) >−1 − (33)
s
In this paper, a new opinion was put forward on the key
problem of state estimation, that is, estimated error comes where < · >−1 represents the inverse function. The R-
from the residual and measurement error, but traditional WLS transform of a sum of matrices is equal to the sum of their
does not take the effect of measurement error into account. respective R-transforms [29]:
Therefore, we proposed a data-driven method to overcome
this problem by cleaning measurement error at first. Combined RA+B (s) = RA (s) + RB (s) (34)
with WLS, a two-stage state estimation model was conducted.
Our method is based on the deterministic property about The R-transform is a great tool to analyze additive Gaussian
eigenvalues distribution of the fully stochastic measurement errors, since by the R-transform we can obtain the spectrum
error in matrix-level, and it is the first time to use RMT based distribution of A + B from A and B.
noise-cleaning method to improve state estimation. Addition- Theorem 5: S-Transform The S-transform allows us to
ally, another innovation is the Hermitian matrix construction, compute the eigenvalues distribution of matrices product. The
which is a kind of extension of previous cleaning models S-transform is defined as:
for covariance matrices, so that they can be applied to clean
s+1
measurement matrices of power systems. Our method not SE (s) = (35)
only has strict mathematical deductions and precise theoretical sΓ−1
s

supports, but also performs well in practical applications. The where Γ(s) = sgE (s) − 1. The S-transform of a product
numerous tests, using different testing systems and various of matrices is identical to the product of their respective S-
hyper-parameters, proved the effectiveness and advantages of transforms:
our method. In the future, we will attempt to clean measure-
ment noise in the iterative process of WLS. SAB (s) = SA (s)SB (s) (36)

From the above basic equations, the eigenvalues distribution

A PPENDIX A of a sum or a product of different or identical matrices can
R ANDOM M ATRICES BASICS be fully computable. So reconsidering the model (18), the
Theorem 1: M-P Law For an N × T matrix G whose eigenvalues of each term can be analytically connected.
entries follow an identical and independent Gaussian distri-
bution N (0, σ 2 ), N → ∞, T → ∞ and T ∼ O(N ). Then
A PPENDIX B
the spectrum distribution of its covariance matrix GGT /T
P ROOF OF F UNCTION (25)
asymptotically follows the M-P distribution [11]:
1 p According to function (22) and (32), and the equation
λ∼ (b − λ)(λ − a) (29) GE (z) = (z(1 − qgE (z)) − (1 − q) − (1 − qgE (z))−1 C)−1 ,
2πλqσ 2
 11

we have: R EFERENCES
N
X [1] Y. Huang, S. Werner, J. Huang, N. Kashyap, and V. Gupta, “State
ξi = cj E[(vlj · uli )2 ] estimation in electric power grids: Meeting new challenges presented by
the requirements of the future grid,” IEEE Signal Processing Magazine,
j=1
Z vol. 29, no. 5, pp. 33–43, 2012.
[2] A. Monticelli, “Electric power system state estimation,” Proceedings of
= cρc N E[(vlj · uli )2 ]dc the IEEE, vol. 88, no. 2, pp. 262–282, 2000.
Z (37) [3] W. Zheng, W. Wu, A. Gomez-Exposito, B. Zhang, and Y. Guo, “Dis-
1 cρC (c) tributed robust bilinear state estimation for power systems with nonlinear
= lim Im[ dc]
πρE (λi ) z→λi −η 1 − qgE (s) − c measurements,” IEEE Transactions on Power Systems, vol. 32, no. 1,
pp. 499–509, 2017.
1
= lim Im[T r(GE (s)C)] [4] Y. Weng, R.Negi, C. Faloutsos, and M. D. Ilic, “Robust data-driven state
πρE (λi ) s→λi −η estimation for smart grid,” IEEE Transactions on Smart Grid, vol. 8,
no. 4, pp. 1956–1967, 2017.
And the right part is: [5] V. Kekatos and G. B. Giannakis, “Distributed robust power system
state estimation,” IEEE Transactions on Power Systems, vol. 28, no. 2,
pp. 1617–1626, 2013.
T r(GE (s)C) = T r(z(1 − qgE (s)) − (1 − q))−1 C [6] M. Gol and A. Abur, “Lav based robust state estimation for systems
− (1 − qgE (s))IN ) (38) measured by pmus,” IEEE Transactions on Smart Grid, vol. 5, no. 4,
pp. 1808–1814, 2014.
= N ((1 − qgE (s))gE (s) − 1) [7] W. Xu, M. Wang, J. Cai, and A. Tang, “Sparse error correction from
nonlinear measurements with applications in bad data detection for
One also has T r(GE (s)C) = N (Z(s)gE (s) − 1), where power networks,” IEEE Transactions on Signal Processing, vol. 61,
no. 24, pp. 6175–6187, 2013.
Z(s) = 1 − qgE (s). So substitution of this and (24) into (38) [8] V. M. Marco A.M. Saran, “State estimation pre-filtering with overlap-
yields: ping tiling of autoencoders,” Electric Power Systems Research, vol. 157,
pp. 261 – 271, 2018.
lim Im[T r(GC (s)C)] [9] S. Zhong and A. Abur, “Combined state estimation and measurement
z→λi −η calibration,” IEEE Transactions on Power Systems, vol. 20, no. 1,
= N πρE (λi )((1 − qhE (λi ))(λi − (1 − q) (39) pp. 458–465, 2005.
[10] C. Gomez-Quiles, A. de la Villa Jaen, and A. Gomez-Exposito, “A
− 2qλi hE (λi )) + qϕ(λi )) factorized approach to wls state estimation,” IEEE Transactions on
Power Systems, vol. 26, no. 3, pp. 1724–1732, 2011.
Then by substituting function (39) into (37), we obtain the [11] V. A. Marcenko and L. A. Pastur, “Distributions of eigenvalues for
some sets of random matrices,” Mathematics of the Ussr-Sbornik, vol. 1,
function (25). pp. 507–536, 1967.
[12] R. C. Qiu and P. Antonik, Smart Grid and Big Data: Theory and
Practice. Hoboken, NJ, USA: Wiley, 2015.
A PPENDIX C [13] R. C. Qiu, “The foundation of big data: Experiments, formulation, and
applications,” CoRR, vol. abs/1412.6570, 2014.
N ORMAL - INVERSE G AUSSIAN D ISTRIBUTION [14] T. Guhr, A. MllerCGroeling, and H. A. Weidenmller, “Random-matrix
theories in quantum physics: common concepts,” Physics Reports,
The p.d.f of a NIG distribution is [35]: vol. 299, no. 4C6, pp. 189–425, 1997.
p [15] L. Chu, F. Wen, and R. C. Qiu, “Eigen-inference precoding for coarsely
αδK1 ( δ 2 + (x − µ)2 ) δγ+β(x−µ) quantized massive mu-mimo system with imperfect csi,” IEEE Trans-
f (x) = p e (40) actions on Vehicular Technology, vol. 68, pp. 8729–8743, Sep. 2019.
π δ 2 + (x − µ)2 [16] A. T. Walden and D. Schneider-Luftman, “Random matrix derived
shrinkage of spectral precision matrices,” IEEE Transactions on Signal
where K1 (·) denotes the modified Bessel function, and in this Processing, vol. 63, no. 17, pp. 4689–4699, 2015.
[17] X. Shi, R. Qiu, Z. Ling, F. Yang, H. Yang, and X. He, “Spatio-temporal
case α = 1, β = 1, δ = 1, γ = 1 and µ = −1. The coefficients correlation analysis of online monitoring data for anomaly detection and
make the mean and variance of this NIG distribution become location in distribution networks,” IEEE Transactions on Smart Grid,
zero and σi . pp. 1–1, 2019.
[18] X. Xu, X. He, Q. Ai, and R. C. Qiu, “A correlation analysis method for
power systems based on random matrix theory,” IEEE Transactions on
Smart Grid, vol. 8, no. 4, pp. 1811–1820, 2017.
A PPENDIX D [19] J. Bun, J. P. Bouchaud, and M. Potters, “Cleaning large correlation
S UPPLEMENTARY M ATERIAL matrices: Tools from random matrix theory,” Physics Reports, vol. 666,
2016.
[20] D. Bartz and K. R. Mller, “Generalizing analytic shrinkage for arbitrary
This supplementary material explains what role the Kol- covariance structures,” in International Conference on Neural Informa-
mogolov limit plays in our work. Normally, the covariance tion Processing Systems, 2013.
H
matrix E = ZZH = (H + G)(H + G) is estimated by [21] O. Ledoit and M. Wolf, “A well-conditioned estimator for large-
dimensional covariance matrices,” Journal of Multivariate Analysis,
E(E) = E(HH + GH + HG + GGH ) = HHH + IN .
H H H
vol. 88, no. 2, pp. 365–411, 2004.
However this well-known model is only accurate in an almost [22] W. A. Fuller, Measurement error models. John Wiley & Sons, 2009.
impossible condition: the number of samplings is much greater [23] K. Mori, “Estimating the variance of measurement errors in run-
than the number of variables, i.e., T >> N . In the case of ning variables of sharp regression discontinuity designs,” CoRR,
vol. abs/1709.05863, 2017.
N ∼ O(T ), this model is no longer fully trusted, since the [24] G. H. Born, Martz, and F. H., “Empirical bayes estimation of observation
eigenvalues distribution converges to the M-P law (29) rather error variances in linear systems,” Aiaa Journal, vol. 10, no. 7, pp. 1183–
than to units [36]. RMT exactly derives some deterministic 1187, 1970.
[25] J. Chen, A. Bandoni, and J. A. Romagnoli, “Robust estimation of
properties of random matrices in the case of N ∼ O(T ), by measurement error variance/covariance from process sampling data,”
which we improve the WLS based state estimation. Computers & Chemical Engineering, vol. 21, no. 21, pp. 593–600, 1997.
 12

[26] G. A. Almasy and R. S. H. Mah, “Estimation of measurement error

variances from process data,” Industrial & Engineering Chemistry
Process Design & Development, vol. 23, no. 4, pp. 779–784, 1984.
[27] J. Bun, J. P. Bouchaud, and M. Potters, “Rotational invariant estimator
for general noisy matrices,” IEEE Transactions on Information Theory,
vol. 62, no. 12, pp. 7475–7490, 2016.
[28] J. Bun, J. P. Bouchaud, and M. Potters, “On the overlaps between eigen-
vectors of correlated random matrices,” CoRR, vol. abs/1603.04364,
2016.
[29] J. A. Mingo and R. Speicher, Free Probability and Random Matrices.
New York, TN, USA: Springer, 2014.
[30] G. Mateos and G. B. Giannakis, “Load curve data cleansing and
imputation via sparsity and low rank,” IEEE Transactions on Smart Grid,
vol. 4, no. 4, pp. 2347–2355, 2013.
[31] P. Gao, M. Wang, S. G. Ghiocel, J. H. Chow, B. Fardanesh, and
G. Stefopoulos, “Missing data recovery by exploiting low-dimensionality
in power system synchrophasor measurements,” IEEE Transactions on
Power Systems, vol. 31, no. 2, pp. 1006–1013, 2016.
[32] C. Josz, S. Fliscounakis, J. Maeght, and P. Panciatici, “AC power flow
data in MATPOWER and QCQP format: itesla, RTE snapshots, and
PEGASE,” arXiv: 1603.01533, 2016.
[33] O. Ledoit and M. Wolf, “Nonlinear shrinkage estimation of large-
dimensional covariance matrices,” Social Science Electronic Publishing,
vol. 40, no. 2, pp. 1024–1060, 2012.
[34] D. Bartz, “Cross-validation based nonlinear shrinkage,” CoRR,
vol. abs/1611.00798, 2016.
[35] C. Roberts, E. M. Stewart, and F. Milano, “Validation of the ornstein-
uhlenbeck process for load modeling based on µpmu measurements,” in
2016 Power Systems Computation Conference (PSCC), pp. 1–7, 2016.
[36] R. B. Dozier and J. W. Silverstein, “On the empirical distribution of
eigenvalues of large dimensional information-plus-noise-type matrices,”
Journal of Multivariate Analysis, vol. 98, no. 4, pp. 678–694, 2007.