This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication.

JOURNAL OF MODERN POWER SYSTEMS AND CLEAN ENERGY, VOL. XX, NO. XX, XX XXXX 1

Data-driven Robust State Estimation Through

Off-line Learning and On-line Matching
Yanbo Chen, Hao Chen, Yang Jiao, Jin Ma, and Yuzhang Lin

—To overcome the shortcomings of model-driven

Abstract— state estimation (SE) is crucial in the development of smart
state estimation methods, this paper proposes a data-driven ro‐ grid development [4]-[6]. The traditional SE methods require
bust state estimation (DDSE) method through off-line learning the establishment of measurement equations based on the
and on-line matching. At the off-line learning stage, a linear re‐
gression equation is presented by clustering historical data from node admittance matrix, and then construct a mathematical
supervisory control and data acquisition (SCADA), which pro‐ optimal model to estimate the state variables [7]. These
vides a guarantee for solving the over-learning problem of the methods are called model-driven SE (MDSE) methods.
existing DDSE methods; then a novel robust state estimation The performance of MDSE methods has been greatly im‐
method that can be transformed into quadratic programming proved during the past fifty years. However, in the emerging
(QP) models is proposed to obtain the mapping relationship be‐
smart grid environments, MDSE methods have the following
tween the measurements and the state variables (MRBMS). The
proposed QP models can well solve the problem of collinearity shortcomings [8].
in historical data. Furthermore, the off-line learning stage is 1) MDSE methods usually use only the current measure‐
greatly accelerated from three aspects including reducing histor‐ ment snapshot while ignoring the massive historical data col‐
ical categories, constructing tree retrieval structure for known lected in smart grids. When the gross errors and the parame‐
topologies, and using sensitivity analysis when solving QP mod‐ ter errors exist simultaneously, the identification ability of
els. At the on-line matching stage, by quickly matching the cur‐
MDSE methods drops [9], [10].
rent snapshot with the historical ones, the corresponding
MRBMS can be obtained, and then the estimation values of the 2) The grid parameters themselves have a certain degree
state variables can be obtained. Simulations demonstrate that of uncertainty [11]. The grid parameters stored in the data‐
the proposed DDSE method has obvious advantages in terms of base of power dispatching control centers may differ from
suppressing over-learning problems, dealing with collinearity their actual values, which will affect the performance of the
problems, robustness, and computation efficiency. real-time MDSE.
Index Terms——Robust state estimation, historical snapshot, off- 3) In smart grids, the power generation and loads often be‐
line learning, on-line matching, collinearity come intermittent and much more uncertain [12], and the to‐
pology also changes frequently, resulting in the significant
state shifts. This will cause the widely-used weighted least
I. INTRODUCTION square (WLS) to converge to a local optimal solution with‐

S MART grids make the huge amounts of system status da‐

ta available; however, at the same time, the requirements
for the data reliability are higher than ever before [1], [2].
out any physical meaning [13] - [16], as WLS usually uses
the estimation result of the last snapshot as the initial guess.
4) Using large amounts of historical data, the malicious at‐
Achieving a comprehensive, real-time, and accurate percep‐ tackers are more likely to accurately know the state of pow‐
tion of smart grids is a prerequisite for the intelligent dis‐ er grids and launch a malicious data injection attack that can‐
patch and control [3]. Therefore, the performance of the not be identified by MDSE [17], [18].
Recently, data-driven SE (DDSE) methods have been pro‐
posed to address the above problems. In [19], a data-driven
Manuscript received: November 30, 2020; accepted: March 10, 2021. Date of
CrossCheck: March 10, 2021. Date of online publication: XX XX, XXXX. scalable approach is proposed to monitor distribution sys‐
This work was supported in part by National Natural Science Foundation of tems by using artificial neural networks (ANNs) for its SE.
China (No. 52077076) and in part by the State Key Laboratory of Alternate Reference [16] proposes an architecture to clean historical
Electrical Power System with Renewable Energy Sources (No. LAPS2021-18).
This article is distributed under the terms of the Creative Commons Attribu‐ data and conduct supervised learning, and then the nonlinear
tion 4.0 International License (http://creativecommons.org/licenses/by/4.0/). relationship between the current measurements and the state
Y. Chen (corresponding author), H. Chen, and Y. Jiao are with the State Key
Laboratory of Alternate Electrical Power System with Renewable Energy Sourc‐
vector are estimated by using the historical data. The robust
es, School of Electrical & Electronic Engineering, North China Electric Power data-driven Kalman filter approaches are used to estimate
University, Beijing 102206, China (e-mail: chenyanbo@ncepu. edu. cn; the rotor angles and the angular velocities in [20], [21]. In
[email protected]; [email protected]).
J. Ma is with the School of Electrical and Information Engineering, Universi‐ [22], a“PaToPaEM”framework is proposed to estimate the
ty of Sydney, Sydney, 2006, Australia (e-mail: [email protected]). topology and parameters simultaneously with the historical
Y. Lin is with the Department of Electrical and Computer Engineering, Uni‐ data. However, in some power grids, the number of phasor
versity of Massachusetts, Lowell, MA 01852, USA (e-mail: yuzhang_lin@uml.
edu). measurement unit (PMU) measurements may not be suffi‐
DOI: 10.35833/MPCE.2020.000835 cient to meet the observability requirement of this method.
2 JOURNAL OF MODERN POWER SYSTEMS AND CLEAN ENERGY, VOL. XX, NO. XX, XX XXXX

In [23], [24], data-driven methods are used to enhance the The contributions of this paper mainly include four as‐
observability of SE for distribution networks. In [25], Gauss- pects.
Newton method is provided with a good initial value 1) By clustering historical data of SE, an LRE is present‐
through a shallow neural network by using the historical or ed, which provides a guarantee for solving the over-learning
simulation-derived data. problem of the existing DDSE methods.
In summary, compared with MDSE methods, the general 2) A novel robust estimation method that can be trans‐
DDSE methods include the following characteristics. formed into QP models is proposed to obtain the MRBMS at
1) A large amount of historical data is stored in the histori‐ the off-line learning stage, and the proposed QP based meth‐
cal database of SE, and these data include measurement vec‐ od can solve the collinearity problem in historical data.
tors and the corresponding estimation values of state vectors 3) The computational efficiency of the off-line learning
given by historical MDSE methods. DDSE methods try to stage is sped up by the reduction of historical categories
use the massive historical data to overcome the shortcom‐ (RHC), the establishment of establishing a tree search struc‐
ings of MDSE methods. ture for known historical topologies, and the usage of sensi‐
2) DDSE methods do not need to know the measurement tivity algorithm for QP models.
equations of the current snapshot exactly like MDSE meth‐ 4) A method of QMCH is proposed, thereby greatly im‐
ods, nor do they need to build an optimal estimation model proving the computation efficiency of the on-line matching
of the state vector of the current snapshot based on the mea‐ stage, which is beneficial to the on-line application of the
surement equations like MDSE methods. proposed DDSE method.
3) DDSE methods generally need to use a large amount The remainder of this paper is organized as follows. The
of historical data or simulation data as the sample data, and LRE for the DDSE method is presented in Section II. To ob‐
construct a learning model (such as a regression model, etc.) tain the MRBMS, Section III proposes a robust estimation
to find the internal mapping relationship between the mea‐ method that can be transformed into QP models based on
surements and the state variables (MRBMS); and then the the proposed LRE. In Section IV, the off-line learning stage
corresponding MRBMS is used to calculate the state vector is sped up from three aspects. In Section V, the on-line
of the current snapshot. matching stage is presented. The performance of the pro‐
Although the development of DDSE methods has been in posed DDSE method is tested in Section VI. Conclusions
steady progress, the over-learning problem and the low-learn‐ are presented in Section VII.
ing efficiency are the main shortcomings or even big obsta‐
cles [19], which can be attributed to the use of nonlinear II. FORMULATION OF LRE FOR DDSE METHOD
MRBMS. Considering the fact that so far, measurements pro‐
vided by the supervisory control and data acquisition (SCA‐ A. Review of Exact Linear Measurement Equations (ELMEs)
DA) system still account for the absolute majority, if a linear for SE
MRBMS can be constructed based on historical data of SE Reference [26] proposes that the nonlinear measurement
and SCADA, it is expected to solve the over-learning prob‐ equations in traditional SE can be exactly linearized by a co‐
lem of the existing DDSE methods, and the computation effi‐ ordinate transformation on the measurements and the state
ciency will be greatly improved. This motivates us to pro‐ variables, resulting in the ELMEs as:
pose a DDSE method based on the linear regression equa‐ z͂ = Jy + ω (1)
tion (LRE).
where z͂ =[u i P i Q i P ij Q ij I ij2 ]T Î Rm is the auxiliary measure‐
In addition to the above general characteristics, the pro‐
posed DDSE method also has the following unique character‐ ment vector, i and j are the node numbers, u i = v i is the
2

istics. square of the voltage amplitude measurement, Pi and Qi are

1) It includes off-line learning stage and on-line matching the measurements of active and reactive injection power at
stage. The former performs off-line analysis and processing node i, respectively, Pij and Qij are the measurements of ac‐
of historical data, while the latter performs on-line calcula‐ tive and reactive power flow from bus i to bus j, I ij2 is the
tion based on the current measurement snapshot. square of the line current magnitude measurement from bus
2) The off-line learning stage only needs to run once, but i to bus j, m n is the total number of measurements; y =
it can be used multiple times; whereas the on-line matching [u i R l l K l l ]Î R is the auxiliary state vector, n = N + 2b, N
i j i j

stage requires online real-time periodic operation. and b are the numbers of nodes and branches, respectively,
3) At the off-line learning stage, an LRE suitable for R l l = v l v l cos θ l l , K l l = v l v l sin θ l l , li and lj are the terminal
i j i j i j i j i j i j

DDSE is formed by clustering the historical data; then based buses of the branch l, v l and v l are the voltage magnitudes
i j
on this LRE, a novel robust estimation method is proposed
of buses li and lj, respectively. θ l and θ l are the voltage an‐
to filter the historical data to obtain the MRBMS. i j

4) At the on-line matching stage, by quickly matching the gles of buses li and lj, respectively. θ l l = θ l - θ l is the angle i j i j

current snapshot with the historical data (QMCH), the corre‐ difference between buses li and lj; ω is the m-dimensional
sponding MRBMS can be obtained quickly, and then the es‐ vector of measurement error with variance R͂ (an m ´ m diag‐
timated state variables of the current snapshot can be ob‐ onal matrix); and J Î Rm ´ n is a constant matrix, in which all
tained quickly based on this MRBMS. elements are determined by the network topology and net‐
CHEN et al.: DATA-DRIVEN ROBUST STATE ESTIMATION THROUGH OFF-lINE LEARNING AND ON-LINE MATCHING 3

work parameters. As for the details of (1), please refer to fects the practicability of the algorithm.
[26]. Equation (1) illustrates that the auxiliary measurement 3) In extreme cases, it may be difficult to find multiple
vector is an exact linear function of the auxiliary state vector. historical snapshots with the same topology as the current
snapshot in the historical database of SE. To this end, a span‐
B. Clustering of Historical Data from SCADA ning tree method is proposed to solve the above problems
1) Problem Description through the following seven steps.
The general modeling method of DDSE is shown in Fig. Step 1: for the studied distribution network, assuming that
1. As shown in Fig. 1, to construct a DDSE learning model, all the branches are put into operation, the network has N
a large number of historical snapshots stored in the historical nodes and b edges (here, multiple edges connected in paral‐
database of SE need to be used as the sample data. These lel between two nodes are treated as one edge). The number
historical snapshots include the historical measurement vec‐ of all spanning trees of this network is assumed to be T, the
tors and the corresponding estimation values of historical value of which can be determined by Kirchhoff’s matrix
state vectors given by the historical MDSE, of which the his‐ tree theorem (KMTT) and will be reduced in Section IV.
torical measurement vectors are the sample input, and the es‐ Step 2: for each spanning tree, create a corresponding
timation values of historical state vectors are the sample out‐ structure including three fields. The first field C links stores all
put. the links corresponding to this spanning tree (in ascending
Input: historical order according to link numbers), which is the flag of the
measurement corresponding spanning tree. The number of elements in
Historical data
vectors
Historical MDSE
Output: historical each C links is l c = b - N + 1. The second field C snapshots stores the
state vectors multiple historical snapshots (the measurement vectors and
Off-line learning the corresponding estimation values of state vectors) with
model of DDSE Predictive value of
historical state vectors the same spanning tree. And the third field C mapping is used to
Match MRBMS store the corresponding two MRBMSs, which will be intro‐
Measurement vector On-line calculation Estimation state vector duced in detail below. When each spanning tree is formed,
of current snapshot of DDSE of current snapshot
the corresponding links are determined, so the first field
Fig. 1. General modeling method of DDSE. C links in the corresponding structure is easy to be determined.
The formation of the second field C snapshots requires process‐
A very important observation that can be made from (1) is ing large numbers of historical snapshots, and the processing
that the constant Jacobian matrix of the ELMEs changes on‐ method is given in Steps 3-7. The formation of the third
ly with respect to the changes in topological structure. There‐ field C mapping corresponding to each spanning tree will be giv‐
fore, the clustering of the historical snapshots can be based en in Section III.
on their topologies, and those snapshots with the same topol‐ Step 3: assuming that there are S historical snapshots avail‐
ogy should be in the same category. Since the above process able, take out the ith (the initial value of i is 1) historical
only needs to be performed off-line, we call it off-line histor‐ snapshot, including the topology, the measurement vector,
ical data clustering (OHDC). and the estimation value of state vector.
Note that when the DDSE learning model is built, the Step 4: for the ith historical snapshot, select a spanning
available historical data generally include the historical mea‐ tree in the distribution network. If the distribution network is
surement vectors and the corresponding estimation values of radial, the corresponding spanning tree is itself; if the distri‐
historical state vectors given by historical MDSE. The corre‐ bution network is meshed, take one of its spanning trees that
sponding historical topology may be known or unknown, has not been selected so far. Store the corresponding links of
and correspondingly, different OHDC methods need to be this spanning tree into a collection (in ascending order ac‐
constructed, which are presented in the following subsec‐ cording to link numbers) and mark it as C imeas, and then
tions. match C imeas with all C links in Step 2. If C imeas and one of C links
2) OHDC When Historical Topologies Are Known in Step 2 are identical, store the ith historical snapshot into
In general, the vast majority (almost all) of different opera‐ the corresponding collection C snapshots in Step 2. The specific
tion modes of the studied power grid are stored in the histor‐ storage method is illustrated as follows: ① take out all the
ical database of SE, i. e., the historical data include almost measurements and the corresponding estimation values of
all the possible topologies with a high probability. As a re‐ the state vector in the ith historical snapshot; ② according to
sult, when the historical topologies are known, it is theoreti‐ the definition in (1), calculate the historical auxiliary mea‐
cally possible to directly cluster the historical data according surement vector of all the measurements and the auxiliary
to the topologies. However, the drawbacks of direct cluster‐ state vector associated with the spanning tree, then store
ing according to the original topologies are as follows. them into the corresponding field C snapshots in Step 2.
1) Since each topology leads to a category, the direct clus‐ Step 5: let i = i + 1. If i £ S, go to Step 4; else, go to Step 6.
tering method will result in too many categories caused by Step 6: if all the second fields C snapshots in Step 2 store at
too many different topologies for the studied power grid, least s historical snapshots (the value of s will be analyzed
which will affect the computation efficiency. and given in Section III), then go to Step 7; otherwise, take
2) A large number of historical snapshots are needed to new historical data and return to Step 3.
form the datasets for the corresponding topologies, which af‐ Step 7: for each spanning tree, there are s historical snap‐
4 JOURNAL OF MODERN POWER SYSTEMS AND CLEAN ENERGY, VOL. XX, NO. XX, XX XXXX

shots in C snapshots, which have the same unknown constant Ja‐ connected to any links.
cobian matrix according to (1). When the historical topologies are known, the advantages
For each snapshot i (i = 12s), mark the historical auxil‐ of the proposed OHDC method based on the spanning tree
iary measurement vectors in C snapshots as z͂ i Î Rm and the histor‐ include four aspects.
ical auxiliary state vectors associated with the spanning tree 1) In theory, the number of spanning trees is less than that
as y i Î Rn, n = N + 2(N - 1) = 3N - 2 (for the spanning tree, b of all possible original topologies of the studied power grid.
is equal to N - 1). Further, the auxiliary measurement vec‐ Therefore, the total number of modes that need to be pro‐
tors and the auxiliary state vectors associated with the span‐ cessed at the off-line learning stage can be reduced by using
ning tree of all the s historical snapshots are aggregated into the spanning tree method.
the following matrix forms. 2) Any operational topology of the studied power grid is
Y = [ y1 y2  y s ]Î Rn ´ s included in all T structures of the field C links, and the corre‐
(2)
sponding historical snapshots needed are stored in the field
Z = [ z͂ 1 z͂ 2  z͂ s ]Î Rm ´ s (3) C snapshots, which lays the data foundation for DDSE.
In most cases, multiple historical snapshots with the same 3) The proposed spanning tree method has no special re‐
topology as the current snapshot are available in the histori‐ quirements for the distribution of historical measurements
cal database of SE. These historical snapshots and the cur‐ and all the historical measurements can be used in the auxil‐
rent snapshot have the same MRBMS. At this time, Z Î Rm ´ s iary measurement vectors in most cases, whereas the number
and Y Î Rn ´ s can be used as the sample input and the sam‐ of auxiliary state variables associated with the spanning tree
ple output to construct the DDSE learning model, respective‐ is n = N + 2(N - 1) = 3N - 2, which is smaller than the number
ly. In this circumstance, all the measurements in each histori‐ of auxiliary state variables corresponding to a mesh network
cal measurement snapshot are used in the DDSE learning (N + 2b). The accuracy of the DDSE model is improved by
model. using the proposed spanning tree method.
In a few extreme cases, historical snapshots with the same 4) It should be emphasized that the off-line learning stage
topology as the current snapshot are not available in the his‐ of the proposed DDSE method is to learn the MRBMSs (i.
torical database of SE. Therefore, there are no historical e., the mapping matrix H) corresponding to all the different
snapshots with the same MRBMS as the current snapshot. spanning trees when the historical topologies are known. As
However, we can always find multiple historical snapshots a result, even if a new topology appears in the current snap‐
with the same spanning tree as the current snapshot; these shot in extreme cases, that is to say, there are no historical
historical/current auxiliary measurement vectors associated snapshots with the same topology as the current snapshot,
with the spanning tree have the same MRBMS. At this time, but we can still find historical snapshots with the same span‐
the sample output is still Y Î Rn ´ s, but the sample input of ning tree as the current snapshot. These historical/current
the DDSE learning model should be historical auxiliary mea‐ auxiliary measurement vectors associated with the spanning
surement vectors associated with the spanning tree, and the tree have the same MRBMS, thus the proposed DDSE meth‐
matrix form is formulated as: od can still work.
3) OHDC When Historical Topologies Are Unknown
Z cut = [ z͂ 1- z͂ 2-  z͂ s- ] (4)
When the historical topologies are unknown, the corre‐
where z͂ i- Î R (m cut £ m) is the historical auxiliary measure‐
m cut
sponding topologies can only be inferred according to the
ment vector associated with the spanning tree for the ith snap‐ measurement vectors.
shot; and Z cut Î R
m ´s
.
cut
Taking out any two historical snapshots, the corresponding
At the off-line learning stage, two MRBMSs should be auxiliary measurement vectors z͂ 1 Î Rm and z͂ 2 Î Rm can be
formed for each spanning tree. For the first MRBMS, the in‐ calculated according to (1). If z͂ 1 and z͂ 2 have a strong linear
put and output of the corresponding DDSE learning model correlation, they can be considered to have the same topolo‐
are Z Î Rm ´ s and Y Î Rn ´ s, respectively, which are suitable gy [16]. The Spearman’s rank correlation coefficient (RCC)
for most cases where the same topologies of historical snap‐ in statistics can be used to measure the correlation between
shots as the current snapshot are available. For the second the two vectors. The RCC of z͂ 1 and z͂ 2 is calculated as:

6∑d i2
MRBMS, the input and output of the corresponding DDSE m

learning model are Z cut Î R and Y Î Rn ´ s, respectively,

m ´s cut

i=1 (5)
which are suitable for a few extreme cases where the same RCC(z͂ 1 z͂ 2 ) = 1-
m(m2 - 1)
topologies of historical snapshots as the current snapshot are
not available. For the convenience of expression, the follow‐ where d i = rg(z͂ 1i ) - rg(z͂ 2i ) is the difference between the two
ing analysis only takes the formation of the first MRBMS as ranks of z͂ 1 and z͂ 2, rg(×) represents the consecutively ranking
an example. By using the same method, it is easy to get the number from small (starting from 1) to large, and z͂ 1i and z͂ 2i
second MRBMS. are the ith elements of z͂ 1 and z͂ 2, respectively; and RCC(z͂ 1 z͂ 2 )
Note that these historical auxiliary measurement vectors is the RCC of z͂ 1 and z͂ 2.
associated with the spanning tree for the second MRBMS in‐ The criterion is that, if R threshold £ RCC(z͂ 1 z͂ 2 )£ 1, then it is
clude the measurements of all node voltage amplitudes, pow‐ considered that z͂ 1 and z͂ 2 have a strong linear correlation,
er flow measurements on the twigs of the spanning tree, and and their corresponding topologies can be considered the
injection power measurements of those nodes that are not same. Obviously, the choice of the threshold R threshold is very
CHEN et al.: DATA-DRIVEN ROBUST STATE ESTIMATION THROUGH OFF-lINE LEARNING AND ON-LINE MATCHING 5

important. In a large number of simulation experiments, we ity problems among different historical snapshots, so it is
have found that even if R threshold is 0.9, the above method can not ideal to use the WLS method to estimate H directly.
still be used to correctly identify the historical measurement This issue will be addressed in Section III.
snapshots with the exact same topology. It should be pointed 3) Vectorization of Mapping Relationships
out that the reason why we use RCC instead of Pearson cor‐ Considering that the variables to be solved in mathemati‐
relation coefficient (PCC) is that RCC does not require the cal planning are often vectors, two methods can be adopted
auxiliary measurement vectors to conform to the normal dis‐ to transform the matrix H into a vector.
tribution, and RCC is robust when gross errors exist in his‐ The first method is to stack the elements of the matrix
torical data. variable H into a vector, then we have
As an analogue to the method proposed in Section II-B, Γ = Φβ + Ξ (8)
all historical snapshots can be clustered according to the
where Γ = vec(Y T ) and Ξ = vec(V T ) are the column vectors
RCC between any two historical snapshots. Here, the three
with F elements, F = s ´ n, and vec(×) represents the vectoriza‐
fields stored in the structure corresponding to each topologi‐
tion of the matrix by the column vectors; β = vec(H T ) is the
cal structure should be RCC, C snapshots, and C mapping. With suffi‐
cient historical snapshots, theoretically, all the topologies of column vector to be solved with M elements, M = m ´ n; and
the studied power grid can be obtained. For each topology, Φ = diag {Z T
Z T
 Z
} is the block diagonal matrix com‐
T

n
assuming that there are s snapshots, the aggregation method posed of n matrices Z T, Φ Î RF ´ M.
of the auxiliary measurement vectors and the auxiliary state The estimation value of β can be obtained based on (8) by
vectors is the same as (2)-(4). using the historical data, and then β can be compressed into
C. LREs for DDSE the matrix H. However, in this method, Φ is a diagonal ma‐
1) Expression for MRBMS trix with a very large order, and is highly sparse, thus the
According to (1), the true value of y must be a linear func‐ memory required in the estimation process might be very
large, which may affect the practicability of the algorithm.
tion of z͂ , so we have
The second method is to use the historical snapshots to
y = Hz͂ + v (6) solve each column of H T. The regression equation for solv‐
where H Î Rn ´ m is the unknown mapping matrix from the ing the ith (i=1, 2,, n) column of H T is as follows.
true value of z͂ to the true value of y; and v Î Rn is the error Θ = Ψα + Ω (9)
vector.
where Ψ = Z T Î Rs ´ m; and Θ = Y iT Î Rs, α = H iT Î Rm, and Ω =
According to (6), Y must also be a linear function of Z.
V iT Î Rs are the ith columns of Y T, H T, and V T, respectively.
Y = HZ + V (7) Solving n models based on (9) by using the historical
where V Î Rn ´ s is the error matrix. snapshots could give each column of H T. Obviously, the
According to (1) and (6), as long as the local topologies computer memory required to solve each model is very
of different snapshots are the same, these local topologies small. Although this method needs to solve (9) n times, the
have the same matrix J and matrix H. This is the reason total off-line calculation takes much less time than the first
why different topologies sharing the same spanning tree can method, which supports the practical application of the algo‐
be clustered into one category. rithm. Equation (8) or (9) is the LRE for the proposed
Obviously, the unknown matrix H corresponds to the third DDSE method.
field C mapping in OHDC. If H can be estimated according to Compared with the existing nonlinear regression equations
(7) by the given Y and Z, the MRBMS can be obtained. Fur‐ and the approximate LREs of existing DDSE methods, the
ther, when the current measurement snapshot is given, by establishment of the LRE in this paper avoids the over-fit‐
matching the current snapshot with the historical snapshots, ting and ill-conditioned problems in the existing DDSE meth‐
we can get the MRBMS corresponding to the current snap‐ ods, and lays a foundation for the establishment of new
shot, and then get the estimation value of the state variables, DDSE methods with good robustness.
thereby constructing a DDSE method.
2) Number of Required Historical Snapshots III. OFF-LINE LEARNING OF MRBMS BY SOLVING QP MODEL
The task of the off-line learning stage of the proposed The most important task at the off-line learning stage of
DDSE method is to estimate H based on (7). Obviously, the the proposed DDSE method is to estimate H based on the
most direct method is to use the WLS method. At this time, LRE, which will be given in this section.
a unique estimation value of H can be obtained only when
the matrix Z T is column full rank. Therefore, for each histori‐ A. Motivation
cal category, it is preferable to ensure that s (s ³ m) historical For each C mapping, it is necessary to estimate the value of
snapshots are available when the WLS method is used, α = H iT (i = 12n) based on (9). An intuitive idea is using
which is obviously a necessary condition. The necessary and WLS, which is very simple in principle. When the noise con‐
sufficient condition is ensuring that s auxiliary measurement forms to the normal distribution, the WLS method is the op‐
vectors are linearly independent. timal estimation; however, it is not a robust method and can‐
In practical systems, we may not be able to obtain so not handle ill-conditioned situations (such as there is a collin‐
many historical snapshots, and worse, there may be collinear‐ earity problem in the historical snapshots).
6 JOURNAL OF MODERN POWER SYSTEMS AND CLEAN ENERGY, VOL. XX, NO. XX, XX XXXX

Considering that the least absolute value (LAV) estimation are all 0; C = [ I 2s 0 ] is a matrix with 2s rows and 2s + m col‐
has good robustness [13], and the ridge regression (RR) has umns, I 2s is the identity matrix (order is 2s), the other ele‐
good adaptability to the ill-conditioned situations, a new ro‐ ments in C are all 0.
bust estimation method is proposed by combing linear least The standard QP model (12) can be solved by mature
squares (LS), LAV, and RR. In this paper, we call this new commercial software, such as GROUBI to obtain the estima‐
robust method the LSAVRR method. tion value α̂ . Z T and Y iT (i = 12n) are respectively intro‐
B. LSAVRR Model for Estimating MRBMSs duced into model (11), then the estimation values of each
column of H can be obtained, thereby obtaining the
Based on (9), the estimation value of α, denoted as α̂ , can MRBMS corresponding to each historical category. That is,
be obtained by solving the following LSAVRR model.

{
the third field C mapping is obtained.

αt ∑ i
t 2 + η 1 | t i | + η 2∑α 2j
( )
s m
min
i=1 j=1 (10) IV. SPEEDING UP OFF-LINE LEARNING STAGE OF DDSE
s.t. t i = Θ i - Ψ i α i = 12s METHOD
where Θ i is the ith element of Θ; Ψ i is the ith row of Ψ; The complete off-line learning stage of the proposed
η 1 ³ 0 and η 2 ³ 0 are tuning parameters; and t Î R s is residual DDSE method has been given above. Further research re‐
vector, t i is the ith element of t. veals that the off-line learning stage can be accelerated
When both η 1 and η 2 are equal to 0, model (10) is the LS through the following three aspects.
method; when η 1 is equal to 0, model (10) is the RR meth‐ A. RHC
od; when η 1 is sufficiently large, model (10) approaches the 1) When Historical Topologies Are Known
LAV method. Based on the extensive simulations, we recom‐ As shown in Section II, when the historical topologies are
mend taking η 1 = 1 and η 2 = 1´ 10-6, respectively. Note that known, all possible spanning trees of the studied power net‐
the proposed model (10) is different from the elastic net re‐ work have been stored in T fields C links, so the spanning tree
gression method [27]. of the current snapshot needs to be matched with T fields
C. Equivalent Model of LSAVRR and Solution Method C links. This may cause the following problems.
1) Although some topologies exist in theory, the probabili‐
1) Equivalent Model of LSAVRR
ty that they appear in the actual operation of the power net‐
Note that the objective function of the proposed model
work is very low. Considering all possible spanning trees
(10) is non-differentiable, so model (10) cannot be solved di‐
will cause the value of T too large (for example, T is 3909
rectly using the gradient-based method. According to the
for IEEE 14-bus system), leading to a very heavy computa‐
same method in [14], model (10) can be rewritten as:
tional load for off-line learning.
∑ ∑
ì s
é( u + v ) 2 + η ( u + v )ù + η
m
2) Likewise, it will also cause too much calculation by
ïmin α 2j
ï αuv i = 1 ë û
i i 1 i i 2
j=1 considering all spanning trees at the on-line matching stage.
í (11)
To this end, we propose a specific method for the RHC:
ïs.t. Θ - Ψα - u + v = 0
ï
î u ³ 0v ³ 0 ① calculate the probability of each spanning tree in the actu‐
al operation of the power grid; ② consider only those span‐
where u Î Rs and v Î Rs are two auxiliary vectors, and u i ning trees with a relatively high probability of occurrence,
and v i are the ith elements of u and v, respectively. so the value of T can be greatly reduced, and the reduced
2) Solution Method value is set to be Tcut.
Model (11) can be further transformed into a standard 2) When Historical Topologies Are Unknown
form of QP model as: When the historical topologies are unknown, we propose
ì ̂
ïX (α̂ ) = arg min
ï
í
(
1 T
2
X QX + X T c ) (12)
to consider only Tcut categories by the RHC, i. e., the RCC
values of all categories from large to small are sorted, and
ïs.t. AX = Θ then only the first Tcut categories are retained.
ï
î CX ³ 0 B. Tree Retrieval Structure for Known Historical Topologies
where X =[u 1 v 1 u 2 v 2  u s v s α 1 α 2  α m ] T is a column When historical topologies are known, for Tcut spanning
vector with 2s + m elements; X̂ (α̂ ) is the estimation value of trees, a tree retrieval structure is established based on the fol‐
X; Q = diag(  q
q 1 q
11 q 2 ) is the semi-definite block diago‐ lowing steps to improve the efficiency of the on-line match‐
s ing stage.
2 2ù
nal matrix, q 1 = éê ú, q 2 represents the identity matrix (or‐ Step 1: for each C links, establish a tree retrieval structure by
ë2 2 û taking l c = b - N + 1 elements (link numbers) in C links as the l c
der is m) multiplied by 2η 2; c T =[1 2s 0 m ], 1 2s is the 2s-dimen‐ nodes, where the smallest number in C links is the root node,
sional row vector, whose elements are all η 1, 0 m represents and each node in the tree structure (TS) corresponds to a
the 0-dimensional row vector, whose elements are all 0; A i = link number in C links. Since the elements in C links have been
[D i Ψ i ] is the ith row of the matrix A with s rows and 2s + m arranged in the ascending order, the child node is always
columns, D i is a row vector with 2s elements, of which the larger than its parent node. At this time, each node has only
(2i - 1)th element is 1, the (2i)th element is -1, other elements one child node (except for the leaf node); the number of lay‐
CHEN et al.: DATA-DRIVEN ROBUST STATE ESTIMATION THROUGH OFF-lINE LEARNING AND ON-LINE MATCHING 7

ers in the TS is l c. A total of Tcut TSs can be obtained corre‐ first MRBMS in the corresponding C mapping stores the target
sponding to all the C links; let i=1. matrix H; if only the spanning tree of the historical snap‐
Step 2: if there are TSs with the same value of the ith lay‐ shots is the same as that of the current snapshot, and their
ers, the ith nodes of these TSs are merged into one node and original topologies are not the same, the second MRBMS in
the number of TSs is reduced. the corresponding C mapping stores the target matrix H.
Step 3: i = i + 1; if i £ l c, go to Step 2; otherwise, go to
B. On-line Matching When Current Topology Is Unknown
Step 4.
Step 4: denote Tmerge as the number of TSs; let j = 1. When the current topology is unknown, the auxiliary mea‐
Step 5: for the jth TS, let k = 2. surement vector of the current snapshot can be used to
Step 6: sort the kth layer of the jth TS in the ascending or‐ match historical snapshots based on (5). Once the RCC be‐
der according to the value of the kth node. tween the historical measurement vectors and the current
Step 7: k = k + 1; if k £ l c, go to Step 6; otherwise, go to measurement vector satisfies 0.9 £ RCC £ 1, the correspond‐
Step 8. ing MRBMS, i.e., the target matrix H, is obtained.
Step 8: j = j + 1; if j £ T merge, go to Step 5; otherwise, go to C. Estimation Value of State Variables
Step 9. After H is obtained, the estimation value of the auxiliary
Step 9: end. state vector for the current snapshot, ŷ c, can be obtained by:
C. Sensitivity Algorithm for QP Models ŷ c = Hz͂ c (13)
To get MRBMSs, the estimation values of the columns of where z͂ c is the auxiliary measurement vector of the current
H can be obtained by solving QP models n times. It is appar‐ snapshot.
ent that only the values of Θ = Y iT are different in these n QP If the current topology is unknown, the power flow can
models; therefore, the sensitivity algorithm for QP models in be further obtained and the estimation process ends; if the
[28] can be used to improve the efficiency of obtaining H. current topology is known, the estimation value of the origi‐
By adopting the above sensitivity algorithm, the calculation nal state variables x̂ c (the voltage amplitudes and angles of
efficiency of the off-line learning stage can be further im‐ all nodes) of the current snapshot can be further obtained
proved. [26], [29] - [31]. The structural framework of the proposed
DDSE method is shown in Fig. 2.
V. ON-LINE MATCHING STAGE OF DDSE METHOD The advantages of the proposed DDSE method are as fol‐
The task of the on-line matching stage of the proposed lows: ① the MRBMSs are obtained based on the LRE,
DDSE method is the QMCH to obtain the corresponding H which avoids the over-learning problems; ② the proposed
matrix. That is to say, if the current snapshot and historical LSAVRR method can well solve the problem of collinearity
snapshots have the same spanning tree (when historical to‐ in historical data, and it also has good robustness; ③ the
pologies are known) or the same original topology (when proposed DDSE method does not require nonlinear itera‐
historical topologies are unknown), then the current snapshot tions, and therefore does not require an initial guess, so that
and historical snapshots have the same H matrix; and fur‐ the proposed method has a strong adaptability to the uncer‐
ther, the original state variables of the current snapshot can tainty of power generation and load in smart grids; ④ the
be estimated based on H. proposed DDSE method does not need to know any network
parameters, so the uncertainty of network parameters does
A. On-line Matching When Current Topology Is Known not have any impacts on the proposed method; meanwhile,
When the current topology is known, the method of the problems of convergence and leverage gross errors
QMCH is as follows. caused by the network parameters also have no effects on
Step 1: choose a spanning tree arbitrarily in the network the proposed method; ⑤ the proposed DDSE method can be
of the current snapshot; store all links in a collection C c with implemented with both known and unknown topologies, so
an ascending order by link numbers. Note that the open or it is adaptive to the frequent changes in the topology of
overhauled branches in the current snapshot should also be smart grids; ⑥ in extreme cases, it may be difficult to find
considered as links. When selecting the spanning tree, avoid multiple historical snapshots with the same topology as the
selecting branches that appear in one tree retrieval structure current snapshot in the historical database of SE, but the pro‐
as twigs at the same time. posed DDSE method can still work by using the spanning
Step 2: take the first element in C c, and denote it as C c (1); tree method.
find the TS whose root node equals C c (1), and denote it as
TSc. VI. CASE STUDIES
Step 3: according to the depth-first search (DFS) algo‐ This section tests the performance of the proposed DDSE
rithm, find a path from the root node to the leaf node in TSc, method on IEEE benchmark systems. In the tests, the load
and the nodes on this path need to be the same as all the ele‐ used in the simulation gradually changes from 90% to 110%
ments in C c. The C links corresponding to this path stores the of the base case. One measurement snapshot is generated ev‐
historical topology that matches the current topology. If the ery 10 s to mimic the sampling period of SCADA and
original topologies and the spanning tree of the historical MDSE runs every 3 minutes to generate the sample data.
snapshots are the same as those of the current snapshot, the The generation method of historical sample data is as fol‐
8 JOURNAL OF MODERN POWER SYSTEMS AND CLEAN ENERGY, VOL. XX, NO. XX, XX XXXX

Off-line learning On-line matching

Clustering (OHDC) LRE LSAVRR When current
When historical topologies topologies are known
are known
Tree Current topology
Match
Spanning trees retrieval Spanning tree
RHC structure
Topology 1 { Clinks, Csnapshots, Cmapping { Current Cmapping

{ Clinks, Csnapshots, Cmapping

{
Topology 2 Current yˆ c

Historical
Topology T { Clinks, Csnapshots, Cmapping { Current xˆ c
data

When historical topologies When current

Match topologies are unknown
are unknown
Whole network Current measurement
RHC { RCC, Csnapshots, Cmapping
{
Cmapping
Comparison { RCC, Csnapshots, Cmapping
{
of historical Current yˆ c

data { RCC, Csnapshots, Cmapping {
Power flow

Fig. 2. Structural framework of DDSE method.

lows: first calculate the power flow, and then superimpose ing to the RHC method, there are three categories (T cut = 3)
the normal distribution random errors (the standard deviation for the most common topologies of this grid, and the corre‐
is 10-3) on the result of the power flow to simulate the mea‐ sponding C links are {③, ④}, {①, ③}, and {②, ④}. The cor‐
surements. For the historical SE, the widely-used WLS meth‐ responding tree retrieval structures after the RHC are shown
od is used to obtain estimation values of the state variables, as Fig. 4(b), and they correspond to three spanning trees,
and the largest normal residual (LNR) method is used to and the three C mapping (i. e., six H matrices) corresponding to
identify gross errors. The generated historical data are stored these three spanning trees need to be solved based on the
for testing. The tests are performed on an Intel® CoreTM i5 PC, historical data. Here, we take the number of historical snap‐
with 2.20 GHz processor and 8 GB RAM. shots as 12, i.e., s = 12.







A. Tests on IEEE 4-bus System












The IEEE 4-bus system is firstly used to show the calcula‐ TS1 TS2 TS3
tion process of the proposed DDSE method in detail.
(a) (b)
1) When Current Topology Is Known
The topology and measurement configuration of the IEEE Fig. 4. Tree retrieval structures. (a) Before RHC. (b) After RHC.
4-bus system are shown in Fig. 3, where 1-4 represent the
node numbers and ①-⑤ represent the branch numbers. The For the first MRBMS, the numbers of auxiliary measure‐
numbers of nodes, branches and links are N = 4, b = 5 and ments and auxiliary state variables in this test system are m =
l c = 2, respectively. Here it is assumed that only the measure‐ 12 and n = 10, respectively. Then, the first three H Î R10 ´ 12
ment vectors and estimation values of state vectors corre‐ matrices corresponding to Fig. 4(b) can be obtained based
sponding to this topology are available in the historical data‐ on the LSAVRR method by solving (12).
base of SE. For the second MRBMS, the number of auxiliary state
variables in this test system are n = 10; whereas the number


G
1 2 of auxiliary measurements corresponding to TS1, TS2, and

TS3 in Fig. 4(b) are m = 6, m = 8, and m = 8, respectively.





Therefore, the three mapping matrixes corresponding to TS1,
3 4
G TS2, and TS3 in Fig. 4(b) are H Î R10 ´ 6, H Î R10 ´ 8, and
Voltage amplitude measurement; Injection power measurement H Î R10 ´ 8, respectively. Then, the second three mapping ma‐
Branch power flow measurement; Load; G Generator trices corresponding to Fig. 4(b) can also be obtained based
on the LSAVRR method by solving (12).
Fig. 3. IEEE 4-bus system with N = 4, b = 5, and l c = 2.
2) On-line matching stage
Suppose there are two topologies of the current snapshot,
1) Off-line learning stage as shown in Fig. 3 (Topology A) and Fig. 5 (Topology B),
According to the KMTT, T = 8 can be obtained. According respectively. We will show how to use the on-line matching
to the proposed OHDC method, all the C links can be obtained stage of the proposed DDSE method to estimate the state
as: {③, ⑤}, {③, ④},{①, ③}, {①, ④},{①, ⑤}, {②, vector.
③}, {②, ④}, and {②, ⑤}. The corresponding tree retriev‐ When the topology of the current snapshot is the same as
al structures before the RHC are shown as Fig. 4(a). Accord‐ Fig. 3 (Topology A), select a spanning tree arbitrary, sup‐
CHEN et al.: DATA-DRIVEN ROBUST STATE ESTIMATION THROUGH OFF-lINE LEARNING AND ON-LINE MATCHING 9

G
1
2 the original state variables as well as and the estimation val‐
ues given by WLS. It can be observed from Table II that


even if there is no historical snapshot with the same topolo‐


G gy as the current snapshot, the proposed DDSE method can
3 4
Voltage amplitude measurement; Injection power measurement
still obtain high-precision SE results. Note that for this case,
Branch power flow measurement; Load; G Generator the existing DDSE methods [16], [19] cannot work.

Fig. 5. Topology B of current snapshot. TABLE II

TRUE AND ESTIMATION VALUES OF STATE VARIABLES FOR TOPOLOGY B

pose the twigs of the selected spanning tree are ②, ④, and True value of state Estimation values of state variables
⑤ , and the corresponding C c is { ① , ③}. After quickly Node
variables WLS Proposed DDSE
matching with the tree retrieval structures in Fig. 4(b), it can 1 1Ð0 1Ð0 1Ð0
be found that it corresponds to TS1, and then the first 2 0.9824Ð - 0.0170 0.9824Ð - 0.0170 0.9823Ð - 0.0171
H Î R10 ´ 12 matrix corresponding to TS1 is the target matrix. 3 0.9690Ð - 0.0327 0.9690Ð - 0.0327 0.9689Ð - 0.0328
All the 12 measurements of the current snapshot can be used
4 1.0200Ð0.0266 1.0200Ð0.0266 1.0199Ð0.0266
at the on-line matching stage. The estimation values of the
original state variables (voltage amplitudes and phase angles
of all nodes) of the current snapshot can be further obtained 2) When Current Topology Is Unknown
easily, as shown in Table I. For comparison, Table I also When the historical topologies of the IEEE 4-bus system
gives the true values of the original state variables as well as are unknown, the performance of the proposed DDSE meth‐
the estimation values given by WLS. It can be seen from Ta‐ od is also tested. It is assumed that the measurement vectors
ble I that the estimation values obtained by the proposed and the estimation values of state vectors given by historical
DDSE method is in good agreement with the true values, MDSE in the historical database of SE correspond to the
thus proving the correctness of the proposed DDSE method. three topological operation modes of the IEEE 4-bus system.
TABLE I
In the test, 1000 historical measurement snapshots are taken
TURE AND ESTIMATION VALUES OF STATE VARIABLES FOR TOPOLOGY A out. The RCCs are calculated and the results show that these
historical measurement snapshots can be clustered into 3 cat‐
True values of state Estimation values of state variables egories, and the intra-cluster RCC values for these 3 catego‐
Node ries are all between 0.9975 and 1; while the average values
variables WLS Proposed DDSE
1 1Ð0 1Ð0 1Ð0 of inter-cluster RCC are less than 0.8.
2 0.9797Ð - 0.0223 0.9797Ð - 0.0223 0.9797Ð - 0.0223
Take ten of the historical auxiliary measurement vectors,
0.9739Ð - 0.0292 0.9739Ð - 0.0292 0.9739Ð - 0.0292
and the RCCs among them are shown in Fig. 6. It can be ob‐
3
served from Fig. 6 that ten historical measurement snapshots
4 1.0200Ð0.0245 1.0200Ð0.0245 1.0200Ð0.0245
can be clustered into 3 categories, i.e., z͂ 1, z͂ 7, z͂ 8, and z͂ 10 be‐
long to the first category; z͂ 2, z͂ 4, and z͂ 9 belong to the second
When the topology of the current snapshot is the same as category; and z͂ 3, z͂ 5, and z͂ 6 belong to the third category. The
Fig. 5 (Topology B), there is no historical snapshot with the identification results are completely consistent with the actu‐
same topology as the current snapshot. According to the pro‐ al topologies, thereby proving the correctness of the cluster‐
posed QMCH method, a spanning tree can be selected for ing using RCC. After using the RCC for on-line matching,
the current snapshot. Suppose the twigs of the selected span‐ the estimation values of the auxiliary state variables and the
ning tree are ① , ② , and ⑤ , and the corresponding C c is power flow of the current snapshot can be further obtained,
{③, ④}. After quickly matching with the tree retrieval struc‐ and the error between the estimation results given by the pro‐
tures in Fig. 4(b), it can be found that it corresponds to TS3, posed DDSE method and those given by WLS is less than
and then the second H Î R10 ´ 8 matrix corresponding to TS3 10-4, thereby proving the correctness of the proposed DDSE
is the target matrix. The estimation values of the auxiliary method when the historical topologies are unknown.
state variables of the current snapshot (denoted as ŷ c) are giv‐
B. Tests on Other IEEE Benchmark Systems
en by:
1) When Historical Topologies Are Known
ŷ c = Hz͂ cut (14)
1) The number of spanning trees before and after RHC
where z͂ cut =[u 1  u 4  P 12  P 43  Q 12  Q 43  P 1  Q 1 ] includes the
T
Before and after the RHC, the number of spanning trees
auxiliary measurements associated with the selected span‐ in each IEEE benchmark system is shown in Table III. It
ning tree. can be known from Table III that, before the RHC, although
Since the twigs of the selected spanning tree (①, ②, and the spanning trees can include all possible topological opera‐
⑤) do not include ④, P 4 and Q 4 are not associate with this tion states of the power grid, the number of spanning trees
spanning tree, and therefore, P 4 and Q 4 are not included is too large, which substantially increases the time of off-
in z͂ cut. line learning and on-line matching. After the RHC, the most
As shown in Table II, the estimation values of the original common topological operation modes of the power grid is re‐
state variables of the current snapshot can be further ob‐ tained, which can greatly improve the computation efficien‐
tained. For comparison, Table II also gives the true values of cy of the proposed DDSE method.
10 JOURNAL OF MODERN POWER SYSTEMS AND CLEAN ENERGY, VOL. XX, NO. XX, XX XXXX

RCC
1.0
insufficient observability prevents from using WLS for esti‐
z~10 mating H, which demonstrates the good performance of the
0.9
proposed LSAVRR method.
z~9

z~8 10-4
0.8

z~7 10-5

|dv|m
0.7
Measurements

10-6
z~6
0.6 10-7
z~5 0
0.4
Ra
z~4 0.5 tio 0.8 30 14 9
1.2 2746 300 118 57 39
Numb er of bu ses
z~3 0.4
Fig. 7. |dv| m with different Ratio for each IEEE benchmark system.
z~2
0.3
z~1 10-4
0.2
z~1 z~2 z~3 z~4 z~5 z~6 z~7 z~8 z~9 z~10 10-5

|dθ|m
Measurements
10-6
Fig. 6. RCC correlation matrix of ten historical measurement snapshots.
10-7
0
0.4
TABLE III Ra
tio 0.8 30 14 9
NUMBER OF SPANNING TREES OF EACH IEEE BENCHMARK SYSTEM 1.2 2746 300 118 57 39
BEFORE AND AFTER RHC Number of buses
Fig. 8. |dθ| m with different Ratio for each IEEE benchmark system.
Number of spanning trees
System
Before RHC After RHC
3) Estimation accuracy with different measurement redun‐
IEEE 9-bus 6 3
dancy
IEEE 14-bus 3909 5
As we all know, the measurement redundancy affects the
IEEE 30-bus 7824000 10
estimation accuracy of SE.
IEEE 39-bus 421380 10 To test the influence of the measurement redundancy on
IEEE 57-bus 61946380490028 12 the estimation accuracy of the proposed DDSE method, let
IEEE 118-bus 9223372036854775807 20 s = m. With different measurement redundancy from 1.5 to 3,
IEEE 300-bus 1020 25 |dv| m and |dθ| m between the true values and the estimation
IEEE 2746-bus 1030 100 values of the proposed DDSE method are given in Figs. 9
and 10, respectively. It can be seen from Figs. 9 and 10 that
as the measurement redundancy increases, both |dv| m and
2) Estimation accuracy with different numbers of histori‐
cal snapshots |dθ| m gradually decrease; when the measurement redundancy
Obviously, the number of historical snapshots s will affect is larger than 2, both |dv| m and |dθ| m obtained by the pro‐
the estimation accuracy of the matrix H at the off-line learn‐ posed DDSE method are smaller than 10-4 for all the test
ing stage, which, in turn, will affect the estimation accuracy systems. The above test results prove the practicability of
of the proposed DDSE method. the proposed DDSE method considering the measurement re‐
To test the influence of the number of historical snapshots dundancy of the actual systems.
on the estimation accuracy of the proposed DDSE method, 4) Comparison of estimation accuracy with other SE meth‐
we denote Ratio = s/m. Here, the measurement redundancy is ods
taken to be 2.5 for each test system. With different Ratio When s is equal to m (Ratio = 1) and the measurement re‐
(from 0.1 to 1.2), the mean absolute error of voltage magni‐ dundancy is 2.5, the estimation accuracies of WLS, the exist‐
tude (denoted as |dv| m) and the mean absolute error of phase 10-3
angle (denoted as |dθ| m) between the true values and the esti‐
mation values of the proposed DDSE method are given in 10-4
|dv|m

Figs. 7 and 8, respectively. It can be observed from Figs. 7 10-5

and 8 that as Ratio increases, both |dv| m and |dθ| m gradually
10-6
decrease; when Ratio = 1, both |dv| m and |dθ| m obtained by
10-7
the proposed DDSE method are smaller than 10-4 for all the 1.5 2.0
test systems, thereby proving the correctness of the proposed Redu 2.5 3.0 30 14 9
ndanc
y 2746 300 118 57 39
DDSE method. Note that even when s is less than m (i. e., Number of buses
Ratio < 1), the estimation accuracy obtained by the proposed Fig. 9. |dv| m with different redundancy for each IEEE benchmark system.
DDSE method is still acceptable; whereas at this time, the
CHEN et al.: DATA-DRIVEN ROBUST STATE ESTIMATION THROUGH OFF-lINE LEARNING AND ON-LINE MATCHING 11

10-3 10-3

10-4 10-4

|dv|m
10-5
|dθ|m

10-5
10-6
10-6
10-7
10-7 10 8
6 9
1.5 2.0
9 PBD 4 2 0 2746 300 118 57 39 30 14
Redu 2.5 3.0 30 14 (%)
ndanc 2746 300 118 57 39 Number of buses
y Number of buses
Fig. 11. |dv| m with different PBD for each IEEE benchmark system.
Fig. 10. |dθ| m with different redundancy for each IEEE benchmark system.

10-3
ing DDSE method in [16], and the proposed DDSE method
10-4
are compared in Table IV. As can be observed from Table

|dθ|m
IV, among the above three SE methods, the estimation accu‐ 10-5
racy of WLS is the highest, followed by the proposed DDSE 10-6
method. The estimation accuracy of the proposed DDSE 10-7
method can meet the requirements of engineering applica‐ 10 8
6 9
tions. PBD 4 2 0 2746 300 118 57 39 30 14
(%)
Number of buses
TABLE IV Fig. 12. |dθ| m with different PBD for each IEEE benchmark system.
COMPARISON OF ESTIMATION ACCURACY WITH DIFFERENT SE METHODS

WLS DDSE in [16] Proposed DDSE than that of WLS, so the proposed DDSE method is very
System
|dv| m |dθ| m |dv| m |dθ| m |dv| m |dθ| m suitable for online applications of large-scale systems.
IEEE 9-bus 2×10-8 1×10-8 4×10-5 1×10-5 9×10-6 3×10-5
Compution time (ms) 104 WLS
IEEE 14-bus 2×10-8 1×10-8 3×10-5 6×10-5 4×10-6 1×10-5
103 On-line of DDSE
IEEE 30-bus 4×10-8 1×10-8 7×10-5 9×10-5 3×10-5 3×10-5
IEEE 39-bus 7×10 -8
2×10 -8
9×10-5
9×10-5
5×10-5
6×10-5 102
IEEE 57-bus 6×10-8 6×10-8 9×10-5 8×10-5 6×10-5 4×10-5 101
-8 -7 -5 -5 -6 -6
IEEE 118-bus 4×10 2×10 6×10 4×10 3×10 2×10 100
IEEE 300-bus 1×10-8 3×10-8 2×10-6 4×10-6 2×10-7 6×10-7 10-1
9 14 30 39 57 118 300 2746
IEEE 2746-bus 1×10-8 1×10-8 1×10-6 2×10-6 1×10-7 4×10-7
Number of buses
Fig. 13. Comparison of calculation efficiency.
5) Robustness of proposed DDSE method
The robustness of the proposed DDSE method is also test‐
ed. For each IEEE benchmark system, the percentage of bad 2) When Historical Topologies Are Unknown
data (PBD) from 0% to 10% in each snapshot is randomly For the IEEE benchmark systems, when historical topolo‐
selected, and then added with 10% relative error. At this gies are unknown, a large number of historical measurement
time, the number of the selected historical measurement snapshots are clustered based on the RCC. The clustering re‐
snapshots meets s = 1.1m (Ratio = 1.1). In 100 tests, the sults show that the historical measurement snapshots have
changes of the average values of |dv| m and |dθ| m with the obvious clustering phenomena, and most of the measurement
changes of PBD obtained by the proposed DDSE method snapshots belong to the most common topological catego‐
are shown in Figs. 11 and 12, respectively. As can be ob‐ ries. And even if there are gross errors in the current snap‐
served from Figs. 11 and 12, with the gradual increase of shot, the matching method based on the proposed RCC also
PBD, the estimation accuracy of the proposed DDSE meth‐ achieves the correct matching results. This proves the ratio‐
od decreases slowly; when the PBD in historical data is as nality of clustering using the RCC and the necessity of
high as 10%, the proposed DDSE method still suppresses RHC. The final estimation error is less than 10-4, which
the gross errors well and obtains highly accurate estimation proves the correctness of the proposed DDSE method. Due
results. to page limitations, the specific results of tests are omitted
6) Computation efficiency test here.
The computation efficiency of the on-line matching stage
of the proposed DDSE method directly determines its engi‐
VII. CONCLUSION
neering usability.
In order to measure the efficiency of the proposed DDSE Aiming at resolving the shortcomings of the traditional
method, the on-line calculation time of the proposed DDSE MDSE methods, this paper proposes a DDSE method which
method is compared with that of WLS, as shown in Fig. 13. includes off-line learning stage and on-line matching stage.
It can be observed from Fig. 13 that the on-line computation The off-line learning stage targets to cluster historical data
efficiency of the proposed DDSE method is much higher and develop the linear MRBMS; while the on-line matching
12 JOURNAL OF MODERN POWER SYSTEMS AND CLEAN ENERGY, VOL. XX, NO. XX, XX XXXX

stage obtains the current MRBMS by QMCH, and further itoring approach for distribution systems,” IEEE Transactions on In‐
strumentation and Measurement, vol. 64, no. 5, pp. 1292-1305, May
quickly obtaining the estimation values of the state variables 2015.
of the current snapshot. The proposed DDSE method does [20] M. Netto and L. Mili, “A robust data-driven koopman kalman filter
not need to know the parameters of the network, and has for power systems dynamic state estimation,” IEEE Transactions on
Power Systems, vol. 33, no. 6, pp. 7228-7237, Nov. 2018.
good robustness and very high computation efficiency, mak‐ [21] W. S. Rosenthal, A. M. Tartakovsky, and Z. Huang, “Ensemble kal‐
ing it very suitable for the online applications of large-scale man filter for dynamic state estimation of power grids stochastically
systems. driven by time-correlated mechanical input power,” IEEE Transactions
In low-voltage distribution networks, the number of mea‐ on Power Systems, vol. 33, no. 4, pp. 3701-3710, Jul. 2018.
[22] J. Yu, Y. Weng, and R. Rajagopal, “PaToPaEM: A data-driven parame‐
surements is very limited (often not enough to ensure observ‐ ter and topology joint estimation framework for time-varying system
ability), and the topology information is difficult to obtain in distribution grids,” IEEE Transactions on Power Systems, vol. 34,
accurately. Next, we will study the DDSE method for low- no. 3, pp. 1682-1692, May 2019.
[23] K. Dehghanpour, Y. Yuan, Z. Wang et al., “A game-theoretic data-driv‐
voltage distribution networks. Also, the proposed DDSE en approach for pseudo-measurement generation in distribution system
method can be extended to the integrated energy systems state estimation,” IEEE Transactions on Smart Grid, vol. 10, no. 6,
(IESs) so as to realize the comprehensive, real-time and ac‐ pp. 5942-5951, Nov. 2019.
[24] Y. Yuan, K. Dehghanpour, and F. Bu, “A multi-timescale data-driven
curate perception of IES in uncertainty circumstances. approach to enhance distribution system observability,” IEEE Transac‐
tions on Power Systems, vol. 34, no. 4, pp. 3168-3177, Jul. 2019.
REFERENCES [25] A. S. Zamzam, X. Fu, and N. D. Sidiropoulos, “Data-driven learning-
based optimization for distribution system state estimation,” IEEE
[1] Nature. (2008, Sept.). Big data (specials). [Online]. Available: http:// Transactions on Power Systems, vol. 34, no. 6, pp. 4796-4805, Nov.
www.nature.com/news/specials/bigdata/index.html. 2019.
[2] R. Qiu and P. Antonik, Smart Grid and Big Data. Hoboken, USA: [26] R. A. Jabr, “Radial distribution load flow using conic programming,”
Wiley, 2014. IEEE Transactions on Power Systems, vol. 21, no. 3, pp. 1458-1459,
[3] X. He, Q. Ai, R. C. Qiu et al., “A big data architecture design for Aug. 2006.
smart grids Based on random matrix theory,” IEEE Transactions on [27] H. Zou, and T. Hastie, “Regularization and variable selection via the
Smart Grid, vol. 8, no. 2, pp. 674-686, Mar. 2017. elastic net,” J. R. Statist. Soc. B (2005). 67, Part 2, pp. 301-320.
[4] F. C. Schweppe and J. Wildes, “Power system static state estimation, [28] A. G. Hadigheh, “Sensitivity analysis in convex quadratic optimiza‐
part I: exact model,” IEEE Transactions on Power Apparatus and Sys‐ tion: simultaneous perturbation of the objective and right-hand-side
tems, vol. PAS-89, no. 1, pp. 120-125, Jan. 1970. vectors,” Algorithmic Operations Research, vol. 2, pp. 94-111, May
[5] F. C. Schweppe and D. B. Rom, “Power system static state estimation, 2007.
part II: approximate model,” IEEE Transactions on Power Apparatus [29] C. Gomez-Quiles, A. Villa Jaen, and A. Gomez-Exposito, “A factor‐
and Systems, vol. PAS-89, no. 1, pp. 125-130, Jan. 1970. ized approach to WLS state estimation,” IEEE Transactions on Power
[6] F. C. Schweppe, “Power system static state estimation, part III: imple‐ Systems, vol. 26, no. 3, pp. 1724-1732, Aug. 2011.
mentation,” IEEE Transactions on Power Apparatus and Systems, vol. [30] Y. Chen, J. Ma, F. Liu et al., “A bilinear robust state estimator,” Inter‐
PAS-89, no. 1, pp. 130-135, Jan. 1970. national Transactions on Electrical Energy Systems, vol. 26, no. 7, pp.
[7] A. Abur and A. G. Expósito, Power System State Estimation: Theory 1476-1492, Jul. 2016.
and Implementation, New York: Marcel Dekker, 2004, pp. 157-184. [31] Y. Chen, Y. Yao, Y. Zhang, “A robust state estimation method based
[8] K. Dehghanpour, Z. Wang, and J. Wang, “A survey on state estimation on SOCP for integrated electricity-heat system,” IEEE Transactions on
techniques and challenges in smart distribution systems,” IEEE Trans‐ Smart Grid, vol. 12, no. 1, pp. 810-820, Jan. 2021.
actions on Smart Grid, vol. 10, no. 2, pp. 2312-2322, Mar. 2019.
[9] Y. Chen, F. Liu, G. He et al., “A Seidel-type recursive Bayesian ap‐
proach and its applications to power systems,” IEEE Transactions on Yanbo Chen received the B. S. degree in electrical engineering from Hua‐
Power Systems, vol. 27, no. 3, pp. 1710-1711, Aug. 2012. zhong University of Science and Technology, Wuhan, China, in 2007, the M. S.
[10] Y. Chen, F. Liu, S. Mei et al., “An improved recursive Bayesian ap‐ degree in electrical engineering from China Electric Power Research Insti‐
proach for transformer tap position estimation,” IEEE Transactions on tute, Beijing, China in 2010, and the Ph.D. degree in electrical engineering
Power Systems, vol. 28, no. 3, pp. 2830-2841, Aug. 2013. from Tsinghua University, Beijing, China, in 2013. He is currently an Asso‐
[11] G. Sivanagaraju, S. Chakrabarti, S. C. Srivastava et al., “Uncertainty ciate Professor of North China Electric Power University, Beijing, China.
in transmission line parameters: estimation and impact on line current He serves as the Associate Editor of IET Generation, Transmission & Distri‐
differential protection,” IEEE Transactions on Instrumentation and bution and IET Renewable Power Generation. His research interests include
Measurement, vol. 63, no. 6, pp. 1496-1504, Jun. 2014. state estimation and power system analysis and control.
[12] Y. Xiang, J. Liu, Y. Liu et al., “Robust energy management of mi‐
crogrid with uncertain renewable generation and load,” IEEE Transac‐
tions on Smart Grid, vol. 7, no. 2, pp. 1034-1043, Mar. 2016. Hao Chen received the B. S. degree in electrical engineering from North
[13] Y. Chen, F. Liu, S. Mei et al., “A robust WLAV state estimation using China Electric Power University, Beijing, China, in 2018, where he is cur‐
optimal transformations,” IEEE Transactions on Power Systems, vol. rently studying for the M.S. degree. His research interests include state esti‐
30, no. 4, pp. 2190-2191, Jul. 2015. mation and power system analysis and control.
[14] Y. Chen, J. Ma, P. Zhang et al., “Robust state estimator based on max‐
imum exponential absolute value,” IEEE Transactions on Smart Grid, Yang Jiao received the B. S. degree in electrical engineering from Shanxi
vol. 8, no. 4, pp. 1537-1544, Jul. 2017. University, Taiyuan, China, in 2019. He is currently pursuing the M.S. de‐
[15] Y. Chen, Z. Zhang, H. Fang et al., “Generalised-fast decoupled state gree at North China Electric Power University, Beijing, China. His research
estimator,” IET Generation, Transmission & Distribution, vol. 12, no. interests include state estimation and power system analysis and control.
22, pp. 5928-5938, Sept. 2018.
[16] Y. Weng, R. Negi, C. Faloutsos et al., “Robust data-driven state esti‐ Jin Ma received the B. S. and M. S. degrees in electrical engineering from
mation for smart grid,” IEEE Transactions on Smart Grid, vol. 8, no. Zhejiang University, Hangzhou, China, in 1997 and 2000, respectively, and
4, pp. 1956-1967, Jul. 2017. the Ph. D. degree in electrical engineering from Tsinghua University, Bei‐
[17] J. Kim, L. Tong, R. J. Thomas et al., “Subspace methods for data at‐ jing, China, in 2004. He is currently with the School of Electrical and Infor‐
tack on state estimation: a data driven approach,” IEEE Transactions mation Engineering, University of Sydney, Sydney, Australia. He is the
on Signal Processing, vol. 63, no. 5, pp. 1102-1114, Mar. 2015. Member of CIGRE W. G. C4.605“Modeling and aggregation of loads in
[18] J. Zhang, Z. Chu, L. Sankar et al., “Can attackers with limited infor‐ flexible power networks”and the Corresponding Member of CIGRE Joint
mation exploit historical data to mount successful false data injection Workgroup C4-C6/CIRED“Modeling and dynamic performance of inverter
attacks on power systems?” IEEE Transactions on Power Systems, based generation in power system transmission and distribution studies” . He
vol. 33, no. 5, pp. 4775-4786, Sept. 2018. is a Registered Chartered Engineer. His research interests include load mod‐
[19] M. Ferdowsi, A. Benigni, and A. Löwen, “A scalable data-driven mon‐ eling, nonlinear control system, dynamic power system, and power system
CHEN et al.: DATA-DRIVEN ROBUST STATE ESTIMATION THROUGH OFF-lINE LEARNING AND ON-LINE MATCHING 13

economics. ty, Boston, USA, in 2018. He is currently an Assistant Professor of Depart‐

ment of Electrical and Computer Engineering, University of Massachusetts,
Yuzhang Lin received the B.S. and M.S. degrees in Electrical Engineering Massachusetts, USA. His research interests include state estimation and
from Tsinghua University, Beijing, China, in 2012 and 2014, respectively, power system analysis and control.
and the Ph.D. degree in Electrical Engineering from Northeastern Universi‐