System Identification
6.435
SET 12
Identification in Practice Error Filtering Order Estimation Model Structure Validation Examples
Munther A. Dahleh
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 1
Practical Identification
Given: Want 1) 2) 3) a model for the plant a model for the noise an estimate of the accuracy
choice of the model structure
flexibility
parsimony
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
What do we know? We know methods for identifying models inside a priori given model structures. How can we use this knowledge to provide a model for the plant, the process noise, with reasonable accuracy.
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
Considerations
Pre-treatment of data Remove the biase (may not be due to inputs) Filter the high frequency noise Outliers Introduce filtered errors. Emphasize certain frequency range. (The filter depends on the criterion).
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
Pick a model structure (or model structures) Which one is better? How can you decide which one reflects the real system? Is there any advantage from picking a model with a large number of parameters, if the input is exciting only a smaller number of frequency points? What are the important quantities that can be computed directly from the data (inputs & outputs), that are important to identification?
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
Pre-treatment of Data
Removing the biase
If input
, then the relation between the static and output is given by
The static component of due to
may not be entirely
, i.e. the noise might be biased.
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
Method I: Subtract the means: Define
New Data:
Method II: Model the offset by an unknown constant and estimate it.
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
High Frequency disturbances in the data record. High means above the frequency of interest. Related to the choice of the sampling period.
sampler
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
Without an anti-aliasing filter, high frequency noise is folded to low frequency.
Wanti-alias
Sample
high frequency noise depends on: a) high frequency noise due to b) aliasing. Problem occurs at both the inputs and outputs.
L LTI LP filter.
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 9
equivalently i.e. multiply the noise filter by Outliers, Bursts Either erroneous or high-disturbed data point. Could have a very bad effect on the estimate. Solution: a) Good choice of a criterion (Robust to changes) b) Observe the residual spectrum. Sometimes it is possible to determine bad data. c) Remove by hand!!! Messing up with real data. d) Failure-detection using hypothesis testing or statistical methods. (Need to define a threshold).
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 10
Role of Filters: Affecting the Biase Distribution
Frequency domain interpretation of parameter estimation:
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
11
If
: independently parametrized model structure
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
12
Heuristically, is chosen as a compromise between minimizing the integral of and matching the error spectrum Input spectrum Noise spectrum With a pre-filter: Can view the pre-filters as weighting functions to emphasize certain frequency ranges. This interpretation may not coincide with getting rid of high frequency components of the data. Depending on the criterion, the choice of L can be different.
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 13
weighting function
OE Model Structures
If
rolls off, then as long as
is small around will be
the contribution of the criterion very small. If , we expect to match
much better at
low-frequency.
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
14
Example (book)
No noise. PSRB
OE:
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
15
Good match at low frequency. Not as good at high frequency. Introduce a high-pass filter (5th order Butterworth filter, cut-off freq = 0.5 rad/sec.
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 16
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
17
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
18
ARX Model Structure
not independently-parametrized.
If rolls off, and is large at high frequency. If looks like , then it will emphasize the high frequency part of the criterion. Conclusions are not as transparent in the noisy case. However, it is in general true for large (SNR). Same example and the frequency response of
Lecture 12
is:
19
6.435, System Identification Prof. Munther A. Dahleh
Not a very good match at low frequency. Better than OE at high frequency. Can change this through a pre-filter. (5th order Butterworth, lowpass with cut-off frequency = 0.5)
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 20
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
21
Better low frequency fit. Another interpretation small at high frequency.
low frequency
Filters: high frequency if L low pass.
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
22
Conclusions
Pre filters can be viewed as design parameters as well as the standard interpretation for noise reduction. Pre-treatment of the data is quite valuable, however should be done with care. Sampling can be quite tricky. Need to estimate the bandwidth of the system.
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
23
Model Structure Determination
Flexible vs Parsimony Lots of trial and error. Usually more than one experiment is available.
Choice of model structure estimated parameters
Validation New data
Theme: Fit the data with the least complex model structure. Avoid over-fitting which amounts to fitting noise. Better to compare similar model structures, although it is necessary to compare different structures at the end.
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 24
Available data dictates the possible model structures and their dimensions. Can noise help identify the parameters? How bad is the effect of noise? Consistency in general is guaranteed for any SNR. What does that mean? Is there a rigorous way of comparing different model structures?
min prediction
model structure
Akaikes Information Theoretic Criterion (AIC). Akaikes final prediction error critierion (FPE).
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 25
Order Estimation
Qualitative Spectral estimate Step response if available. Otherwise, step response of spectral estimate. Quantitative Covariance matrices Information matrix Residual-input correlation Residual whiteness All methods are limited by the input used.
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 26
Covariance Matrix
Two basic results
u is p.e of order 2n and A, B are coprime u is p.e of order n
white or persistent
To determine n, obtain estimates of
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
27
Case 1: u is
WN, v is WN
is singular.
Increase s until
Use SVD, robust rank tests. , observe a sudden drop in the rank. Case 2: u is p.e of order If , Noise is white. is singular.
you really cannot estimate the order of the system if it is larger than .
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 28
Case 3: u is p.e of order
, Noise free case.
cannot be determined. Not a likely hypothesized model structure.
is a bad estimate of (data length). Enhanced criterion
. Of course N is fixed
estimated noise contribution.
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
29
If noise level is high, use an instrumental variable
test:
rank
If u is p.e , then generically
Other tests: Estimates of
non singular
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 30
Examples
unknown. Study possible conclusions for different experiments and different SNR. Model structure
Inputs
WN
Can determine from the spectrum or u (or simply FFT).
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
31
SNR:
All examples, you can access both the inputs and outputs.
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
32
First Experiment
Test for model order:
From data
is singular for (noticed a sudden drop)
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
33
Estimated system
small
(ARX)
Comment: If m is the correct structure estimates of the parameters. Plot shows different SNR.
, the above are good
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
34
random Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 35
ARX (ARX) Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 36
Second Experiment
u is p.e of order 2.
1 0.1 0.01
-0.0014 -1.38 -1.398
0.0005 0.479 0.4885
-0.0520 1.255 0.9631
0.054 0.356 0.5437
high high low
2.4x10-6 3.03x10-7 2.67x10-7
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
37
Theoretical Analysis: Data is informative (although det is small)
regardless of the estimates were quite bad for comparison to WN inputs. Even though noise helps in obtaining asymptotic convergence (through providing excitation), it is not helpful for finite-data records, since its effect cannot be averaged. Accuracy depends on in
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
38
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
39
=1 =1 =0 =0
ARX (with ARX plant) 2nd input
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
40
Third Experiment
u is p.e of order 4.
1 0.1 0.01
-1.475 -1.39 -1.395
0.5317 0.478 0.4862
2.38 1.186 1.022
-1.134 0.333 0.487
high smaller low
0.25 0.07 0.069
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
41
Theoretical Analysis:
Data is informative w. r. to
Results for
are better in this case than
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
42
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
43
ARX (of an ARX plant) Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 44
SNR = 1
ARX (of an ARX plant) Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 45
SNR = 0.01
ARX (of an ARX plant) Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 46
Conclusions
Accuracy of estimates depend on If ARX, Estimate of accuracy Large is large. is singular. is large.
u is not rich
u is rich (but close to not rich)
small better accuracy.
Lecture 12
is large.
6.435, System Identification Prof. Munther A. Dahleh
47
Explanation (Proof of HW#3 problem 2)
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
48
is near singular Low accuracy for estimates. This can be countered .
by a small
How about other structures?
same conclusions as long as v is persistent. OE, ARMAX,
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
49
Comparisons of Different Model Structures
There is a trade off between the model structure complexity and the min. error
min V
dim of model structure
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
50
Akaikes Final Prediction Error criterion (FPE)
model structure
Based on minimum max likelihood. Tradeoff dm vs
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
51
A natural way to evaluate a model structure
is by
is a random variable Obtain estimates of both Result and J
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
52
Proof: expand
around
also
Notice:
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
53
and
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
54
Akaikes Information Theoretic Criterion
Let Assume The matrix (Log likelihood function) true system is invertible (identifiability).
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
55
Model structure determination problem
For every fixed
does not affect the min.
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
56
Example
Assume that the innovations unknown variance. are Gaussian with
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
57
Approximately minimize
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
58
Akaikes Final Prediction Error Criterion
Let
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
59
estimate of
(FPE)
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
60
Example 2
is unknown. 3 experiments
Consider 2 - model structures
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
61
Experiment 1
u = rand sequence
system has dim = 2
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
62
Estimated parameters
ARX OE
-1.4 -1.4
0.49 0.49
1 1
0.5 0.5
0 0
Try different structures
AIC or FPE
Lecture 12
is the best choice.
6.435, System Identification Prof. Munther A. Dahleh 63
Experiment 2
u is p.e of order 2
ARX: OE:
Lecture 12
64
6.435, System Identification Prof. Munther A. Dahleh
Structure
(1, 1, 1) (2, 2, 1) (3, 3, 1)
ARX Parameters
(-0.8627, 2.3182) (-1.4, 0.49, 1, 0.5) (- * * * * ) ( 0.034) 0 170
OE Parameters
(-0.85, 2.51) (-1.4, 0.49, 1, 0.5) ( * * * * ) 0.214 0 0.3317
Loss of identifiability.
Clearly
is the preferred model structure
Remark: Both experiments were generated from the model
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
65
Experiment 3
Structure
(2 2 1) (3 3 1)
ARX Parameters
(-1.4004, 0.4903, 0.98, 0.51) singular 3.4x10-5
OE Parameters
(-1.40, 0.49, 0.95, 0.54) singular 9.2x10-6
OE model is preferred.
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
66
Structure
(2 2 1) (3 3 1)
ARX Parameters
(-1.4, 0.49, 1.0031, 0.4944) ( * * * ) 1.6x10-5 1.3x10-5
OE Parameters
(1.399, 0.489, 0.99, 0.50) ( * * * ) 1.3x10-5 4.2x10-5 Num. errors
AIC for ARX AIC for ARX and OE
{In fact
does better}. OE structure.
Remark: Data generated by
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
67
Validation
Use different sets of data to validate the model structure and the estimated model. You can obtain different estimates using the data and then average them. OR you can construct new input-output pairs and re-estimate.
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
68
Conclusions
Criterion contains a penalty function for the dimension of the system. AIC is one way of doing that. (FPE) is an estimate of the AIC with a quadratic objective. AIC has connections with information theory observation assumed PDF true Entropy of PDF
w. r. to
Lecture 12
6.435, System Identification Prof. Munther A. Dahleh
69
over the observation = information distance
AIC is the average information distance
after some simplification
Careful about Numerical errors!?
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 70
