System Identification

6.435
SET 12
Identification in Practice Error Filtering Order Estimation Model Structure Validation Examples

Munther A. Dahleh
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 1

Practical Identification
Given: Want 1) 2) 3) a model for the plant a model for the noise an estimate of the accuracy

choice of the model structure

flexibility

parsimony

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

 What do we know? We know methods for identifying models inside a priori given model structures. How can we use this knowledge to provide a model for the plant, the process noise, with reasonable accuracy.

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

Considerations
Pre-treatment of data Remove the biase (may not be due to inputs) Filter the high frequency noise Outliers Introduce filtered errors. Emphasize certain frequency range. (The filter depends on the criterion).

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

 Pick a model structure (or model structures) Which one is better? How can you decide which one reflects the real system? Is there any advantage from picking a model with a large number of parameters, if the input is exciting only a smaller number of frequency points? What are the important quantities that can be computed directly from the data (inputs & outputs), that are important to identification?

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

Pre-treatment of Data
Removing the biase

If input

, then the relation between the static and output is given by

The static component of due to

may not be entirely

, i.e. the noise might be biased.

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

 Method I: Subtract the means: Define

New Data:

Method II: Model the offset by an unknown constant and estimate it.

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

 High Frequency disturbances in the data record. High means above the frequency of interest. Related to the choice of the sampling period.
sampler

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

 Without an anti-aliasing filter, high frequency noise is folded to low frequency.

Wanti-alias

Sample

high frequency noise depends on: a) high frequency noise due to b) aliasing. Problem occurs at both the inputs and outputs.

L LTI LP filter.

Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 9

equivalently i.e. multiply the noise filter by Outliers, Bursts Either erroneous or high-disturbed data point. Could have a very bad effect on the estimate. Solution: a) Good choice of a criterion (Robust to changes) b) Observe the residual spectrum. Sometimes it is possible to determine bad data. c) Remove by hand!!! Messing up with real data. d) Failure-detection using hypothesis testing or statistical methods. (Need to define a threshold).
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 10

Role of Filters: Affecting the Biase Distribution

Frequency domain interpretation of parameter estimation:

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

11

If

: independently parametrized model structure

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

12

 Heuristically, is chosen as a compromise between minimizing the integral of and matching the error spectrum Input spectrum Noise spectrum With a pre-filter: Can view the pre-filters as weighting functions to emphasize certain frequency ranges. This interpretation may not coincide with getting rid of high frequency components of the data. Depending on the criterion, the choice of L can be different.
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 13

weighting function

OE Model Structures

If

rolls off, then as long as

is small around will be

the contribution of the criterion very small. If , we expect to match

much better at

low-frequency.

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

14

 Example (book)

No noise. PSRB

OE:

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

15

 Good match at low frequency. Not as good at high frequency. Introduce a high-pass filter (5th order Butterworth filter, cut-off freq = 0.5 rad/sec.
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 16

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

17

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

18

ARX Model Structure

If rolls off, and is large at high frequency. If looks like , then it will emphasize the high frequency part of the criterion. Conclusions are not as transparent in the noisy case. However, it is in general true for large (SNR). Same example and the frequency response of
Lecture 12

is:
19

6.435, System Identification Prof. Munther A. Dahleh

 Not a very good match at low frequency. Better than OE at high frequency. Can change this through a pre-filter. (5th order Butterworth, lowpass with cut-off frequency = 0.5)
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 20

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

21

 Better low frequency fit. Another interpretation small at high frequency.

Filters: high frequency if L low pass.

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

22

Conclusions
Pre filters can be viewed as design parameters as well as the standard interpretation for noise reduction. Pre-treatment of the data is quite valuable, however should be done with care. Sampling can be quite tricky. Need to estimate the bandwidth of the system.

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

23

Model Structure Determination

Validation New data

Theme: Fit the data with the least complex model structure. Avoid over-fitting which amounts to fitting noise. Better to compare similar model structures, although it is necessary to compare different structures at the end.
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 24

 Available data dictates the possible model structures and their dimensions. Can noise help identify the parameters? How bad is the effect of noise? Consistency in general is guaranteed for any SNR. What does that mean? Is there a rigorous way of comparing different model structures?
min prediction

model structure

Akaikes Information Theoretic Criterion (AIC). Akaikes final prediction error critierion (FPE).
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 25

Order Estimation
Qualitative Spectral estimate Step response if available. Otherwise, step response of spectral estimate. Quantitative Covariance matrices Information matrix Residual-input correlation Residual whiteness All methods are limited by the input used.
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 26

Covariance Matrix

Two basic results

u is p.e of order 2n and A, B are coprime u is p.e of order n

To determine n, obtain estimates of

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

27

Case 1: u is

WN, v is WN
is singular.

Increase s until

Use SVD, robust rank tests. , observe a sudden drop in the rank. Case 2: u is p.e of order If , Noise is white. is singular.

you really cannot estimate the order of the system if it is larger than .
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 28

Case 3: u is p.e of order

, Noise free case.

cannot be determined. Not a likely hypothesized model structure.

is a bad estimate of (data length). Enhanced criterion

. Of course N is fixed

estimated noise contribution.

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

29

 If noise level is high, use an instrumental variable

test:

rank

If u is p.e , then generically

Other tests: Estimates of

non singular
Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 30

Examples
unknown. Study possible conclusions for different experiments and different SNR. Model structure

Inputs

WN
Can determine from the spectrum or u (or simply FFT).

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

31

 SNR:

All examples, you can access both the inputs and outputs.

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

32

First Experiment

Test for model order:

From data

is singular for (noticed a sudden drop)

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

33

 Estimated system

small
(ARX)

Comment: If m is the correct structure estimates of the parameters. Plot shows different SNR.

, the above are good

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

34

random Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 35

ARX (ARX) Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 36

Second Experiment

u is p.e of order 2.

1 0.1 0.01

-0.0014 -1.38 -1.398

0.0005 0.479 0.4885

-0.0520 1.255 0.9631

0.054 0.356 0.5437

high high low

2.4x10-6 3.03x10-7 2.67x10-7

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

37

Theoretical Analysis: Data is informative (although det is small)

regardless of the estimates were quite bad for comparison to WN inputs. Even though noise helps in obtaining asymptotic convergence (through providing excitation), it is not helpful for finite-data records, since its effect cannot be averaged. Accuracy depends on in

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

38

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

39

=1 =1 =0 =0

ARX (with ARX plant) 2nd input

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

40

Third Experiment

u is p.e of order 4.

1 0.1 0.01

-1.475 -1.39 -1.395

0.5317 0.478 0.4862

2.38 1.186 1.022

-1.134 0.333 0.487

high smaller low

0.25 0.07 0.069

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

41

Theoretical Analysis:

Data is informative w. r. to

Results for

are better in this case than

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

42

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

43

ARX (of an ARX plant) Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 44

SNR = 1

ARX (of an ARX plant) Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 45

SNR = 0.01

ARX (of an ARX plant) Lecture 12 6.435, System Identification Prof. Munther A. Dahleh 46

Conclusions
Accuracy of estimates depend on If ARX, Estimate of accuracy Large is large. is singular. is large.

u is not rich

u is rich (but close to not rich)

is large.

6.435, System Identification Prof. Munther A. Dahleh

47

 Explanation (Proof of HW#3 problem 2)

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

48

is near singular Low accuracy for estimates. This can be countered .

by a small

How about other structures?

same conclusions as long as v is persistent. OE, ARMAX,

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

49

Comparisons of Different Model Structures

dim of model structure

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

50

 Akaikes Final Prediction Error criterion (FPE)

Based on minimum max likelihood. Tradeoff dm vs

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

51

A natural way to evaluate a model structure

is by

is a random variable Obtain estimates of both Result and J

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

52

Proof: expand

around

also

Notice:

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

53

and

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

54

Akaikes Information Theoretic Criterion

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

55

 Model structure determination problem

For every fixed

does not affect the min.

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

56

Example
Assume that the innovations unknown variance. are Gaussian with

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

57

Approximately minimize

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

58

Akaikes Final Prediction Error Criterion

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

59

estimate of

(FPE)

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

60

Example 2
is unknown. 3 experiments

Consider 2 - model structures

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

61

Experiment 1
u = rand sequence

system has dim = 2

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

62

 Estimated parameters

ARX OE

-1.4 -1.4

0.49 0.49

1 1

0.5 0.5

0 0

Try different structures

AIC or FPE
Lecture 12

is the best choice.

Experiment 2

u is p.e of order 2

ARX: OE:
Lecture 12

64

6.435, System Identification Prof. Munther A. Dahleh

Structure
(1, 1, 1) (2, 2, 1) (3, 3, 1)

ARX Parameters
(-0.8627, 2.3182) (-1.4, 0.49, 1, 0.5) (- * * * * ) ( 0.034) 0 170

OE Parameters
(-0.85, 2.51) (-1.4, 0.49, 1, 0.5) ( * * * * ) 0.214 0 0.3317

Loss of identifiability.

Clearly

is the preferred model structure

Remark: Both experiments were generated from the model

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

65

Experiment 3
Structure
(2 2 1) (3 3 1)

ARX Parameters
(-1.4004, 0.4903, 0.98, 0.51) singular 3.4x10-5

OE Parameters
(-1.40, 0.49, 0.95, 0.54) singular 9.2x10-6

OE model is preferred.

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

66

 Structure
(2 2 1) (3 3 1)

ARX Parameters
(-1.4, 0.49, 1.0031, 0.4944) ( * * * ) 1.6x10-5 1.3x10-5

OE Parameters
(1.399, 0.489, 0.99, 0.50) ( * * * ) 1.3x10-5 4.2x10-5 Num. errors

AIC for ARX AIC for ARX and OE

{In fact

does better}. OE structure.

Remark: Data generated by

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

67

Validation
Use different sets of data to validate the model structure and the estimated model. You can obtain different estimates using the data and then average them. OR you can construct new input-output pairs and re-estimate.

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

68

Conclusions
Criterion contains a penalty function for the dimension of the system. AIC is one way of doing that. (FPE) is an estimate of the AIC with a quadratic objective. AIC has connections with information theory observation assumed PDF true Entropy of PDF

w. r. to

Lecture 12

6.435, System Identification Prof. Munther A. Dahleh

69

over the observation = information distance

AIC is the average information distance

after some simplification

Careful about Numerical errors!?