Full download Identification of Nonlinear Physiological Systems IEEE Press Series on Biomedical Engineering 1st Edition David T. Westwick pdf docx

Visit https://ebookgate.
com to download the full version and

explore more ebooks
Identification of Nonlinear Physiological Systems

IEEE Press Series on Biomedical Engineering 1st
Edition David T. Westwick
_____ Click the link below to download _____

https://ebookgate.com/product/identification-of-
nonlinear-physiological-systems-ieee-press-series-on-
biomedical-engineering-1st-edition-david-t-westwick/
Explore and download more ebooks at ebookgate.com

Here are some recommended products that might interest you.
You can download now and explore!
Nonlinear Dynamic Modeling of Physiological Systems IEEE

Press Series on Biomedical Engineering 1st Edition
Professor Vasilis Z. Marmarelis
https://ebookgate.com/product/nonlinear-dynamic-modeling-of-
physiological-systems-ieee-press-series-on-biomedical-engineering-1st-
edition-professor-vasilis-z-marmarelis/
ebookgate.com
Elements of Tidal Electric Engineering IEEE Press Series

on Power Engineering 1st Edition Robert H. Clark
https://ebookgate.com/product/elements-of-tidal-electric-engineering-
ieee-press-series-on-power-engineering-1st-edition-robert-h-clark/
ebookgate.com
Communication and Control in Electric Power Systems

Applications of Parallel and Distributed Processing IEEE
Press Series on Power Engineering 1st Edition Mohammad
Shahidehpour
https://ebookgate.com/product/communication-and-control-in-electric-
power-systems-applications-of-parallel-and-distributed-processing-
ieee-press-series-on-power-engineering-1st-edition-mohammad-
shahidehpour/
ebookgate.com
Electricity Power Generation The Changing Dimensions IEEE

Press Series on Power Engineering 1st Edition Digambar M.
Tagare
https://ebookgate.com/product/electricity-power-generation-the-
changing-dimensions-ieee-press-series-on-power-engineering-1st-
edition-digambar-m-tagare/
ebookgate.com
Reliability Wearout Mechanisms in Advanced CMOS
Technologies IEEE Press Series on Microelectronic Systems
1st Edition Alvin W. Strong
https://ebookgate.com/product/reliability-wearout-mechanisms-in-
advanced-cmos-technologies-ieee-press-series-on-microelectronic-
systems-1st-edition-alvin-w-strong/
ebookgate.com
Introduction to FACTS Controllers Theory Modeling and

Applications IEEE Press Series on Power Engineering 1st
Edition Kalyan K. Sen
https://ebookgate.com/product/introduction-to-facts-controllers-
theory-modeling-and-applications-ieee-press-series-on-power-
engineering-1st-edition-kalyan-k-sen/
ebookgate.com
Sourcebook of ATM and IP Internetworking IEEE Press Series

on Network Management 1st Edition Khalid Ahmad
https://ebookgate.com/product/sourcebook-of-atm-and-ip-
internetworking-ieee-press-series-on-network-management-1st-edition-
khalid-ahmad/
ebookgate.com
Perceptual Computing Aiding People in Making Subjective

Judgments IEEE Press Series on Computational Intelligence
1st Edition Jerry Mendel
https://ebookgate.com/product/perceptual-computing-aiding-people-in-
making-subjective-judgments-ieee-press-series-on-computational-
intelligence-1st-edition-jerry-mendel/
ebookgate.com
System of Systems Engineering Wiley Series in Systems

Engineering and Management 1st Edition Mohammad Jamshidi
https://ebookgate.com/product/system-of-systems-engineering-wiley-
series-in-systems-engineering-and-management-1st-edition-mohammad-
jamshidi/
ebookgate.com
IDENTIFICATION OF
NONLINEAR
PHYSIOLOGICAL SYSTEMS
IEEE Press Series in Biomedical Engineering
The focus of our series is to introduce current and emerging technologies to biomedical and electrical engineer-
ing practitioners, researchers, and students. This series seeks to foster interdisciplinary biomedical engineering
education to satisfy the needs of the industrial and academic areas. This requires an innovative approach that
overcomes the difficulties associated with the traditional textbook and edited collections.
Metin Akay, Series Editor

Dartmouth College
Advisory Board
Thomas Budinger Simon Haykin Richard Robb
Ingrid Daubechies Murat Kunt Richard Satava
Andrew Daubenspeck Paul Lauterbur Malvin Teich
Murray Eden Larry McIntire Herbert Voigt
James Greenleaf Robert Plonsey Lotfi Zadeh
Editorial Board
Eric W. Abel Gabor Herman Kris Ropella
Dan Adam Helene Hoffman Joseph Rosen
Peter Adlassing Donna Hudson Christian Roux
Berj Bardakjian Yasemin Kahya Janet Rutledge
Erol Basar Michael Khoo Wim L. C. Rutten
Katarzyna Blinowska Yongmin Kim Alan Sahakian
Bernadette Bouchon-Meunier Andrew Laine Paul S. Schenker
Tom Brotherton Rosa Lancini G. W. Schmid-Schönbein
Eugene Bruce Swamy Laxminarayan Ernest Stokely
Jean-Louis Coatrieux Richard Leahy Ahmed Tewfik
Sergio Cerutti Zhi-Pei Liang Nitish Thakor
Maurice Cohen Jennifer Linderman Michael Unser
John Collier Richard Magin Eugene Veklerov
Steve Cowin Jaakko Malmivuo Al Wald
Jerry Daniels Jorge Monzon Bruce Wheeler
Jaques Duchene Michael Neuman Mark Wiederhold
Walter Greenleaf Banu Onaral William Williams
Daniel Hammer Keith Paulsen Andy Yagle
Dennis Healy Peter Richardson Yuan-Ting Zhang
A list of books in the IEEE Press Series in Biomedical Engineering can be found
on page 262.
IDENTIFICATION OF
NONLINEAR
PHYSIOLOGICAL SYSTEMS
DAVID T. WESTWICK
ROBERT E. KEARNEY
A JOHN WILEY & SONS, INC., PUBLICATION

IEEE Press
445 Hoes Lane
Piscataway, NJ 08854
IEEE Press Editorial Board

Stamatios V. Kartalopoulos, Editor in Chief
M. Ajay R. J. Herrick M. S. Newman

J .B. Anderson R. F. Hoyt M. Padgett
J. Baker D. Kirk W. D. Reeve
J. E. Brewer R. Leonardi S. Tewksbury
M. E. El-Hawary G. Zobrist
Kenneth Moore, Director of IEEE Press
Catherine Faduska, Senior Acquisitions Editor
Christina Kuhnen, Associate Acquisitions Editor
IEEE Engineering in Medicine and Biology Society, Sponsor

EMB-S Liaison to IEEE Press, Metin Akay
Technical Reviewers
Metin Akay
Robert F. Kirsch
John A. Daubenspeck
Copyright
c 2003 by the Institute of Electrical and Electronics Engineers. All rights reserved.
Published simultaneously in Canada.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means,
electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the
1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through
payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923,
978-750-8400, fax 978-750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be
addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011,
fax (201) 748-6008, e-mail: [email protected].
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book,
they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and
specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created
or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable
for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable
for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or
other damages. For general information on our other products and services please contact our Customer Care Department
within the U.S. at 877-762-2974, outside the U.S. at 317-572-3993 or fax 317-572-4002. Wiley also publishes its books in a
variety of electronic formats. Some content that appears in print, however, may not be available in electronic format.
Library of Congress Cataloging-in-Publication Data:
Westwick, D. T. (David T.)

Identification of nonlinear physiological systems / D.T. Westwick, R.E. Kearney.
p. cm.—(IEEE Press series on biomedical engineering)
“IEEE Engineering in Medicine and Biology Society, Sponsor.”
Includes bibliographical references and index.
ISBN 0-471-27456-9 (cloth)
1. Physiology—Mathematical models. 2. Nonlinear systems. I. Kearney, Robert E.,
1947- II. IEEE Engineering in Medicine and Biology Society. III. Title. IV. IEEE Press
series in biomedical engineering.
QP33.6.M36W475 2003
612 .01 5118—dc21
2003043255
Printed in the United States of America
10 9 8 7 6 5 4 3 2 1
CONTENTS
Preface xi
1 Introduction 1
1.1 Signals / 1
1.1.1 Domain and Range / 2
1.1.2 Deterministic and Stochastic Signals / 2
1.1.3 Stationary and Ergodic Signals / 3
1.2 Systems and Models / 3
1.2.1 Model Structure and Parameters / 4
1.2.2 Static and Dynamic Systems / 5
1.2.3 Linear and Nonlinear Systems / 6
1.2.4 Time-Invariant and Time-Varying Systems / 7
1.2.5 Deterministic and Stochastic Systems / 7
1.3 System Modeling / 8
1.4 System Identification / 8
1.4.1 Types of System Identification Problems / 9
1.4.2 Applications of System Identification / 11
1.5 How Common are Nonlinear Systems? / 11
2 Background 13
2.1 Vectors and Matrices / 13

2.2 Gaussian Random Variables / 14
2.2.1 Products of Gaussian Variables / 15
2.3 Correlation Functions / 16
v
vi CONTENTS
2.3.1 Autocorrelation Functions / 16

2.3.2 Cross-Correlation Functions / 18
2.3.3 Effects of Noise / 20
2.3.4 Estimates of Correlation Functions / 21
2.3.5 Frequency Domain Expressions / 22
2.3.6 Applications / 23
2.3.7 Higher-Order Correlation Functions / 25
2.4 Mean-Square Parameter Estimation / 25
2.4.1 Linear Least-Squares Regression / 26
2.4.2 Properties of Estimates / 27
2.5 Polynomials / 29
2.5.1 Power Series / 29
2.5.2 Orthogonal Polynomials / 30
2.5.3 Hermite Polynomials / 31
2.5.4 Tchebyshev Polynomials / 32
2.5.5 Multiple-Variable Polynomials / 33
2.6 Notes and References / 35
2.7 Problems / 36
2.8 Computer Exercises / 36
3 Models of Linear Systems 39
3.1 Linear Systems / 39

3.2 Nonparametric Models / 40
3.2.1 Time Domain Models / 41
3.2.2 Frequency Domain Models / 43
3.3 Parametric Models / 46
3.3.1 Parametric Frequency Domain Models / 46
3.3.2 Discrete-Time Parametric Models / 48
3.4 State-Space Models / 52
3.4.1 Example: Human Ankle Compliance—Discrete-Time,
State-Space Model / 54
3.6 Theoretical Problems / 55
4 Models of Nonlinear Systems 57
4.1 The Volterra Series / 57

4.1.1 The Finite Volterra Series / 59
4.1.2 Multiple-Input Systems / 62
4.1.3 Polynomial Representation / 64
4.1.4 Convergence Issues(† ) / 65
CONTENTS vii
4.2 The Wiener Series / 67

4.2.1 Orthogonal Expansion of the Volterra Series / 68
4.2.2 Relation Between the Volterra and Wiener Series / 70
4.2.3 Example: Peripheral Auditory Model—Wiener Kernels / 71
4.2.4 Nonwhite Inputs / 73
4.3 Simple Block Structures / 73
4.3.1 The Wiener Model / 73
4.3.2 The Hammerstein Model / 77
4.3.3 Sandwich or Wiener–Hammerstein Models / 79
4.3.4 NLN Cascades / 83
4.3.5 Multiple-Input Multiple-Output Block Structured Models / 87
4.4 Parallel Cascades / 87
4.4.1 Approximation Issues(† ) / 89
4.5 The Wiener–Bose Model / 91
4.5.1 Similarity Transformations and Uniqueness / 92
4.5.2 Approximation Issues(† ) / 94
4.5.3 Volterra Kernels of the Wiener–Bose Model / 94
4.5.4 Wiener Kernels of the Wiener–Bose Model / 95
4.5.5 Relationship to the Parallel Cascade Model / 97
4.7 Theoretical Problems / 100
5 Identification of Linear Systems 103
5.1 Introduction / 103

5.1.1 Example: Identification of Human Joint Compliance / 103
5.1.2 Model Evaluation / 105
5.2 Nonparametric Time Domain Models / 107
5.2.1 Direct Estimation / 107
5.2.2 Least-Squares Regression / 108
5.2.3 Correlation-Based Methods / 109
5.3 Frequency Response Estimation / 115
5.3.1 Sinusoidal Frequency Response Testing / 115
5.3.2 Stochastic Frequency Response Testing / 116
5.3.3 Coherence Functions / 117
5.4 Parametric Methods / 119
5.4.1 Regression / 119
5.4.2 Instrumental Variables / 120
5.4.3 Nonlinear Optimization / 121
viii CONTENTS
6 Correlation-Based Methods 125
6.1 Methods for Functional Expansions / 125

6.1.1 Lee–Schetzen Cross-Correlation / 125
6.1.2 Colored Inputs / 140
6.1.3 Frequency Domain Approaches / 144
6.2 Block-Structured Models / 149
6.2.1 Wiener Systems / 150
6.2.2 Hammerstein Models / 155
6.2.3 LNL Systems / 162
6.3 Problems / 167
7 Explicit Least-Squares Methods 169
7.1 Introduction / 169

7.2 The Orthogonal Algorithms / 169
7.2.1 The Orthogonal Algorithm / 171
7.2.2 The Fast Orthogonal Algorithm / 173
7.2.3 Variance of Kernel Estimates / 180
7.2.4 Example: Fast Orthogonal Algorithm Applied to Simulated Fly
Retina Data / 182
7.2.5 Application: Dynamics of the Cockroach Tactile Spine / 186
7.3 Expansion Bases / 187
7.3.1 The Basis Expansion Algorithm / 190
7.3.2 The Laguerre Expansion / 191
7.3.3 Limits on α / 192
7.3.4 Choice of α and P / 194
7.3.5 The Laguerre Expansion Technique / 195
7.3.6 Computational Requirements / 195
7.3.7 Variance of Laguerre Kernel Estimates / 195
7.3.8 Example: Laguerre Expansion Kernels of the Fly
Retina Model / 196
7.4 Principal Dynamic Modes / 198
7.4.1 Example: Principal Dynamic Modes of the Fly
Retina Model / 200
7.4.2 Application: Cockroach Tactile Spine / 201
7.5 Problems / 205
8 Iterative Least-Squares Methods 207
8.1 Optimization Methods / 207

8.1.1 Gradient Descent Methods / 208
CONTENTS ix
8.1.2 Identification of Block-Structured Models / 209

8.1.3 Second-Order Optimization Methods / 212
8.1.4 Jacobians for Other Block Structures / 216
8.1.5 Optimization Methods for Parallel Cascade Models / 219
8.1.6 Example: Using a Separable Volterra Network / 220
8.2 Parallel Cascade Methods / 223
8.2.1 Parameterization Issues / 226
8.2.2 Testing Paths for Significance / 228
8.2.3 Choosing the Linear Elements / 230
8.2.4 Parallel Wiener Cascade Algorithm / 242
8.2.5 Longer Cascades / 242
8.2.6 Example: Parallel Cascade Identification / 243
8.3 Application: Visual Processing in the Light-Adapted Fly Retina / 246
8.4 Problems / 249
References 251
Index 259
IEEE Press Series in Biomedical Engineering 262

PREFACE
Since it first appeared in 1978, Advanced Methods in Physiological Modeling: The White
Noise Approach by P. Z. Marmarelis and M. Z. Marmarelis has been the standard ref-
erence for the field of nonlinear system identification, especially as applied in biomed-
ical engineering and physiology. Despite being long out of print, Marmarelis and Mar-
marelis is still, in many cases, the primary reference. Over the years, dramatic advances
have been made in the field, many of which became practical only with the advent of
widespread computing power. Many of these newer developments have been described
in the three volumes of the series Advanced Methods in Physiological Modeling, edited
by V. Z. Marmarelis. While these volumes have been an invaluable resource to many
researchers, helping them to stay abreast of recent developments, they are all collections
of research articles. As a resource for someone starting out in the field, they are some-
what lacking. It is difficult for a newcomer to the field to see the relationships between
myriad contributions. Choosing which approach is best for a given application can be an
arduous task, at best.
This textbook developed out of a review article (Westwick and Kearney, 1998) on the
same subject. The goal of the review article was to bring the various analyses that have
been developed by several groups of researchers into a common notation and framework,
and thus to elucidate the relationships between them. The aim of this book was to go one
step farther and to provide this common framework along with the background necessary
to bring the next generation of systems physiologists into the fold.
In this book, we have attempted to provide the student with an overview of many of
the techniques currently in use, and some of the earlier methods as well. Everything is
presented in a common notation and from a consistent theoretical framework. We hope
that the relationships between the methods and their relative strengths and weaknesses
will become apparent to the reader. The reader should be well-equipped to make an
informed decision as to which techniques to try, when faced with an identification or
modeling problem.
xi
xii PREFACE
We have assumed that readers of this book have a background in linear signals and
systems equivalent to that given by a junior year signals and systems course. Back-
ground material beyond that level is summarized, with references given to more detailed,
pedagogical treatments.
Each chapter has several theoretical problems, which can be solved with pencil and
paper. In addition, most of the chapters conclude with some computer exercises. These
are intended to give the reader practical experience with the tools described in the text.
These computer exercises make use of MATLAB∗ and the nonlinear system identifica-
tion (NLID) toolbox (Kearney and Westwick, 2003). More information regarding the NLID
toolbox can be found at www.bmed.mcgill.ca. In addition to implementing all of the
system identification tools as MATLAB m-files, the toolbox also contains the data and
model structures used to generate the examples that run throughout the text.
Although our primary goal is to educate informed users of these techniques, we have
included several theoretical sections dealing with issues such as the generality of some
model structures, convergence of series-based models, and so on. These sections are
marked with a dagger, †, and they can be skipped by readers interested primarily in
practical application of these methods, with little loss in continuity.
The dedication in Marmarelis and Marmarelis reads “To an ambitious breed: Systems
Physiologists.” We feel that the sentiment reflected in those words is as true today as it
was a quarter century ago. The computers are (much) faster, and they will undoubtedly
be faster still in a few years. As a result, the problems that we routinely deal with
today would have been inconceivable when M & M was first published. However, with
increased computational abilities come more challenging problems. No doubt, this trend
will continue. We hope that it is an interesting ride.
DAVID T. WESTWICK
ROBERT E. KEARNEY
Calgary, Alberta, Canada

Montreal, Quebec, Canada
May, 2003
∗ MATLAB is a registered trademark of the MathWorks, Inc.

CHAPTER 1
INTRODUCTION
The term “Biomedical Engineering” can refer to any endeavor in which techniques from
engineering disciplines are used to solve problems in the life sciences. One such under-
taking is the construction of mathematical models of physiological systems and their
subsequent analysis. Ideally the insights gained from analyzing these models will lead to
a better understanding of the physiological systems they represent.
System identification is a discipline that originated in control engineering; it deals with
the construction of mathematical models of dynamic systems using measurements of their
inputs and outputs. In control engineering, system identification is used to build a model
of the process to be controlled; the process model is then used to construct a controller.
In biomedical engineering, the goal is more often to construct a model that is detailed
enough to provide insight into how the system operates. This text deals with system
identification methods that are commonly used in biomedical engineering. Since many
physiological systems are highly nonlinear, the text will focus on methods for nonlinear
systems and their application to physiological systems. This chapter will introduce the
concepts of signals, systems, system modeling, and identification. It also provides a brief
overview of the system identification problem and introduces some of the notation and
terminology to be used in the book. The reader should be acquainted with most of the
material covered in this chapter. If not, pedagogical treatments can be found in most
undergraduate level signals and systems texts, such as that by Kamen (1990).
1.1 SIGNALS
The concept of a signal seems intuitively clear. Examples would include speech, a televi-
sion picture, an electrocardiogram, the price of the NASDAQ index, and so on. However,
formulating a concise, mathematical description of what constitutes a signal is somewhat
involved.
Identification of Nonlinear Physiological Systems, By David T. Westwick and Robert E. Kearney

ISBN 0-471-27456-9 c 2003 Institute of Electrical and Electronics Engineers
1
2 INTRODUCTION
1.1.1 Domain and Range

In the examples above, two sets of values were required to describe each “signal”; these
will be termed the domain and range variables of the signal. Simply put, a signal may be
viewed as a function that assigns a value in the range set for each value in the domain
set; that is, it represents a mapping from the domain to the range. For example, with
speech, the domain is time while the range could be one of a variety of variables: the air
pressure near the speaker’s mouth, the deflection of the listener’s ear drum, or perhaps
the voltage produced by a microphone.
This concept can be defined formally by describing a signal, s(t), as a mapping from
a domain set, T , which is usually time, to a range set, Y . Thus,
s:T →Y
where t ∈ T is a member of the domain set, usually time. In continuous time, T is the
real line; in discrete time, it is the set of integers. In either case, the value of the signal
is in the range set, Y . The range of the signal is given by applying the mapping to the
domain set, and is therefore s(T ).
The above definition really describes a function. A key point regarding the domain
set of a signal is the notion that it is ordered and thus has a direction. Thus, if x1 and x2
are members of the domain set, there is some way of stating x1 > x2 , or the reverse. If
time is the domain, t1 > t2 is usually taken to mean that t1 is later than t2 .
The analysis in this book will focus on signals with one-dimensional domains—usually
time. However, most of the ideas can be extended to signals with domains having
two dimensions (e.g., X-ray images), three dimensions (e.g., MRI images), or more
(e.g., time-varying EEG signals throughout the brain).
1.1.2 Deterministic and Stochastic Signals

A signal is deterministic if its future values can be generated based on a set of known
rules and parameters, perhaps expressed as a mathematical equation. For example, the
sinusoid
yd (t) = cos(2πf t + φ)
can be predicted exactly, provided that its frequency f and phase φ are known. In
contrast, if yr (k) is generated by repeatedly tossing a fair, six-sided die, there is no way
to predict the kth value of the output, even if all other output values are known. These
represent two extreme cases: yd (t) is purely deterministic while yr (k) is completely
random, or stochastic.
The die throwing example is an experiment where each repetition of the experiment
produces a single random variable: the value of the die throw. On the other hand, for
a stochastic process the result of each experiment will be a signal whose value at each
time is a random variable. Just as a single throw of a die produces a single realization of
a random variable, a random signal is a single realization of a stochastic process. Each
experiment produces a different time signal or realization of the process. Conceptually,
the stochastic process is the ensemble of all possible realizations.
In reality, most signals fall between these two extremes. Often, a signal may be
deterministic but there may not be enough information to predict it. In these cases, it
SYSTEMS AND MODELS 3
may be necessary to treat the deterministic signal as if it were a single realization of

some underlying stochastic process.
1.1.3 Stationary and Ergodic Signals

The statistical properties of a random variable, such as its mean and variance, are deter-
mined by integrating the probability distribution function (PDF) over all possible range
values. Thus, if f (x) is the PDF of a random variable x, its mean and variance are
given by
∞
µx = xf (x) dx
−∞
∞
σx2 = (x − µx )2 f (x) dx
−∞
Similar integrals are used to compute higher-order moments. Conceptually, these integrals
can be viewed as averages taken over an infinite ensemble of all possible realizations of
the random variable, x.
The value of a random signal at a point in time, considered as a random variable,
will have a PDF, f (x, t), that depends on the time, t. Thus, any statistic obtained by
integrating over the PDF will be a function of time. Alternately, the integrals used to
compute the statistics can be viewed as averages taken over an infinite ensemble of
realizations of the stochastic process, at a particular point in time. If the PDF, and hence
statistics, of a stochastic process is independent of time, then the process is said to be
stationary.
For many practical applications, only a single realization of a stochastic process will
be available; therefore, averaging must be done over time rather than over an ensemble
of realizations. Thus, the mean of a stochastic process would be estimated as
T
1
µ̂x = x(t) dt
T 0
Many stochastic process are ergodic, meaning that the ensemble and time averages are
equal.
1.2 SYSTEMS AND MODELS
Figure 1.1 shows a block diagram of a system in which the “black box,” N, transforms
the input signal, u(t), into the output y(t). This will be written as
y(t) = N(u(t)) (1.1)
to indicate that when the input u(t) is applied to the system N, the output y(t) results. Note
that the domain of the signals need not be time, as shown here. For example, if the system
operates on images, the input and output domains could be two- or three-dimensional
spatial coordinates.
This book will focus mainly on single-input single-output (SISO) systems whose
domain is time. Thus u(t) and y(t) will be single-valued functions of t. For multiple-input
4 INTRODUCTION
Input(s) Output(s)
u(t) y(t)
N
Figure 1.1 Block diagram of a “black box” system, which transforms the input(s) u(t), into the
output(s), y(t). The mathematical description of the transformation is represented by the operator N.
multiple-output (MIMO) systems, Figure 1.1, equation (1.1), and most of the develop-
ment to follow will not change; the input and output simply become vector-valued func-
tions of their domains. For example, a multidimensional input signal may be written as
a time-dependent vector,

u(t) = u1 (t) u2 (t) . . . un (t) (1.2)
1.2.1 Model Structure and Parameters

Using M to indicate a mathematical model of the physical system, N, the model output
can be written as
ŷ(t) = M(u(t)) (1.3)
where the caret, or “hat,” indicates that ŷ(t) is an estimate of the system output, y(t).
In general, a model will depend on a set of parameter parameters contained in the
parameter vector θ . For example, if the model, M(θ), was a third-degree polynomial,
ŷ(θ, t) = M(θ, u(t))

= c(0) + c(1) u(t) + c(2) u2 (t) + c(3) u3 (t) (1.4)
the parameter vector, θ , would contain the polynomial coefficients,

T
θ = c(0) c(1) c(2) c(3)
Note that in equation (1.4) the dependence of the output, ŷ(θ , t), on the parameter vector,
θ , is shown explicitly.
Models are often classified as being either parametric or nonparametric. A parametric
model generally has relatively few parameters that often have direct physical interpreta-
tions. The polynomial in equation (1.4) is an example of a parametric model. The model
structure comprises the constant, linear, quadratic and third-degree terms; the parameters
are the coefficients associated with each term. Thus each parameter is related to a par-
ticular behavior of the system; for example, the parameter c(2) defines how the output
varies with the square of the input.
In contrast, a nonparametric model is described by a curve or surface defined by its
values at a collection of points in its domain, as illustrated in Figure 1.2. Thus, a set
of samples of the curve defined by equation (1.4) would be a nonparametric model of
the same system. Here, the model structure would contain the domain values, and the
“parameters” would be the corresponding range values. Thus, a nonparametric model
usually has a large number of parameters that do not in themselves have any direct
physical interpretation.
Output: y(t) = N(u(t)) 4
−2
−4
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2
Input: u(t)
Figure 1.2 A memoryless nonlinear system. A parametric model of this system is y(t) =
−3 − u(t) + u2 (t) − 0.5u3 (t). A nonparametric model of the same system could include a list
of some of the domain and range values, say those indicated by the dots. The entire curve is
also a nonparametric model of the system. While the parametric model is more compact, the
nonparametric model is more flexible.
1.2.2 Static and Dynamic Systems

In a static, or memoryless, system, the current value of the output depends only on the
current value of the input. For example, a full-wave rectifier is a static system since its
output, y(t) = |u(t)|, depends only on the instantaneous value of its input, u(t).
On the other hand, in a dynamic system, the output depends on some or all of the
input history. For example, the output at time t of the delay operator,
y(t) = u(t − τ )
depends only on the value of the input at the previous time, t − τ .

In contrast, the output of the peak-hold operation
y(t) = max(u(τ ))
τ ≤t
retains the largest value of the past input and consequently depends on the entire history
of the input.
Dynamic systems can be further classified according to whether they respond to the
past or future values of the input, or both. The delay and peak-hold operators are both
examples of causal systems, systems whose outputs depend on previous, but not future,
values of their inputs. Systems whose outputs depend only on future values of their
inputs are said to be anti-causal or anticipative. If the output depends on both the past
and future inputs, the system said to be noncausal or mixed causal anti-causal.
Although physical systems are causal, there are a number of situations where noncausal
system descriptions are needed. For example, behavioral systems may display a predictive
ability if the input signal is deterministic or a preview is available. For example, the
dynamics of a tracking experiment may show a noncausal component if the subject is
permitted to see future values of the input as well as its current value.
Sometimes, feedback can produce behavior that appears to be noncausal. Consider the
system in Figure 1.3. Suppose that the experimenter can measure the signals labeled u(t)
and y(t), but not w1 (t) and w2 (t). Let both N1 and N2 be causal systems that include
delays. The effect of w1 (t) will be measured first in the “input,” u(t), and then later in the
6 INTRODUCTION
w2 (t)

w1 (t) u(t) y(t)
N1

N2
Figure 1.3 A feedback loop with two inputs. Depending on the relative power of the inputs
w1 (t) and w2 (t), the system N1 , or rather the relationship between u(t) and y(t), may appear to
be either causal, anti-causal, or noncausal.
“output,” y(t). However, the effect of the other input, w2 (t), will be noted in y(t) first,
followed by u(t). Thus, the delays in the feedback loop create what appears to be non-
causal system behavior. Of course the response is not really noncausal, it merely appears
so because neither u(t) nor y(t) was directly controlled. Thus, inadequate experimental
design can lead to the appearance of noncausal relationships between signals.
In addition, as will be seen below, there are cases where it is advantageous to reverse
the roles of the input and output. In the resulting analysis, a noncausal system description
must be used to describe the inverse system.
1.2.3 Linear and Nonlinear Systems

Consider a system, N, and let y(t) be the response of the system due to the input u(t).
Thus,
y(t) = N(u(t))
Let c be a constant scalar. Then if the response to the input c · u(t) satisfies
N(c · u(t)) = c · y(t) (1.5)
for any constant c, the system is said to obey the principle of proportionality or to have
the scaling property.
Consider two pairs of inputs and their corresponding outputs,
y1 (t) = N(u1 (t))

y2 (t) = N(u2 (t))
If the response to the input u1 (t) + u2 (t) is given by
N(u1 (t) + u2 (t)) = y1 (t) + y2 (t) (1.6)
then the operator N is said to obey the superposition property. Systems that obey both
superposition and scaling are said to be linear.
Nonlinear systems do not obey superposition and scaling. In many cases, a system
will obey the superposition and scaling properties approximately, provided that the inputs
lie within a restricted class. In such cases, the system is said to be operating within its
“linear range.”
1.2.4 Time-Invariant and Time-Varying Systems

If the relationship between the input and output does not depend on the absolute time,
then the system is said to be time-invariant. Thus, if y(t) is the response to the input
u(t) generated by a time-invariant system, its response due to u(t − τ ), for any real τ ,
will be y(t − τ ). Thus, a time-invariant system must satisfy
N(u(t)) = y(t) ⇒ N(u(t − τ )) = y(t − τ ) ∀τ ∈ IR (1.7)
Systems for which equation (1.7) does not hold are said to be time-varying.
1.2.5 Deterministic and Stochastic Systems

In a deterministic system, the output, y(t), depends only on the input, u(t). In many
applications, the output measurement is corrupted by additive noise,
z(t) = y(t) + v(t) = N(u(t)) + v(t) (1.8)
where v(t) is independent of the input, u(t). Although the measured output, z(t), has both
deterministic and random components, the system (1.8) is still referred to as deterministic,
since the “true” output, y(t), is a deterministic function of the input.
Alternatively, the output may depend on an unmeasurable process disturbance, w(t),
y(t) = N(u(t), w(t)) (1.9)
where w(t) is a white, Gaussian signal that cannot be measured. In this case, the system is
said to be stochastic, since there is no “noise free” deterministic output. The process noise
term, w(t), can be thought of as an additional input driving the dynamics of the system.
Measurement noise, in contrast, only appears additively in the final output. Clearly, it
is possible for a system to have both a process disturbance and measurement noise, as
illustrated in Figure 1.4, leading to the relation
z(t) = y(t) + v(t) = N(u(t), w(t)) + v(t) (1.10)
w(t) v(t)

u(t) y(t)
z(t)
N

Figure 1.4 Block diagram of a system including a process disturbance, w(t), and measurement
noise, v(t).
8 INTRODUCTION
1.3 SYSTEM MODELING
In many cases, a mathematical model of a system can be constructed from “first princi-
ples.” Consider, for example, the problem of modeling a spring. As a first approxima-
tion, it might be assumed to obey Hooke’s law and have no mass so that it could be
described by
y = −ku (1.11)
where the output, y, is the force produced, the input, u, is the displacement, and k
is the spring constant. If the spring constant were known, then equation (1.11) would
constitute a mathematical model of the system. If the spring constant, k, was unknown, it
could be estimated experimentally. Whether or not the assumptions hold, equation (1.11)
is a model of the system (but not necessarily a good model). If it yields satisfactory
predictions of the system’s behavior, then, and only then, can it be considered to be
a good model. If it does not predict well, then the model must be refined, perhaps by
considering the mass of the spring and using Newton’s second law to give
d 2 u(t)
y(t) = −ku(t) + m (1.12)
dt 2
Other possibilities abound; the spring might be damped, behave nonlinearly, or have
significant friction. The art of system modeling lies in determining which terms are likely
to be significant, and in limiting the model to relevant terms only. Thus, even in this
simple case, constructing a mathematical model based on “first principles” can become
unwieldy. For complex systems, the approach can become totally unmanageable unless
there is a good understanding of which effects should and should not be incorporated
into the model.
1.4 SYSTEM IDENTIFICATION
The system identification approach to constructing a mathematical model of the system

is much different. It assumes a general form, or structure, for the mathematical model
and then determines the parameters from experimental data. Often, a variety of model
structures are evaluated, and the most successful ones are retained. For example, consider
the spring system described in the previous section. If it were assumed to be linear, then
a linear differential equation model, such as
d n y(t) d n−1 y(t) dy(t)

+ an−1 + · · · + a1 + a0 y(t)
dt n dt n−1 dt
d m u(t) d m−1 u(t) du(t)
= bm m
+ bm−1 m−1
+ · · · + b1 + b0 u(t) (1.13)
dt dt dt
could be postulated. It would then be necessary to perform an experiment, record u(t) and
y(t), compute their derivatives, and determine the coefficients a0 . . . an−1 and b0 . . . bm .
Under ideal conditions, many of the coefficients would be near zero and could be removed
from the model. Thus, if the system could be described as a massless linear spring,
then equation (1.13) would reduce to equation (1.11) once all extraneous terms were
removed.
SYSTEM IDENTIFICATION 9
The scheme outlined in the previous paragraph is impractical for a number of reasons.
Most importantly, numerical differentiation amplifies high-frequency noise. Thus, the
numerically computed derivatives of the input and output, particularly the high-order
derivatives, will be dominated by high-frequency noise that will distort the parameter
estimates. Thus, a more practical approach to estimating the system dynamics from
input–output measurements is required.
First, note that a system need not be represented as a differential equation. There
are many possible parametric and nonparametric representations or model structures for
both linear and nonlinear systems. Parameters for many of these model structures can be
estimated reliably from measured data. In general, the model structure will be represented
by an operator, M, having some general mathematical form capable of representing a
wide variety of systems. The model itself will depend on a list of parameters, the vector θ .
From this viewpoint, the system output may be written as
y(t, θ) = M(θ, u(t)) (1.14)
where it is assumed that the model structure, M, and parameter vector, θ , exactly rep-
resent the physical system. Thus, the physical system, N, can be replaced with an exact
model, M(θ).
The objective of system identification is to find a suitable model structure, M, and
corresponding parameter vector, θ , given measurements of the input and output. Then,
the identified model will have a parameter vector, θ̂ , and generate
ŷ(t) = M(θ̂, u(t)) (1.15)
where ŷ(t) is an estimate of the system output, y(t). Similarly, M(θ̂ , u(t)) represents the
model structure chosen together with a vector of estimated parameters. The system iden-
tification problem is then to choose the model structure, M, and find the corresponding
parameter vector, θ̂ , that produces the model output, given by equation (1.15), that best
predicts the measured system output.
Often, instead of having the system output, y(t), only a noise corrupted measurement
will be available. Usually, this measurement noise is assumed to be additive, random,
and statistically independent of the system’s inputs and outputs. The goal, then, is to find
the model, M(θ̂, u(t)), whose output, ŷ(t, θ̂ ), “best approximates” the measured output,
z(t). The relationship between the system, model, and the various signals, is depicted in
Figure 1.5.
1.4.1 Types of System Identification Problems

Figure 1.6 gives a more complete view of a typical system identification experiment.
First, the desired test input, labeled µ(t), is applied to an actuator. In some applications,
such as the study of biomechanics, the actuator dynamics may restrict the types of test
input which can be applied. In addition, the actuator may be influenced by the noise term,
n1 (t). Thus, instead of using the desired input, µ(t), in the identification, it is desirable
to measure the actuator output, u(t), and use it as the input instead. Note, however, that
measurements of the input, û(t), may contain noise, n2 (t).
Many system identification methods are based, either directly or indirectly, on solving
an ordinary least-squares problem. Such formulations are well suited to dealing with
10 INTRODUCTION
v(t)
y(t)
z(t)
N

u(t)

ŷ(t, θ)
M(θ)
Figure 1.5 The deterministic system identification problem in the presence of measurement noise.

û(t)
n1 (t) n2 (t) w(t)
v(t)

µ(t) u(t) y(t)
z(t)
Actuator N
dynamics

Figure 1.6 A more realistic view of the system being identified, including the actuator, which
transforms the ideal input, µ, into the applied input, u(t), which may contain the effects of the
process noise term, n1 (t). Furthermore, the measured input, û(t), may contain noise, n2 (t). As
before, the plant may be affected by process noise, w(t), and the output may contain additive
noise, v(t).
noise in the output signals. However, to deal with noise at the input, it is necessary to
adopt a “total least-squares” or “errors in the variables” framework, both of which are
much more computationally demanding. To avoid this added complexity, identification
experiments are usually designed to minimize the noise in the input measurements. In
some cases, it may be necessary to adopt a noncausal system description so that the
measurement with the least noise may be treated as the input. Throughout this book it
will be assumed that n2 (t) is negligible, unless otherwise specified.
The system may also include an unmeasurable process noise input, w(t), and the
measured output may also contain additive noise, v(t). Given this framework, there are
three broad categories of system identification problem:
• Deterministic System Identification Problem. Find the relationship between u(t)
and y(t), assuming that the process noise, w(t), is zero. The measured output,
z(t), may contain additive noise, v(t). The identification of deterministic systems
is generally pursued with the objective of gaining insight into the system function
and is the problem of primary interest in this text.
• Stochastic System Identification Problem. Find the relationship between w(t) and
y(t), given only the system output, z(t), and assumptions regarding the statistics
of w(t). Usually, the exogenous input, u(t), is assumed to be zero or constant.
This formulation is used where the inputs are not available to the experimenter, or
HOW COMMON ARE NONLINEAR SYSTEMS? 11
where it is not evident which signals are inputs and which are outputs. The myriad
approaches to this problem have been reviewed by Brillinger (1975) and Caines
(1988).
• Complete System Identification Problem. Given both the input and the output, esti-
mate both the stochastic and deterministic components of the model. This problem
formulation is used when accurate output predictions are required, for example in
model-based control systems (Ljung, 1999; Söderström and Stoica, 1989).
1.4.2 Applications of System Identification

There are two general areas of application for models produced by system identification
that will be referred to as “control” and “analysis.”
In “control” applications, the identified model will be used to design a controller,
or perhaps be embedded in a control system. Here, the chief requirements are that the
model be compact and easy to manipulate, so that it produces output predictions with
little computational overhead. Many control applications use the model “online” to pre-
dict future outputs from the histories of both the input and output. Such predictions
commonly extend only one time-step into the future. At each time-step the model uses
the previous output measurement to correct its estimate of the model’s trajectory. Such
one-step-ahead predictions are often all that is required of a control model. As a result,
low-order, linear parametric models of the complete system (i.e., both stochastic and
deterministic parts) are often adequate. Since the model’s output trajectory is corrected
at each sample, the model need not be very accurate. The effects of missing dynam-
ics, or weak nonlinearities, can usually be removed by modeling them as process noise.
Similarly, more severe nonlinearities can handled using an adaptive, time-varying linear
model. Here, the measured output is used to correct the model by varying its parameters
on-line, to track gradual changes in the linearized model.
In “analysis” applications the model is usually employed for off-line simulation of
the system to gain insight into its functioning. For these applications, the model must
be simulated as a free run—that is, without access to past output measurements. With
no access to prediction errors, and hence no means to reconstruct process noise, the
model must be entirely deterministic. Moreover, without the recursive corrections used
in on-line models, an off-line model must be substantially more accurate than the on-line
models typically used in control applications. Thus, in these applications it is critical for
the nonlinearities to be described exactly. Moreover, since simulations are done off-line,
there is less need to minimize the mathematical/arithmetic complexity of the model and
consequently large, nonlinear models may be employed.
1.5 HOW COMMON ARE NONLINEAR SYSTEMS?
Many physiological systems are highly nonlinear. Consider, for example, a single joint
and its associated musculature. First, the neurons that transmit signals to and from the
muscles fire with an “all or nothing” response. The geometry of the tendon insertions
is such that lever arms change with joint angle. The muscle fibers themselves have
nonlinear force–length and force–velocity properties as well as being only able exert
force in one direction. Nevertheless, this complex system is often represented using a
simple linear model.
12 INTRODUCTION
In many biomedical engineering applications, the objective of an identification experi-

ment is to gain insight into the functioning of the system. Here, the nonlinearities may
play a crucial role in the internal functioning of the system. While it may be possible
to linearize the system about one or more operating points, linearization will discard
important information about the nonlinearities. Thus, while a controller may perform
adequately using a linearized model, the model would provide little insight into the
functional organization of the system. Thus, in biomedical applications, it is both common
and important to identify nonlinear systems explicitly.
For these reasons, nonlinear system analysis techniques have been applied to a wide
variety of biomedical systems. Some of these applications include:
• Sensory Systems. These include primitive sensory organs such as the cockroach
tactile spine (French and Korenberg, 1989, 1991; French and Marmarelis, 1995;
French and Patrick, 1994; French et al., 2001), as well as more evolved sensors
such as the auditory system (Carney and Friedman, 1996; Eggermont, 1993; Shi and
Hecox, 1991) and the retina (Citron et al., 1988; Juusola et al., 1995; Korenberg
and Paarmann, 1989; Naka et al., 1988; Sakuranaga et al., 1985a).
• Reflex Loops. Nonlinear system identification techniques have been used to study
reflex loops in the control of limb (Kearney and Hunter, 1988; Kearney et al.,
1997; Westwick and Kearney, 2001; Zhang and Rymer, 1997) and eye position
(the vestibulo-ocular reflex) (Galiana et al., 1995, 2001).
• Organ Systems. Similarly, nonlinear feedback loops have been investigated in mod-
els of heart rate variability (Chon et al., 1996) and in renal auto-regulation (Chon
et al., 1998; Marmarelis et al., 1993, 1999).
• Tissue Mechanics. Biological tissues themselves can exhibit nonlinearities. Strips of
lung tissue (Maksym et al., 1998; Yuan et al., 1999) and the whole lung (Maksym
and Bates, 1997; Zhang et al., 1999) have been shown to include nonlinearities.
Skeletal muscle (Crago, 1992; Hunt et al., 1998; Munih et al., 2000) also has
strongly nonlinear behavior.
Given the prevalence of nonlinearities in physiological systems, along with the
requirement in many biomedical engineering applications to deal explicitly with those
nonlinearities, the need for nonlinear system identification methods is clear. Sample
applications included in Chapters 6–8 will present results from some of the studies
cited above.
CHAPTER 2
BACKGROUND
This chapter will review a number of important mathematical results and establish the
notation to be used throughout the book. Material is drawn from diverse areas, some of
which are not well known and thus extensive references are provided with each section.
2.1 VECTORS AND MATRICES
Many of the techniques presented in this text use numerical methods derived from linear
algebra. This section presents a brief overview of some important results to be used in
the chapters that follow. For a more thorough treatment, the reader should consult the
canonical reference by Golub and Van Loan (1989).
Vectors will be represented using lowercase, boldface letters. The same letter, in
lightface type, will be used for the elements of the vector, subscripted with its position
in the vector. Thus, an M element vector will be written as follows:
T
θ = θ1 θ2 . . . θM
where the superscript T denotes transposition (i.e., θ is a column vector).

Bold uppercase letters will denote matrices. Depending on the context, individual
elements of a matrix will be referenced by placing the row and column number in
parentheses or by using a lowercase, lightface letter with a double subscript. Thus, both
R(i,j ) and ri,j represent the element in the ith row and j th column of the matrix R.
MATLAB’s convention will be used for referencing rows and columns of a matrix, so
that R(:, j ) will be the j th column of R.
Two matrix decompositions will be used extensively: the QR factorization and the
singular value decomposition (SVD). The QR factorization takes a matrix X (i.e., either
Identification of Nonlinear Physiological Systems, By David T. Westwick and Robert E. Kearney

ISBN 0-471-27456-9 c 2003 Institute of Electrical and Electronics Engineers
13
14 BACKGROUND
square or tall) and constructs
X = QR
where R is upper triangular, so that ri,j = 0 for i > j , and Q is an orthogonal matrix,
QT Q = I. Note that the columns of Q are said to be orthonormal (i.e., orthogonal and
normalized); however, the matrix itself is said to be orthogonal.
The singular value decomposition (SVD) takes a matrix, X, of any size and shape and
replaces it with the product,
X = USVT
where S is a diagonal matrix with non-negative diagonal elements. By convention, the

elements of S, the singular values, are arranged in decreasing order along the diagonal.
Thus, s1,1 ≥ s2,2 ≥ · · · ≥ sn,n . U and V are orthogonal matrices containing the left and
right singular vectors, respectively.
Throughout the book, the approximate computational cost of algorithms will be given
in flops. A flop, or floating point operation, represents either the addition or the mul-
tiplication of two floating point numbers. Note that a flop was originally defined as a
floating point multiplication optionally followed by a floating point addition. Thus, the
first edition of Golub and Van Loan (1989) used this earlier definition, whereas subse-
quent editions used the later definition. Indeed, they Golub and Van Loan (1989) noted
that supercomputer manufacturers were delighted by this change in terminology, since
it gave the appearance of an overnight doubling in performance. In any case, the pre-
cise definition is not important to this text, so long as it is applied consistently; the flop
count will be used only as a first-order measurement of the computation requirements of
algorithms.
2.2 GAUSSIAN RANDOM VARIABLES
Gaussian random variables, and signals derived from them, will play a central role in
much of the development to follow. A Gaussian random variable has the probability
density (Bendat and Piersol, 1986; Papoulis, 1984)

1 (x − µ)2
f (x) = √ exp − (2.1)
2π σ 2 2σ 2
that is completely defined by two parameters: the mean, µ, and variance, σ 2 . By con-
vention, the mean, variance, and other statistical moments, are denoted by symbols sub-
scripted by the signal they describe. Thus,
µx = E[x]

σx2 = E (x − µx )2
Figure 2.1 shows a single realization of a Gaussian signal, the theoretical probability
density function (PDF) of the process that generated the signal, and an estimate of the
PDF derived from the single realization.
GAUSSIAN RANDOM VARIABLES 15
(A)
10
5 m+s
Range
m
0
m−s
−5
0 0.1 0.2 0.3 0.4 0.5
Time (s)
(B) (C)
0.2 0.2
m m
Probability
m−s m+s m−s m+s

0.1 0.1
0 0
−10 −5 0 5 10 −10 −5 0 5 10
Range Range
Figure 2.1 Gaussian random variable, x, with mean µx = 2 and standard deviation σx = 3.
(A) One realization of the random process: x(t). (B) The ideal probability distribution of the
sequence x. (C) An estimate of the PDF obtained from the realization shown in A.
2.2.1 Products of Gaussian Variables

The probability density, f (x) defined in equation (2.1), of a Gaussian random variable is
completely specified by its first two moments; all higher-order moments can be derived
from them. Furthermore, filtering a Gaussian signal with a linear system produces an out-
put that is also Gaussian (Bendat and Piersol, 1986; Papoulis, 1984). Consequently, only
the first two moments are required for linear systems analysis. However, the situation is
more complex for nonlinear systems since the response to a Gaussian input is not Gaus-
sian; rather it is the sum of products of one or more Gaussian signals. Consequently, the
expected value of the product of n zero-mean, Gaussian random variables, E[x1 x2 . . . xn ],
is an important higher-order statistic that will be used frequently in subsequent chapters.
The expected value of the product of an odd number of zero-mean Gaussian signals
will be zero. However, the result is more complex for the product of an even number of
Gaussian variables. The expected value may be obtained as follows (Bendat and Piersol,
1986):
1. Form every distinct pair of random variables and compute the expected value of
each pair. For example, when n is 4, the expected values would be
E[x1 x2 ], E[x1 x3 ], E[x1 x4 ], E[x2 x3 ], E[x2 x4 ], E[x3 x4 ]
2. Form all possible distinct combinations, each involving n/2 of these pairs, such
that each variable is included exactly once in each combination. For n = 4, there
16 BACKGROUND
are three combinations
(x1 x2 )(x3 x4 ), (x1 x3 )(x2 x4 ), (x1 x4 )(x2 x3 )
3. For each combination, compute the product of the expected values of each pair,
determined from step 1. Sum the results of all combinations to get the expected
value of the overall product. Thus, for n = 4, the combinations are
E[x1 x2 x3 x4 ] = E[x1 x2 ]E[x3 x4 ] + E[x1 x3 ]E[x2 x4 ] + E[x1 x4 ]E[x2 x3 ] (2.2)
Similarly, when n is 6:
E[x1 x2 x3 x4 x5 x6 ] = E[x1 x2 ]E[x3 x4 ]E[x5 x6 ] + E[x1 x2 ]E[x3 x5 ]E[x4 x6 ] + · · ·
For the special case where all signals are identically distributed, this yields the relation
E[x1 · x2 · . . . · xn ] = (1 · 3 · 5 · . . . · n − 1)E[xi xj ]n/2

n!
= E[xi xj ]n/2 i = j (2.3)
2n/2 (n/2)!
2.3 CORRELATION FUNCTIONS
Correlation functions describe the sequential structures of signals. In signal analysis, they
can be used to detect repeated patterns within a signal. In systems analysis, they are used
to analyze relationships between signals, often a system’s input and output.
2.3.1 Autocorrelation Functions

The autocorrelation function characterizes the sequential structure of a signal, x(t), by
describing its relation to a copy of itself shifted by τ time units. The correlation will be
maximal at τ = 0 and change as the lag, τ , is increased. The change in correlation as a
function of lag characterizes the sequential structure of the signal.
Three alternative correlation functions are used commonly: the autocorrelation func-
tion, the autocovariance function, and the autocorrelation-coefficient function.
To illustrate the relationships between these functions, consider a random signal, x(t),
and let x0 (t) be a zero-mean, unit variance random variable derived from x(t),
x(t) − µx
x0 (t) = (2.4)
σx
The autocorrelation function of the signal, x(t), is defined by
φxx (τ ) = E[x(t − τ )x(t)] (2.5)
Figure 2.2 illustrates time records of several typical signals together with their autocorre-
lations. Evidently, the autocorrelations reveal structures not apparent in the time records
of the signals.
CORRELATION FUNCTIONS 17
(A) (B)
10
1
0
−10
(C) (D)
4
1
0
−4
(E) (F)
1
1
0 0
−1
−1
(G) (H)
10
1
0
−10
0 5 −1 0 1
Time (s) Lag (s)
Figure 2.2 Time signals (left column) and their autocorrelation coefficient functions (right col-
umn). (A, B) Low-pass filtered white-noise signal. (C, D) Low-pass filtered white noise with a
lower cutoff. The resulting signal is smoother and the autocorrelation peak is wider. (E, F) Sine
wave. The autocorrelation function is also a sinusoid. (G, H) Sine wave buried in white noise. The
sine wave is more visible in the autocorrelation than in the time record.
Substituting equation (2.4) into equation (2.5) gives
φxx (τ ) = E[(σx x0 (t − τ ) + µx )(σx x0 (t) + µx )] (2.6)

= σx2 E[x0 (t − τ )x0 (t)] + µ2x (2.7)
18 BACKGROUND
Thus, the autocorrelation, φxx , at any lag, τ , depends on both the mean, µx , and the
variance, σx2 , of the signal.
In many applications, particularly where systems have been linearized about an oper-
ating point, signals will not have zero means. It is common practice in these cases
to remove the mean before calculating the correlation, resulting in the autocovariance
function:
Cxx (τ ) = E[(x(t − τ ) − µx )(x(t) − µx )]

= φxx (τ ) − µ2x (2.8)
Thus, if µx = 0, the autocorrelation and autocovariance functions will be identical.

At zero lag, the value of the autocovariance function is
Cxx (0) = E[(x(t) − µx )2 ]

= σx2
which is the signal’s variance. Dividing the autocovariance by the variance gives the
autocorrelation coefficient function,
Cxx (τ )
rxx (τ ) =
Cxx (0)
= E[x0 (t − τ )x0 (t)] (2.9)
The values of the autocorrelation coefficient function may be interpreted as correla-

tions in the statistical sense. Thus the autocorrelation coefficient function ranges from 1
(i.e., complete positive correlation) to 0 (i.e., no correlation), through −1 (i.e., complete
negative correlation).
It is not uncommon, though it can be very confusing, for the autocovariance function
and the autocorrelation coefficient function to be referred to as the autocorrelation func-
tion. The relationships between the different autocorrelation functions are illustrated in
Figure 2.3.
Finally, note that all the autocorrelation formulations are even and thus are symmetric
about zero lag:
φxx (−τ ) = φxx (τ )

Cxx (−τ ) = Cxx (τ ) (2.10)
rxx (−τ ) = rxx (τ )
2.3.2 Cross-Correlation Functions

Cross-correlation functions measure the sequential relation between two signals. It is
important to remember that two signals may each have considerable sequential structure
yet have no correlation with each other.
The cross-correlation function between two signals x(t) and y(t) is defined by
φxy (τ ) = E[x(t − τ )y(t)] (2.11)

Signal Correlation Covariance Coefficient

4 1 1 1
−4 0 0 0
4 5 1 1
−4 0 0 0
40 100 100 1
−40 0 0 0
400 10 K 10 K 1
−400 0 0 0
0 1 −10 10 −10 10 −10 10
Time (s) Lag (ms) Lag (ms) Lag (ms)
Figure 2.3 Examples of autocorrelation functions. The first column shows the four time signals,
the second column shows their autocorrelation functions, the third column shows the corresponding
autocovariance functions, and the fourth column the equivalent autocorrelation coefficient functions.
First Row: Low-pass filtered sample of white, Gaussian noise with zero mean and unit variance.
Second Row: The signal from the first row with an offset of 2 added. Note that the mean is clearly
visible in the autocorrelation function but not in the auto-covariance or autocorrelation coefficient
function. Third Row: The signal from the top row multiplied by 10. Bottom Row: The signal from
the top row multiplied by 100. Scaling the signal changes the values of the autocorrelation and
autocovariance function but not of the autocorrelation coefficient function.
As before, removing the means of both signals prior to the computation gives the
cross-covariance function,
Cxy (τ ) = E[(x(t − τ ) − µx )(y(t) − µy )]

= φxy (τ ) − µx µy (2.12)
where µx is the mean of x(t), and µy is the mean of y(t). Notice that if either µx = 0
or µx = 0, the cross-correlation and the cross-covariance functions will be identical.
The cross-correlation coefficient function of two signals, x(t) and y(t), is defined by
Cxy (τ )
rxy (τ ) = (2.13)
Cxx (0)Cyy (0)
The value of the cross-correlation coefficient function at zero lag, rxy (0), will be unity
only if the two signals are identical to within a scale factor (i.e., x(t) = ky(t)). In this
20 BACKGROUND
case, the cross-correlation coefficient function will be the same as the autocorrelation
coefficient function of either signal.
As with the autocorrelation coefficient function, the values of the cross-correlation
coefficient function can be interpreted as correlations in the statistical sense, ranging
from complete positive correlation (1) through 0 to complete negative correlation (−1).
Furthermore, the same potential for confusion exists; the cross-covariance and cross-
correlation coefficient functions are often referred to simply as cross-correlations.
The various cross-correlation formulations are neither even nor odd, but do satisfy the
interesting relations:
φxy (τ ) = φyx (−τ ) (2.14)
and

φxy (τ ) ≤ φxx (0)φyy (0) (2.15)
Finally, consider the cross-correlation between x(t) and y(t) where
y(t) = αx(t − τ0 ) + v(t)
that is, y(t) is a delayed, scaled version of x(t) added to an uncorrelated noise signal,
v(t). Then,
φxy (τ ) = αφxx (τ − τ0 )
That is, the cross-correlation function is simply the autocorrelation of the input signal,
x(t), displaced by the delay τ0 , and multiplied by the gain α. As a result, the lag at which
the cross-correlation function reaches its maximum provides an estimate of the delay.
2.3.3 Effects of Noise

Frequently, the auto- and cross-correlations of the signals x(t) and y(t) must be estimated
from measurements containing additive noise. Let
w(t) = x(t) + n(t)

z(t) = y(t) + v(t)
where n(t) and v(t) are independent of x(t) and y(t) and of each other.
First, consider the autocorrelation of one signal,

φzz (τ ) = E (y(t − τ ) + v(t − τ ))(y(t) + v(t))
= φyy (τ ) + φyv (τ ) + φvy (τ ) + φvv (τ )
The terms φyv ≡ φvy ≡ 0 will disappear because y and v are independent. However, the
remaining term, φvv , is the autocorrelation of the noise sequence and will not be zero.
As a result, additive noise will bias autocorrelation estimates,
φzz (τ ) = φyy (τ ) + φvv (τ )

In contrast, for the cross-correlation function
φwz (τ ) = E[w(t)z(t + τ )]

= E (x(t − τ ) + n(t − τ ))(y(t) + v(t))
= φxy (τ ) + φxv (τ ) + φny (τ ) + φnv (τ )
and φxv ≡ φny ≡ φnv ≡ 0, since the noise signals are independent of each other by
assumption. Thus,
φwz (τ ) = φxy (τ )
and estimates of the cross-correlation with additive noise will be unbiased.
2.3.4 Estimates of Correlation Functions

Provided that x(t) and y(t) are realizations of ergodic processes,∗ the expected value in
equation (2.11) may be replaced with an infinite time-average:
T
1
φxy (τ ) = lim x(t − τ )y(t) dt (2.16)
T →∞ 2T −T
In any practical application, x(t) and y(t) will be finite-length, discrete-time signals.
Thus, it can be assumed that x(t) and y(t) have been sampled every t units from
t = 0, t , . . . , (N − 1)t , giving the samples x(i) and y(i) for i = 1, 2, . . . , N .
By using rectangular integration, the cross-correlation function (2.16) can be approxi-
mated as
N
1
φ̂xy (τ ) = x(i − τ )y(i) (2.17)
N −τ
i=τ
This is an unbiased estimator, but its variance increases with lag τ . To avoid this, it is
common to use the estimator:
N
1
φ̂xy (τ ) = x(i − τ )y(i) (2.18)
N
i=τ
which is biased, because it underestimates correlation function values at long lags, but
its variance does not increase with lag τ .
Similar estimators of the auto- and cross-covariance and correlation coefficient func-
tions may be constructed. Note that if N is large with respect to the maximum lag, the
biased and unbiased estimates will be very similar.
∗ Strictly speaking, a deterministic signal, such as a sinusoid, is nonstationary and is certainly not ergodic.
Nevertheless, computations based on time averages are routinely employed with both stochastic and determin-
istic signals. Ljung (1999) defines a class of quasi-stationary signals, together with an alternate expected value
operator, to get around this technicality.
22 BACKGROUND
2.3.5 Frequency Domain Expressions

The discrete Fourier transform of a discrete-time signal of length N is defined as
N
U (f ) = F(u(t)) = u(t)e−2πjf t/N (2.19)
t=1
Note that in this definition, f takes on integer values f = 0, 1, . . . , N − 1. If t is the
sampling increment in the time domain, then the sampling increment in the frequency
domain will be
1
f =
N t
The inverse Fourier transform is given by∗ ,
N
1
u(t) = F−1 (U (f )) = U (f )e2πjf t/N (2.20)
N
f =1
Direct computation of either equation (2.19) or equation (2.20) requires about 4N 2

flops, since the computations involve complex numbers. However, if the fast Fourier
transform (FFT) algorithm (Bendat and Piersol, 1986; Oppenheim and Schafer, 1989;
Press et al., 1992) is used, the cost can be reduced to about N log2 (N ) flops (Oppenheim
and Schafer, 1989). Note that these figures are for real valued signals. FFTs of complex-
valued signals will require twice as many flops.
The Fourier transform of the autocorrelation function is called the power spectrum.
For a discrete-time signal,
N −1
Suu (f ) = φuu (τ )e−2πjf τ/N
τ =0
N −1
=E u(t)u(t − τ )e−2πjf τ/N (2.21)
τ =0
This expression, like the definition of the autocorrelation, is in terms of an expected
value. To estimate the power spectrum, u(t) is treated as if it were ergodic, and the
expected value is replaced with a time average. There are several ways to do this.
One possibility is to estimate the correlation in the time domain using a time average
(2.18) and then Fourier transform the result:
N N
1
F φ̂uu (τ ) = u(i − τ )u(i)e−j 2π τf/N
N
τ =0 i=0
Multiplying by e−j 2π(i−i)f/N = 1 and then simplifying gives

N N
1
F φ̂uu (τ ) = u(i − τ )e j 2π(i−τ )f/N
u(i)e−j 2π if/N
N
τ =0 i=0
1
= U ∗ (f )U (f ) (2.22)
N
√
∗ Some authors (Ljung, 1999) include a factor of 1/ N in both the forward and inverse Fourier transform.
Taking the inverse Fourier transform of (2.22) yields (2.18), the biased estimate of the
cross-correlation. In practice, correlation functions are often computed this way, using
the FFT to transform to and from the frequency domain.
An alternative approach, the averaged periodogram (Oppenheim and Schafer, 1989),
implements the time average differently. Here, the signal is divided into D segments of
length ND , and the ensemble of segments is averaged to estimate the expected value.
Thus, the averaged periodogram spectral estimate is
D
1
Ŝuu (f ) = Ud∗ (f )Ud (f ) (2.23)
DND
d=1
where Ud (f ) is the Fourier transform of the dth segment of ND points of the signal
u(t), and the asterisk, U ∗ (f ), denotes the complex conjugate.
It is common to overlap the data blocks to increase the number of blocks averaged.
Since the FFT assumes that the data are periodic, it is also common to window the
blocks before transforming them. The proper selection of the window function, and of
the degree of overlap between windows, is a matter of experience as well as trial and
error. Further details can be found in Bendat and Piersol (1986).
The averaged periodogram (2.23) is the most commonly used nonparametric spectral
estimate, and it is implemented in MATLAB’s spectrum command. Parametric spectral
estimators have also been developed and are described in Percival and Walden (1993).
2.3.5.1 The Cross-Spectrum The Fourier transform of the cross-correlation func-

tion is called the cross-spectrum,
N −1
Suy (f ) = φuy (τ )e−2πjf τ/N (2.24)
τ =0
Replacing the autocorrelation with the cross-correlation in the preceding derivations

gives the Fourier transform of the cross-correlation,
1 ∗
F φ̂uy (τ ) = U (f )Y (f ) (2.25)
N
and an averaged periodogram estimate of the cross-spectrum,
D
1
Ŝuy (f ) = Ud∗ (f )Yd (f ) (2.26)
DND
d=1
2.3.6 Applications
The variance of a biased correlation estimate (2.18) is proportional to 1/N and thus
decreases as N increases. Furthermore, the effect of the bias is a scaling by a factor
of N/(N − τ ), which decreases as N increases with respect to τ . Thus, in general the
length of a correlation function should be much shorter than the data length from which
it is estimated; that is, N τ . As a rule of thumb, correlation functions should be no
more than one-fourth the length of the data and should never exceed one-half of the data
length.
Autocovariance functions determined from stochastic signals tend to “die out” at
longer lags. The lag at which the autocovariance function has decreased to values that
24 BACKGROUND
cannot be distinguished from zero provides a subjective measure of the extent of the
sequential structure in a signal, or its “memory.”
In contrast, if there is an underlying periodic component, the autocorrelation function
will not die out but will oscillate at the frequency of the periodic component. If there
is substantial noise in the original signal, the periodicity may be much more evident in
the correlation function than in the original data. This periodicity will be evident in the
autocorrelation function at large lags, after the contribution from the noise component
has decayed to zero. An example of this is presented in the bottom row of Figure 2.2.
A common use for the cross-covariance function, as illustrated in the middle panel
of Figure 2.4, is to estimate the delay between two signals. The delay is the lag at which
(A) (B) (C) 1

3 3
0.5
0 0
−3 −3
−0.5
(D) (E) (F) 1

3 3
0.5
0 0
−3 −3
−0.5
(G) (H) (I) 1

3 3
0.5
0 0
−3 −3
−0.5
0 5 0 5 −0.2 0 0.2
Time (s) Time (s) Lag (s)
Figure 2.4 Examples of the cross-correlation coefficient function. Top Row: The signals in A and
B are uncorrelated with each other. Their cross-correlation coefficient function, C, is near zero at
all lags. Middle Row: E is a delayed, scaled version of D with additive noise. The peak in the
cross-correlation coefficient function, F, indicates the delay between input and output. Bottom Row:
G is a low-pass filtered version of H, also with additive noise. The filtering appears as ringing in I.
MEAN-SQUARE PARAMETER ESTIMATION 25
the cross-covariance function is maximal. For example, the delay between two signals
measured at two points along a nerve can be determined from the cross-covariance
function (Heetderks and Williams, 1975). It should be remembered that dynamics can
also give rise to delayed peaks in the cross-correlation function.
2.3.7 Higher-Order Correlation Functions

Second-order cross-correlation functions will be represented by φ with three subscripts.
Thus,
φxxy (τ1 , τ2 ) = E[x(t − τ1 )x(t − τ2 )y(t)] (2.27)
while the second-order autocorrelation function is
φxxx (τ1 , τ2 ) = E[x(t − τ1 )x(t − τ2 )x(t)] (2.28)
Note that there is some confusion in the literature about the terminology for this function.
Some authors (Korenberg, 1988) use the nomenclature “second-order,” as does this book;
others have used the term “third-order” (Marmarelis and Marmarelis, 1978) to describe
the same relation.
2.4 MEAN-SQUARE PARAMETER ESTIMATION
Given a model structure and a set of input–output measurements, it is often desirable

to identify the parameter vector that generates the output that “best approximates” the
measured output. Consider a model, M, with a parameter set, θ, whose response to the
input u(t) will be written
ŷ(θ , t) = M(θ, u(t))
A common definition for the “best approximation” is that which minimizes the mean-
square error between the measured output, z(t), and the model output, ŷ(θ, t):
2
MSE(M, θ, u(t)) = E z(t) − ŷ(θ, t) (2.29)
If the signals are ergodic, then this expectation can be evaluated using a time average
over a single record. In discrete time, this results in the summation:
N
1 2
VN (M, θ, u(t)) = z(t) − ŷ(θ , t) (2.30)
N
t=1
which is often referred to as the “mean-square error,” even though it is computed using
a time average rather than an ensemble average.
Note that the MSE depends on the model structure, M, the parameter vector, θ , and
the test input, u(t). For a particular structure, the goal is to find the parameter vector, θ̂ ,
that minimizes (2.30). That is (Ljung, 1999):
θ̂ = arg min VN (θ, u(t)) (2.31)

θ
26 BACKGROUND
In general, there is no closed-form solution to this minimization problem, so the “param-

eter space” must be searched using iterative methods as discussed in Chapter 8.
2.4.1 Linear Least-Squares Regression

If the model output, ŷ(t), is a linear function of the parameters, the model may be
formulated as a matrix equation (Beck and Arnold, 1977),
ŷ(θ) = Uθ (2.32)
where ŷ is an N element vector containing ŷ(t), and U is a matrix with N rows

(one per data point) and M columns (one per model parameter). Equation (2.30) then
becomes
1
VN (θ ) = (z − Uθ)T (z − Uθ )
N
1 T
= z z − 2θ T UT z + θ T UT Uθ
N
which may be solved analytically as follows. First, differentiate with respect to θ to get
the gradient
∂VN 2
= (UT Uθ − UT z)
∂θ N
The minimum mean-square error solution, θ̂, is found by setting the gradient to zero and
solving:
UT Uθ̂ = UT z
(2.33)
θ̂ = (UT U)−1 UT z
Thus, for any model structure where the output is linear in the parameters, as
in equation (2.32), the optimal parameter vector may be determined directly by from
equations (2.33), called the normal equations.∗ Many of the model structures examined
in this text are linear in their parameters even though they describe nonlinear systems.
Thus, solutions of the normal equations and their properties are fundamental to much of
what will follow.
2.4.1.1 Example: Polynomial Fitting Consider, for example, the following prob-
lem. Given N measurements of an input signal, u1 , u2 , . . . , uN and output z1 , z2 , . . . , zN ,
find the third-order polynomial that best describes the relation between uj and zj . To do
so, assume that
yj = c(0) + c(1) uj + c(2) u2j + c(3) u3j

zj = yj + vj
∗ In the MATLAB environment, the normal equation, (2.33), can be solved using the “left division” operator.
θ̂ = U\z.
MEAN-SQUARE PARAMETER ESTIMATION 27
where v is a zero-mean, white Gaussian noise sequence. First, construct the regression
matrix
 
1 u1 u21 u31
 
1 u2 u22 u32 
 
U = . . . .  (2.34)
 .. .. .. .. 
 
1 uN u2N u3N
Then, rewrite the expression for z as the matrix equation
z = Uθ + v (2.35)
T
where θ = c(0) c(1) c(2) c(3) , and use equations (2.33) to solve for θ.
2.4.2 Properties of Estimates

The solution of equations (2.33) gives the model parameters that provide the minimum
mean-square error solution. It is important to know how close to the true parameters
these estimated values are likely to be. Consider the following conditions:
1. The model structure is correct; that is, there is a parameter vector such that the
system can be represented exactly as y = Uθ .
2. The output, z = y + v, contains additive noise, v(t), that is zero-mean and statisti-
cally independent of the input, u(t).
If conditions 1 and 2 hold, then the expected value of the estimated parameter vector is
E[θ̂ ] = E[(UT U)−1 UT z]

= θ + E[(UT U)−1 UT v] (2.36)
However, since v is independent of u, it will be independent of all the columns of U
and so
E[θ̂ ] = θ (2.37)
That is, the least-squares parameter estimate is unbiased (Beck and Arnold, 1977).
The covariance matrix for these parameter estimates is
Cθ̂ = E[(θ̂ − E[θ̂ ])(θ̂ − E[θ̂ ])T ] (2.38)

= (UT U)−1 UT E[vvT ]U(UT U)−1 (2.39)
Usually, equation (2.39) cannot be evaluated directly since E[vvT ], the covariance
matrix of the measurement noise, is not known. However, if the measurement noise is
white, then the noise covariance in equation (2.39) reduces to E[vvT ] = σv2 IN , where
σv2 is the variance of v(t), and IN is an N × N identity matrix. Furthermore, an unbiased
estimate of the noise variance (Beck and Arnold, 1977) is given by
N
1
σ̂v2 = (z(t) − ŷ(t))2 (2.40)
N −M
t=1
28 BACKGROUND
where M is the number of model parameters. Thus the covariance matrix for the param-
eter estimates, Cθ̂ , reduces to
Cθ̂ = σ̂v2 (UT U)−1 (2.41)
which can be evaluated directly.
2.4.2.1 Condition of the Hessian Instead of using probabilistic considerations,

as in the previous section, it is also instructive to examine the numerical properties of
the estimate. Thus, instead of taking an expected value over a hypothetical ensemble of
records, consider the sensitivity of the computed estimate, θ̂ , to changes in the single,
finite-length, input–output record, [u(t) z(t)].
From the normal equations (2.33), it is evident that the error in the parameter esti-
mate is
θ̃ = θ − θ̂
= θ − (UT U)−1 UT z
= −(UT U)−1 UT v (2.42)
where v(t) is the noise in the measured output, z = Uθ + v.

The error is the product of two terms. The first term, (UT U)−1 , is the inverse of the
Hessian, denoted H, an M × M matrix containing the second-order derivatives of the
cost function with respect to the parameters.
∂ 2 VN (θ)
H(i, j ) = = UT U (2.43)
∂θi ∂θj
Furthermore, if the measurement noise is white, the inverse of the Hessian is proportional
to the parameter covariance matrix, Cθ̂ , given in equation (2.41).
Let ν = UT v be the second term in equation (2.42). Substituting these two expres-
sions gives
θ̃ = −H−1 ν
Now, consider the effect of a small change in ν on the parameter estimate. The Hessian
is a non-negative definite matrix, so its singular value decomposition (SVD) (Golub and
Van Loan, 1989) can be written as
H = VSVT
where V = [v1 v2 . . . vM ] is an orthogonal matrix, VT V = I, and S is a diagonal matrix,
S = diag[s1 , s2 . . . , sM ], where s1 ≥ s2 ≥ · · · ≥ sM ≥ 0. Using this, the Hessian can be
expanded as
M
H= si vi vi T
i=1
POLYNOMIALS 29
and the estimation error, θ̃ , becomes

M
1
θ̃ = − vi vi T ν (2.44)
si
i=1
Notice that if the noise term, ν, changes in the direction parallel to the kth singular
vector, vk , then the change in θ̂ will be multiplied by 1/sk . Consequently, the ratio
of the largest to smallest singular values will determine the relative sensitivity of the
parameter estimates to noise. This ratio is referred to as the condition number (Golub
and Van Loan, 1989) of the matrix and ideally should be close to 1.
2.5 POLYNOMIALS
Polynomials provide a convenient means to model the instantaneous, or memoryless,

relationship between two signals. Section 2.4.1.1 demonstrated that fitting a polynomial
between two data sequences can be done by solving a linear least-squares problem.
2.5.1 Power Series

The power series is a simple polynomial representation involving a sum of monomials
in the input signal,
Q
m(u) = c(q) uq
q=0
Q
= c(q) M(q) (u)
q=0
= c(0) + c(1) u + c(2) u2 + c(3) u3 + · · ·

where the notation M(q) (u) = uq is introduced to prepare for the discussion of orthogonal
polynomials to follow. Uppercase script letters will be used throughout to represent
polynomial systems.
The polynomial coefficients, c(q) , can be estimated by solving the linear least-squares
problem defined by equations (2.34) and (2.35). This approach may give reasonable
results provided that there is little noise and the polynomial is of low order. However, this
problem often becomes badly conditioned, resulting in unreliable coefficient estimates.
This is because in power-series formulations the Hessian will often have a large condition
number; that is the ratio of the largest and smallest singular values will be large. As a
result, the estimation problem is ill-conditioned, since as equation (2.44) shows, small
singular values in the Hessian will amplify errors in the coefficient estimates. The large
condition number arises for two reasons:
1. The columns of U will have widely different amplitudes, particularly for high-order
polynomials, unless σu ≈ 1. As a result, the singular values of U, which are the
square roots of the singular values of the Hessian, will differ widely.
30 BACKGROUND
2. The columns of U will not be orthogonal. This is most easily seen by examining
the Hessian, UT U, which will have the form
 
1 E[u] . . . E[uq ]
E[u2 ]
 E[u] E[u2 ] E[u3 ] . . . E[uq+1 ]
 
 2 3 . . . E[uq+2 ]
4
H = N E[u ] E[u ] E[u ] 
 . . . .. .. 
 .. .. .. . . 
q q+1 q+2
E[u ] E[u ] E[u ] . . . E[u ] 2q
Since H is not diagonal, the columns of U will not be orthogonal to each other.
Note that the singular values of U can be viewed as the lengths of the semiaxes of
a hyperellipsoid defined by the columns of U. Thus, nonorthogonal columns will
stretch this ellipse in directions more nearly parallel to multiple columns and will
shrink it in other directions, increasing the ratio of the axis lengths, and hence the
condition number of the estimation problem (Golub and Van Loan, 1989).
2.5.2 Orthogonal Polynomials

Ideally, the regressors should be mutually orthogonal, so the Hessian will be diagonal
with elements of similar size. This can be achieved by replacing the power series basis
functions with another polynomial function P (q) (u)
Q
m(u) = c(q) P (q) (u) (2.45)
q=0
chosen to make the estimation problem well-conditioned. The objective is to find a

polynomial function that makes the Hessian diagonal. That is, the (i, j )th element of the
Hessian should be
H(i, j ) = N · E[P (i+1) (u)P (j +1) (u)]

= δi,j
which demonstrates that the terms of P (q) (u) must be orthonormal.

The expected value of a function, g, of a random variable, u, with probability density
f (u), is given by (Bendat and Piersol, 1986; Papoulis, 1984)
∞
E[g(u)] = f (u)g(u) du
−∞
Consequently, the expected values of the elements of the Hessian will be

∞
E[P (i) (u)P (j ) (u)] = f (u)P (i) (u)P (j ) (u) du
−∞
This demonstrates that a particular polynomial basis function, P (q) (u), will be orthogonal
only for a particular input probability distribution. Thus each polynomial family will
be orthogonal for a particular input distribution. Figure 2.5 shows the basis functions
corresponding to three families of polynomials: the ordinary power series, as well as the
Hermite and Tchebyshev families of orthogonal polynomials to be discussed next.
POLYNOMIALS 31
Power Hermite Tcheb

2 2 1
Order 0
1 1 0
0 0 −1
10 5 1
Order 1
0 0 0
10 −5 −1
100 10 1
Order 2
50 0 0
0 −10 −1
1k 20 1
Order 3
0 0 0
−1 k −20 −1
10 k 50 1
5k 0 0 Order 4
0 −50 −1
100 k 50 1
Order 5
0 0 0
−100 k −50 −1
−10 0 10 −3 0 3 −1 0 1
Figure 2.5 Power series, Hermite and Tchebyshev polynomials of orders 0 through 5. Left Col-
umn: Power series polynomials over the arbitrarily chosen domain [−10 10]. Middle Column:
Hermite polynomials over the domain [−3 3] corresponding to most of the range of the unit-
variance, normal random variable. Right Column: Tchebyshev polynomials over their full domain
[−1 1].
2.5.3 Hermite Polynomials

The Hermite polynomials H(q) (u) result when the orthogonalization is done for inputs
with a zero-mean, unit-variance, Gaussian distribution. Thus, Hermite polynomials are
constructed such that
∞
exp(−u2 )H(i) (u)H(j ) (u) du = 0 for i = j (2.46)
−∞
32 BACKGROUND
Using equation (2.46) and the results for the expected value of products of Gaussian
variables (2.3), the Hermite polynomials can be shown to be
n/2
(−1)m
H(n) (u) = n! u(n−2m) (2.47)
m!2m (n − 2m)!
m=0
The first four elements are
H(0) (u) = 1
H(1) (u) = u
H(2) (u) = u2 − 1
H(3) (u) = u3 − 3u
The Hermite polynomials may also be generated using the recurrence relation,
H(k+1) (u) = uH(k) (u) − kH(k−1) (u) (2.48)
Note that these polynomials are only orthogonal for zero-mean, unit variance, Gaus-
sian inputs. Consequently, input data are usually transformed to zero mean and unit
variance before fitting Hermite polynomials. The transformation is retained as part of the
polynomial representation and used to transform any other inputs that may be applied to
the polynomial.
2.5.4 Tchebyshev Polynomials

A Gaussian random variable can take on any value between −∞ and +∞ (although the
probability of attaining the extreme values is negligible). Real data, on the other hand,
always have a finite range since they are limited by the dynamic range of the recording
apparatus, if nothing else. Thus, it is logical to develop a set of polynomials that are
orthogonal over a finite range.
The Tchebyshev polynomials, T , are orthogonalized over the range [−1 1] for the
probability distribution (1 − u2 )−1/2 .
1 (i) π
T (u)T (j ) (u) δ for i = 0, j = 0
√ du = 2 i,j (2.49)
−1 1 − u2 π for i = j = 0
This probability density tends to infinity at ±1, and thus does not correspond to the PDF
of any realistic data set. Thus, the Tchebyshev polynomials will not be exactly orthogonal
for any data. However, all Tchebyshev polynomial basis functions are bounded between
−1 and +1 for inputs between −1 and +1 (see Figure 2.5). In addition, each goes through
all of its local extrema (which are all ±1) in this range. Thus, although the regressors
will not be exactly orthogonal, they will have similar variances, and so the estimation
problem should remain well-scaled. Hence, Tchebyshev polynomials are used frequently
since they usually lead to well-conditioned regressions.
The general expression for the order-q Tchebyshev polynomial is
q/2
(q) q (−1)m q − m
T (u) = (2u)q−2m (2.50)
2 q−m m
m=0
POLYNOMIALS 33
The first four Tchebyshev polynomials are
T (0) (u) = 1
T (1) (u) = u
T (2) (u) = 2u2 − 1
T (3) (u) = 4u3 − 3u
The Tchebyshev polynomials are also given by the recursive relationship
T (q+1) (u) = 2uT (q) (u) − T (q−1) (u) (2.51)
In practice, input data are transformed to [−1 1] prior to fitting the coefficients, and
the scale factor is retained as part of the polynomial representation.
2.5.5 Multiple-Variable Polynomials

Polynomials in two or more variables will also be needed to describe nonlinear systems.
To establish the notation for this section, consider the types of terms involved in a
multivariable power series. The order-zero term will be a constant, as in the single
variable case. Similarly, the first-order terms will include only a single input, raised to
the first power, and thus will have the same form as their single-input counterparts,
M(1) (uk ) = uk
However, the second-order terms will involve products of two inputs, either two copies
of the same signal or two distinct signals. Thus, the second-order terms will be of two
types,
M(2) (uk , uk ) = u2k

M(2) (uj , uk ) = uj uk
For example, the second-order terms in a three-input, second-order polynomial will

involve three single-input terms, u21 , u22 , and u23 , and three two-input terms, u1 u2 , u1 u3 ,
and u2 u3 . To remain consistent with the notation for single-variable polynomials, let
M(2) (uk ) = M(2) (uk , uk ) = u2k
Similarly, the order-q terms will involve from one to q inputs, raised to powers such
that the sum of their exponents is q. For example, the third-order terms in a three-input
polynomial are
u31 u32 u33 u21 u2 u21 u3

u22 u1 u22 u3 u23 u1 u23 u2 u1 u2 u3
2.5.5.1 Orthogonal Multiple-Input Polynomials Next, consider the construction

of orthogonal, multiple-input polynomials. Let u1 , u2 , . . . , un be mutually orthonormal,
zero-mean Gaussian random variables. These may be measured signals that just happen
to be orthonormal, but will more likely result from orthogonalizing and normalizing a
set of more general (Gaussian) signals with a QR factorization or SVD.
Random documents with unrelated
content Scribd suggests to you:
All through the winter, against the dark background of my new
knowledge of evil, I had seen him, strong in body and alert in mind,
with a heart like the heart of a little child. Often, in thinking of him, I
had said: “God now and then sends a man into the world who stands
as a promise to the race.”
I thought of Janet, and I cried out to know the meaning of the
world’s great waste of human pain.
The Altruist explained it all to me the next day.
He came to ask me to visit Janet. I had not dared to go. He was
surprised and grieved by my mood.
“The meaning of this sorrow is very clear,” he said gently, with the
old ecstatic gleam in his eyes.
“You explained everything very differently a few weeks ago,” I said
rebelliously, when he had finished. “You told me then, and I believed
you, that God was leading that girl out of her mental tangles into
simple human happiness.”
“Did I?” said the Altruist, dreamily. “It all looks different to me
now.”
“I can see that it does,” I retorted in anger.
“The shock will carry Janet out of her old, cheap pessimism into
conviction and into action of some kind. She will merge her
individual experience in the general life. She will lose herself in great
ideas. Now, at the crisis of so many great questions, she will find her
work. I can see a career for her infinitely more lofty than she could
have had if this sad event had not occurred.”
Here the Doctor entered, interrupting the words of prophecy.
CHAPTER XLI
I was sorry that the Doctor had arrived in time to catch the
Altruist’s last remarks. She waited until he was gone, then sank
wearily into a chair.
“How the angels in heaven must smile at that man’s assurance,”
she exclaimed. “I wish, I wish he could tell the difference between his
voice and the voice of God!”
I was in no mood to defend the Altruist, and so said nothing.
“If the Altruist knows what all this trouble means, he knows a great
deal more than I do,” she went on grimly. “I cannot see, I cannot see
how the Lad could so forget all the people who cared for him.”
The sentence ended in a half sob that almost frightened me. It had
never occurred to me that the Doctor could shed tears.
“Have you seen Janet?” I asked, attempting to change the subject.
I succeeded only in turning the Doctor’s wrath back upon the
Altruist.
“Yes,” she said, “I have seen Janet, and I wish the Altruist were in
Timbuctoo! He has been at the house and has utterly unnerved her.”
“How?” I asked.
“It is hard to believe, even of the Altruist. How do you suppose he
greeted that hurt child? ‘Janet,’ he said, ‘I have always had an
intuition that you were not meant for mere happiness.’”
I groaned. “He doesn’t mean to be cruel,” I said, “but he has not
the simple instinct—”
“A few of the simpler human instincts are really necessary,”
interrupted the Doctor, “in any attempt to help human beings. If the
Altruist had more feeling and less transcendentalism, it would be
better.”
“It isn’t a week,” I responded, “since he had an intuition of a
directly opposite kind. And then I was trying to help him,” I
confessed, for a sudden sense of guilt overcame me as I met the
Doctor’s clear eyes, “in his attempt to explain to God what He
means.”
The fierce expression in her face was changing into a look of
tenderness.
“Go to see the child,” she said huskily, “to-morrow, not to-day. She
will be quieter then.”
But I waited two long days. The hours were tedious and dull and
heavy, full of cloud and rain. No birds were singing in the sunless air,
and the grass had forgotten to grow. It seemed to me that in the
ending of a life dear to me, all life had paused.
CHAPTER XLII
“For the agony of the world’s struggle is the very life of God. Were He mere
spectator, perhaps He too would call life cruel. But in the unity of our lives with
His, our joy is His joy; our pain is His.”
I do not know what incoherent words I was saying. Janet stopped

me.
“No, don’t,” she said. “I do not feel like that. You need not be sorry
for me.”
Her voice was very quiet, and her face was firm with the exalted,
unnatural self-control of extreme grief.
“Do you know?” she said, “the sorrow almost rests me. I have had
so much of the bitter and meaningless pain. Perhaps my quarrel with
life is over.”
“But this is so inexplicable,” I cried, taking the girl’s hands in mine
and forgetting that I was there to comfort her.
“It doesn’t need to be explained, because it hurts, and the hurt is
life, and life is good. Oh, I tell you,” she added proudly, drawing her
hands away and going over to seat herself by the window; “it is only
when you are standing outside, looking at life, talking about it and
thinking about it, that you can say it is cruel. When you are really
living, the very hurt is glorious.”
I sat and watched the tearless face. The girl had been carried
beyond me, out into the deeps of life where my words of help could
not reach her.
“I have always been trying to reason out the meaning of things,”
she said, turning quickly toward me, “and nobody even told me that
it is only what cannot be said that makes life worth while.”
“People have tried to, Janet,” I said softly, “but that is one of the
things that cannot be told.”
“There isn’t any kind of pain,” she said slowly, “that can equal the
joy of simple human love.”
I forgot my rebellion of the night before. I bowed my head in the
presence of this power for whose better apprehending we covet the
very agony and pain of life. We follow swiftly to let even its shadow
fall upon us, for if ‘in its face is light, in its shadow there is healing
too.’
The sunshine falling through the window turned Janet’s hair into a
halo of waving strands.
“Child,” I whispered, “it is true. It is good just to live. But
remember also that the old faith may be true. God may be, and may
be love.”
“I don’t know,” said the girl, looking up. “I haven’t any opinions.”
Then a mist came over her eyes, for even her new comfort was
swept away by the waves of her sorrow; and she bowed her head
upon her hands with the cry that has ever been the one irrefutable
witness to His presence: “O my God!”
CHAPTER XLIII
We are all busy still, and yet the world is not saved.
The Anarchist is perfecting the process that shall bring his
millennium to be, and the young Socialists in Barnet House are
working out the details of their new economic order. The Altruist still
translates the infinite into finite terms; the Young Reformer is on the
platform; I toil daily in the self-same Cause, but the world is not
saved.
Many times since we closed ranks and marched onward over the
Lad’s grave I have paused, disheartened. Full assurance has not been
granted me, and it is my lot in doing battle to strike often in the dark.
Yet I have moments when I know that the strife is not in vain. In
these I wonder why we are so troubled about our duty to our fellow-
man, and about our knowledge of God. The one command in regard
to our neighbour is not obscure. And our foreboding lest our faith in
God shall escape us seems futile, inasmuch as we cannot escape from
our faith.
THE END
THE IRIS SERIES
OF ILLUSTRATED COPYRIGHT NOVELS.
TRYPHENA IN LOVE.
BY
WALTER RAYMOND,
Author of “Love and Quiet Life,” Etc.
ILLUSTRATED BY
J. WALTER WEST.
16mo. Artistic Cloth Binding. 75 cents.
“Fresh and quaint and wholesome as the scent of the homely

flowers.”—London Daily News.
“Full of freshness and life, of vivid touches of local color and
picturesque details, while written with tenderness, sympathy, and
artistic discernment.”
“Nothing more daintily charming in style, more tenderly pathetic
in matter, or more exquisitely balanced as a story, has come to our
table for a long time than ‘Tryphena in Love.’ It is a simple tale of
humble life in the Somerset district of England.”—Boston Traveler.
“A sweet little English tale, idyllic in subject and manner. It is
simply the story of an invalid boy who lies in the old manor house, in
‘the room where the king hid,’ and lives in a world of romance, and of
the buxom little cousin who loves and serves him. The picture is
delicately painted, yet firmly and clearly, and with a poetic
atmosphere that is very charming.”—Philadelphia Times.
“No gentler or sweeter tale has appeared in years than Walter
Raymond’s ‘Tryphena in Love.’”
“A delicious, dreamy love story, told for the love of telling.”—
Chicago Times-Herald.
FURTHER ISSUES IN THE IRIS SERIES
WILL BE
A LOST ENDEAVOUR.
BY
GUY BOOTHBY,
Author of “A Bid for Fortune.”
ILLUSTRATED BY
STANLEY L. WOOD.
16mo. Linen. 75 cents.
MAUREEN’S FAIRING.
BY
JANE BARLOW,
Author of “Irish Idylls,” “The End of Elfintown,” Etc.
Illustrated. 16mo. Linen. 75 cents.

THE WINGS OF ICARUS.
Being the Life of one Emilia Fletcher, as revealed by herself
in
I. Thirty-live Letters written to Constance Norris between July 16, 188–, and
March 26 of the following year.
II. A Fragmentary Journal.
III. A Postscript.
BY
LAURENCE ALMA TADEMA.

18mo. Buckram, gilt top. $1.25.
“It is a study of the inner workings of the human heart, and if the
motives of a soul were ever laid bare, it has been done in ‘The Wings
of Icarus.’... A good story, told in an intensely natural and interesting
manner.”—Providence News.
“In ‘The Wings of Icarus,’ Laurence Alma Tadema has given us a
book which, for its literary excellence and for its exquisite pen
coloring and finish in every detail, is as artistic a piece of work as
ever her distinguished father has produced with his brush.”—Boston
Home Journal.
“It is at once delicate and forcible, and holds in its story a depth of
passion whose expression is yet kept within the limits of a true
refinement.”—Philadelphia Evening Bulletin.
“It is exquisite in style, spontaneous and well-sustained in
movement.”
“It is a story of Italian coloring delicately suggestive, artistic rather
than strong, dreamy rather than aggressive.”—Chicago Evening
Post.
UNIFORM WITH “THE WINGS OF ICARUS.”
MAD SIR UCHTRED OF THE HILLS.

BY
S. R. CROCKETT,
Author of “The Raiders,” “The Stickit Minister,” Etc., Etc.
16mo. Buckram. $1.25.

“Mr. Crockett is surely the poet-laureate of Galloway. The scene of
his latest tale (‘Mad Sir Uchtred’) is laid among the hills with which
we became familiar in ‘The Raiders.’ It is a brief tale, not a novel, and
it can be read through in an hour; indeed, if one begins it, one must
read it through, so compelling is the charm of it. The Lady of
Garthland makes a gracious and pathetic figure, and the wild and
terrible Uchtred, the wrong done him, the vengeance which he did
not take,—all these things are narrated in a style of exquisite
clearness and beauty. Mr. Crockett need not fear comparison with
any of the young Scotsmen who are giving to English literature, just
now, so much that is fresh, and wholesome, and powerful.”—Boston
Courier.
THE SILVER CHRIST,

AND
A LEMON TREE.
BY OUIDA,
Author of “Under Two Flags,” “Two Little Wooden Shoes,”

Etc., Etc.
16mo. Buckram. $1.25.

“Two charming stories by ‘Ouida’ are included in a dainty little
volume (‘The Silver Christ’; ‘A Lemon Tree’). Comparatively few
persons—so at least it seems to us—appreciate this writer at her true
value. We have not the highest opinion of much of her work; it is
meretricious and even vulgar. But at her best she is capable of truly
exquisite writing, and it is in shorter tales, dealing with an episode—
brief studies of character—that she is at her best.”—Boston Courier.
“Is pathetic, simple, and beautifully told, and those who have
classed Ouida among the forbidden fruits of literature, should read it
to understand what an artist with the pen she is.”—Boston Times.
MACMILLAN & CO.,
66 FIFTH AVENUE, NEW YORK.
TRANSCRIBER’S NOTES
1. Silently corrected obvious typographical errors and
variations in spelling.
2. Retained archaic, non-standard, and uncertain spellings
as printed.
*** END OF THE PROJECT GUTENBERG EBOOK AN EXPERIMENT IN
ALTRUISM ***
Updated editions will replace the previous one—the old editions

will be renamed.
Creating the works from print editions not protected by U.S.

copyright law means that no one owns a United States
copyright in these works, so the Foundation (and you!) can copy
and distribute it in the United States without permission and
without paying copyright royalties. Special rules, set forth in the
General Terms of Use part of this license, apply to copying and
distributing Project Gutenberg™ electronic works to protect the
PROJECT GUTENBERG™ concept and trademark. Project
Gutenberg is a registered trademark, and may not be used if
you charge for an eBook, except by following the terms of the
trademark license, including paying royalties for use of the
Project Gutenberg trademark. If you do not charge anything for
copies of this eBook, complying with the trademark license is
very easy. You may use this eBook for nearly any purpose such
as creation of derivative works, reports, performances and
research. Project Gutenberg eBooks may be modified and
printed and given away—you may do practically ANYTHING in
the United States with eBooks not protected by U.S. copyright
law. Redistribution is subject to the trademark license, especially
commercial redistribution.
START: FULL LICENSE

THE FULL PROJECT GUTENBERG LICENSE
PLEASE READ THIS BEFORE YOU DISTRIBUTE OR USE THIS WORK
To protect the Project Gutenberg™ mission of promoting the

free distribution of electronic works, by using or distributing this
work (or any other work associated in any way with the phrase
“Project Gutenberg”), you agree to comply with all the terms of
the Full Project Gutenberg™ License available with this file or
online at www.gutenberg.org/license.
Section 1. General Terms of Use and

Redistributing Project Gutenberg™
electronic works
1.A. By reading or using any part of this Project Gutenberg™
electronic work, you indicate that you have read, understand,
agree to and accept all the terms of this license and intellectual
property (trademark/copyright) agreement. If you do not agree
to abide by all the terms of this agreement, you must cease
using and return or destroy all copies of Project Gutenberg™
electronic works in your possession. If you paid a fee for
obtaining a copy of or access to a Project Gutenberg™
electronic work and you do not agree to be bound by the terms
of this agreement, you may obtain a refund from the person or
entity to whom you paid the fee as set forth in paragraph 1.E.8.
1.B. “Project Gutenberg” is a registered trademark. It may only

be used on or associated in any way with an electronic work by
people who agree to be bound by the terms of this agreement.
There are a few things that you can do with most Project
Gutenberg™ electronic works even without complying with the
full terms of this agreement. See paragraph 1.C below. There
are a lot of things you can do with Project Gutenberg™
electronic works if you follow the terms of this agreement and
help preserve free future access to Project Gutenberg™
electronic works. See paragraph 1.E below.
1.C. The Project Gutenberg Literary Archive Foundation (“the
Foundation” or PGLAF), owns a compilation copyright in the
collection of Project Gutenberg™ electronic works. Nearly all the
individual works in the collection are in the public domain in the
United States. If an individual work is unprotected by copyright
law in the United States and you are located in the United
States, we do not claim a right to prevent you from copying,
distributing, performing, displaying or creating derivative works
based on the work as long as all references to Project
Gutenberg are removed. Of course, we hope that you will
support the Project Gutenberg™ mission of promoting free
access to electronic works by freely sharing Project Gutenberg™
works in compliance with the terms of this agreement for
keeping the Project Gutenberg™ name associated with the
work. You can easily comply with the terms of this agreement
by keeping this work in the same format with its attached full
Project Gutenberg™ License when you share it without charge
with others.
1.D. The copyright laws of the place where you are located also
govern what you can do with this work. Copyright laws in most
countries are in a constant state of change. If you are outside
the United States, check the laws of your country in addition to
the terms of this agreement before downloading, copying,
displaying, performing, distributing or creating derivative works
based on this work or any other Project Gutenberg™ work. The
Foundation makes no representations concerning the copyright
status of any work in any country other than the United States.
1.E. Unless you have removed all references to Project

Gutenberg:
1.E.1. The following sentence, with active links to, or other

immediate access to, the full Project Gutenberg™ License must
appear prominently whenever any copy of a Project
Gutenberg™ work (any work on which the phrase “Project
Gutenberg” appears, or with which the phrase “Project
Gutenberg” is associated) is accessed, displayed, performed,
viewed, copied or distributed:
This eBook is for the use of anyone anywhere in the United

States and most other parts of the world at no cost and
with almost no restrictions whatsoever. You may copy it,
give it away or re-use it under the terms of the Project
Gutenberg License included with this eBook or online at
www.gutenberg.org. If you are not located in the United
States, you will have to check the laws of the country
where you are located before using this eBook.
1.E.2. If an individual Project Gutenberg™ electronic work is

derived from texts not protected by U.S. copyright law (does not
contain a notice indicating that it is posted with permission of
the copyright holder), the work can be copied and distributed to
anyone in the United States without paying any fees or charges.
If you are redistributing or providing access to a work with the
phrase “Project Gutenberg” associated with or appearing on the
work, you must comply either with the requirements of
paragraphs 1.E.1 through 1.E.7 or obtain permission for the use
of the work and the Project Gutenberg™ trademark as set forth
in paragraphs 1.E.8 or 1.E.9.
1.E.3. If an individual Project Gutenberg™ electronic work is

posted with the permission of the copyright holder, your use and
distribution must comply with both paragraphs 1.E.1 through
1.E.7 and any additional terms imposed by the copyright holder.
Additional terms will be linked to the Project Gutenberg™
License for all works posted with the permission of the copyright
holder found at the beginning of this work.
1.E.4. Do not unlink or detach or remove the full Project

Gutenberg™ License terms from this work, or any files
containing a part of this work or any other work associated with
Project Gutenberg™.
1.E.5. Do not copy, display, perform, distribute or redistribute

this electronic work, or any part of this electronic work, without
prominently displaying the sentence set forth in paragraph 1.E.1
with active links or immediate access to the full terms of the
Project Gutenberg™ License.
1.E.6. You may convert to and distribute this work in any binary,
compressed, marked up, nonproprietary or proprietary form,
including any word processing or hypertext form. However, if
you provide access to or distribute copies of a Project
Gutenberg™ work in a format other than “Plain Vanilla ASCII” or
other format used in the official version posted on the official
Project Gutenberg™ website (www.gutenberg.org), you must,
at no additional cost, fee or expense to the user, provide a copy,
a means of exporting a copy, or a means of obtaining a copy
upon request, of the work in its original “Plain Vanilla ASCII” or
other form. Any alternate format must include the full Project
Gutenberg™ License as specified in paragraph 1.E.1.
1.E.7. Do not charge a fee for access to, viewing, displaying,

performing, copying or distributing any Project Gutenberg™
works unless you comply with paragraph 1.E.8 or 1.E.9.
1.E.8. You may charge a reasonable fee for copies of or

providing access to or distributing Project Gutenberg™
electronic works provided that:
• You pay a royalty fee of 20% of the gross profits you derive
from the use of Project Gutenberg™ works calculated using the
method you already use to calculate your applicable taxes. The
fee is owed to the owner of the Project Gutenberg™ trademark,
but he has agreed to donate royalties under this paragraph to
the Project Gutenberg Literary Archive Foundation. Royalty
payments must be paid within 60 days following each date on
which you prepare (or are legally required to prepare) your
periodic tax returns. Royalty payments should be clearly marked
as such and sent to the Project Gutenberg Literary Archive
Foundation at the address specified in Section 4, “Information
about donations to the Project Gutenberg Literary Archive
Foundation.”
• You provide a full refund of any money paid by a user who

notifies you in writing (or by e-mail) within 30 days of receipt
that s/he does not agree to the terms of the full Project
Gutenberg™ License. You must require such a user to return or
destroy all copies of the works possessed in a physical medium
and discontinue all use of and all access to other copies of
Project Gutenberg™ works.
• You provide, in accordance with paragraph 1.F.3, a full refund of

any money paid for a work or a replacement copy, if a defect in
the electronic work is discovered and reported to you within 90
days of receipt of the work.
• You comply with all other terms of this agreement for free
distribution of Project Gutenberg™ works.
1.E.9. If you wish to charge a fee or distribute a Project

Gutenberg™ electronic work or group of works on different
terms than are set forth in this agreement, you must obtain
permission in writing from the Project Gutenberg Literary
Archive Foundation, the manager of the Project Gutenberg™
trademark. Contact the Foundation as set forth in Section 3
below.
1.F.
1.F.1. Project Gutenberg volunteers and employees expend

considerable effort to identify, do copyright research on,
transcribe and proofread works not protected by U.S. copyright
law in creating the Project Gutenberg™ collection. Despite these
efforts, Project Gutenberg™ electronic works, and the medium
on which they may be stored, may contain “Defects,” such as,
but not limited to, incomplete, inaccurate or corrupt data,
transcription errors, a copyright or other intellectual property
infringement, a defective or damaged disk or other medium, a
computer virus, or computer codes that damage or cannot be
read by your equipment.
1.F.2. LIMITED WARRANTY, DISCLAIMER OF DAMAGES - Except

for the “Right of Replacement or Refund” described in
paragraph 1.F.3, the Project Gutenberg Literary Archive
Foundation, the owner of the Project Gutenberg™ trademark,
and any other party distributing a Project Gutenberg™ electronic
work under this agreement, disclaim all liability to you for
damages, costs and expenses, including legal fees. YOU AGREE
THAT YOU HAVE NO REMEDIES FOR NEGLIGENCE, STRICT
LIABILITY, BREACH OF WARRANTY OR BREACH OF CONTRACT
EXCEPT THOSE PROVIDED IN PARAGRAPH 1.F.3. YOU AGREE
THAT THE FOUNDATION, THE TRADEMARK OWNER, AND ANY
DISTRIBUTOR UNDER THIS AGREEMENT WILL NOT BE LIABLE
TO YOU FOR ACTUAL, DIRECT, INDIRECT, CONSEQUENTIAL,
PUNITIVE OR INCIDENTAL DAMAGES EVEN IF YOU GIVE
NOTICE OF THE POSSIBILITY OF SUCH DAMAGE.
1.F.3. LIMITED RIGHT OF REPLACEMENT OR REFUND - If you

discover a defect in this electronic work within 90 days of
receiving it, you can receive a refund of the money (if any) you
paid for it by sending a written explanation to the person you
received the work from. If you received the work on a physical
medium, you must return the medium with your written
explanation. The person or entity that provided you with the
defective work may elect to provide a replacement copy in lieu
of a refund. If you received the work electronically, the person
or entity providing it to you may choose to give you a second
opportunity to receive the work electronically in lieu of a refund.
If the second copy is also defective, you may demand a refund
in writing without further opportunities to fix the problem.
1.F.4. Except for the limited right of replacement or refund set

forth in paragraph 1.F.3, this work is provided to you ‘AS-IS’,
WITH NO OTHER WARRANTIES OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR ANY PURPOSE.
1.F.5. Some states do not allow disclaimers of certain implied

warranties or the exclusion or limitation of certain types of
damages. If any disclaimer or limitation set forth in this
agreement violates the law of the state applicable to this
agreement, the agreement shall be interpreted to make the
maximum disclaimer or limitation permitted by the applicable
state law. The invalidity or unenforceability of any provision of
this agreement shall not void the remaining provisions.
1.F.6. INDEMNITY - You agree to indemnify and hold the

Foundation, the trademark owner, any agent or employee of the
Foundation, anyone providing copies of Project Gutenberg™
electronic works in accordance with this agreement, and any
volunteers associated with the production, promotion and
distribution of Project Gutenberg™ electronic works, harmless
from all liability, costs and expenses, including legal fees, that
arise directly or indirectly from any of the following which you
do or cause to occur: (a) distribution of this or any Project
Gutenberg™ work, (b) alteration, modification, or additions or
deletions to any Project Gutenberg™ work, and (c) any Defect
you cause.
Section 2. Information about the Mission

of Project Gutenberg™
Project Gutenberg™ is synonymous with the free distribution of
electronic works in formats readable by the widest variety of
computers including obsolete, old, middle-aged and new
computers. It exists because of the efforts of hundreds of
volunteers and donations from people in all walks of life.
Volunteers and financial support to provide volunteers with the

assistance they need are critical to reaching Project
Gutenberg™’s goals and ensuring that the Project Gutenberg™
collection will remain freely available for generations to come. In
2001, the Project Gutenberg Literary Archive Foundation was
created to provide a secure and permanent future for Project
Gutenberg™ and future generations. To learn more about the
Project Gutenberg Literary Archive Foundation and how your
efforts and donations can help, see Sections 3 and 4 and the
Foundation information page at www.gutenberg.org.
Section 3. Information about the Project

Gutenberg Literary Archive Foundation
The Project Gutenberg Literary Archive Foundation is a non-
profit 501(c)(3) educational corporation organized under the
laws of the state of Mississippi and granted tax exempt status
by the Internal Revenue Service. The Foundation’s EIN or
federal tax identification number is 64-6221541. Contributions
to the Project Gutenberg Literary Archive Foundation are tax
deductible to the full extent permitted by U.S. federal laws and
your state’s laws.
The Foundation’s business office is located at 809 North 1500

West, Salt Lake City, UT 84116, (801) 596-1887. Email contact
links and up to date contact information can be found at the
Foundation’s website and official page at
www.gutenberg.org/contact
Section 4. Information about Donations to
the Project Gutenberg Literary Archive
Foundation
Project Gutenberg™ depends upon and cannot survive without
widespread public support and donations to carry out its mission
of increasing the number of public domain and licensed works
that can be freely distributed in machine-readable form
accessible by the widest array of equipment including outdated
equipment. Many small donations ($1 to $5,000) are particularly
important to maintaining tax exempt status with the IRS.
The Foundation is committed to complying with the laws

regulating charities and charitable donations in all 50 states of
the United States. Compliance requirements are not uniform
and it takes a considerable effort, much paperwork and many
fees to meet and keep up with these requirements. We do not
solicit donations in locations where we have not received written
confirmation of compliance. To SEND DONATIONS or determine
the status of compliance for any particular state visit
www.gutenberg.org/donate.
While we cannot and do not solicit contributions from states

where we have not met the solicitation requirements, we know
of no prohibition against accepting unsolicited donations from
donors in such states who approach us with offers to donate.
International donations are gratefully accepted, but we cannot

make any statements concerning tax treatment of donations
received from outside the United States. U.S. laws alone swamp
our small staff.
Please check the Project Gutenberg web pages for current

donation methods and addresses. Donations are accepted in a
number of other ways including checks, online payments and
credit card donations. To donate, please visit:
www.gutenberg.org/donate.
Section 5. General Information About

Project Gutenberg™ electronic works
Professor Michael S. Hart was the originator of the Project
Gutenberg™ concept of a library of electronic works that could
be freely shared with anyone. For forty years, he produced and
distributed Project Gutenberg™ eBooks with only a loose
network of volunteer support.
Project Gutenberg™ eBooks are often created from several

printed editions, all of which are confirmed as not protected by
copyright in the U.S. unless a copyright notice is included. Thus,
we do not necessarily keep eBooks in compliance with any
particular paper edition.
Most people start at our website which has the main PG search
facility: www.gutenberg.org.
This website includes information about Project Gutenberg™,

including how to make donations to the Project Gutenberg
Literary Archive Foundation, how to help produce our new
eBooks, and how to subscribe to our email newsletter to hear
about new eBooks.
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
Let us accompany you on the journey of exploring knowledge and

personal growth!
ebookgate.com

Full download Identification of Nonlinear Physiological Systems IEEE Press Series on Biomedical Engineering 1st Edition David T. Westwick pdf docx

Uploaded by

Copyright:

Available Formats

Full download Identification of Nonlinear Physiological Systems IEEE Press Series on Biomedical Engineering 1st Edition David T. Westwick pdf docx

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Full download Identification of Nonlinear Physiological Systems IEEE Press Series on Biomedical Engineering 1st Edition David T. Westwick pdf docx

Uploaded by

Copyright:

Available Formats

Visit https://ebookgate.

com to download the full version and

Identification of Nonlinear Physiological Systems

_____ Click the link below to download _____

Explore and download more ebooks at ebookgate.com

Nonlinear Dynamic Modeling of Physiological Systems IEEE

Elements of Tidal Electric Engineering IEEE Press Series

Communication and Control in Electric Power Systems

Electricity Power Generation The Changing Dimensions IEEE

Introduction to FACTS Controllers Theory Modeling and

Sourcebook of ATM and IP Internetworking IEEE Press Series

Perceptual Computing Aiding People in Making Subjective

System of Systems Engineering Wiley Series in Systems

Metin Akay, Series Editor

A JOHN WILEY & SONS, INC., PUBLICATION

IEEE Press Editorial Board

M. Ajay R. J. Herrick M. S. Newman

IEEE Engineering in Medicine and Biology Society, Sponsor

Published simultaneously in Canada.

Library of Congress Cataloging-in-Publication Data:

Westwick, D. T. (David T.)

Printed in the United States of America

2.1 Vectors and Matrices / 13

2.3.1 Autocorrelation Functions / 16

3 Models of Linear Systems 39

3.1 Linear Systems / 39

4 Models of Nonlinear Systems 57

4.1 The Volterra Series / 57

4.2 The Wiener Series / 67

5 Identiﬁcation of Linear Systems 103

5.1 Introduction / 103

6 Correlation-Based Methods 125

6.1 Methods for Functional Expansions / 125

7 Explicit Least-Squares Methods 169

7.1 Introduction / 169

8 Iterative Least-Squares Methods 207

8.1 Optimization Methods / 207

8.1.2 Identiﬁcation of Block-Structured Models / 209

IEEE Press Series in Biomedical Engineering 262

Calgary, Alberta, Canada

∗ MATLAB is a registered trademark of the MathWorks, Inc.

Identiﬁcation of Nonlinear Physiological Systems, By David T. Westwick and Robert E. Kearney

1.1.1 Domain and Range

1.1.2 Deterministic and Stochastic Signals

may be necessary to treat the deterministic signal as if it were a single realization of

1.1.3 Stationary and Ergodic Signals

1.2 SYSTEMS AND MODELS

y(t) = N(u(t)) (1.1)

1.2.1 Model Structure and Parameters

ŷ(t) = M(u(t)) (1.3)

ŷ(θ, t) = M(θ, u(t))

the parameter vector, θ , would contain the polynomial coefﬁcients,

Output: y(t) = N(u(t)) 4

1.2.2 Static and Dynamic Systems

depends only on the value of the input at the previous time, t − τ .

1.2.3 Linear and Nonlinear Systems

N(c · u(t)) = c · y(t) (1.5)

y1 (t) = N(u1 (t))

If the response to the input u1 (t) + u2 (t) is given by

N(u1 (t) + u2 (t)) = y1 (t) + y2 (t) (1.6)

_ Click the link below to download _