Stochastic Systems Analysis and Simulations

Alejandro Ribeiro
Dept. of Electrical and Systems Engineering
University of Pennsylvania
[email protected]
http://www.seas.upenn.edu/users/~aribeiro/

August 29, 2018

Stoch. Systems Analysis Introduction 1

Presentations

Presentations

Class description and contents

Gambling

Stoch. Systems Analysis Introduction 2

Who are us, where to find me, lecture times

I Alejandro Ribeiro
I Professor, Electrical and Systems Engineering
I Walnut 3401 floor 4B. http://alliance.seas.upenn.edu/~aribeiro/wiki
I Teaching assistants: Harshat Kumar and Luana Rubini Ruiz
I Class email: [email protected]. Don’t write to our personal
addresses. Please.

I We also have a separate grader

I We meet on the Berger auditorium in Skirkanich Hall

I Mondays, Wednesdays, Fridays 10 am to 11 am
I My office hours, Wednesdays and Fridays from 11 am to 12 pm
I Anytime, as long as you have something interesting to tell me
I http://alliance.seas.upenn.edu/~ese303/wiki

Stoch. Systems Analysis Introduction 3

Prerequisites

I Probability theory
I Stochastic processes are time-varying random entities
I If unknown, need to learn as we go
I Will cover in first seven lectures

I Linear algebra
I Vector matrix notation, systems of linear equations, eigenvalues

I Programming in Matlab
I Needed for homework.
I If you know programming you can learn Matlab in one afternoon
I But it has to be this afternoon

I Differential equations, Fourier transforms

I Appear here and there. Should not be a problem

Stoch. Systems Analysis Introduction 4

Homework and grading

I 14 homework sets in 14 weeks

I Collaboration accepted, welcomed, and encouraged
I Sets graded as 0 (bad), 1 (good), 2 (very good) and 3 (outstanding)
I We’ll use the 3 sparingly. Goal is to earn 28 homework points

I First Midterm examination starts on Friday October 12, 36 points

I Take home due on Monday October 15
I Work independently. No collaboration, no discussion

I Second midterm examination on December 10 worth 36 points

I At least 60 points are required for passing.

I C requires at least 70 points. B at least 80. A at least 90
I Goal is for everyone to earn an A

Stoch. Systems Analysis Introduction 5

Textbooks

I Textbook for the class is (older or newer editions acceptable)

I Sheldon M. Ross ”Introduction to Probability Models”,
Academic Press, whatever ed.

I Same topics at advanced level (more rigor, includes proofs)

I Sheldon Ross ”Stochastic Processes”, John Wiley & sons, 2nd ed.

I Stohastic processes in systems biology

I Darren J. Wilkinson ”Stochastic Modelling for Systems Biology”,
Chapman & Hall/CRC, 1st ed.
I Part on simulation of chemical reactions taken from here

I Use of stochastic processes in finance

I Masaaki Kijima ”Stochastic Processes with Applications to
Finance”, Chapman & Hall/CRC, 1st ed.

Stoch. Systems Analysis Introduction 6

This is not a programming class

I Just that. Not a programming class

Stoch. Systems Analysis Introduction 7

Difficulty

I This class has a reputation for been hard and demanding

⇒ I do not entirely agree but I take the point
I On the other hand, the quality ratings are very good
⇒ S. Reid Warren Jr. Award (2012)
⇒ Lindback Award for Distinguished Teaching (2017)
⇒ Collected the two possible teaching awards in 8 years

I This is, really, a great class. You will do things that look like magic

I Also, the class is front loaded. It will become easier after the break.
⇒ Don’t drop it! You will enjoy it.

Stoch. Systems Analysis Introduction 8

Some remarks on teaching methods

I I ask questions to individual students (cold calling).

⇒ Absolutely zero premium (penalty) on right (wrong) answer

I Do these questions serve any purpose?

⇒ I need to gauge what you understand of what I say
⇒ There are different ways of explaining ideas
⇒ Your spatial memory associates parts of the room with
concepts
⇒ People remember conversations better than lectures

I Can anyone explain to me why this is called cold calling?

Stoch. Systems Analysis Introduction 9

Class contents

Presentations

Class description and contents

Gambling

Stoch. Systems Analysis Introduction 10

Stochastic systems

I Anything random that evolves in time

I Time can be discrete (0, 1, . . . ) or continuous

I More formally, assign a function to a random event

I Compare with “random variable assigns a value to a random event”
I Generalizes concept of random vector to functions
I Or generalizes the concept of function to random settings

I Can interpret a stochastic process as a set of random variables

I Not always the most appropriate way of thinking

Stoch. Systems Analysis Introduction 11

A voice recognition system

I Random event ∼ word spoken. Stochastic process ∼ the waveform

I Try the file speech signals.m
“Hi” “Good”
1 1

0.8 0.8

0.6 0.6

0.4 0.4

0.2 0.2

Amplitude
Amplitude

0 0

−0.2 −0.2

−0.4 −0.4

−0.6 −0.6

−0.8 −0.8

−1 −1
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4
Time [sec] Time [sec]

“Bye” ‘S”
1 1

0.8 0.8

0.6 0.6

0.4 0.4

0.2 0.2
Amplitude

Amplitude

0 0

−0.2 −0.2

−0.4 −0.4

−0.6 −0.6

−0.8 −0.8

−1 −1
0 0.05 0.1 0.15 0.2 0.25 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14
Time [sec] Time [sec]

Stoch. Systems Analysis Introduction 12

Five blocks

I Probability theory review (6 lectures)

I Probability spaces
I Conditional probability: time n + 1 given time n, future given past ...
I Limits in probability, almost sure limits: behavior as t → ∞ ...
I Common probability distributions (binomial, exponential, Poisson, Gaussian)

I Stochastic processes are complicated entities

I Restrict attention to particular classes that are somewhat tractable

I Markov chains (9 lectures)

I Continuous time Markov chains (12 lectures)
I Stationary random processes (9 lectures)

I Midterm covers up to Markov chains

Stoch. Systems Analysis Introduction 13

Markov chains

I A set of states 1, 2, . . .. At time n, state is Xn

I Memoryless property
⇒ Probability of next state Xn+1 depends on current state Xn
⇒ But not on past states Xn−1 , Xn−2 , . . .

I Can be happy (Xn = 0) or sad (Xn = 1) 0.8 0.3

I Happiness tomorrow affected by 0.2
happiness today only
I Whether happy or sad today, likely to H S
be happy tomorrow 0.7
I But when sad, a little less likely so
I Classification of states, ergodicity, limiting distributions
I Google’s page rank, machine learning, virus propagation, queues ...

Stoch. Systems Analysis Introduction 14

Continuous time Markov chains

I A set of states 1, 2, . . .. Continuous time index t

I Transition between states can happen at any time
I Future depends on present but is independent of the past

0.2dt

I Probability of changing state in H S

an infinitesimal time dt
0.7dt

I Poisson processes, exponential distributions, transition probabilities,

Kolmogorov equations, limit distributions
I Chemical reactions, queues, communication networks, weather
forecasting ...

Stoch. Systems Analysis Introduction 15

Stationary random processes

I Continuous time t, continuous state x(t), not necessarily memoryless

I System has a steady state in a random sense
I Prob. distribution of x(t) constant or becomes constant as t grows

I Brownian motion, white noise, Gaussian processes, autocorrelation,

power spectral density.

I Black Scholes model for option pricing, speech, noise in electric

circuits, filtering and equalization ...

Stoch. Systems Analysis Introduction 16

Gambling

Presentations

Class description and contents

Gambling

Stoch. Systems Analysis Introduction 17

An interesting betting game

I There is a certain game in a certain casino in which ...

⇒ your chances of winning are p > 1/2
I You place $1 bets
(a) With probability p you gain $1 and
(b) With probability (1 − p) you lose your $1 bet

I The catch is that you either

(a) Play until you go broke (lose all your money)
(b) Keep playing forever

I You start with an initial wealth of $w0

I Shall you play this game?

Stoch. Systems Analysis Introduction 18

Modeling

I Let t be a time index (number of bets placed)

I Denote as x(t) the outcome of the bet at time t
I x(t) = 1 if bet is won (with probability p)
I x(t) = 0 if bet is lost (probability (1 − p))
I x(t) is called a Bernoulli random varible with parameter p

I Denote as w (t) the player’s wealth at time t

I At time t = 0, w (0) = w0
I At times t > 0 wealth w (t) depends on past wins and losses
I More specifically we have
I When bet is won w (t) = w (t − 1) + 1
I When bet is lost w (t) = w (t − 1) − 1

Stoch. Systems Analysis Introduction 19

Coding

t = 0; w (t) = w0 ; maxt = 103 ; // Initialize variables

% repeat while not broke up to time maxt
while (w (t) > 0) & (t < maxt ) do
x(t) = random(’bino’,1,p); % Draw Bernoulli random variable
if x(t) == 1 then
w (t + 1) = w (t) + b; % If x = 1 wealth increases by b
else
x(t + 1) = w (t) − b; % If x = 0 wealth decreases by b
end
t = t + 1;
end
I Initial wealth w0 = 20, bet b = 1, win probability p = 0.55

I Shall we play?

Stoch. Systems Analysis Introduction 20

One lucky player

I She didn’t go broke. After t = 1000 bets, her wealth is w (t) = 109
I Less likely to go broke now because wealth increased

200

180

160

140

120
wealth (in $)

100

80

60

40

20

0
0 100 200 300 400 500 600 700 800 900 1000
bet index

Stoch. Systems Analysis Introduction 21

Two lucky players

I Wealths are w1 (t) = 109 and w2 (t) = 139

I Increasing wealth seems to be a pattern

200

180

160

140

120
wealth (in $)

100

80

60

40

20

0
0 100 200 300 400 500 600 700 800 900 1000
bet index

Stoch. Systems Analysis Introduction 22

Ten lucky players

I Wealths wj (t) between 78 and 139

I Increasing wealth is definitely a pattern

200

180

160

140

120
wealth (in $)

100

80

60

40

20

0
0 100 200 300 400 500 600 700 800 900 1000
bet index

Stoch. Systems Analysis Introduction 23

One unlucky player

I But this does not mean that all players will turn out as winners
I The twelfth player j = 12 goes broke

200

180

160

140

120
wealth (in $)

100

80

60

40

20

0
0 100 200 300 400 500 600 700 800 900 1000
bet index

Stoch. Systems Analysis Introduction 24

One unlucky player

I But this does not mean that all players will turn out as winners
I The twelfth player j = 12 goes broke

40

35

30

25
wealth (in $)

20

15

10

0
0 50 100 150 200 250
bet index

Stoch. Systems Analysis Introduction 25

One hundred players

I Only one player (j = 12) goes broke

I All other players end up with substantially more money

200

180

160

140

120
wealth (in $)

100

80

60

40

20

0
0 100 200 300 400 500 600 700 800 900 1000
bet index

Stoch. Systems Analysis Introduction 26

Average tendency

I It is not difficult to find a line estimating the average of w (t)

I w̄ (t) ≈ w0 + (2p − 1)t ≈ w0 + 0.1t

200

180

160

140

120
wealth (in $)

100

80

60

40

20

0
0 100 200 300 400 500 600 700 800 900 1000
bet index

Stoch. Systems Analysis Introduction 27

Where does the average tendency comes from?

I To discover average tendency notice that for all times t we can write
 
W (t + 1) = W (t) + 2X (t) − 1

I Taking expectation on both sides and using linearity of expectations

 
E [W (t + 1)] = E [W (t)] + 2E [X (t)] − 1

I The expected value of X (t) is

E [X (t)] = 1 × P (X (t) = 1) + 0 × P (X (t) = 1) = p

I Which yields ⇒ E [W (t + 1)] = E [W (t)] + (2p − 1)

I Applying recursively ⇒ E [W (t + 1)] = w0 + (2p − 1)t

Stoch. Systems Analysis Introduction 28

Analysis of outcomes: mean

I For a more accurate analysis analyze simulation’s outcome

I Consider J experiments
I For each experiment, there is a wealth history wj (t)
I We can estimate the average outcome as
J
1X
w̄J (t) = wj (t)
J
j=1

I w̄J (t) is called the sample average

I Do not confuse w̄J (t) with E [w (t)]
I w̄J (t) is computed from experiments, it is a random quantity in itself
I E [w (t)] is a property of the random variable w (t)
I We will see later that for large J, w̄J (t) → E [w (t)]

Stoch. Systems Analysis Introduction 29

Analysis of outcomes: mean

I Expected value E [w (t)] in black (approximation)

I Sample average for J = 10 (blue), J = 20 (red), and J = 100 (magenta)

65

60

55

50
wealth (in $)

45

40

35

30

25

20
0 50 100 150 200 250 300 350 400
bet index

Stoch. Systems Analysis Introduction 30

Histogram

I There is more information in the simulation’s output

I Estimate the probability distribution function (pdf) ⇒ Histogram

I Consider a set of points w (0) , . . . , w (N)

I Indicator function of the event w (n) ≤ wj < w (n+1)
h i
I I w (n) ≤ wj < w (n+1) = 1 when w (n) ≤ wj < w (n+1)
h i
I I w (n) ≤ wj < w (n+1) = 0 else

I Histogram is then defined as

h i 1X J h i
H t; w (n) , w (n+1) = I w (n) ≤ wj (t) < w (n+1)
J
j=1

I Fraction of experiments with wealth wj (t) between w (n) and w (n+1)

Stoch. Systems Analysis Introduction 31

Histogram

I The pdf broadens and shifts to the right (t = 10, 50, 100, 200)

0.25 0.25

0.2 0.2
frequency

frequency
0.15 0.15

0.1 0.1

0.05 0.05

0 0
0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90
wealth (in $) wealth (in $)

0.25 0.25

0.2 0.2
frequency

frequency

0.15 0.15

0.1 0.1

0.05 0.05

0 0
0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90
wealth (in $) wealth (in $)

Stoch. Systems Analysis Introduction 32

What is this class about

I Analysis and simulation of stochastic systems

⇒ A system that evolves in time with some randomness

I They are usually quite complex ⇒ Simulations

I We will learn how to model stochastic systems, e.g.,
I x(t) Bernoulli with parameter p
I w (t) = w (t − 1) + 1 when x(t) = 1
I w (t) = w (t − 1) − 1 when x(t) = 0

I ... how to analyze, e.g., E [W (t)] = w0 + t(2p − 1)

I ... and how to interpret simulations and experiments, e.g,
I Average tendency through sample average

Stoch. Systems Analysis Introduction 33