CE 502

Advanced Probability and Statistics for Civil Engineers

Spring 2022

Taner Yılmaz, PhD

Özyeğin University
Department of Civil Engineering

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 WEEK 4
Subjects:
• Multiple Random Variables
• Joint Probability Function
• Covariance and Correlation Coefficient

References
Ang, A.H-S. and W.H. Tang, Probability Concepts in Engineering: Emphasis on Applications to Civil and Environmental
Engineering, 2nd edition, John Wiley & Sons, 2007.
Haldar, A. and Mahadevan, S. Probability, Reliability, and Statistical Methods in Engineering Design, John Wiley & Sons,
2000.
Montgomery, D.C. And Runger G.C. Applied Statistics and Probability for Engineers, Wiley, 5th Edition (SI), 2011.
2

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Multiple Random Variables
The previous discussion was on modeling of uncertainty in a single random variable.
It may be necessary to consider more than one random variable to formulate a particular problem.

For example: Let’s take;

• L = Load applied to a structure
• R = Structural response
L and R can be modeled separately as random variables, however it would be better to model the
uncertainties jointly and more information can be extracted from the joint distribution.

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Joint Probability Distributions
(The following sections are explained for two random variables, but the discussion can be extended for more than two
variables.)

As any pair of values of the random variables X and Y represents events, there are probabilities associated with given
values of x and y; accordingly the probabilities for all possible pairs of x and y may be described with the joint
distribution function CDF of the random variables X and Y .

𝐹𝑋,𝑌 𝑥, 𝑦 = 𝑃(𝑋 ≤ 𝑥, 𝑌 ≤ 𝑦)

which is the CDF of the joint occurrences of the events identified by 𝑋 ≤ 𝑥 and 𝑌 ≤ 𝑦.

Complying with the fundamental axioms of probability, the above CDF must satisfy the following:

𝐹𝑋,𝑌 −∞, −∞ = 0, 𝐹𝑋,𝑌 +∞, +∞ = 1.0

𝐹𝑋,𝑌 −∞, 𝑦 = 0, 𝐹𝑋,𝑌 𝑥, −∞ = 0

𝐹𝑋,𝑌 𝑥, +∞ = 𝐹𝑋 (𝑥), 𝐹𝑋,𝑌 +∞, 𝑦 = 𝐹𝑌 (𝑦)

𝐹𝑋,𝑌 is nonnegative and is a nondecreasing function of X and Y.

The joint probability distribution of two random variables is sometimes referred to as the bivariate probability
distribution of the random variables.

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Joint Probability Distributions for Discrete Random Variables
Suppose X and Y are two discrete random variables.

• The joint probability distribution of X and Y is a description of the set of point (x, y) in the range of (X, Y) along
with the probability of each point.
• One way to describe the joint probability distribution of two discrete random variables is through a joint
probability mass function.

The joint probability mass function, pX,Y(x, y) for the discrete random variables X and Y satistfies the following:

(1) 𝑝𝑋,𝑌 𝑥, 𝑦 = 𝑃(𝑋 = 𝑥, 𝑌 = 𝑦)

(2) 𝑝𝑋,𝑌 𝑥, 𝑦 ≥ 0

𝟑 ෍ ෍ 𝑝𝑋,𝑌 𝑥𝑖 , 𝑦𝑗 = 1.0
𝑥𝑖 , 𝑦 𝑗

The joint cumulative distribution function of X and Y: 𝐹𝑋,𝑌 𝑥, 𝑦 = 𝑃 𝑋 ≤ 𝑥, 𝑌 ≤ 𝑦 = ෍ ෍ 𝑝𝑋,𝑌 𝑥, 𝑦

𝑥𝑖 ≤𝑥 𝑦𝑗 ≤𝑦

The joint probability mass function of X and Y is assumed to be zero at values for which a probability is not specified.

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Joint Probability Distributions for Continuous Random Variables
Suppose X and Y are two continuous random variables.

• The joint probability distribution of two continuous random variables X and Y can be specified by providing a
method for calculating the probability that X and Y assume a value in any region R of two-dimensional space.
• The joint probability distribution of two continuous random variables can be described through a joint
probability density function.

The joint probability density function, fX,Y(x, y) for the continuous random variables X and Y satistfies the following:

(1) 𝑓𝑋,𝑌 𝑥, 𝑦 ≥ 0 for all x, y

∞ ∞

(2) න න 𝑓𝑋,𝑌 𝑥, 𝑦 𝑑𝑥 𝑑𝑦 = 1.0

−∞ −∞

(3) For any region R of two-dimensional space,

This integral can be interpreted as the volume

𝑃((𝑋, 𝑌) ∈ 𝑅) = න න 𝑓𝑋,𝑌 𝑥, 𝑦 𝑑𝑥 𝑑𝑦 under the surface over the region R.
𝑅

𝑥 𝑦

The joint cumulative distribution function of X and Y: 𝐹𝑋 𝑥, 𝑦 = 𝑃 𝑋 ≤ 𝑥, 𝑌 ≤ 𝑦 = න න 𝑓𝑋,𝑌 𝑢, 𝑣 𝑑𝑣 𝑑𝑢

−∞ −∞

Typically, fX,Y(x, y) is defined over all of two-dimensional space by assuming that fX,Y(x, y) = 0 for all points for which
fX,Y(x, y) is not specified.
6

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Joint Probability Distributions

For two random variables, PDF or PMF

can be shown on a three-dimensional
plot.

This figure is taken from Haldar and Mahedevan, 1999

7

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Joint Probability Distributions

The following probability is the volume under the surface f(x, y).

𝑏 𝑑

𝑃 𝑎 < 𝑋 ≤ 𝑏, 𝑐 < 𝑌 ≤ 𝑑 = න න 𝑓𝑋,𝑌 𝑢, 𝑣 𝑑𝑣 𝑑𝑢

𝑎 𝑐

The above figure is taken from

Ang and Tang, 2007.
8

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Conditional Probability Distributions
In many engineering problems, the values a random variable takes may be statistically dependent on the values of
another random variable.

Conditional PMF (probability mass function) for discrete random variables:

𝑝𝑋,𝑌 𝑥𝑖 , 𝑦𝑗
𝑝𝑋|𝑌 𝑥𝑖 |𝑦𝑗 = 𝑃(𝑋 = 𝑥𝑖 |𝑌 = 𝑦𝑗 ) = 𝑓𝑜𝑟 𝑝𝑌 𝑦𝑗 ≠ 0
𝑝𝑌 𝑦𝑗

𝑝𝑋,𝑌 𝑥𝑖 , 𝑦𝑗
𝑝𝑌|𝑋 𝑦𝑗 |𝑥𝑖 = 𝑃 𝑌 = 𝑦𝑗 𝑋 = 𝑥𝑖 = 𝑓𝑜𝑟 𝑝𝑋 𝑥𝑖 ≠ 0
𝑝𝑋 𝑥𝑖

Conditional PDF (probability density function) for continuous random variables:

𝑓𝑋,𝑌 𝑥, 𝑦 𝑓𝑜𝑟 𝑓𝑌 𝑦 ≠ 0
𝑓𝑋|𝑌 𝑥 | 𝑦 =
𝑓𝑌 𝑦

𝑓𝑋,𝑌 𝑥, 𝑦
𝑓𝑌|𝑋 𝑦 | 𝑥 = 𝑓𝑜𝑟 𝑓𝑋 𝑥 ≠ 0
𝑓𝑋 𝑥

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Statistical Independence
Considering the equations for conditional probability, we can write:

𝑓𝑋,𝑌 𝑥, 𝑦 = 𝑓𝑋|𝑌 𝑥 | 𝑦 𝑓𝑌 𝑦 or 𝑓𝑋,𝑌 𝑥, 𝑦 = 𝑓𝑌|𝑋 𝑥 | 𝑦 𝑓𝑋 𝑦

If the random variables X and Y are statistically independent, the condition has no meaning;

𝑓𝑋|𝑌 𝑥 | 𝑦 = 𝑓𝑋 𝑥 and 𝑓𝑌|𝑋 𝑦 |𝑥 = 𝑓𝑌 𝑦

Then using the upmost equations;

𝑓𝑋,𝑌 𝑥, 𝑦 = 𝑓𝑋 𝑦 𝑓𝑌 𝑦 for continuous random variables X and Y.

Similarly, 𝑝𝑋,𝑌 𝑥, 𝑦 = 𝑝𝑋 𝑥 𝑝𝑌 𝑦 for discrete random variables X and Y.

10

10

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Marginal Probability Distributions
• If more than one random variable is defined in a random experiment, it is important to distinguish between the joint
probability distribution of X and Y and the probability distribution of each variable individually.
• The individual probability distribution of a random variable is referred to as its marginal probability distribution.

Marginal PDF and PMF can be derived by using the theorem of total probability:

For discrete random variables X and Y;

Summing all the joint probabilities

𝑝𝑋 𝑥 = ෍ 𝑝𝑋,𝑌 𝑥, 𝑦𝑗 for which X=x.
𝑎𝑙𝑙 𝑦𝑗

Summing all the joint probabilities

𝑝𝑌 𝑦 = ෍ 𝑝𝑋,𝑌 𝑥𝑖 , 𝑦 for which Y=y.
𝑎𝑙𝑙 𝑥𝑖

For continuous random variables X and Y;

𝑓𝑋 𝑥 = න 𝑓𝑋,𝑌 𝑥, 𝑦 𝑑𝑦 This integral is over all points in

−∞ the range of (X, Y) for which X= x:

𝑓𝑌 𝑦 = න 𝑓𝑋,𝑌 𝑥, 𝑦 𝑑𝑥 This integral is over all points in

the range of (X, Y) for which Y= y:
−∞

11

11

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Marginal Probability Distributions

The above figure is taken from

Ang and Tang, 2007. 12

12

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Example 3.1
Consider the random variables X and Y. The joint occurrence probability of these random varibles are given in the table
below.
pX,Y(x, y) x=0 x=1 x=2 a) Find the marginal PMF for both X and Y.
y=0 1/16 1/3 1/12 b) Find the conditional probability of X when y = 1.

y=1 2/9 1/6 0

c) Check whether X and Y are statistically independent or not.

y=2 1/36 0 0

a)
pX,Y(x, y) x=0 x=1 x=2 pY(y)
pX(x) pY(y)
y=0 1/16 1/3 1/12 7/12 x=0 5/12 y=0 7/12

y=1 2/9 1/6 0 7/18 x=1 1/2 y=1 7/18

y=2 1/36 0 0 1/36 x=2 1/12 y=2 1/36

pX(x) 5/12 1/2 1/12

b) 𝑝𝑋,𝑌 𝑥, 1 c) 𝑝𝑋, 𝑌 𝑥, 𝑦 = 𝑝𝑋 𝑥 𝑝𝑌 𝑦
𝑝 𝑥|𝑦 = 1 = Check whether
𝑝𝑌 1
for statistically independency for all x, y combinations.
2/9
𝑝 𝑥 = 0|𝑦 = 1 = = 4/7
7/18
Check for (x, y) = (0, 0) 𝑝𝑋, 𝑌 0, 0 = 1/16
1/6
𝑝 𝑥 = 1|𝑦 = 1 = = 3/7
7/18 𝑝𝑋 0 = 5/12 𝑝𝑌 0 = 7/12

0 5 7 35 1
𝑝 𝑥 = 2|𝑦 = 1 = =0 𝑝𝑋 0 𝑝𝑌 0 = = ≠ X and Y are dependent.
7/18 12 12 144 16
13

13

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Example 3.2
The following joint probability function is given for the random variables X and Y.

𝑓𝑋,𝑌 𝑥, 𝑦 = 4𝑥𝑦 0 ≤ 𝑥 ≤ 1, 0≤𝑦≤1 a) Find the marginal PDF for both X and Y.
b) Check whether X and Y are statistically independent or not.
𝑓𝑋,𝑌 𝑥, 𝑦 = 0 𝑒𝑙𝑠𝑒𝑤ℎ𝑒𝑟𝑒

1 1
a)
𝑓𝑋 𝑥 = න 4𝑥𝑦 𝑑𝑦 = 2𝑥 For 0 ≤ 𝑥 ≤ 1 𝑓𝑌 𝑦 = න 4𝑥𝑦 𝑑𝑥 = 2𝑦 For 0 ≤ 𝑦 ≤ 1
0 0

𝑓𝑋 𝑥 = 0 For 𝑒𝑙𝑠𝑒𝑤ℎ𝑒𝑟𝑒 𝑓𝑌 𝑦 = 0 For 𝑒𝑙𝑠𝑒𝑤ℎ𝑒𝑟𝑒

b) Check whether 𝑓𝑋,𝑌 𝑥, 𝑦 = 𝑓 𝑥 𝑓𝑌 𝑦

𝑓 𝑥, 𝑦 = 4𝑥𝑦

𝑓𝑋 𝑥 𝑓𝑌 𝑦 = 2𝑥 2𝑦 = 4𝑥𝑦 = 𝑓𝑋,𝑌 𝑥, 𝑦 X and Y are statistically independent.

14

14

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Covariance and Correlation
• The problem may be more cumbersome if the probability needs to be calculated using the joint distribution of many
random variables.
• Furthermore, the available information may not be adequate to develop the joint distribution of the multiple
random variables .
• For practical applications, it could be adventageous to use the information on the dependence or independence
between two random variables to extract as much information as possible. This can be accomplished by covariance
and correlation analyses.

Covariance of two random variables, X and Y, denoted as Cov(X, Y)

E(XY) can be calculated as: E(XY) is also called joint second moment.

If X and Y are statistically independent, then:

• For statistically independent X and Y, Cov(X, Y) = 0.

• Otherwise (not statistically independent), it can be positive or negative and has the unit that is the square of the
unit of the mean.
• Cov(X, Y) indicates the degree of linear relationship between the two random variables.

15

15

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Correlation Coefficient
Nondimensionalizing the covariance will result in the correlation coefficient, denoted as ρX, Y :

𝐶𝑜𝑣(𝑋, 𝑌)
𝜌𝑋,𝑌 =
𝜎𝑋 𝜎𝑌

• Values of ρX, Y range between -1 and +1. It also represents the degree of linear dependence between two random
variables.
• If the correlation coefficient needs to be calculated from observed sample values, it is rare to obtain values of precisely
zero, +1, or -1.
• The two random variables can be considered to be statistically independent if the correlation coefficient is less than
±0.3; they can be considered to be perfectly correlated if the correlation coefficient is greater than ±0.9.

No linear relationship between X and Y. Positive relationship between X and Y. Y increases as X increases
“Uncorrelated” “The relationship is not perfectly linear ”

16

16

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Correlation Coefficient

Positive relationship between X and Y. Y increases as X increases Negative relationship between X and Y. Y decreased as X increases
“The relationship is perfectly linear ” “The relationship is perfectly linear ”

There could be some nonlinear relationship between the two

random variables, but since the relationship is not linear ρX, Y = 0

17

17

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Example 3.3 (Example 3.36 in Ang and Tang, 2007)
A cantilever beam is subjected to two random loads, S1 and S2, that are statistically independent. The shear force Q and
bending moment M at the fixed support of the beam are both functions of the two loads. Find the correlation coefficient
between Q and M, ρQ, M .
𝑆1: 𝜇1 , 𝜎1 𝑆2: 𝜇2 , 𝜎2 Note:
S1 S1
M 𝜎𝑋 2 = 𝐸 𝑋 2 − 𝜇𝑋 2
𝑄 = 𝑆1 + 𝑆2 Therefore;
a a
𝐸 𝑋 2 = 𝜎𝑋 2 + 𝜇𝑋 2
𝑀 = 𝑎 𝑆1 + 2𝑎 𝑆2

𝐸 𝑄 = 𝜇𝑄 = 𝐸 𝑆1 + 𝐸 𝑆2 = 𝜇1 + 𝜇2 𝐸 𝑄𝑀 = 𝐸[(𝑆1 + 𝑆2)(𝑎 𝑆1 + 2𝑎 𝑆2)] = 𝑎 𝐸 𝑆12 + 3𝑎 𝐸 𝑆1 ∙ 𝑆2 + 2𝑎 𝐸(𝑆22 )

𝐸 𝑀 = 𝜇𝑀 = 𝑎 𝜇1 + 2𝑎 𝜇2 𝜇1 𝜇2 (since
= 𝑎 (𝜎1 2 + 𝜇1 2 ) + 3𝑎 (𝜇1 𝜇2 ) + 2𝑎 (𝜎2 2 + 𝜇2 2 ) independent)
𝑉𝑎𝑟 𝑄 = (𝜎𝑄 )2 = 𝜎1 2 +𝜎2 2
= 𝑎 (𝜎1 2 + 2 𝜎2 2 ) + 𝑎 𝜇1 2 + 2𝑎 𝜇2 2 + 3𝑎 (𝜇1 𝜇2 )
2 2
𝑉𝑎𝑟 𝑀 = (𝜎𝑀 )2 = 𝑎2 𝜎1 +(2𝑎)2 𝜎2
2 2 = 𝑎 (𝜎1 2 + 2 𝜎2 2 ) + (𝜇1 + 𝜇2 ) (𝑎 𝜇1 + 2𝑎 𝜇2 )
= 𝑎2 𝜎1 +4 𝑎2 𝜎2
𝜇𝑄 𝜇𝑀
𝐶𝑜𝑣 𝑄, 𝑀 = 𝐸 𝑄𝑀 − 𝐸 𝑄 𝐸(𝑀)
If it is assumed that 𝜎1 = 𝜎2 = 𝜎
= 𝑎 𝜎1 2 + 2 𝜎2 2 + 𝜇𝑄 𝜇𝑀 − 𝜇𝑄 𝜇𝑀
𝑎 𝜎 2 + 2𝜎 2 3𝑎 𝜎 2 3
𝜌𝑋,𝑌 = = =
= 𝑎 𝜎1 2 + 2 𝜎2 2 + 𝜇𝑄 𝜇𝑀 − 𝜇𝑄 𝜇𝑀 𝜎2 +𝜎 2 𝑎2 𝜎 2 +4𝑎2 𝜎 2 10 𝑎 𝜎2 10

𝐶𝑜𝑣 𝑄, 𝑀 = 𝑎 𝜎1 2 + 2 𝜎2 2 𝜌𝑋,𝑌 = 0.948

𝑎 𝜎1 2 + 2 𝜎2 2
𝜌𝑋,𝑌 = Q and M are highly correlated since both of them depend on
𝜎𝑄 𝜎𝑀
S1 and S2. 18

18

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz
 Multivariate Distributions
Multivariate distributions are mathematically more complicated, but sometimes they can be used if the joint PDF is
known.

Bivariate Normal Distribution

For example, below the PDF of a bivariate normal distribution is given (it requires 5 parameters: μX, μY, σX, σY, ρX,Y)

The above figure is taken from Montgomery and Runger, 2011. 19

19

Özyeğin University – CE 502 Advanced Probability and Statistics for Civil Engineers – Dr. Taner Yılmaz