MODULE-2

RANDOM VARIABLES AND PROCESSES

Contents:

Introduction, Probability, Random variables, Statistical averages,

Random processes, Mean, correlation, and Covariance functions
Power spectral density, Gaussian process, Noise, Narrowband noise.

Probability:

Probability theory is rooted in phenomena that, explicitly or implicitly,

can be modelled by an experiment with an outcome that is subject to
chance. Moreover, if the experiment is repeated, the outcome can
differ because of the influence of an underlying random phenomenon
or chance mechanism. Such an experiment is referred to as a random
experiment.

For example, the experiment may be the observation of the result of

tossing a fair coin. In this experiment, the possible outcomes of a trial
are “heads” or “tails.”

There are two approaches to the definition of probability. The first

approach is based on the relative frequency of occurrence: in n trials
of a random experiment, if we expect an event A to occur m times,
then we assign the probability m/n to the event A. This definition of
probability is straightforward to apply in games of chance and many
engineering situations.

However, there are situations where experiments are not repeatable

and yet the concept of probability has intuitive applicability. In this
second case, we use a more general definition of probability based on
set theory and a set of related mathematical axioms. In situations
where the experiment is repeatable, the set theory approach agrees
completely with the relative frequency of occurrence approach.

In general, when we perform a random experiment, it is natural for us

to be aware of the various outcomes that are likely to arise. In this
context, it is convenient to think of an experiment and its possible
outcomes as defining a space and its points. If an experiment has K
possible outcomes, then for the kth possible outcome we have a point
called the sample point, which we denote by sk. With this basic
framework, we make the following definitions:

• The set of all possible outcomes of the experiment is called the

sample space, which we denote by S.
• An event corresponds to either a single sample point or a set of
sample points in the space S.
• A single sample point is called an elementary event.
• The entire sample space S is called the sure event, and the null
set f is called the null or impossible event.
 • Two events are mutually exclusive if the occurrence of one event
precludes the occurrence of the other event.

The sample space S may be discrete with a countable number of

outcomes, such as the outcomes when tossing a die. Alternatively, the
sample space may be continuous, such as the voltage measured at the
output of a noise source.

A probability measure P is a function that assigns a non-negative

number to an event A in the sample space S and satisfies the following
three properties (axioms):

This abstract definition of a probability system is illustrated in Figure 1.

 Figure 1 Illustration of the relationship between sample space,
events, and probability.

The sample space S is mapped to events via the random experiment.

The events may be elementary outcomes of the sample space or larger
subsets of the sample space. The probability function assigns a value
between 0 and 1 to each of these events. The probability value is not
unique to the event; exclusive events may be assigned the same
probability. However, the probability of the union of all events— that
is, the sure event— is always unity.

The three axioms and their relationship to the relative frequency

approach may be illustrated by the Venn diagram of Figure 2.
 Figure 2 Venn diagram presenting a geometric interpretation of the
three axioms of probability.

If we equate P to a measure of the area in the Venn diagram with the

total area of S equal to one, then the axioms are simple statements of
familiar geometric results regarding area.

The following properties of probability measure P may be derived

from the above axioms:
CONDITIONAL PROBABILITY:

Suppose we perform an experiment that involves a pair of events A

and B. Let P[B|A] denote the probability of event B, given that event
A has occurred. The probability P[B|A] is called the conditional
probability of B given A. Assuming that A has nonzero probability, the
conditional probability P[B|A] is defined by

where P[AՈB] is the joint probability of A and B. We may write Eq. as

we may also write

Accordingly, we may state that the joint probability of two events may
be expressed as the product of the conditional probability of one event
given the other, and the elementary probability of the other.

Situations may exist where the conditional probability P[A|B] and the
probabilities P[A] and P[B] are easily determined directly, but the
conditional probability P[B|A] is desired. From the above Eqs., it
follows that, provided P[A] # 0, we may determine P[B|A] by using
the relation

This relation is a special form of Bayes’ rule.

Suppose that the conditional probability P[B|A] is simply equal to the

elementary probability of occurrence of event B, that is, P[B|A] =
P[B]
Under this condition, the probability of occurrence of the joint event
A DB is equal to the
product of the elementary probabilities of the events A and B:
P[AՈB] = P[A]P[B]
so that P[A|B] = P[A]

That is, the conditional probability of event A, assuming the

occurrence of event B, is simply equal to the elementary probability of
event A. We thus see that in this case a knowledge of the occurrence
of one event tells us no more about the probability of occurrence of
the other event than we knew without that knowledge. Events A and
B that satisfy this condition are said to be statistically independent.

EXAMPLE: Binary Symmetric Channel

Consider a discrete memoryless channel used to transmit binary data.
The channel is said to be discrete in that it is designed to handle
discrete messages. It is memoryless in the sense that the channel
output at any time depends only on the channel input at that time.
Owing to the unavoidable presence of noise in the channel, errors are
made in the received binary data stream. Specifically, when symbol 1
is sent, occasionally an error is made and symbol 0 is received and vice
versa. The channel is assumed to be symmetric, which means that the
probability of receiving symbol 1 when symbol 0 is sent is the same as
the probability of receiving symbol 0 when symbol 1 is sent.

Sol:
To describe the probabilistic nature of this channel fully, we need two
sets of probabilities.
Accordingly, we may use the transition probability diagram shown in
Figure 3 to represent the binary communication channel specified in
this example; the term “transition probability” refers to the
conditional probability of error. Figure 3 clearly depicts the (assumed)
symmetric nature of the channel; hence, the name “binary symmetric
channel.”
Figure 3 Transition probability diagram of binary symmetric channel.

From Figure 3, we deduce the following results:

RANDOM VARIABLES:

A function whose domain is a sample space and whose range is a set

of real numbers is called a random variable of the experiment. That is,
for events in ξ, a random variable assigns a subset of the real line. Thus,
if the outcome of the experiment is s, we denote the random variable
as X(s) or just X. Note that X is a function, even if it is, for historical
reasons, called a random variable. We denote a particular outcome of
a random experiment by x; that is, X(sk)=x.

The concept of a random variable is illustrated in Figure, where we

have suppressed the events but show subsets of the sample space
being mapped directly to a subset or the real line. The probability
function applies to this random variable in exactly the same manner
that it applies to the underlying events.
 Figure 4 Illustration of the relationship between sample space,
random variable, and probability.

let us consider the random variable X and the probability of the event
X ≤ x. We denote this probability by P[X ≤ x]. It is apparent that this
probability is a function of the dummy variable x. To simplify our
notation, we write Fx(x)= f [X ≤ x].

The function Fx(x) is called the cumulative distribution function (cdf )

or simply the distribution function of the random variable X. Note that
Fx(x) is a function of x, not of the random variable
X. However, it depends on the assignment of the random variable X,
which accounts for the use of X as subscript. For any point x, the
distribution function Fx(x) expresses a probability.

The distribution function Fx(x) has the following properties, which

follow directly from Eq. Fx(x)= f [X ≤ x]:
 • The distribution function Fx(x) is bounded between zero and one.
• The distribution function Fx(x) is a monotone-nondecreasing
function of x; that is,

STATISTICAL AVERAGES:
The expected value or mean of a random variable X is defined by

where E denotes the statistical expectation operator. That is, the

mean µx locates the center of gravity of the area under the
probability density curve of the random variable X.

FUNCTION OF A RANDOM VARIABLE:

Let X denote a random variable, and let g(X) denote a real-valued
function defined on the
real line. The quantity obtained by letting the argument of the
function g( X ) be a random
variable is also a random variable which we denote as Y = g(X )
To find the expected value of the random variable Y, we could of
course find the probability density function fY{ y) and then apply the
standard formula
A simpler procedure, however, is to write

MOMENTS:
For the special case of g( X) = Xn, using Eq. we obtain the nth moment
of the probability distribution of the random variable A; that is,

Mean-square value of X:

We may also define central moments, which are simply the moments
of the difference between a random variable X and its mean µx. Thus,
the nth central moment is

For n = 1, the central moment is, of course, zero, whereas for n = 2 the
second central moment is referred to as the variance of the random
variable X, written as
JOINT MOMENTS:
Consider next a pair of random variables X and Y.A set of statistical
averages of importance in this case are the joint moments, namely,
the expected value of XiYk, where i and k may assume any positive
integer values. We may thus write

A joint moment of particular importance is the correlation defined by

E[XY], which corresponds to i = k = 1 in above Eq.
The correlation of the centered random variables X - E[X| and Y - E[Y],
that is, the joint moment

is called the covariance of X and Y. Letting µx = E[X] and µy = E[Y], we

may expand above Eq. to obtain the result

Let 𝜎𝑋2 and 𝜎𝑌2 denote the variances of X and Y, respectively. Then the
covariance of X and Y, normalized with respect to σxσy,is called the
correlation coefficient of X and Y:
We say that the two random variables X and Y are uncorrelated if and
only if their covariance is zero, that is, if and only if

We say that they are orthogonal if and only if their correlation is zero,
that is, if and only if

RANDOM PROCESSES:
MEAN, CORRELATION, AND COVARIANCE FUNCTIONS:

Consider a random process X(t). We define the mean of the process

X(t) as the expectation of the random variable obtained by observing
the process at some time t,as shown by

Autocorrelation function of the process X(t)

PROPERTIES OF THE AUTOCORRELATION FUNCTION:

For convenience of notation, we redefine the autocorrelation function

of a stationary process
X(t) as

This autocorrelation function has several important properties:

1. The mean-square value of the process may be obtained from
Rx(τ) simply by putting x = 0 in Eq., as shown by

2. The autocorrelation function Rx(τ) is an even function of x, that

is,

3. The autocorrelation function Rx(τ) has its maximum magnitude

at x = 0, that is,

POWER SPECTRAL DENSITY:

The Fourier transform of the autocorrelation function is called the

power spectral density Sx(f) of the random process X(t).
The power spectral density Sx( f ) and the autocorrelation function
RX(τ) of a widesense
stationary random process X(t) form a Fourier-transform pair with f
and τ as the variables of interest, as shown by the pair of relations:

PROPERTIES OF THE POWER SPECTRAL DENSITY:

PROPERTY 1
The zero-frequency value of the power spectral density of a wide-
sense stationary random process equals the total area under the
graph of the autocorrelation function; that is,

PROPERTY 2
The mean-square value of a wide-sense stationary random process
equals the total area under the graph of the power spectral density;
that is,
PROPERTY 3
The power spectral density of a wide-sense stationary random
process is always nonnegative; that is,

PROPERTY 4
The power spectral density of a real-valued random process is an
even function of frequency; that is,

GAUSSIAN PROCESS:

PROPERTY 1
If a Gaussian process X(t) is applied to a stable linear filter, then the
random process Y(t) developed at the output of the filter is also
Gaussian.
Let ℎ(𝑡) represent the impulse response of the linear filter. The output
𝑌(𝑡) is given by the convolution of the input 𝑋(𝑡) with the filter's
impulse response:

Since the input X(t) is Gaussian and the filter is linear and stable,
Y(t) will also be a Gaussian process, as it is a weighted sum of
Gaussian random variables.
This property plays a fundamental role in noise filtering, modulation,
and demodulation in communication systems where Gaussian
processes like thermal noise are often encountered.

PROPERTY 2
Consider the set of random variables or samples X(t{), X(t2 ), X( t n),
obtained by observing a random process X(t) at times th t2,..., If the
process X(t) is Gaussian, then this set of random variables is jointly
Gaussian for any n, with their n-fold joint probability density function
being completely determined by specifying the set of means

The mean of each random variable X(ti) observed at time tit_iti is

denoted as μX(ti) and is given by:

and the set of autocovariance functions

The autocovariance function between any two random variables X(tk)

and X(ti) observed at times tk and ti is denoted as CX(tk,ti). It
measures the covariance between these two random variables:
This equation quantifies the linear relationship between the
fluctuations of X(tk) and X(ti) around their respective means.

For a Gaussian process X(t), the set of random variables

X(t1),X(t2),…,X(tn)X(t_1), X(t_2), , which are obtained by observing
the process at times t1,t2,…,tn, , are jointly Gaussian. This means
their joint probability density function (PDF) is fully determined by
two things:
1. The set of means μX(ti)}, which are the expected values of the
variables at the respective times.
2. The set of autocovariance functions CX(tk,ti), which describe
how the values of the process at different times are correlated.
Because the process is Gaussian, knowing the mean vector and the
autocovariance matrix for the random variables is sufficient to
completely specify their joint distribution.

PROPERTY 3
If a Gaussian process is wide-sense stationary, then the process is also
stationary in the strict sense.
PROPERTY 4
If the random variables X(t{), X(t2 ), X(t n), obtained by sampling a
Gaussian process X( t) at times t1, t2,……, tm are uncorrelated, that
is,
NOISE:
The term noise is used customarily to designate unwanted waves that
tend to disturb the transmission and processing of signals in
communication systems and over which we have incomplete control.

SHOT NOISE:
Shot noise arises in electronic devices such as diodes and transistors
because of the discrete nature of current flow in these devices. For
example, in a photodetector circuit a current pulse is generated every
time an electron is emitted by the cathode due to incident light from
a source of constant intensity. The electrons are naturally emitted at
random times denoted by tk, where -oo < k < oo. It is assumed that
the random emissions of electrons have been going on for a long
time. Thus, the total current flowing through the photodetector may
be modeled as an infinite sum of current pulses, as shown by

where h(t - τk ) is the current pulse generated at time tk. The process
X(t) defined by above Eq. is a stationary process, called shot noise.
THERMAL NOISE:
Thermal noise is the name given to the electrical noise arising from
the random motion of electrons in a conductor.
Thermal noise, also known as Johnson-Nyquist noise, is an inherent
electrical noise generated by the random motion of electrons in a
conductor due to thermal agitation. This type of noise is present in all
electrical components and circuits, and it increases with temperature.
Thermal noise is characterized by a flat frequency spectrum, meaning
it is equally distributed across all frequencies, making it a type of
white noise.

The power of thermal noise in a resistor can be quantified using the

formula:
P = kTB
Where:
P is the noise power in watts,
-k is Boltzmann's constant (\(1.38 \times 10^{-23} \, \text{J/K}\)),
- T is the absolute temperature of the conductor in kelvins (K),
- B is the bandwidth in hertz over which the noise is measured.

This noise sets a fundamental limit on the performance of

communication systems, particularly in terms of signal-to-noise ratio
(SNR).
WHITE NOISE:
The noise analysis of communication systems is customarily based on
an idealized form of noise called white noise, the power spectral
density of which is independent of the operating frequency.
White noise is an idealized form of noise used in the analysis of
communication systems. It is characterized by having a constant power
spectral density (PSD) across all frequencies, meaning that its power is
uniformly distributed over the entire frequency spectrum. This makes
white noise a theoretical model of random noise that is independent
of frequency, which simplifies the analysis of noise in systems.

In practice, real-world noise sources can approximate white noise over

certain bandwidths, but no physical noise source can have infinite
bandwidth, as white noise theoretically suggests. Nevertheless, white
noise serves as a useful model in many applications, particularly in
systems design and signal processing.

In communication systems, white noise is often used to model thermal

noise or other background noise sources, allowing engineers to
calculate key metrics like the signal-to-noise ratio (SNR) and to predict
the performance of receivers and filters.

The power spectral density of white noise is typically represented as:

S(f) = N0 / 2
Where:
- S(f) is the power spectral density,
- N0 is the noise power per unit of bandwidth (in watts per hertz).

This constant PSD simplifies the mathematical treatment of noise in

signal processing and communication theory.

NARROWBAND NOISE:
The receiver of a communication system usually includes some
provision for preprocessing the received signal. The preprocessing may
take the form of a narrowband filter whose bandwidth is just large
enough to pass the modulated component of the received signal
essentially undistorted but not so large as to admit excessive noise
through the receiver. The noise process appearing at the output of such
a filter is called narrowband noise.

Narrowband noise refers to the noise present at the output of a

narrowband filter in a communication system. When a receiver applies
preprocessing, such as passing the received signal through a
narrowband filter, the filter's purpose is to eliminate unnecessary noise
while allowing the essential modulated signal to pass with minimal
distortion. The resulting noise that passes through this filter, due to its
reduced bandwidth, is what is known as narrowband noise.

Unlike white noise, which is spread across a wide frequency range,

narrowband noise is confined to the specific bandwidth of the filter,
typically centered around the carrier frequency of the received signal.
This type of noise often takes on a more structured form, making it
more predictable and, in some cases, easier to handle in
communication systems.

Narrowband noise can be represented as a combination of two

components:
- In-phase (I) component: Corresponds to the noise component that is
in phase with the carrier signal.
- Quadrature (Q) component: Corresponds to the noise component
that is 90 degrees out of phase with the carrier signal.

Together, these components create a complex noise process that can

be modeled as:
n(t) = nI(t) \cos(2 \pi fct) - n_Q(t) \sin(2 pi fc t)
Where:
- n(t) is the narrowband noise,
- nI(t) and nQ(t) are the in-phase and quadrature components,
respectively,
- fc is the carrier frequency.

By focusing the receiver's bandwidth on a narrow range, narrowband

noise allows for improved performance in communication systems by
reducing the total amount of noise admitted, though it remains
present and must be managed effectively for optimal system
operation.