Stochastic Process Definition A stochastic process, also known as a random process, is a collection of random variables that are indexed by some mathematical set. Each probabiity and random process are uniquely associated with an element in the sat. The index set is the set used to index the random variables. The index set was traditionally a subset of the real line, such as the natural numbers, which provided the index set with time interpretation ‘Stochastic Process Meaning is one that has a system for which there are observations at certain times, and that the outcome, that is, the observed value at each time is a random variable. Each random variable in the collection of the values is taken from the same mathematical space, known as the state space This state-space could be the integers, the real line, or n-dimensional Euclidean space, for example. A stochastic process's increment is the amount that a stochastic process changes between two. index values, which are frequently interpreted as two points in time. Because of its randomness, a stochastic process can have many outcomes, and a single outcome of a stochastic process is known as, among ather things, a sample function or realization Classification A stochastic process can be classified in a variety of ways, such as by its state space, index set, or the dependence among random variables and stochastic proces ses are classified in a single way, the cardinality of the index set and the state space. When expressed in terms of time, a stochastic process is said to be in discrete-time if its index set contains a finite or countable number of elements, such as a finite set of numbers, the set of integers, or the natural numbers. Time is said to be continuous if the mdex set is some interval of the real line Discrete-tme stochastic processes and continuous-time stochastic processes are the twa types of stochastic processes The continuous-time stochastic processes require more advanced mathematical techniques and knowledge, particularly because the index set is uncountable, discrete-time stochastic processes are Considered easier to study If the index set consists of integers or a subset of them, the stochastic process ts also known as a random sequence. If the state space is made up of integers or natural numbers, the stochastic process is known as @ discrete or integer-valued stochastic process. If the state space is the real line, the stochastic process is known as a real-valued stochastic process or a process with continuous state space if the state space is n- dimensional Euclidean space, the stochastic process is known as a f-dimensional vector process of 1 vector process. Examples You can study all the theory of probability and random processes mentioned below in the brief, by referring to the book Essentials of stochastic processes. Types of Stochastic Processes The probability of any event depends upon vanous extemal factors The mathematical interpretation of these factors and using it to calculate the possibility of such an event is studied under the chapter of Probability in Mathematics. According to probability theory to find a definite number for the occurrence of any event all the random variables are counted These random variables are put together in a set then It is called a stochastic process. For mathematical models used for understanding any phenomenon or system that results from a very random behavior, Stochastic processes are used. Such phenomena can occur anywhere anytime in this constantly active and changing world To make the learning of the Stochastic process easier it has been classified into various categories. If the sample space consists of @ finite set of numbers or a countable number of elements such as integers or the natural numbers or any real values then ft remains in a discrete time. For an uncountable index set, the process gets more complex It will be taught in higher classes in this article, we will deal with discrete-time stochastic processes. Various types of processes that constitute the Stochastic processes are as follows Bernoulli Process ‘The Bemoulli process is one of the simplest stochastic processes. It is a sequence of independent and identically distributed (iid) random variables, where each random variable has a probability of one or ze10, say one with probability P and zero with probability 1-P This process ts analogous to repeatedly flipping a coin, where the probability of getting a head is P and its value is one, and the probability of getting a tall js zero. In other words, a Bernoulli process is a series of iid Bernoulli random variables, with each coin flip representing a Bernoulli tial[signin L download POF | NCERTSelutens CBSE SY COBEStutyMateral Y= Te: > Random Walk Random walks are stochastic processes that are typically defined as sums of id random variables or random vectors in Euclidean space, implying thet they are discrete-time processes. However, some people use the term to refer to processes that change in real-time, such as the Wiener process used in finance, which has caused some confusion and led to criticism Other types of random walks are defined so that their state spaces can be other mathematical objects, such as lattices and groups, and they are widely studied and used in a variety of disciplines. The simple random walk is @ classic example of a random walk. It is a stochastic process in discrete time with integers as the state space and is based on a Bernoulli process, with each Bernoulli variable taking either a postive or negative value In other words, the simple random walk oceurs on integers, and its value increases by one with probability or decreases by one with probabilay 1-p, so the mdex set of this random walk is natural numbers, while fs state space Is mtegers If p=0'5, This random walk is referred to as an asymmetric random walk Wiener Process The Wiener process Is a stationary stochastic process with independently distributed mcrements that are usually distributed depending on their size The Wiener process is named after Norbert Wiener, who demonstrated ts mathematical existence, but @ is also known as the Brownian motion process or simply Brownian motion due to its historical significance as a model for Brownian movement in liquids (Image will be Uploaded Soon) The Wiener process, which plays a central role in probability theory, is frequently regarded as the most important and studied stochastic process, with connections to other stochastic processes. It has a continuous index set and states space because its index set and state spaces are nonnegative numbers and real numbers, respectively However, the process can be defined more broadly so that its state space is -dimensional Euclidean space, The resulting Wiener or Brownian motion process is said to have zero drift if the mean of any increment Is zero. If the mean of the increment between any two pomts in time equals the time difference multiplied by some constant j, that ts a real number, the resulting stochastic process is said tohave drift y Almost certainly, a Wiener process sample path is continuous everywhere but differentiable nowhere It can be thought of as a continuous variation on the simple random walk Donsker's theorem or invariance principle, also known as the functional central lima theorem, is concerned wah the mathematical limt of other stochastic processes, such as certain random walks rescaled The Wiener process belongs to several important families of stochastic processes, including the Markov, Lévy, and Gaussian families The process has 3 wide range of applications and is the primary stochastic process in stochastic calculus. It is crucial im quantitative finance, where i is used in models such as the Black-Scholes-Merton The process is also used as a mathematical model for various random phenomene in a variety of fields, including the majorty of natural sciences and some branches of social sciences. Poisson Process ‘The Poisson process is @ stochastic process with various forms and definitions. is a counting process, which is a stochastic process that represents the random number of points or events up to a certain time. The number of process points located in the interval from zero to some given time is a Poisson random variable that is dependent on that time and some parameter This process's state space is made up of natural numbers, and its index set is made up of non-negative numbers. This process is also known as the Poisson counting process because it can be interpreted as a counting process. ‘A homogeneous Poisson process is ane in which 2 Poisson process is defined by @ single postive constant, The homogeneous Poisson pracess belongs to the same class of stochastic processes as the Markov and Lévy processes There are several ways to define and generalize the homogeneous Poisson process. This stochastic process is also known as the Poisson stationary process because tts index set is the real fine If the Poisson process's parameter constant is replaced wah a nonnegative mtegrable function of t. The resulting process is known as an inhomogeneous or nonhomogeneous Poisson process because the average density of the process's points is no longer constant. The Poisson process, which is a fundamental process in queueing theory, is an important process for mathematical models, where t finds applications for models, of events randomly occurring in certain time windows.Correlation Coefficient The correlation coefficient is a statistical concept which helps in establishing a relation between predicted and actual values obtained in a statistical experiment. The calculated value of the correlation coefficient explains the exactness between the predicted and actual values. The correlation coefficient value always lies between -] and +1. If the correlation coefficient value is positive, then there is a similar and identical relation between the two variables. Else, it indicates the dissimilarity between the two variables. The covariance of two variables divided by the product of their standard deviations gives Pearson's correlation coefficient. It is usually represented by p (rho). p (X,Y) = cov (X,Y) / oX.o. Here, cov is the covariance. ox is the standard deviation of X, and oY is the standard deviation of Y. The given equation for the correlation coefficient can be expressed in terms of means and expectations. X=w.MY=n, p(x,y) = penton ux and py are the mean of x and the mean of y, respectively. E is the expectation.Assumptions of Karl Pearson’‘s Correlation Coefficient The assumptions and requirements for calculating Pearson’s correlation coefficient are as follows: 1. The data set which is to be correlated should approximate the normal distribution. If the data is normally distributed, then the data points tend to lie closer to the mean. 2. ‘'Homoscedastic’ is a Greek word meaning ‘able to disperse’. Homoscedasticity means ‘equal variances’. For all the values of the independent variable, the error term is the same. Suppose the error term is smaller for a certain set of values of the independent variable and larger for another set of values; then, homoscedasticity is violated. It can be checked visually through a scatter plot. The data is said to be homoscedastic if the points lie equally on both sides of the line of best fit. 3. When the data follow a linear relationship, it is said to be linear. If the data points are in the form of a straight line on the scatter plot, then the data satisfies the condition of linearity. 4. The variables which can take any value in an interval are continuous variables. The data set must contain continuous variables to compute the Pearson correlation coefficient. If one of the data sets is ordinal,Pearson Correlation Coefficient Formula The linear correlation coefficient defines the degree of relation between two variables and is denoted by “r”’. It is also called a cross-correlation coefficient, as it predicts the relation between two quantities. Now, let us proceed to a statistical way of calculating the correlation coefficient. If x & y are the two variables of discussion, then the correlation coefficient can be calculated using the formula ES WE EW) hy P= Fy FW Here, n = Number of values or elements Xx = Sum of Ist values list Ly = Sum of 2nd values list >xy = Sum of the product of Ist and 2nd values 2x? = Sum of squares of I°* values dy? = Sum of squares of 2™4 values How to Find the Correlation Coefficient Correlation is used almost everywhere inHow to Find the Correlation Coefficient Correlation is used almost everywhere in statistics. Correction illustrates the relationship between two or more variables. It is expressed in the form of a number that is known as the correlation coefficient. There are mainly two types of correlations: ¢ Positive Correlation ¢ Negative Correlation - Positive The value of one Correlation variable increases linearly with an increase in another variable. This indicates asimilar relation between both variables. So its correlation coefficient would be positive or1in this case. Negative When thereis a Correlation decrease in the values SS of one variable with an =, increase in the values of another variable, In that case, the correlation coefficient would be negative. Zero There is one more Correlation | situation when there is |= orNo no specific relation ee Correlation between two variables. Correlation CoefficientCorrelation Coefficient Properties The correlation coefficient is all about establishing relationships between two variables. Some properties of the correlation coefficient are as follows: 1) The correlation coefficient remains in the same measurement as in which the two variables. 2) The sign that correlations of coefficient have will always be the same as the variance. 3) The numerical value of the correlation of coefficient will be between -1 to + 1. It is known as the real number value. 4) The negative value of the coefficient suggests that the correlation is strong and negative. And if ‘r’ goes on approaching -1, then it means that the relationship is going towards the negative side. When ‘r approaches the side of + 1, then it means the relationship is strong and positive. By this, we can say that if +1 is the result of the correlation, then the relationship is in a positive state. 5) The weak correlation is signalled when the coefficient of correlation approaches zero. When ‘r is near zero, then we can deduce that the relationship is weak. 6) Correlation coefficient can be very dicey because we cannot sav whether the6) Correlation coefficient can be very dicey because we cannot say whether the participants are truthful or not. The coefficient of correlation is not affected when we interchange the two variables. 7) The coefficient of correlation is a pure number without the effect of any units on it. It also does not get affected when we add the same number to all the values of one variable. We can multiply all the variables by the same positive number. It does not affect the correlation coefficient. As we discussed, ‘r is not affected by any unit because 'r’ is a scale-invariant. 8) We use correlation for measuring the association, but that does not mean we are talking about causation. By this, we simply mean that when we are correlating the two variables, then it might be the possibility that the third variable may be influencing them. Examples on_ Correlation Coefficient Example 1: Calculate the correlation coefficient of the given data. x 50 51 52 53 54 | | y 31 32 3.3 3.4 3.5 | Solution: Here,n =5Example 1: Calculate the correlation coefficient of the given data. x 50 51 52 53 54 | y |3a 32 33 34 3.5 Solution: Here,n =5 x 50 5I 52 53 54 y 31 3.2 33 3.4 3.5 xy | 155 163.2 171.6 180.2 189 x2 | 2500 ‘| 2601 2704 | 2809 | 2916 y2 | 9.61 10.24 | 1089 | 11.56 12.25 Xx = 260 dy =16.5 xy = 859 rx? = 13530 dy? = 54.55 By substituting all the values in the formula, we get r = 1. This shows a positive correlation coefficient. Example 2: Calculate the correlation coefficient of the given data. ly |. lon 12 91 7Cramer's V Correlation Cramer’s V correlation is identical to the Pearson correlation coefficient. Pearson correlation coefficient is used to find the correlation between variables, whereas Cramer's V is used to calculate correlation in tables with more than 2 x 2 columns and rows. It varies between 0 and 1. 0 indicates less association between the variables, whereas | indicates a very strong association. Cramer's V .25 or higher - Very strong relationship 15 to .25 — Strong relationship 1 to .15 - Moderate relationship .06 to 10 — Weak relationship .01 to .05 — No or negligible relationship Other types of correlation are as follows: 1] Concordance Correlation Coefficient It measures the bivariate pairs of observations comparative to a “gold standard” measurement. 2] Intraclass Correlation It measures the reliability of the data that are collected as groups. 3] Kendall's Tau It is a non-parametric measure of relationships between the columns of ranked data. Al Maran‘e |Other types of correlation are as follows: 1] Concordance Correlation Coefficient It measures the bivariate pairs of observations comparative to a “gold standard” measurement. 2] intraclass Correlation It measures the reliability of the data that are collected as groups. 3] Kendall's Tau It is a non-parametric measure of relationships between the columns of ranked data. 4] Moran’s | It measures the overall spatial autocorrelation of the data set. 5] Partial Correlation It measures the strength of a relationship between two variables while controlling for the effect of one or more other variables. 6] Phi Coefficient It measures the association between two binary variables. 7] Point Biserial Correlation: It is a special case of Pearson's correlation coefficient. It measures the relationship between two variables: a] One continuous variable. b] One naturally binary variable. 8] Spearman Rank Correlation It is the nonparametric version of theCovariance In statistics and probability theory, covariance deals with the joint variability of two random variables: x and y. Generally, it is treated as a statistical tool used to define the relationship between two variables. In this article, covariance meaning, formula, and its relation with correlation are given in detail. Covariance Meaning Covariance is a measure of the relationship between two random variables and to what extent, they change together. Or we can say, in other words, it defines the changes between the two variables, such that change in one variable is equal to change in another variable. This is the property of a function of maintaining its form when the variables are linearly transformed. Covariance is measured in units, which are calculated by multiplying the units of the two variables. ¢ Variance « Covariance Formula ¢ Covariance Matrix Formula ¢ Correlation Types of Covariance Covariance can have both positive and negative values. Based on this, it has twoTypes of Covariance Covariance can have both positive and negative values. Based on this, it has two types: 1, Positive Covariance 2. Negative Covariance Positive Covariance If the covariance for any two variables is positive, that means, both the variables move in the same direction. Here, the variables show similar behaviour. That means, if the values (greater or lesser) of one variable corresponds to the values of another variable, then they are said to be in positive covariance. Negative Covariance If the covariance for any two variables is negative, that means, both the variables move in the opposite direction. It is the opposite case of positive covariance, where greater values of one variable correspond to lesser values of another variable and vice-versa. Covariance Formula Covariance formula is a statistical formula, used to evaluate the relationship between two variables. It is one of the statistical measurements to know the relationship between the variance between the two variables. Let us say X and Y are any two variables, whose relationship has to beCovariance Formula Covariance formula is a statistical formula, used to evaluate the relationship between two variables. It is one of the statistical measurements to know the relationship between the variance between the two variables. Let us say X and Y are any two variables, whose relationship has to be calculated. Thus the covariance of these two variables is denoted by Cov(X,Y). The formula is given below for both population covariance and sample covariance. mays Population Covariance Formula Cov(x,y) = £(%; - X(y;- Y) N Sampie Covariance Cov(x,y) = (x, - x)(y; - y) N-1 Where, « xi = data value of x ¢ yi = data value of y « x = mean of x *° y=meanofy * N= number of data values. Covariance of X and Y Below figure shows the covariance of X and Y. mews a2 Vibes,Covariance of X and Y Below figure shows the covariance of X and Y. Besyu's lf, 1% 1& x xX 4 cov(X,Y)>0 cov(X,Y)=0 cov(X,Y)<0 If cov(X, y) is greater than zero, then we can say that the covariance for any two variables is positive and both the variables move in the same direction. If cov(X, Y) is less than zero, then we can say that the covariance for any two variables is negative and both the variables move in the opposite direction. If cov(X, Y) is zero, then we can say that there is no relation between two variables. Correlation Coefficient Formula We have already discussed covariance, which is the evaluation of changes between any two variables. Correlation estimates the depth of the relationship between variables. It is the estimated measure of covariance and is dimensionless. In other words, the correlation coefficient is a constant value always and does not have any units. The relationship between the correlation coefficient and covariance is given by;coefficient and covariance is given by; | cometation.9(.¥) = Cov(X,Y)/ox oy Where: p(X,Y) = correlation between the variables X andy ¢ Cov(X,Y) = covariance between the variables X and Y ¢ oX = standard deviation of the X variable ¢ oY = standard deviation of the Y variable Based on the value of correlation coefficient, we can estimate the type of correlation between the given two variables. Also, the graphical representation of correlation among two variables is given in the below figure. Bowus Strong positive Weak positive Strong negative correlation correlation correlation Weak negative’ Moderate negative No correlation correlation correlation Covariance and Correlation Below table shows the comparison among covariance and correlation in brief. ra r ~Covariance and Correlation Below table shows the comparison among covariance and correlation in brief. Covariance Correlation It ls a Measure to show the extent to which given two random variables change with respect to each other. It is a measure used to describe how strongly the given two random variables are related to each other. It is a measure of correlation. It is defined as the scaled form of covariance. The value of covariance lies between -oo and +oo. The value of correlation lies between -1 and +1. It indicates the direction of the linear relationship between the given two variables. It measures the direction and strength of the linear relationship between the given two variables. Covariance and Variance Covariance and variance both are the terms used in statistics. Variance is the measure of spread of data around its mean value but covariance measures the relation between two random variables. Learn Variance in statistics at BYJU'S. Covariance ExampleCovariance Example Below example helps in better understanding of the covariance of among two variables. Question: Calculate the coefficient of covariance for the following data: | x 2 8 18 | 20 28 30 | | Y 5 12 18 | 23 45 50 | Solution: Number of observations = 6 Mean of X = 17.67 Mean of Y = 25.5 Cov(X, Y) = (%) [(2 - 17.67)(5 - 25.5) + (8 - 17.67) (12 — 25.5) + (18 - 17.67) (18 - 25.5) + (20 - 17.67)(23 - 25.5) + (28 - 17.67)(45 - 25.5) + (30 - 17.67)(50 — 25.5)] = 157.83 Covariance Matrix In statistics and probability theory, a square matrix provides the covariance between each pair of components (or elements) of a given random vector is called a covariance matrix. Any covariance matrix is symmetric and positive semi-definite. The principal diagonal or main diagonal (sometimes a primary diagonal) of this matrix containsCovariance Matrix In statistics and probability theory, a square matrix provides the covariance between each pair of components (or elements) of a given random vector is called a covariance matrix. Any covariance matrix is symmetric and positive semi-definite. The principal diagonal or main diagonal (sometimes a primary diagonal) of this matrix contains variances. That means the covariance of each element with itself. A covariance matrix is also known as the auto-covariance matrix, variance matrix, dispersion matrix, or variance-covariance matrix. Frequently Asked Questions on Covariance Ql What does Covariance mean? Covariance, in statistics, is the measurement of relationships between two random variables. Q2 What is the difference between covariance and correlation? Covariance describes the extent to which one variable is related to another whereas correlation states how strongly the given two random variables are related to each other. Q3 Howto calculate the covariance? To find the covariance, first find the mean of the set of data for two random warinkioeWhat is Autocorrelation? The autocorrelation function of a signal is defined as the measure of similarity or coherence between a signal and its time delayed version. Thus, the autocorrelation is ‘the correlation of a signal with itself. The autocorrelation function is defined separately for energy or aperiodic signals and power or periodic signals. Autocorrelation Function for Energy Signals The autocorrelation function of an energy signal «(t) is defined as - Ru(r) =R)= [20 x (t-7) de f a(ter) a (1) de 00 Where, the variable r is called the delay parameter. Properties of Autocorrelation Function of Energy Signals The properties of autocorrelation function for energy signals are given as follows — Property 1 The autocorrelation function of energy signals exhibits complex conjugate symmetry, which means the real part of autocorrelation function F (7) is an even function of delay parameter | 7) and the imaginary part of R(r) is an odd function of the parameter 7. Thus, R(t) =R' (-7) Property 2 When the delay parameter 7 is increased in either direction, the autocorrelation function R (7) of an energy signal reduces. Hence, when the parameter 7 reduces, the autocorrelation R(7) increases and tt is maximum at 7 = 0 (or at origin), Therefore, |R(r)| << R(0); forallr Property 3 The value of autocorrelation function of an energy signal at the origin (ie, at 7 = 0) is equal to total energy of that signal, ie., R()|,o=B= f le(oPa Property 4 The autocorrelation function R(r) and the ESD (Energy Spectral Density) function (cv) of an energy signal form a Fourier transform pair, ie,R(t) + vw) Autocorrelation Function for Power Signals The autocorrelation function of a power or periodic signal x (#) with any time period T is defined as — am if meee Ro)= jm 5 f 20 r)dt Properties of Autocorrelation Function for Power Signals The properties of autocorrelation function for power signals are given as follows — Property 1 The autocorrelation function of power or periodic signals exhibits complex conjugate symmetry property, that is, R(r) = (7) Property 2 The value of the autocorrelation function for a power signal at origin jie., at r = 0) is equal to the average power (P) of that signal, ie. R(0)=P Property 3 When the delay parameter r reduces, the autocorrelation R(r) of the power signal increases and it is maximum at the origin, ie., |R(r)| < R(0) Property 4 The autocorrelation function R(r) be periodic with the same time period as the power {or periodic) signal itself ie, R(r)=R(r£ nT); where, n=, 2,3, --- Property 5 The autocorrelation function R(r) and the PSD (Power Spectral Density) function S (w) of @ power signal form a Fourier transform pair, ie., BIl, A RANDOM BINARY PROCESS A random binary process V(t) with unit amplitude may be represented as [8] V(t) = (-1)'O; ti (1-2ng|t|), [tT] < 20 (3) i.e. the autocorrelation function remains above its tangent drawn at the origin. Hence, the width of the autocorrelation function Ry(7) depends on the mean zero-crossing rate no, and the function Ry(7) becomes narrower as the rate ng is increasing. The mean zero-crossing rate, m9, of a random binary process V(t) can be determined from arr no = im, Ry(r) = 3 jim Ry (r)- 4 Since 4,(0+)=—2n9 and Ri(0-) =2no (5) the second derivative Ry,(0) at the origin of the autocorrelation function Ry (7) can be represented by —4n94(7), where 5(-) is the Dirac delta function.10.2.1 Power Spectral Density So far, we have studied random processes in the time domain. Itis often very useful to study random processes in the frequency domain as well To do this, we need to use the Fourier transform. Here, we will assume that you are familiar with the Fourier transform. A brief review of the Fourier transform and its properties is given in the appendix. Consider a WSS random process X(t) with autocorrelation function Ry (7). We define the Power Spectral Density (PSD) of X(t) as the Fourier transform of R(T). We show the PSD of X(t), by Sx(f). More specifeally, we can write Sx(f) = F{Rx(r)} = / Rx(r)e 2% dr, where j = V1 Power Spectral Density(PSD). Sx(f) = F{Rx(7)} = / ” Ry(rje 20 dr, where j = V/ From this definition, we can conclude that Ry (7) can be obtained by the inverse Fourier transform f Sx(f) Thatis 2 Ry(7) = F'{Sx(f} = [seiner af. As we have seen before, if X(t) is a real-valued random process, then Rx (r) is an even, real-valued function of T. From the properties of the Fourier transform, we conclude that S( f) is alsoreal- valued and an even function of f Also, from what wewill discuss later on, we can conclude that Sx(f) isnon-negatve for all f 1. Sx(—f) = Sx(f), for all fi, 2. Sx(f) > 0, for all f Before going any further, let's try to understand the idea behind the PSD Todo so, let's choose T = 0. Weknow that expected power in X(#) 1s given by B[X(t)?] = Rx(0) = [ Sx( flew! df = [= seins. We conclude that the expected power in X(t) can be obtained by integrating the PSDof X(t). This faci helps us to understand why Sx (f) is called the power spectral density In fact, as we will see shortly, we can find the expected power of X(t) in a specific frequency range by integrating the PSD. over that specific range. The expected power in X(t) can be obtained as B(x] = Rx(0) = [sein ay. Example 10.13 Consider a WSS random process X(t) with Rx(r where @ is a positive real number. Find the PSD of X(t)What is Markov Chain? Markov chain refers to mathematical system that undergoes transitions from one state to other as per some probabilistic rules. The defining characteristic of this chain is that no matter how system arrived at its current state, the possible future states are fixed. The probability of transitioning to any particular state relies solely on current state and time elapsed. For example, consider a weather model that predicts the weather for a particular region over the next week. Each day, the weather can be either sunny, cloudy, or rainy. We can represent the weather for each day as a state in the Markov chain. If we know that the weather today is sunny, we can use the probabilistic rules of the Markov chain to predict the weather tomorrow. For example, if the weather has a 60% chance of being sunny, a 30% chance of being cloudy, anda 10% chance of being rainy, then we can say that there is a 60% chance that the weather will be sunny tomorrow. Explore deep learning courses Types of Markov Chain There are several different types of Markov chains, including:Types of Markov Chain There are several different types of Markov chains, including: 1. Finite Markov chains: These are Markov chains with a finite number of states. The transition probabilities between states are fixed, and system will then eventually reach a steady state in which the probabilities of being in each state become constant. An example of a finite state Markov chain might be a model of a traffic light, with three states (red, yellow, green) and transitions governed by the rules of the traffic light. 2. Infinite Markov chains: These are Markov chains with an infinite number of states. The transition probabilities between states are fixed, but the system may not reach a steady state. For example, a model of the spread of a virus through a population. The states of the Markov chain represent the number of people who have been infected at any given time, and the transitions between states are governed by the rate of infection and the rate of recovery. 3. Continuous-time Markov chains (CTMC): These are Markov chains in which the transitions between states occur at random times rather than at discrete time intervals. The transition probabilities are defined by rate functions rather than probabilities. For example, the state of a CTMC could represent the number of customers in a store at a giventime, and the transitions between states could represent the arrival and departure of customers. The probability of a system being in a particular state (e.g., the probability that there are a certain number of customers in the store) can be calculated using the CTMC model. 4. Discrete-time Markov chains (DTMC): These are Markov chains in which the transitions between states occur at discrete time intervals. The transition probabilities are defined by a transition matrix. For example, the state of a DTMC could represent the weather on a particular day (e.g., sunny, cloudy, or rainy), and the transitions between states could represent the change in weather from one day to the next. The probability of a system being in a particular state (e.g., the probability of it being sunny on a particular day) can be calculated using the DTMC model. Properties of Markov Chain There are also several properties that Markov chains can have, including: 1. Irreducibility: Markov chain is irreducible when it is possible to reach any state from any other state in a finite number of steps. 2. Aperiodicity: A Markov chain is aperiodic when it is possible to reach any state from any other state in a finite number of steps, regardless of the starting state.3. Recurrence: A state in a Markov chain is recurrent if it is possible to return to that state in a finite number of steps. 4. Transience: A state in a Markov chain is transient if it is not possible to return to that state in a finite number of steps. 5. Ergodicity: A Markov chain is ergodic if it is both irreducible and aperiodic and if the long- term behaviour of the system is independent of the starting state. 6. Reversibility: A Markov chain is reversible if probability of transitioning from one state to another is equal to the probability of transitioning from that state back to the original state. Applications of Markov Chains One common application of Markov chains is in the field of economics, where they can be used to model the behaviour of stock prices. In this case, the possible states of the system might represent upward or downward trends in the market, and the transitions between states might be influenced by various economic factors such as interest rates and the performance of individual stocks. By analyzing the probability of transitioning between different states, it is possible to make predictions about the future behaviour of the market. Markov chains can be used for modeling biological systems, such as the spread of a disease througha population. In such case, the states of the system might represent the number of individuals inapopulation who are susceptible, infected, or immune to the disease, and the transitions between states might be influenced by several factors such as the rate of infection and the effectiveness of a vaccine. By analyzing the probability of transitioning between different states, it is possible to make predictions about the future spread of the disease and the effectiveness of various control measures, In order to analyze and predict the behaviour of a Markov chain, it is necessary to define the states of the system and the transitions between those states, as well as the probability of transitioning between each pair of states. This information can be represented using a transition matrix, which is a matrix of numbers representing the probability of transitioning from one state to another. For example, consider a simple Markov chain with three states A, B, and C, and the following transition matrix:transition matrix: This matrix represents the probability of transitioning from one state to another. For example, the value in the second row and third column (0.7) represents the probability of transitioning from state B to state C. In order to analyse the behaviour of this Markov chain over time, it is necessary to define a starting state and calculate the probabilities of transitioning to other states at each time step. This can be done using matrix multiplication. For example, if the Fe ee i Eee nlusing matrix multiplication. For example, if the starting state is A, the probability of being in state B after one time step can be calculated as follows: 0.5 This is obtained by multiplying the transition matrix by the starting state vector:To calculate the probability of being in state B after two-time steps, we can simply multiply the transition matrix by itself and then multiply the result by the starting state vector: afalc ala|c alalc ofo ojo alo alo 5] 5 s|s alo | oas | 3 0 o} * 0 o}| = 8 o| . 8 o|. B/ °F} o | %8 3 7 3 7 7 o}ojo o;ol}o c 2 0.56 ° c : c 2]}6]2 2]}6}2 a}le|ec A alo jou |] x A = > 8| 0. 2a 03 06 a ee ee 02 02 c| ° | 066 | ° This process can be repeated to calculate the probabilities of being in each state at any time step. One important property of Markov chains is that they will eventually reach a steady state, where the probabilities of being in each state no longer change. This steady state can be calculated using matrix multiplication as well, by finding the eigenvectors of the transition matrix and normalizing them. The resulting vector represents the long-term behaviour of the Markov chain. sepa traumas oecety amdation pwam.temsranaite aiittmnm, aotehMarkov chains have many other properties and applications, and there are many algorithms and techniques for analysing and manipulating them. They are a useful tool for modelling and analysing systems with a finite number of states and transitions between those states, and have many applications in fields such as economics, biology, and computer science. Markov Chain In Python Here is an example of a simple function that implements a Markov chain: _chain(transition_matrix, state, nu Perform transitions of a Markov for iin range(num_steps) state = np.random ce(len(transition_m| return state This function takes as input a transition matrix and a starting state, and performs a specified number of transitions according to the probabilities in the transition matrix. It uses the ‘np.random.choice’ function from the NumPy library to select the next state according to the probabilities in the transition matrix. Here is an example of how this function could be ticad to cimislate a Markoany chain with three ctatecHere is an example of how this function could be used to simulate a Markov chain with three states: transition_matrix = [[0, ie 1 Cs poh E beech C3 ofc} _chain(transition_matrix, state, num_{ This would perform 5 transitions of the Markov chain starting from state 0, using the transition matrix defined above. The function would return the final state after the 5 transitions. Explore free Python courses Advantages and Disadvantages of Markov Chain A Markov chain undergoes transitions from one state to another as per specific probabilistic rules. The defining characteristic of a Markov chain is that no matter how the system arrived at its current state, the possible future states are fixed. This specific kind of memory-lessness is known as the “Markov property”. Some advantages of Markov chains include: 1. They can be used to model and analyse a wide variety of systems in many different fields, including economics, biology, and computer2. They are relatively simple to understand and use, making them a popular choice for modelling complex systems. 3. They can be used to predict the long-term behaviour of a system, even if the system is subject to random fluctuations in the short term. Some disadvantages of Markov chains include: 1. They are only able to model systems that exhibit the Markov property, which means that the future state of the system is dependent only on the current state and not on the sequence of events that led to the current state. 2. They can be computationally intensive, especially for large systems or systems with many possible states. 3. They may not accurately capture the behaviour of systems that exhibit complex dependencies or long-range correlations. 4, They may not be able to capture the full complexity of real-world systems, which can have many interacting variables and non-linear relationships.Properties of State Transition Matrix A rectangular arrangement of numbers in rows and columns is called a matrix. Control theory refers to the control of continuously operating dynamical systems in engineered processes and machines. The state- transition matrix is a matrix whose product with the state vector x at the time tp gives x at a time t, where tg denotes the initial time. This matrix is used to obtain the general solution of linear dynamical systems. It is represented by ®. It is an important part of both the zero input and the zero state solutions of systems represented in state space. In this article, we will learn the important properties of the state transition matrix. 10 Important Properties of State Transition Matrix 1. It has continuous derivatives. 2. It is continuous. 3. It cannot be singular. ©”! (t, r) = 0(z, t) ©" (t, 7) O(t, 7) =L lis the identity matrix. 4.0(t,t) =Ivt 5. © (tzt}) O(t, to) = O(ty to) V tos tis te 6. &(t, 7) = U(t) U(z). U(t) is an nxn matrix. It is the fundamental solution matrix that satisfies the followingsolution matrix that satisfies the following equation with initial condition U(to) = |. U(t) = A(t)U(t) 7.\t also satisfies the following differential equation with initial conditions (tg, tp) =! Boltto) — A(t) 9(t, to) 8. x(t) = 0(t, z) x(z) 9. The inverse will be the same as that of the state transition matrix just by replacing ‘t’ by -t.o1(t) = o(-t). 10. 0(t) + to) = ©(t)) (ta) If t = t) + ta, the resulting state transition matrix is equal to the multiplication of the two state transition matrices at t = t; and t = ty. Frequently Asked Questions Q1. What do you mean by a Matrix? A matrix is an arrangement of numbers in rows and columns. Q2 What do you understand by the State Transition Matrix? A matrix whose product with the state vector x at the time to gives x at a time t, (tp denotes the initial time) is called the State Transition Matrix. Q3 List two properties of the State Transition Matrix. Inverse of the state transition matrix is equal to that of the state transition matrix just by replacing t by -t. The State Transition Matrix cannot be singular. It le pantiniipiic and hac continiiogicNn a a Calculation of n step transition probability The n-step transition probability refers to the probability of transitioning from one state to another in a Markov chain after n steps. The calculation of n-step transition Probabilities depends on the specific characteristics of the Markov chain and the transition probabilities between states. To calculate the n-step transition probability, you need to raise the transition matrix to the power of n, where each element of the matrix represents the probability of transitioning from one state to another. The resulting matrix will represent the probabilities of transitioning between states after n steps. Here's a step-by-step guide on how to calculate the n-step transition probability: Start with the transition matrix, which is a square matrix where each element represents the probability of transitioning from one state to another. Let's denote the transition matrix as P. Raise the transition matrix to the power of n. You can use matrix multiplication to perform this operation. For example, if you want to calculate the 2-step transition probability, you would multiply the transition matrix by itself: P*2 = P * P. Repeat the matrix multiplication n times, each time multiplying the resulting matrix by the original transition matrix. For example, to calculate the 3-step transition probability, you would multiply P*2 by P: P*3 = P*2 * P. After performing the matrix multiplication n times, you will obtain the n-step transition probability matrix, denoted as P*n. The elements of the resulting matrix P*n represent the probabilities of transitioning between states after n steps. Each element P“n[i][j] represents the probability of transitioning from state i to state j after n steps. Note that the above steps assume a discrete-time Markov chain. If you are dealing with a continuous-time Markov chain, the calculations involve the transition rate matrix and the matrix exponential function. It's important to have the transition matrix defined for the Markov chain before attempting to calculate the n-step transition probabilities.