Machine Learning Models and Theories
Machine Learning Models and Theories
theories
Content
Common names:
Bayesian network,
Bayes network
Belief network
Bayes(ian) model
probabilistic directed acyclic graphical model
Bayesian Belief Network
P(A) P()
So how to solve this ?
Query 1:
Rain occur
Sprinkler Sprinkler
On Off
Grass not
Grass wet
wet
Solution
• Select probability of Rain occurrence = (.2)
• Select probability of Rain occurrence and Sprinkler On = ( 0.01)
• Select probability of Rain occurrence and Sprinkler On and Wet grass (.99)
P (B | AC) =
Likelihood Test (LR test)
• In statistics, a likelihood ratio test (LR test) is a statistical test used for comparing the
goodness of fit of two statistical models — a null model against an alternative model.
The test is based on the likelihood ratio, which expresses how many times more likely
the data are under one model than the other. This likelihood ratio, or equivalently its
logarithm, can then be used to compute a p-value, or compared to a critical value to
decide whether or not to reject the null model.
Discrete probability distribution
Getting a no. 1 2 3 4 5 6
Probability 1/6 1/6 1/6 1/6 1/6 1/6
• Suppose that we want to classify two kind of fish (A) Sea Bass and (B)
Salmon . We have a fish which has some properties w1, similar to Sea Bass
and some properties w2 similar to Salmon and we don’t know whether it is
Sea Bass or Salmon actually.
Select the region which is showing minimum error among given decision areas
So again problem is same “ How to find min[P (w1|x), P(W2|x)] ” and can we get specific bound of this
problem?
There are different methods are available in statistics to solve this optimization
problem. Traditionally we solve it by applying global minima and Dynamic
programming but here we will go for Chernoff bound to estimate
Chernoff bound
• Lemma :
Let us discuss it. Assume that we have two cases either a is greater than b and b is greater than a then
Assume a> b then we have to prove only
b<
1<
1<
Bhattacharyya Bound (An extension of Chernoff
bound which is slightly less tight and developed for )
Error probabilities in probabilistic classifier Confusion
estimations matrix
1: Chernoff Bound
2: Bhattacharyya Bound
The concept of bounds are directly taken from moment generating
function of a random variable X.
Chernoff Bound
• Suppose X1, ..., Xn are independent random variables taking values in {0,
1}. Let X denote their sum and let μ = E[X] denote the sum's expected
value. Then for any δ > 0
• Consider a biased coin with probability p = 1/3 of landing heads and probability 2/3 of
landing tails. Suppose the coin is flipped some number n of times, and let Xi be a
random variable denoting the ith flip, where Xi = 1 means heads, and Xi = 0 means
tails. Use the Chernoff bound to determine a value for n so that the probability that
more than half of the coin flips come out heads is less that 0.001
Solution
Expectation–maximization (EM) algorithm