Bayes Theorem in Machine learning

Machine Learning is one of the most emerging technology of Artificial Intelligence. We are living in
the 21th century which is completely driven by new technologies and gadgets in which some are yet
to be used and few are on its full potential. Similarly, Machine Learning is also a technology that is
still in its developing phase. There are lots of concepts that make machine learning a better
technology such as supervised learning, unsupervised learning, reinforcement learning, perceptron
models, Neural networks, etc. In this article "Bayes Theorem in Machine Learning", we will discuss
another most important concept of Machine Learning theorem i.e., Bayes Theorem. But before
starting this topic you should have essential understanding of this theorem such as what exactly is
Bayes theorem, why it is used in Machine Learning, examples of Bayes theorem in Machine Learning
and much more. So, let's start the brief introduction of Bayes theorem.

Introduction to Bayes Theorem in Machine Learning

Bayes theorem is given by an English statistician, philosopher, and Presbyterian minister named Mr.
Thomas Bayes in 17th century. Bayes provides their thoughts in decision theory which is extensively
used in important mathematics concepts as Probability. Bayes theorem is also widely used in
Machine Learning where we need to predict classes precisely and accurately. An important concept
of Bayes theorem named Bayesian method is used to calculate conditional probability in Machine
Learning application that includes classification tasks. Further, a simplified version of Bayes theorem
(Naïve Bayes classification) is also used to reduce computation time and average cost of the projects.

Bayes theorem is also known with some other name such as Bayes rule or Bayes Law. Bayes
theorem helps to determine the probability of an event with random knowledge. It is used to
calculate the probability of occurring one event while other one already occurred. It is a best method
to relate the condition probability and marginal probability.

In simple words, we can say that Bayes theorem helps to contribute more accurate results.

Bayes Theorem is used to estimate the precision of values and provides a method for calculating the
conditional probability. However, it is hypocritically a simple calculation but it is used to easily
calculate the conditional probability of events where intuition often fails. Some of the data scientist
assumes that Bayes theorem is most widely used in financial industries but it is not like that. Other
than financial, Bayes theorem is also extensively applied in health and medical, research and survey
industry, aeronautical sector, etc.

What is Bayes Theorem?

Bayes theorem is one of the most popular machine learning concepts that helps to calculate the
probability of occurring one event with uncertain knowledge while other one has already occurred.

Bayes' theorem can be derived using product rule and conditional probability of event X with known
event Y:

o According to the product rule we can express as the probability of event X with known event
Y as follows;

1. P(X ? Y)= P(X|Y) P(Y) {equation 1}

o Further, the probability of event Y with known event X:

1. P(X ? Y)= P(Y|X) P(X) {equation 2}

Mathematically, Bayes theorem can be expressed by combining both equations on right hand side.
We will get:

Here, both events X and Y are independent events which means probability of outcome of both
events does not depends one another.

The above equation is called as Bayes Rule or Bayes Theorem.

o P(X|Y) is called as posterior, which we need to calculate. It is defined as updated probability

after considering the evidence.

o P(Y|X) is called the likelihood. It is the probability of evidence when hypothesis is true.

o P(X) is called the prior probability, probability of hypothesis before considering the evidence

o P(Y) is called marginal probability. It is defined as the probability of evidence under any
consideration.

Hence, Bayes Theorem can be written as:

posterior = likelihood * prior / evidence

Prerequisites for Bayes Theorem

While studying the Bayes theorem, we need to understand few important concepts. These are as
follows:

1. Experiment

An experiment is defined as the planned operation carried out under controlled condition such as
tossing a coin, drawing a card and rolling a dice, etc.

2. Sample Space

During an experiment what we get as a result is called as possible outcomes and the set of all
possible outcome of an event is known as sample space. For example, if we are rolling a dice, sample
space will be:

S1 = {1, 2, 3, 4, 5, 6}

Similarly, if our experiment is related to toss a coin and recording its outcomes, then sample space
will be:

S2 = {Head, Tail}

3. Event
Event is defined as subset of sample space in an experiment. Further, it is also called as set of
outcomes.

Assume in our experiment of rolling a dice, there are two event A and B such that;

A = Event when an even number is obtained = {2, 4, 6}

B = Event when a number is greater than 4 = {5, 6}

o Probability of the event A ''P(A)''= Number of favourable outcomes / Total number of

possible outcomes
P(E) = 3/6 =1/2 =0.5

o Similarly, Probability of the event B ''P(B)''= Number of favourable outcomes / Total number
of possible outcomes
=2/6
=1/3
=0.333

o Union of event A and B:

A∪B = {2, 4, 5, 6}

o Intersection of event A and B:

A∩B= {6}
 o Disjoint Event: If the intersection of the event A and B is an empty set or null then such
events are known as disjoint event or mutually exclusive events also.

4. Random Variable:

It is a real value function which helps mapping between sample space and a real line of an
experiment. A random variable is taken on some random values and each value having some
probability. However, it is neither random nor a variable but it behaves as a function which can either
be discrete, continuous or combination of both.

5. Exhaustive Event:

As per the name suggests, a set of events where at least one event occurs at a time, called
exhaustive event of an experiment.

Thus, two events A and B are said to be exhaustive if either A or B definitely occur at a time and both
are mutually exclusive for e.g., while tossing a coin, either it will be a Head or may be a Tail.

6. Independent Event:

Two events are said to be independent when occurrence of one event does not affect the occurrence
of another event. In simple words we can say that the probability of outcome of both events does
not depends one another.

Mathematically, two events A and B are said to be independent if:

P(A ∩ B) = P(AB) = P(A)*P(B)

7. Conditional Probability:

Conditional probability is defined as the probability of an event A, given that another event B has
already occurred (i.e. A conditional B). This is represented by P(A|B) and we can define it as:

P(A|B) = P(A ∩ B) / P(B)

8. Marginal Probability:

Marginal probability is defined as the probability of an event A occurring independent of any other
event B. Further, it is considered as the probability of evidence under any consideration.

P(A) = P(A|B)*P(B) + P(A|~B)*P(~B)

Here ~B represents the event that B does not occur.

How to apply Bayes Theorem or Bayes rule in Machine Learning?

Bayes theorem helps us to calculate the single term P(B|A) in terms of P(A|B), P(B), and P(A). This
rule is very helpful in such scenarios where we have a good probability of P(A|B), P(B), and P(A) and
need to determine the fourth term.

Naïve Bayes classifier is one of the simplest applications of Bayes theorem which is used in
classification algorithms to isolate data as per accuracy, speed and classes.

Let's understand the use of Bayes theorem in machine learning with below example.

Suppose, we have a vector A with I attributes. It means

A = A1, A2, A3, A4……………Ai

Further, we have n classes represented as C1, C2, C3, C4…………Cn.

These are two conditions given to us, and our classifier that works on Machine Language has to
predict A and the first thing that our classifier has to choose will be the best possible class. So, with
the help of Bayes theorem, we can write it as:

P(Ci/A)= [ P(A/Ci) * P(Ci)] / P(A)

Here;

P(A) is the condition-independent entity.

P(A) will remain constant throughout the class means it does not change its value with respect to
change in class. To maximize the P(Ci/A), we have to maximize the value of term P(A/Ci) * P(Ci).

With n number classes on the probability list let's assume that the possibility of any class being the
right answer is equally likely. Considering this factor, we can say that:

P(C1)=P(C2)-P(C3)=P(C4)=…..=P(Cn).

This process helps us to reduce the computation cost as well as time. This is how Bayes theorem
plays a significant role in Machine Learning and Naïve Bayes theorem has simplified the conditional
probability tasks without affecting the precision. Hence, we can conclude that:

P(Ai/C)= P(A1/C)* P(A2/C)* P(A3/C)*……*P(An/C)

Hence, by using Bayes theorem in Machine Learning we can easily describe the possibilities of
smaller events.

What is Naïve Bayes Classifier in Machine Learning

Naïve Bayes theorem is also a supervised algorithm, which is based on Bayes theorem and used to
solve classification problems. It is one of the most simple and effective classification algorithms in
Machine Learning which enables us to build various ML models for quick predictions. It is a
probabilistic classifier that means it predicts on the basis of probability of an object. Some popular
Naïve Bayes algorithms are spam filtration, Sentimental analysis, and classifying articles.

Advantages of Naïve Bayes Classifier in Machine Learning:

 o It is one of the simplest and effective methods for calculating the conditional probability and
text classification problems.

o A Naïve-Bayes classifier algorithm is better than all other models where assumption of
independent predictors holds true.

o It is easy to implement than other models.

o It requires small amount of training data to estimate the test data which minimize the
training time period.

o It can be used for Binary as well as Multi-class Classifications.

Disadvantages of Naïve Bayes Classifier in Machine Learning:

The main disadvantage of using Naïve Bayes classifier algorithms is, it limits the assumption of
independent predictors because it implicitly assumes that all attributes are independent or unrelated
but in real life it is not feasible to get mutually independent attributes.

Conclusion

Though, we are living in technology world where everything is based on various new technologies
that are in developing phase but still these are incomplete in absence of already available classical
theorems and algorithms. Bayes theorem is also most popular example that is used in Machine
Learning. Bayes theorem has so many applications in Machine Learning. In classification related
problems, it is one of the most preferred methods than all other algorithm. Hence, we can say that
Machine Learning is highly dependent on Bayes theorem. In this article, we have discussed about
Bayes theorem, how can we apply Bayes theorem in Machine Learning, Naïve Bayes Classifier, etc

Concept learning in machine learning refers to the task of learning a general concept or target
concept from examples. This is a foundational aspect of supervised learning, where the goal is to
infer a general rule or pattern that can correctly classify new instances based on the examples
provided during training. In concept learning, the system tries to learn a concept (a target concept or
category) from a set of examples that are either positive (examples that belong to the concept) or
negative (examples that do not belong to the concept).

Key Ideas of Concept Learning:

1. Concept: A concept in this context is a general category or pattern that can be used to
classify objects or instances. For example, the concept could be "dog," and the task would be
to learn what characteristics (features) make an instance a "dog" or not.

2. Examples:

o Positive examples: These are instances that belong to the concept. For example, if
the concept is "dog," then a set of images containing dogs would be positive
examples.

o Negative examples: These are instances that do not belong to the concept. Using the
same example of "dog," images of cats, cars, or trees would be negative examples.

3. Hypothesis: In concept learning, the system builds a hypothesis or model that describes the
concept based on the provided examples. The hypothesis tries to generalize from the
positive examples while excluding the negative ones.
Steps in Concept Learning:

1. Input Examples: A set of labeled examples is provided, where each example is labeled as
either a positive or negative example of the concept.

2. Hypothesis Generation: The learning algorithm uses these labeled examples to generate a
hypothesis that captures the defining characteristics of the concept. In other words, it builds
a rule or decision boundary that distinguishes positive examples from negative examples.

3. Generalization: The hypothesis aims to generalize from the provided examples so that it can
correctly classify unseen examples. This is achieved by considering common patterns in the
positive examples while ensuring the hypothesis excludes the negative examples.

4. Evaluation: Once the hypothesis is generated, it is evaluated on new examples (test set) to
assess how well it generalizes. If the hypothesis works well on unseen data, it is considered a
good representation of the concept.

Example of Concept Learning:

Let’s say we are trying to learn the concept of "Fruit." We have the following examples:

 Positive examples (Fruit):

o A red apple (features: color = red, shape = round, texture = smooth)

o A yellow banana (features: color = yellow, shape = elongated, texture = smooth)

 Negative examples (Non-Fruit):

o A green leaf (features: color = green, shape = jagged, texture = rough)

o A stone (features: color = gray, shape = irregular, texture = hard)

From these examples, the learning algorithm might infer the following hypothesis for the concept
"Fruit":

 Fruit is something that has a smooth texture and could be round or elongated in shape, and
the color could vary (but the negative examples generally don't match this pattern).

This hypothesis might then be used to classify new, unseen examples, like another red apple or a
pear, based on the features.

Types of Concept Learning:

1. Version Space: Version space is the set of all hypotheses that are consistent with the
provided examples. It starts as a broad set of hypotheses and gets narrowed down as more
examples are added. The learning process aims to find the hypothesis that best explains the
positive and negative examples.

2. Inductive Learning: Concept learning is often referred to as inductive learning because the
algorithm generalizes from specific examples to broader rules or concepts. Inductive learning
uses specific instances (examples) to form a general understanding (concept).

3. Search-Based Learning: In some cases, concept learning involves searching through a

hypothesis space to find the hypothesis that best fits the examples. This search can be
 guided by techniques like decision trees, which recursively divide the data into subsets based
on feature values.

4. Learning from Positive and Negative Examples: In concept learning, there are usually two
types of learning:

o Learning from both positive and negative examples: The learner tries to find a
concept that explains the positive examples and excludes the negative ones.

o Learning from only positive examples: This type of learning is more difficult and
requires the learner to infer the boundaries of the concept based solely on the
positive instances (without knowing what does not belong to the concept).

Concept Learning Algorithms:

Several algorithms have been developed to perform concept learning:

1. Version Space Algorithm: This algorithm maintains a version space, which represents the
hypotheses that are consistent with the training examples. As new examples are introduced,
the version space is updated to eliminate inconsistent hypotheses.

2. Find-S Algorithm: The Find-S (Find-Specific Hypothesis) algorithm is a simple concept

learning algorithm that starts with the most specific hypothesis (representing the first
positive example) and generalizes it by incorporating additional positive examples.

3. Candidate Elimination Algorithm: This algorithm maintains a version space of hypotheses

and systematically eliminates hypotheses that do not fit the examples, either positive or
negative.

4. Decision Trees: Decision tree learning algorithms like ID3 and CART can also be viewed as
concept learning methods. They iteratively divide the feature space into smaller, more
homogeneous regions based on the values of input features, forming a tree that classifies
new examples.

Challenges in Concept Learning:

1. Noise and Ambiguity: If the examples contain noise (i.e., mislabeled examples) or if the
concept is ambiguous, it becomes harder to learn an accurate concept.

2. Overfitting: A concept learned from very specific examples might not generalize well to
unseen data (overfitting). Striking a balance between fitting the examples and generalizing is
crucial.

3. Complexity of the Concept: Some concepts might be inherently more complex and harder to
learn, especially if they depend on intricate relationships between features.

Conclusion:

Concept learning is a fundamental task in machine learning where the goal is to learn a general rule
or pattern from a set of positive and negative examples. It plays a central role in supervised learning,
where the system tries to generalize from labeled training data to make predictions on new, unseen
data. The process involves hypothesis generation, generalization, and evaluation, with the challenge
of ensuring that the learned concept generalizes well to new examples without overfitting to the
training data.
The Least Squares Method

The least squares method used in fitting models is more of a pragmatic approach to fit formulas or
models using data. We formulate an error function which is the difference between the model
predictions and the actual value, and we find the model parameters that will minimize the sum of
the squares of the errors calculated from the fitting data.

For example, we want to find a linear equation that calculates the weight of high school students in
terms of their age. This means that we want to find the best values of the parameters a, b such that
we could estimate the weight of any student as

The hat over the “Weight” signifies that it is the estimated value, and not the observed one. Fitting
the regression model means that we try to find the values of the parameters a and b such that the
estimated values of the Weight are as close as possible to the actual values in the known
observations. For any of those we define the error as the difference between the actual “Weight”
and the model estimate:

Because the error could be positive or negative, depending on whether the estimated value is more
or less than the actual value, we use the square of the error as a more objective measure of error. If
we sum these squares of errors for all the observations, we get what is known as the total sum of
squares errors (SSE).

In the least squares method, we find the model parameters (a,b) such that the total sum of squares
of errors is minimum. There is no specific reason why we assumed that this criterion would lead to
the “best” model. We could for example have selected the objective measure of error to be the
absolute value of the difference between the actual and estimated value:

In this case, we will not need to square the errors before summing them to formulate the function
we minimize to find the parameters (a,b). In fact, this way of defining the model criterion is called L1
regression.

The point is, finding the model parameters by minimizing the sum of the squares of the errors is a
heuristic pragmatic approach that does not have theoretical justification, other than it makes some
sense and it works!

However, it can be demonstrated that the least squares method is susceptible to outliers. For
example, if we consider the data in figure 1 below showing some observations for Weight and Age.
Figure 1: effect of outliers on least squares models

Figure 1(a) on the left shows the least squares model with the outliers (circled in red) included in the
data. Figure 1(b) shows the least squares model when we remove these outliers. The reason why we
called these observations outliers is that we can see in the figure that the “Weight” of these
individuals is very much higher than the weights of the rest of the individuals in the sample. This is a
common issue in least squares regression models.

However, there are many ways to overcome this problem, such as assigning different weights
(sample weights) to the observations while fitting the model or removing the outliers.

The Maximum Likelihood Method

The maximum likelihood method is based on a kind of a general logical principle. It goes like this: If
we start with some data and want to fit a model to represent the behavior of this data, then
the best model we could get is when this data is the most likely data to be generated by this model.
That is that fitting data has the highest likelihood of being created by the model.

All we need to do is to define a meaningful likelihood function and use it to calculate the probability
that the model generates data. Then, this function will attain its maxim value at the fitting data. Let’s
demonstrate that with simple linear regression of the “Weight” and “Age” as we did with the least
squares method.

We start by assuming that the estimate of “Weight” of high school students is related to their “Age”
by the equation:

And the estimation error is give by:

If we further assume that this error e is normally distributed, then the probability that we get an
error of value e is given by the equation of the normal distribution, i.e.,
Where the e with a bar on top is the average error, and the Greek symbol sigma is the standard
deviation of the error. This equation defines the probability of the error taking a specific value for any
observation. The choice of the normal distribution is convenient because the probability of the error
is maximum when the error is zero. Therefore, when we maximize this probability, we are in effect
reducing the estimation error.

If we assume that the errors resulting from the fitting data observations are independent, then the
probability that we get these errors from all the observations at the same time would be their
product. This will be the definition of our likelihood function to maximize. Let’s call this function L:

L = Product of the probabilities of the errors calculated from all observations of the fitting data

To keep the math simple, I will assume that we have only three observations to fit the linear equation
to estimate the Weight. Let’s denote the estimation errors at these three observations e1, e2, and
e3. And assume also that the errors are independent and have a zero mean value and a standard
deviation of 1. Then we have

And the likelihood function L would then be written as

Because the logarithmic function is monotonic (i.e., it preserves the order of values) we can use “Log
L” instead of L, or

Finally, we can ignore the first term on the right-hand side of Log L (because it is constant), and we
rearrange the equation to express -2 Log L as:
The maximum likelihood principle requires L to be maximized. Equivalently, we will minimize -2 Log
L. Thus, we have demonstrated that, in this case, the maximum likelihood method is equivalent to
the least squares method. This result can be generalized for any number of observations, and for any
constant value of standard deviation. But we need to keep the assumptions of errors being
independent with zero mean.

Therefore, the maximum likelihood method could (under certain assumptions) be equivalent to the
least squares method. This is the relationship between these two methods.

ML and LS Error Hypothesis

Maximum Likelihood Estimation (MLE) and Least-Squares Error Hypothesis in Machine Learning

In machine learning, Maximum Likelihood Estimation (MLE) and Least-Squares Error (LSE) are two
fundamental concepts used for parameter estimation in statistical models. While they share some
similarities, they differ in their underlying assumptions and objectives.

Maximum Likelihood Estimation (MLE)

MLE is a method for estimating model parameters by finding the values that maximize the likelihood
of observing the given data. In other words, it seeks to find the parameters that make the observed
data most probable. The likelihood function is defined as the probability of observing the data given
the model parameters.

In machine learning, MLE is often used for:

1. Parameter estimation: MLE is used to estimate model parameters, such as weights and
biases in neural networks, coefficients in linear regression, and hyperparameters in Bayesian
models.

2. Model selection: MLE can be used to compare the performance of different models and
select the one with the highest likelihood.

Least-Squares Error (LSE)

LSE is a method for estimating model parameters by minimizing the sum of squared errors between
the predicted and actual values. It is a widely used technique in linear regression, neural networks,
and other machine learning models.

In machine learning, LSE is often used for:

 1. Regression: LSE is used to estimate model parameters in linear regression, where the goal is
to minimize the squared error between predicted and actual values.

2. Neural networks: LSE is used as an objective function in some neural network architectures,
such as mean squared error (MSE) or mean absolute error (MAE).

Relationship between MLE and LSE

Under certain assumptions, MLE and LSE are equivalent. Specifically:

1. Normal errors: If the errors in the model are normally distributed, MLE and LSE will produce
the same estimates.

2. Linear models: In linear models, such as linear regression, MLE and LSE are equivalent when
the errors are normally distributed.

However, when the errors are non-normal or the model is non-linear, MLE and LSE may produce
different estimates. In such cases, MLE may be more robust and provide better estimates, while LSE
may be more sensitive to outliers and non-normality.

Key differences

1. Objective: MLE aims to maximize the likelihood of observing the data, while LSE aims to
minimize the sum of squared errors.

2. Assumptions: MLE assumes a probabilistic model, while LSE assumes a deterministic

relationship between the predicted and actual values.

3. Robustness: MLE is generally more robust to non-normal errors and outliers, while LSE is
more sensitive to these issues.

In summary, MLE and LSE are both widely used techniques in machine learning, but they differ in
their underlying assumptions and objectives. While they are equivalent under certain conditions,
MLE may be more robust and suitable for non-linear or non-normal models, while LSE is often used
in linear regression and neural networks.

Maximum likelihood

Introduction

Maximum likelihood is an approach commonly used for such density estimation problems, in which a
likelihood function is defined to get the probabilities of the distributed data. It is imperative to study
and understand the concept of maximum likelihood as it is one of the primary and core concepts
essential for learning other advanced machine learning and deep learning techniques and
algorithms.

In this article, we will discuss the likelihood function, the core idea behind that, and how it works
with code examples. This will help one to understand the concept better and apply the same when
needed.

Let us dive into the likelihood first to understand the maximum likelihood estimation.

What is the Likelihood?

In machine learning, the likelihood is a measure of the data observations up to which it can tell us
the results or the target variables value for particular data points. In simple words, as the name
suggests, the likelihood is a function that tells us how likely the specific data point suits the existing
data distribution.

For example. Suppose there are two data points in the dataset. The likelihood of the first data point
is greater than the second. In that case, it is assumed that the first data point provides accurate
information to the final model, hence being likable for the model being informative and precise.

After this discussion, a gentle question may appear in your mind, If the working of the likelihood
function is the same as the probability function, then what is the difference?

Difference Between Probability and Likelihood

Although the working and intuition of both probability and likelihood appear to be the same, there is
a slight difference, here the possibility is a function that defines or tells us how accurate the
particular data point is valuable and contributes to the final algorithm in data distribution and how
likely is to the machine learning algorithm.

Whereas probability, in simple words is a term that describes the chance of some event or thing
happening concerning other circumstances or conditions, mostly known as conditional probability.

Also, the sum of all the probabilities associated with a particular problem is one and can not exceed
it, whereas the likelihood can be greater than one.

What is Maximum Likelihood Estimation?

After discussing the intuition of the likelihood function, it is clear to us that a higher likelihood is
desired for every model to get an accurate model and has accurate results. So here, the term
maximum likelihood represents that we are maximizing the likelihood function, called
the Maximization of the Likelihood Function.

Let us try to understand the same with an example.

Let us suppose that we have a classification dataset in which the independent column is the marks of
the students that they achieved in the particular exam, and the target or dependent column is
categorical, which has yes and No attributes representing if students are placed on the campus
placements or not.

Noe here, if we try to solve the same problem with the help of maximum likelihood estimation, the
function will first calculate the probability of every data point according to every suitable condition
for the target variable. In the next step, the function will plot all the data points in the two-
dimensional plots and try to find the line that best fits the dataset to divide it into two parts. Here
the best-fit line will be achieved after some epochs, and once achieved, the line is used to classify the
data point by simply plotting it to the graph.

Maximum Likelihood: The Base

The maximum likelihood estimation is a base of some machine learning and deep learning
approaches used for classification problems. One example is logistic regression, where the algorithm
is used to classify the data point using the best-fit line on the graph. The same approach is known as
the perceptron trick regarding deep learning algorithms.
As shown in the above image, all the data observations are plotted in a two-dimensional diagram
where the X-axis represents the independent column or the training data, and the y-axis represents
the target variable. The line is drawn to separate both data observations, positives and negatives.
According to the algorithm, the observations that fall above the line are considered positive, and
data points below the line are regarded as negative data points.

Maximum Likelihood Estimation: Code Example

We can quickly implement the maximum likelihood estimation technique using logistic regression on
any classification dataset. Let us try to implement the same.

import pandas as pd

import numpy as np

import seaborn as sns

from sklearn.linear_model import LogisticRegression

lr=LogisticRegression()

lr.fit(X_train,y_train)

lr_pred=lr.predict(X_test)

sns.regplot(x="X",y='lr_pred',data=df_pred ,logistic=True, ci=None)

The above code will fit the logistic regression for the given dataset and generate the line plot for the
data representing the distribution of the data and the best fit according to the algorithm.

Key Takeaways

 Maximum Likelihood is a function that describes the data points and their likeliness to the
model for best fitting.

 Maximum likelihood is different from the probabilistic methods, where probabilistic methods
work on the principle of calculation probabilities. In contrast, the likelihood method tries o
maximize the likelihood of data observations according to the data distribution.

 Maximum likelihood is an approach used for solving the problems like density distribution
and is a base for some algorithms like logistic regression.

 The approach is very similar and is predominantly known as the perceptron trick in terms of
deep learning methods.
The Minimum Description Length (MDL) principle is a concept in information theory and machine
learning that aims to strike a balance between model complexity and the fit of a model to data. It is
used for model selection and regularization, ensuring that a model is simple enough to avoid
overfitting but complex enough to capture the underlying patterns in the data.

Key Idea:

MDL is based on the idea of data compression. In essence, it tries to minimize the total length of two
parts:

1. The description length of the model (complexity): This is the length of the description or
encoding of the model itself.

2. The description length of the data (given the model): This is the length of the encoded data,
assuming the model is correct and used for compression.

In other words, the MDL principle suggests that the best model is the one that minimizes the total
number of bits required to describe both the model and the data it explains.

Components:

1. Model Encoding: This represents how complex the model is, often expressed in terms of the
number of parameters or the structure of the model (e.g., the size of a neural network,
number of features in linear regression).

2. Data Encoding (Error or Residual): This is the amount of information required to represent
the data after the model has been applied. It’s usually related to the model’s errors or the
likelihood of the data under the model.

Formally:

The MDL principle seeks to minimize the following quantity:

MDL=Length of model encoding+Length of error encoding (given the model)\text{MDL} = \

text{Length of model encoding} + \text{Length of error encoding (given the model)}

Steps in MDL:

1. Choose a model family: First, you select a set of candidate models (e.g., linear models,
decision trees, neural networks).

2. Estimate the model parameters: For each model, fit the parameters that minimize the error
on the training data.

3. Compute the description length: For each model, calculate the sum of the length of the
model description (its complexity) and the length of the residuals or errors (the data
description, given the model).

4. Select the model: The model that minimizes the MDL score (the sum of both description
lengths) is chosen as the best model.

MDL and Overfitting:

MDL is inherently related to the idea of regularization. A model that is too complex will have a long
description length for the model, while a model that is too simple will have large errors, resulting in a
large error encoding. MDL tries to balance these two factors. By penalizing overly complex models,
MDL avoids overfitting, while still allowing enough complexity to capture meaningful patterns in the
data.

MDL in Machine Learning:

In machine learning, MDL can be used in various ways:

 Model Selection: By comparing models and selecting the one that minimizes the MDL
criterion, one can ensure that the chosen model is neither too simple (underfitting) nor too
complex (overfitting).

 Regularization: In settings like regression, MDL can help to prevent overfitting by penalizing
overly complex models (e.g., through a term that reduces the number of parameters).

 Clustering and Classification: MDL is also used in unsupervised learning, for example, in
model-based clustering methods or in decision tree learning.

Example:

For a simple linear regression model, the description length of the model would depend on the
number of coefficients in the regression equation (e.g., the number of features and the intercept
term). The error encoding would depend on how well the model fits the data (i.e., the residuals). A
simple model might fit the data poorly, leading to a large error encoding, while a more complex
model might fit the data well but require more bits to describe.

Advantages:

 Principled approach: MDL is based on a solid theoretical foundation in information theory,

providing a well-defined way to compare models.

 Balancing complexity and fit: It naturally avoids both overfitting and underfitting.

 Model agnostic: MDL can be applied to a wide range of models, including both supervised
and unsupervised learning tasks.

Disadvantages:

 Computational complexity: Calculating the exact MDL can be computationally expensive,

especially for complex models or large datasets.

 Requires model choice: The MDL criterion depends on the choice of model family, which
could lead to challenges in selecting appropriate models.

Conclusion:

The MDL principle offers a powerful and theoretically sound method for model selection and
regularization. By balancing model complexity with data fit, it helps avoid both overfitting and
underfitting, making it an important concept in machine learning, especially in scenarios requiring
model selection.

The Bayes Optimal Classifier is a theoretical concept in machine learning and statistics that
represents the most accurate classifier possible based on probabilistic reasoning. It is optimal in the
sense that it minimizes the classification error or maximizes the posterior probability for each class
given the available data. However, it is a theoretical classifier and is generally not feasible to
implement directly in practice because it requires complete knowledge of the true probability
distributions of the data.

Key Concepts:

1. Conditional Probability: The Bayes Optimal Classifier makes predictions based on conditional
probabilities of the target class given the feature values. It uses Bayes' Theorem to update
beliefs about the class label after observing the data (features).

2. Posterior Probability: For each class, the classifier calculates the posterior probability, which
represents the probability of a class CkC_k given the feature vector XX:

P(Ck∣X)=P(X∣Ck)P(Ck)P(X)P(C_k | X) = \frac{P(X | C_k) P(C_k)}{P(X)}

o P(X∣Ck)P(X | C_k): The likelihood, or the probability of observing the features XX

given class CkC_k.

o P(Ck)P(C_k): The prior probability of class CkC_k, i.e., the probability of that class
occurring before observing the data.

o P(X)P(X): The evidence, or the total probability of observing XX across all classes.
This is used for normalization and can be computed as: P(X)=∑kP(X∣Ck)P(Ck)P(X) = \
sum_{k} P(X | C_k) P(C_k)

3. Decision Rule: The Bayes Optimal Classifier assigns the class with the highest posterior
probability:

C^=arg⁡max⁡CkP(Ck∣X)\hat{C} = \arg\max_{C_k} P(C_k | X)

In other words, it predicts the class that maximizes the conditional probability given the features.

Bayes' Theorem Recap:

Bayes' Theorem is the foundation of the Bayes Optimal Classifier. It allows us to update our beliefs
about the class given new data:

P(Ck∣X)=P(X∣Ck)P(Ck)P(X)P(C_k | X) = \frac{P(X | C_k) P(C_k)}{P(X)}

Where:

 P(Ck∣X)P(C_k | X) is the posterior probability (the probability of class CkC_k given the
observed data XX).

 P(X∣Ck)P(X | C_k) is the likelihood (the probability of observing XX given the class CkC_k).

 P(Ck)P(C_k) is the prior probability (the initial belief about class CkC_k).

 P(X)P(X) is the evidence (the total probability of observing XX across all classes).

How the Bayes Optimal Classifier Works:

1. Prior: First, it considers the prior probability P(Ck)P(C_k) of each class CkC_k. This can be
estimated from the relative frequencies of classes in the training data.
 2. Likelihood: Then, it computes the likelihood P(X∣Ck)P(X | C_k), which is the probability of
observing the data XX given a particular class CkC_k. This is typically estimated using the
training data.

3. Posterior: It calculates the posterior probability P(Ck∣X)P(C_k | X) using Bayes' Theorem. The
class with the highest posterior probability is then chosen as the predicted class for the new
input XX.

4. Prediction: The classifier predicts the class that has the maximum posterior probability.

Example:

Let's say we have a dataset for a binary classification problem with two classes, Spam and Not Spam,
and we want to predict whether a new email is Spam or Not Spam based on certain features like the
presence of specific words in the email. The Bayes Optimal Classifier would:

 Compute the prior probability of Spam and Not Spam emails.

 Compute the likelihood of observing the features (words) in the new email for both classes.

 Use Bayes' Theorem to compute the posterior probability for each class and predict the class
with the highest posterior.

Limitations of the Bayes Optimal Classifier:

1. Infeasible in Practice: In most real-world scenarios, we do not have access to the true
distribution of the data. For example, we cannot easily compute P(X∣Ck)P(X | C_k) or
P(Ck)P(C_k) directly. We usually need to estimate these from data, which is why the classifier
is theoretical.

2. Assumptions: The Bayes Optimal Classifier assumes that we know the true probability
distributions of both the features and the classes. In practice, these distributions are often
unknown and need to be approximated, which introduces errors.

3. Data Requirements: To make accurate predictions, the classifier requires a large amount of
data to estimate the true probability distributions well. In cases where the data is sparse, the
estimates of the probabilities might be inaccurate, leading to suboptimal predictions.

Practical Implementation:

In practice, we often use Naive Bayes classifiers, which are a simplified version of the Bayes Optimal
Classifier. The Naive Bayes classifier makes the assumption that the features are conditionally
independent given the class. While this assumption is rarely true in real-world data, the Naive Bayes
classifier can still perform well and is computationally efficient.

Advantages of the Bayes Optimal Classifier:

 Theoretically optimal: It provides the best possible classification performance when the true
distributions are known.

 Probabilistic predictions: It not only gives a classification but also provides a measure of
confidence in its predictions through posterior probabilities.

 Works well with uncertainty: Because it is based on probability theory, it can naturally
handle uncertainty and noisy data.
Conclusion:

The Bayes Optimal Classifier is a powerful theoretical model for classification that minimizes
classification error by maximizing the posterior probability of the classes given the data. However,
due to the need for true probability distributions, it is generally impractical in real-world scenarios,
leading to the use of approximations like the Naive Bayes classifier.

The Gibbs Sampling algorithm is a powerful statistical method used in Markov Chain Monte Carlo
(MCMC) techniques to generate samples from a complex multivariate distribution. In the context of
machine learning, Gibbs Sampling is often used to approximate the posterior distribution of
parameters in a probabilistic model when direct sampling or analytical solutions are infeasible.

Key Concepts:

1. Markov Chain Monte Carlo (MCMC): MCMC methods, including Gibbs Sampling, are used to
sample from a probability distribution. The idea is to create a Markov chain whose
equilibrium distribution matches the target distribution. By sampling from this Markov chain,
we can approximate the target distribution.

2. Gibbs Sampling: Gibbs Sampling is a specific MCMC technique that generates samples from
the joint distribution of multiple variables by iteratively sampling from their conditional
distributions.

In simpler terms, Gibbs Sampling simplifies the problem of sampling from a complex joint
distribution by breaking it down into easier-to-sample conditional distributions for each variable
(given the others). This iterative process is repeated until convergence to the target distribution.

Steps in the Gibbs Sampling Algorithm:

Suppose we have a set of random variables X1,X2,…,XnX_1, X_2, \dots, X_n and we want to sample
from their joint distribution P(X1,X2,…,Xn)P(X_1, X_2, \dots, X_n). The idea is to sample each variable
conditioned on the others.

1. Initialize the variables X1(0),X2(0),…,Xn(0)X_1^{(0)}, X_2^{(0)}, \dots, X_n^{(0)} (initial

guesses or values for the variables).

2. Iterate over each variable XiX_i (for i=1,2,…,ni = 1, 2, \dots, n):

o Sample Xi(t+1)X_i^{(t+1)} from the conditional distribution P(Xi∣X1(t),

…,Xi−1(t),Xi+1(t),…,Xn(t))P(X_i | X_1^{(t)}, \dots, X_{i-1}^{(t)}, X_{i+1}^{(t)}, \dots,
X_n^{(t)}), where all other variables are held fixed at their current values.

3. Repeat the process for a sufficient number of iterations. The idea is that after many
iterations, the samples will converge to the target joint distribution P(X1,X2,…,Xn)P(X_1, X_2,
\dots, X_n).

4. Extract samples after convergence, which are drawn from the target distribution.

Example:

Consider a simple case with two variables X1X_1 and X2X_2, and we want to sample from the joint
distribution P(X1,X2)P(X_1, X_2). The Gibbs sampling steps would look like this:
 1. Initialize X1(0)X_1^{(0)} and X2(0)X_2^{(0)}.

2. Sample X1(1)X_1^{(1)} from the conditional distribution P(X1∣X2(0))P(X_1 | X_2^{(0)}).

3. Sample X2(1)X_2^{(1)} from the conditional distribution P(X2∣X1(1))P(X_2 | X_1^{(1)}).

4. Repeat steps 2 and 3 for several iterations.

Mathematical Foundation:

The key idea behind Gibbs Sampling is to exploit the conditional independence in the joint
distribution. Given a set of variables X1,X2,…,XnX_1, X_2, \dots, X_n, the joint distribution can often
be factorized into conditional distributions:

P(X1,X2,…,Xn)=P(X1∣X2,…,Xn)P(X2∣X1,X3,…,Xn)…P(Xn∣X1,X2,…,Xn−1)P(X_1, X_2, \dots, X_n) = P(X_1 |

X_2, \dots, X_n) P(X_2 | X_1, X_3, \dots, X_n) \dots P(X_n | X_1, X_2, \dots, X_{n-1})

The algorithm uses these conditional distributions to update the value of each variable based on the
current state of the others, effectively constructing the Markov chain that approximates the joint
distribution.

When to Use Gibbs Sampling:

Gibbs Sampling is useful when:

 The joint distribution is difficult to sample from directly, but the conditional distributions are
easier to handle.

 The model has a complex structure where multiple variables interact, and it is easier to
sample from their conditional distributions rather than the joint distribution.

 It is used in Bayesian inference for approximating the posterior distribution of model

parameters, especially when the model is too complicated for direct computation.

Common Applications of Gibbs Sampling:

1. Bayesian Inference: In Bayesian models, we often need to compute the posterior

distribution P(θ∣D)P(\theta | D), where θ\theta is a set of model parameters and DD is the
observed data. Gibbs Sampling is used to sample from this posterior when it is difficult to
compute analytically. The algorithm iteratively samples each parameter θi\theta_i
conditioned on the others, helping to approximate the full posterior distribution.

2. Hidden Markov Models (HMMs): In Hidden Markov Models, Gibbs Sampling can be used to
estimate the latent state sequence given observed data. This involves sampling the hidden
states conditioned on the observations and other states.

3. Latent Variable Models: Gibbs Sampling is commonly applied in models with latent
variables, such as Latent Dirichlet Allocation (LDA) for topic modeling, where the latent
topics are sampled conditionally on the observed words and other topics.

4. Gaussian Mixture Models (GMMs): In GMMs, where the goal is to estimate the mixture
components (such as the mean and covariance of each Gaussian), Gibbs Sampling can be
used to iteratively update the parameters of each Gaussian component.

Advantages of Gibbs Sampling:

  Simplicity: It is easy to implement, especially when the conditional distributions are known
and easy to sample from.

 Convergence to the target distribution: After a large number of iterations, the samples
generated by Gibbs Sampling converge to the target distribution, making it useful for
approximating complex posterior distributions.

 No need for gradient computation: Unlike optimization algorithms, Gibbs Sampling does not
require gradient information, which is useful in high-dimensional or complex models where
gradients are hard to compute.

Disadvantages of Gibbs Sampling:

 Convergence issues: The algorithm may take a long time to converge, especially in high-
dimensional or complex models. The rate of convergence depends on how strongly the
variables are correlated.

 Burn-in period: The early iterations may not represent the target distribution well, so these
initial samples (called the "burn-in" period) are often discarded.

 Limited to conditional distributions: Gibbs Sampling requires that the conditional

distributions for each variable are easy to sample from. If this is not the case, Gibbs Sampling
may not be applicable.

Conclusion:

Gibbs Sampling is a powerful and widely used MCMC technique that helps in approximating complex
probability distributions by iteratively sampling from conditional distributions. It is particularly useful
in Bayesian inference, latent variable models, and other probabilistic machine learning models where
exact sampling is intractable. While it is simple and efficient in many cases, careful consideration of
convergence and burn-in periods is necessary to ensure accurate sampling.

The Naïve Bayes classifier is a simple yet powerful probabilistic classifier based on Bayes' Theorem
with a strong (naïve) assumption of conditional independence between the features given the class
label. Despite its simplicity, it often performs surprisingly well, especially in text classification
problems like spam detection, sentiment analysis, and document classification.

Key Concepts:

1. Bayes' Theorem: Bayes' Theorem is a fundamental theorem in probability theory that

describes the probability of an event based on prior knowledge of conditions related to the
event. It is expressed as:

P(C∣X)=P(X∣C)P(C)P(X)P(C | X) = \frac{P(X | C) P(C)}{P(X)}

o P(C∣X)P(C | X): The posterior probability of class CC given the feature vector XX.

o P(X∣C)P(X | C): The likelihood, or the probability of observing features XX given the
class CC.

o P(C)P(C): The prior probability of class CC.

o P(X)P(X): The evidence, or the total probability of observing XX across all classes (this
can be computed as P(X)=∑CP(X∣C)P(C)P(X) = \sum_{C} P(X | C) P(C)).
 2. Conditional Independence Assumption: The "naïve" aspect of the Naïve Bayes classifier
comes from the assumption that all features (attributes) are conditionally independent
given the class label. That is, given the class CC, the features X1,X2,…,XnX_1, X_2, \dots, X_n
are assumed to be independent of each other. This simplifies the computation of the
likelihood P(X∣C)P(X | C) because:

P(X∣C)=P(X1∣C)P(X2∣C)…P(Xn∣C)P(X | C) = P(X_1 | C) P(X_2 | C) \dots P(X_n | C)

This simplification makes the Naïve Bayes classifier computationally efficient and easier to
implement.

How Naïve Bayes Works:

The general steps for using a Naïve Bayes classifier are:

1. Model Assumptions:

o The classifier assumes the data is generated from a probabilistic model where the
class label and features are connected via conditional probabilities.

o Each class CC has a prior probability P(C)P(C), which is estimated from the frequency
of each class in the training data.

o The conditional probabilities P(Xi∣C)P(X_i | C) for each feature XiX_i given the class
CC are also learned from the training data.

2. Training:

o Estimate Prior Probabilities: For each class CC, compute the prior probability
P(C)P(C), which is the proportion of samples in class CC in the training set:

P(C)=Number of samples in class CTotal number of samplesP(C) = \frac{\text{Number of samples in

class } C}{\text{Total number of samples}}

o Estimate Conditional Probabilities: For each feature XiX_i and each class CC,
compute the likelihood P(Xi∣C)P(X_i | C), which is the probability of feature XiX_i
given class CC. This can be done using various distributions depending on the type of
data:

 Categorical data: Use the relative frequency of XiX_i in class CC.

 Continuous data: Often assume that the features follow a Gaussian (normal)
distribution and estimate the mean and variance for each class.

3. Prediction: Given a new instance X=(X1,X2,…,Xn)X = (X_1, X_2, \dots, X_n), the classifier
computes the posterior probability for each class CC using Bayes' Theorem:

P(C∣X)=P(C)∏i=1nP(Xi∣C)P(C | X) = P(C) \prod_{i=1}^{n} P(X_i | C)

o The class with the highest posterior probability is chosen as the predicted class for
XX.

o In practice, it's common to compute the log of the probabilities to avoid numerical
underflow, and the class is assigned as: C^=arg⁡max⁡C[log⁡P(C)+∑i=1nlog⁡P(Xi∣C)]\hat{C}
= \arg\max_C \left[ \log P(C) + \sum_{i=1}^{n} \log P(X_i | C) \right]
Types of Naïve Bayes Classifiers:

There are several variations of the Naïve Bayes classifier, depending on the type of data being
handled:

1. Gaussian Naïve Bayes:

o Used for continuous data. It assumes that the features X1,X2,…,XnX_1, X_2, \dots,
X_n are normally distributed within each class.

o For each feature XiX_i, the conditional probability P(Xi∣C)P(X_i | C) is computed using
the probability density function of a Gaussian distribution: P(Xi∣C)=12πσ2exp⁡(−
(Xi−μ)22σ2)P(X_i | C) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{(X_i - \mu)^2}
{2 \sigma^2}\right) where μ\mu and σ2\sigma^2 are the mean and variance of XiX_i
for class CC.

2. Multinomial Naïve Bayes:

o Used for discrete count data, such as word counts in text classification (e.g., spam
detection or sentiment analysis). Here, the likelihood P(Xi∣C)P(X_i | C) is computed
using the multinomial distribution: P(Xi∣C)=(Ni)!∏i=1n(Ni,j)!∏j=1nP(wj∣C)NijP(X_i | C)
= \frac{(N_i)!}{\prod_{i=1}^n (N_{i,j})!} \prod_{j=1}^n P(w_j | C)^{N_{ij}} where NiN_i
is the total count of words in class CC, and NijN_{ij} is the count of the j-th word.

3. Bernoulli Naïve Bayes:

o Used when the features are binary (e.g., in binary text classification tasks, where the
presence or absence of a word is recorded). In this case, P(Xi∣C)P(X_i | C) is modeled
as a Bernoulli distribution: P(Xi∣C)=pXi(1−p)1−XiP(X_i | C) = p^{X_i} (1-p)^{1-X_i}
where pp is the probability of a feature being present given class CC.

Advantages of Naïve Bayes:

 Simple and fast: Naïve Bayes is computationally efficient and easy to implement, especially
when dealing with large datasets.

 Works well with high-dimensional data: It performs well in scenarios where the number of
features (dimensions) is high, such as text classification tasks.

 Handles missing data: It can handle missing data well because it works on conditional
probabilities and can easily ignore features with missing values.

 Scalable: It scales well with large datasets because of its simplicity and low computational
cost.

Disadvantages of Naïve Bayes:

 Naïve independence assumption: The assumption that features are conditionally

independent given the class is often unrealistic, especially in real-world data, which may lead
to suboptimal performance if the features are highly correlated.

 Requires large amounts of data: In cases with limited data, the probability estimates can be
inaccurate, leading to poor performance.
  Poor with continuous features: Although Gaussian Naïve Bayes can work with continuous
features, the assumption of normality may not hold in all cases.

Applications of Naïve Bayes:

 Text Classification: Naïve Bayes is widely used in applications like spam detection, sentiment
analysis, topic modeling, and document categorization.

 Medical Diagnosis: It can be used to classify diseases based on various symptoms and
medical test results.

 Recommendation Systems: In collaborative filtering, Naïve Bayes can be used to predict user
preferences based on past behavior.

Conclusion:

The Naïve Bayes classifier is a simple, efficient, and interpretable probabilistic model that works well
for a variety of classification tasks, especially when the assumption of feature independence holds or
when working with high-dimensional datasets. Despite its simplicity and the strong independence
assumption, it often provides competitive performance, particularly in domains like text classification
and natural language processing.

An Example Learning to Classify Test

Let's walk through an example of how a machine learning model can be trained to classify data, step
by step, using a supervised learning approach. In this case, we'll use a simple Naïve Bayes classifier
to classify test data based on some features.

Problem Setup

Imagine you are tasked with building a classifier that can classify whether an email is spam or not
spam based on the email's content. The features for the email might include certain words or
phrases appearing in the email.

Step 1: Collecting the Data

You have a training dataset that consists of labeled examples of emails. Each email has a set of
features (e.g., presence of words like "free", "offer", "buy") and a label indicating whether the email
is spam or not spam.

Training Data Example:

Email ID "Free" "Buy" "Offer" "Money" Class (Label)

1 1 0 1 0 Spam

2 0 1 0 0 Not Spam

3 1 1 1 0 Spam

4 0 0 1 1 Not Spam

5 1 0 1 0 Spam

 In this dataset, each row represents an email, and the columns "Free", "Buy", "Offer", and
"Money" represent whether these words appear (1) or do not appear (0) in the email.
  The Class column is the label, indicating whether the email is Spam or Not Spam.

Step 2: Preprocessing the Data

Before applying any machine learning model, the data might need to be preprocessed. In this case,
the data is already numeric (binary features), so no major preprocessing steps are needed. However,
in a real-world scenario, you would often perform tasks like:

 Tokenizing text.

 Removing stop words (like "the", "is", etc.).

 Normalizing/standardizing data if necessary (though in our binary feature case, this isn't
required).

Step 3: Training the Naïve Bayes Classifier

We will use the Naïve Bayes classifier to learn from this dataset and make predictions. The classifier
assumes that the features are conditionally independent given the class (which may or may not hold
in practice). We will calculate the probabilities P(C)P(C) and P(Xi∣C)P(X_i | C) for each feature in the
training data.

1. Prior Probability of Each Class P(C)P(C):

o From the dataset, we see that there are 3 spam emails and 2 non-spam emails. So:
P(Spam)=35,P(Not Spam)=25P(\text{Spam}) = \frac{3}{5}, \quad P(\text{Not Spam}) =
\frac{2}{5}

2. Likelihood P(Xi∣C)P(X_i | C):

o For each feature (word in the email), we calculate the likelihood of the feature given
the class. For example, the likelihood of the word "Free" given that the email is Spam
is:
P(Free∣Spam)=Number of times "Free" appears in Spam emailsTotal number of Spam
emails=33=1P(\text{Free} | \text{Spam}) = \frac{\text{Number of times "Free"
appears in Spam emails}}{\text{Total number of Spam emails}} = \frac{3}{3} = 1

o Similarly, we calculate the likelihood for all the features:

P(Buy∣Spam)=13,P(Offer∣Spam)=1,P(Money∣Spam)=0P(\text{Buy} | \text{Spam}) = \
frac{1}{3}, \quad P(\text{Offer} | \text{Spam}) = 1, \quad P(\text{Money} | \
text{Spam}) = 0

o For the Not Spam class, we calculate similarly:

P(Free∣Not Spam)=0,P(Buy∣Not Spam)=12,P(Offer∣Not Spam)=12,P(Money∣Not Spam)
=12P(\text{Free} | \text{Not Spam}) = 0, \quad P(\text{Buy} | \text{Not Spam}) = \
frac{1}{2}, \quad P(\text{Offer} | \text{Not Spam}) = \frac{1}{2}, \quad P(\
text{Money} | \text{Not Spam}) = \frac{1}{2}

Step 4: Making Predictions

Now that we have trained the classifier, we can use it to classify a new email based on its features.

Test Data:

Suppose we have a new email with the following features:

  "Free" = 1

 "Buy" = 0

 "Offer" = 1

 "Money" = 0

We need to predict whether this email is Spam or Not Spam.

1. Calculate Posterior Probability for Spam: Using Bayes' Theorem, the posterior probability for
Spam is:

P(Spam∣X)∝P(Spam)⋅P(Free∣Spam)⋅P(Buy∣Spam)⋅P(Offer∣Spam)⋅P(Money∣Spam)P(\text{Spam} | X) \
propto P(\text{Spam}) \cdot P(\text{Free} | \text{Spam}) \cdot P(\text{Buy} | \text{Spam}) \cdot P(\
text{Offer} | \text{Spam}) \cdot P(\text{Money} | \text{Spam})

Substituting the values:

P(Spam∣X)∝35⋅1⋅13⋅1⋅0=0P(\text{Spam} | X) \propto \frac{3}{5} \cdot 1 \cdot \frac{1}{3} \cdot 1 \

cdot 0 = 0

2. Calculate Posterior Probability for Not Spam: Similarly, for Not Spam:

P(Not Spam∣X)∝P(Not Spam)⋅P(Free∣Not Spam)⋅P(Buy∣Not Spam)⋅P(Offer∣Not Spam)⋅P(Money∣Not Sp

am)P(\text{Not Spam} | X) \propto P(\text{Not Spam}) \cdot P(\text{Free} | \text{Not Spam}) \cdot
P(\text{Buy} | \text{Not Spam}) \cdot P(\text{Offer} | \text{Not Spam}) \cdot P(\text{Money} | \
text{Not Spam})

Substituting the values:

P(Not Spam∣X)∝25⋅0⋅12⋅12⋅12=0P(\text{Not Spam} | X) \propto \frac{2}{5} \cdot 0 \cdot \frac{1}{2} \

cdot \frac{1}{2} \cdot \frac{1}{2} = 0

Since both the probabilities are 0, it appears something went wrong in our calculation. This is often
due to the problem of zero probabilities when certain features are missing in a particular class. This
can be handled by Laplace smoothing (additive smoothing), which ensures that no probability is
exactly zero.

Step 5: Model Evaluation

After training the model, it's essential to evaluate how well the classifier performs. This can be done
using:

 Confusion Matrix: This shows the number of true positives, false positives, true negatives,
and false negatives.

 Accuracy, Precision, Recall, F1-Score: These metrics help assess the classifier's performance.

Conclusion

In this example, we've learned how a Naïve Bayes classifier works, from training on labeled data to
making predictions on new, unseen data. Despite its simplicity, Naïve Bayes can perform surprisingly
well on text classification tasks, especially when the features are conditionally independent, or when
the conditional independence assumption does not dramatically violate the real-world structure of
the data.
Bayesian Belief Networks (BBNs) in Machine Learning

A Bayesian Belief Network (BBN), also known as a Bayesian Network or Probabilistic Graphical
Model, is a probabilistic model that represents a set of variables and their conditional dependencies
via a directed acyclic graph (DAG). Each node in the graph represents a random variable, and the
edges between the nodes represent conditional dependencies. These networks are used for
reasoning under uncertainty, making predictions, and understanding the relationships between
variables in complex systems.

Key Components of a Bayesian Network:

1. Nodes: Each node represents a random variable in the model. These variables can be
discrete (e.g., "rain" as yes/no) or continuous (e.g., "temperature" as a real number).

2. Edges: The directed edges between nodes represent conditional dependencies. An edge
from node AA to node BB implies that BB is conditionally dependent on AA. In other words,
the value of BB is influenced by the value of AA.

3. Conditional Probability Distributions (CPDs): Each node has a Conditional Probability

Distribution (CPD) that specifies the probability of the node given its parents in the graph. If
a node has no parents (i.e., it is a root node), its CPD is simply its marginal probability
distribution.

4. Directed Acyclic Graph (DAG): The structure of the network is a DAG, meaning that there are
no cycles in the graph. This ensures that the relationships between the variables are causal
or directional, and it allows for efficient computation of joint probabilities.

How Bayesian Networks Work:

A Bayesian Network enables us to represent a set of random variables and their probabilistic
relationships. Given this, the joint probability distribution of all variables in the network can be
factored into the product of the conditional probabilities:

P(X1,X2,...,Xn)=∏i=1nP(Xi∣Parents(Xi))P(X_1, X_2, ..., X_n) = \prod_{i=1}^{n} P(X_i | \text{Parents}

(X_i))

Where:

 P(Xi∣Parents(Xi))P(X_i | \text{Parents}(X_i)) is the conditional probability of node XiX_i given

its parents (the nodes with direct edges pointing to XiX_i).

Example: Medical Diagnosis

Imagine we want to build a Bayesian Network for diagnosing a disease based on symptoms. Let’s
consider three variables:

1. Disease: Whether the person has the disease or not (binary: yes/no).

2. Fever: Whether the person has a fever (binary: yes/no).

3. Cough: Whether the person has a cough (binary: yes/no).

We could model this problem using a simple Bayesian Network where:

 The Disease variable has a direct influence on the Fever and Cough variables.
  The Fever and Cough variables are conditionally independent given the Disease.

The network might look like this:

Disease --> Fever

--> Cough

Conditional Probability Distributions:

 P(Disease)P(Disease) might represent the probability of having the disease (e.g., 0.1 for
having the disease, 0.9 for not having it).

 P(Fever∣Disease)P(Fever | Disease) represents the probability of having a fever given the

disease (e.g., if the person has the disease, they have a 70% chance of having a fever).

 P(Cough∣Disease)P(Cough | Disease) represents the probability of having a cough given the

disease.

Example CPDs:

1. P(Disease)={Disease=True:0.1,Disease=False:0.9}P(Disease) = \{ \text{Disease} = \text{True}:

0.1, \text{Disease} = \text{False}: 0.9 \}

2. P(Fever∣Disease)={Fever=True,Disease=True:0.7,Fever=False,Disease=True:0.3}P(Fever |
Disease) = \{ \text{Fever} = \text{True}, \text{Disease} = \text{True}: 0.7, \text{Fever} = \
text{False}, \text{Disease} = \text{True}: 0.3 \}

3. P(Cough∣Disease)={Cough=True,Disease=True:0.8,Cough=False,Disease=True:0.2}P(Cough |
Disease) = \{ \text{Cough} = \text{True}, \text{Disease} = \text{True}: 0.8, \text{Cough} = \
text{False}, \text{Disease} = \text{True}: 0.2 \}

Using these CPDs, we can calculate the joint probability of different combinations of symptoms and
disease status, and also use Bayes' Theorem to make inferences or predictions.

Inferences in Bayesian Networks:

A key advantage of Bayesian Networks is their ability to perform inference: once a model is trained
(by specifying the structure and the CPDs), you can update the probabilities of certain variables given
evidence about others. This process is known as probabilistic inference and involves calculating
posterior probabilities.

For example, if we observe that a person has a fever, we can compute the probability that they have
the disease, given this evidence. The general process of inference is:

1. Forward Inference: Given prior information (e.g., prior probabilities and evidence), compute
the likelihood of unobserved variables.

2. Backward Inference: Given observed evidence, update the posterior probabilities of

unknown or hidden variables.

This is particularly useful in diagnostic problems where new evidence (e.g., the appearance of new
symptoms) can update the likelihood of a disease.
Learning from Data in Bayesian Networks:

Bayesian Networks can be learned from data in two main ways:

1. Structure Learning:

o Involves determining the structure of the network (i.e., which nodes are connected
to which others) from data.

o Methods include constraint-based approaches (e.g., PC algorithm), score-based

approaches (e.g., Bayesian Dirichlet equivalence), and hybrid approaches.

2. Parameter Learning:

o Once the structure is known, the next step is to estimate the parameters (the
conditional probability distributions) of the network from the data.

o This can be done using Maximum Likelihood Estimation (MLE) or Bayesian

Estimation.

Advantages of Bayesian Networks:

 Uncertainty representation: BBNs naturally handle uncertainty by modeling random

variables with probability distributions.

 Flexibility: They can model complex, real-world dependencies among variables.

 Interpretability: The graphical structure provides an intuitive understanding of the

relationships between variables.

 Inference: BBNs allow for efficient probabilistic inference, even with incomplete data.

 Decision Support: Can be used to make informed decisions by updating beliefs based on new
evidence.

Challenges of Bayesian Networks:

 Scalability: For very large networks with many variables, the computations can become
expensive.

 Learning from data: Learning both the structure and parameters from data can be
challenging, especially if the data is sparse or noisy.

 Complexity: Building and maintaining a large Bayesian Network can be complex, especially
when dealing with many variables and interactions.

Applications of Bayesian Networks:

1. Medical Diagnosis: Used to model the relationships between symptoms and diseases,
helping doctors make diagnostic decisions.

2. Risk Management: In industries like finance and insurance, BBNs can model uncertainties in
economic systems and predict risks.

3. Robotics: Used in robot decision-making, where various sensors provide uncertain data.
 4. Natural Language Processing (NLP): BBNs can be used for probabilistic parsing and speech
recognition.

5. Bioinformatics: Used to model gene interactions and other biological networks where
uncertainty is inherent.

Conclusion:

Bayesian Belief Networks (BBNs) are a powerful tool for reasoning under uncertainty and modeling
complex dependencies between random variables. Their ability to represent conditional
independence and perform probabilistic inference makes them widely applicable in fields such as
medical diagnosis, robotics, and risk management. However, designing and training large Bayesian
Networks can be computationally expensive, especially for large datasets. Despite these challenges,
BBNs remain one of the most popular techniques for probabilistic modeling in machine learning and
artificial intelligence.

The EM Algorithm in Machine Learning

The Expectation-Maximization (EM) Algorithm is an iterative optimization technique used to

estimate the parameters of statistical models, particularly when the data is incomplete or has latent
(hidden) variables. It is commonly used in machine learning and statistics for models where the
observations are governed by unobserved or missing data.

The EM Algorithm is especially useful in situations where directly computing the likelihood of the
data is difficult due to the presence of latent variables. The algorithm consists of two main steps:

1. Expectation (E-step): Given the current estimate of the parameters, the algorithm computes
the expected value of the hidden (latent) variables, based on the observed data.

2. Maximization (M-step): Using the expected values of the latent variables from the E-step,
the algorithm then maximizes the likelihood of the complete data (observed data + latent
data) with respect to the model parameters.

The key idea is to iteratively improve the estimates of the model parameters by alternately
performing these two steps, with the overall goal of maximizing the likelihood of the observed data.

Steps of the EM Algorithm

The EM Algorithm alternates between the E-step and the M-step until convergence. The steps can
be outlined as follows:

1. Initialization: Start with an initial guess for the model parameters θ(0)\theta^{(0)}.

2. E-Step (Expectation Step):

o Given the current parameters θ(t)\theta^{(t)}, calculate the expected log-likelihood

of the missing data (latent variables), conditioned on the observed data and the
current parameters.

o Essentially, this step computes an expectation over the latent variables given the
current parameter values.

3. M-Step (Maximization Step):

 o Maximize the expected log-likelihood found in the E-step with respect to the model
parameters θ\theta.

o The result of this step is a new set of parameters θ(t+1)\theta^{(t+1)}.

4. Convergence Check:

o Check for convergence of the algorithm, typically by evaluating if the change in the
log-likelihood or parameter values between iterations is below a certain threshold.

5. Repeat: If convergence is not reached, update the parameters and repeat the E-step and M-
step until convergence.

Mathematical Formulation of the EM Algorithm

Let XX denote the observed data and ZZ denote the latent (hidden) variables. The likelihood of the
observed data XX is:

P(X∣θ)=∫P(X,Z∣θ)dZP(X | \theta) = \int P(X, Z | \theta) dZ

where P(X,Z∣θ)P(X, Z | \theta) is the joint likelihood of the observed data XX and latent variables ZZ,
and θ\theta represents the model parameters.

E-Step:

In the E-step, we compute the expected value of the log-likelihood of the latent variables ZZ given
the observed data XX and the current parameters θ(t)\theta^{(t)}. This is done using the conditional
distribution of the latent variables:

Q(θ∣θ(t))=EZ∣X,θ(t)[log⁡P(X,Z∣θ)]Q(\theta | \theta^{(t)}) = \mathbb{E}_{Z | X, \theta^{(t)}}[\log P(X, Z

| \theta)]

Where EZ∣X,θ(t)\mathbb{E}_{Z | X, \theta^{(t)}} denotes the expectation of the complete-data log-

likelihood with respect to the posterior distribution of the latent variables.

M-Step:

In the M-step, we maximize the expected log-likelihood found in the E-step with respect to the
parameters θ\theta:

θ(t+1)=arg⁡max⁡θQ(θ∣θ(t))\theta^{(t+1)} = \arg \max_{\theta} Q(\theta | \theta^{(t)})

Example of the EM Algorithm: Gaussian Mixture Model (GMM)

One of the most common applications of the EM algorithm is in Gaussian Mixture Models (GMMs),
which are probabilistic models used to represent a mixture of multiple Gaussian distributions.

Model Setup:

 We assume the data points X={x1,x2,...,xN}X = \{x_1, x_2, ..., x_N\} are generated from a
mixture of KK Gaussian distributions, each with its own mean μk\mu_k and covariance
matrix Σk\Sigma_k.

 The latent variable ZZ indicates which Gaussian component generated each data point.

The model can be expressed as:

P(X∣θ)=∑k=1KπkN(X∣μk,Σk)P(X | \theta) = \sum_{k=1}^{K} \pi_k \mathcal{N}(X | \mu_k, \Sigma_k)

Where:

 πk\pi_k is the weight of the kk-th Gaussian component (the mixing coefficient).

 N(X∣μk,Σk)\mathcal{N}(X | \mu_k, \Sigma_k) is the Gaussian distribution with mean μk\mu_k

and covariance matrix Σk\Sigma_k.

E-Step in GMM:

In the E-step, we compute the responsibility γik\gamma_{ik}, which represents the probability that
data point xix_i was generated by the kk-th Gaussian component:

γik=πkN(xi∣μk,Σk)∑j=1KπjN(xi∣μj,Σj)\gamma_{ik} = \frac{\pi_k \mathcal{N}(x_i | \mu_k, \Sigma_k)}{\

sum_{j=1}^{K} \pi_j \mathcal{N}(x_i | \mu_j, \Sigma_j)}

M-Step in GMM:

In the M-step, we update the parameters based on the responsibilities computed in the E-step:

1. Update the mixing coefficients:

πk=1N∑i=1Nγik\pi_k = \frac{1}{N} \sum_{i=1}^{N} \gamma_{ik}

2. Update the means:

μk=∑i=1Nγikxi∑i=1Nγik\mu_k = \frac{\sum_{i=1}^{N} \gamma_{ik} x_i}{\sum_{i=1}^{N} \gamma_{ik}}

3. Update the covariances:

Σk=∑i=1Nγik(xi−μk)(xi−μk)T∑i=1Nγik\Sigma_k = \frac{\sum_{i=1}^{N} \gamma_{ik} (x_i - \mu_k)(x_i

- \mu_k)^T}{\sum_{i=1}^{N} \gamma_{ik}}

Iteration:

 These steps are repeated until convergence (i.e., when the parameters stop changing
significantly between iterations).

Advantages of the EM Algorithm:

1. Handles Missing Data: The EM algorithm is very effective when there is incomplete or
missing data, as it treats the missing data as latent variables.

2. General Framework: The algorithm can be applied to a wide variety of problems involving
latent variables, not just Gaussian mixtures.

3. Simple to Implement: The algorithm is conceptually simple and does not require complex
optimization techniques.

Disadvantages of the EM Algorithm:

1. Local Convergence: The EM algorithm is sensitive to the initial values of the parameters. It
can converge to a local maximum of the likelihood rather than the global maximum, so good
initialization is important.

2. Slow Convergence: The algorithm can sometimes converge slowly, especially when the
likelihood function has many parameters.
 3. Requires Good Model Assumptions: The EM algorithm assumes a specific probabilistic
model. If the assumptions about the underlying distribution are incorrect, the algorithm may
not perform well.

Applications of the EM Algorithm:

 Gaussian Mixture Models (GMMs): Clustering and density estimation.

 Hidden Markov Models (HMMs): In speech recognition, bioinformatics, and time-series

modeling.

 Missing Data Imputation: Estimating missing values in datasets.

 Factor Analysis: In data compression, signal processing, and exploratory data analysis.

 Image Segmentation: In computer vision tasks where pixel classifications might be hidden.

Conclusion:

The Expectation-Maximization (EM) algorithm is a powerful tool for parameter estimation in the
presence of latent variables and incomplete data. It is widely used in machine learning for tasks like
clustering, density estimation, and missing data imputation. Despite its simplicity and versatility, it
requires careful initialization and can suffer from slow convergence or local maxima, making it
important to combine it with techniques like random restarts or cross-validation.

Computational Learning Theory in Machine Learning

Computational Learning Theory (CLT) is a field within computer science and artificial intelligence
that focuses on the study of the theoretical aspects of learning algorithms. It provides a framework
for understanding the limits, capabilities, and efficiency of machine learning algorithms. CLT aims to
answer fundamental questions about how algorithms can learn from data and how their
performance can be quantified in terms of computation, sample complexity, and generalization.

In essence, CLT attempts to formalize the process of learning, providing mathematical foundations to
the idea of a machine learning model that improves its performance over time based on the data it is
exposed to.

Key Concepts in Computational Learning Theory

1. Learning Models: CLT examines different models of learning. A learning model typically
consists of a class of possible hypotheses or functions that the learning algorithm can choose
from. The goal of the learner is to select the best hypothesis that accurately predicts or
classifies unseen data based on the observed data.

2. Sample Complexity: This is the number of examples (samples) that are required to train a
model to a certain level of accuracy. A key question in CLT is how many samples are needed
for a learning algorithm to generalize well, i.e., how well it can perform on unseen data.

3. Generalization: This refers to a model’s ability to perform well on unseen data, after having
learned from a finite set of training data. In CLT, generalization is mathematically defined in
terms of PAC learning (Probably Approximately Correct), which provides a framework for
understanding how a model can generalize from a finite set of examples.

4. Hypothesis Space: The set of all possible hypotheses that the learner can consider. The size
and complexity of the hypothesis space directly affect the difficulty of learning. In CLT, the
 hypothesis space is typically assumed to be finite or countably infinite, and the goal is to find
the best hypothesis within this space.

5. VC Dimension: The Vapnik-Chervonenkis (VC) dimension is a measure of the capacity (or

complexity) of a hypothesis space. It tells us how complex a model is by quantifying the
largest number of points that can be shattered (correctly classified in all possible ways) by
the hypothesis space. The VC dimension plays a crucial role in the theory of generalization,
influencing how well an algorithm can perform on new, unseen data.

6. PAC Learning: Probably Approximately Correct (PAC) Learning is a formal model of learning
that provides guarantees on how well a learning algorithm can perform given a set of
assumptions. In PAC learning, the learner aims to produce a hypothesis that is approximately
correct with high probability, based on a finite set of training examples. The goal is to achieve
a hypothesis that is close to the true function (with respect to some performance measure,
like accuracy) with high probability, given a limited number of training examples.

o PAC Learnability: A concept that formalizes whether a concept class (a set of

possible hypotheses) can be learned efficiently (in polynomial time) from a finite
number of samples.

7. Overfitting and Underfitting: These are fundamental concerns in machine learning that are
studied within CLT.

o Overfitting occurs when a model learns the noise or irrelevant patterns in the
training data, resulting in poor performance on new, unseen data.

o Underfitting occurs when a model is too simple to capture the underlying patterns in
the data.

8. Bias-Variance Tradeoff: In CLT, the bias-variance tradeoff describes how a model’s

performance depends on its complexity and how it generalizes to new data. Models with
high bias are often simple and underfit the data, while models with high variance are more
complex and overfit the data.

Important Theoretical Results in CLT

1. The PAC Learning Framework:

o A concept class CC is said to be PAC learnable if there exists a learning algorithm

that, for any distribution over the data, can learn an approximately correct
hypothesis with high probability after seeing a polynomial number of examples.

o The algorithm must output a hypothesis that is approximately correct with a

probability of at least 1 - δ\delta, where δ\delta is a small error probability, and the
hypothesis should have an error of at most ϵ\epsilon, where ϵ\epsilon is a small
error tolerance.

2. The No-Free-Lunch Theorem:

o The No-Free-Lunch (NFL) theorem states that no single learning algorithm performs
best for all possible problems. In other words, every learning algorithm has its
strengths and weaknesses depending on the specific problem and data distribution.
 o The theorem emphasizes that generalization is dependent on the problem at hand,
and no algorithm will universally outperform others on every task.

3. Vapnik-Chervonenkis (VC) Dimension:

o The VC dimension is a measure of the capacity of a hypothesis class. It is defined as

the largest set of points that can be classified in all possible ways by hypotheses from
the class.

o A high VC dimension suggests that a hypothesis class is more flexible (and potentially
more prone to overfitting), while a low VC dimension implies that the model is
simpler and less likely to overfit.

4. Shattering and Learnability:

o A concept class is said to be shattered by a hypothesis class if, for any set of data
points, the hypothesis class can perfectly classify them in all possible ways.

o The ability of a hypothesis class to shatter sets of points directly relates to its VC
dimension, and the more points it can shatter, the higher the capacity of the
hypothesis class.

5. Sample Complexity in PAC Learning:

o In PAC learning, the sample complexity refers to the number of training examples
required to learn an accurate hypothesis with high probability.

o The sample complexity is typically proportional to the VC dimension of the

hypothesis class and the inverse of the error tolerance ϵ\epsilon and the confidence
1−δ1 - \delta. More formally, the number of examples required is roughly
O(1ϵ⋅VC dimension⋅log⁡(1δ))O\left(\frac{1}{\epsilon} \cdot \text{VC dimension} \
cdot \log\left(\frac{1}{\delta}\right)\right).

Applications of Computational Learning Theory

1. Supervised Learning:

o CLT provides formal guarantees about the learnability of supervised learning

algorithms, such as decision trees, linear classifiers, and neural networks. For
example, the PAC framework can guarantee that a learner can find an approximately
correct hypothesis given a sufficient number of labeled training samples.

2. Unsupervised Learning:

o While CLT is more focused on supervised learning, concepts like clustering and
density estimation can also be studied in the context of learnability and the sample
complexity of finding good clustering structures in data.

3. Online Learning:

o CLT extends to online learning scenarios, where data arrives sequentially, and the
learner must update its hypothesis incrementally as new data arrives. This is relevant
for applications like recommendation systems or stock price prediction.

4. Reinforcement Learning:
 o Computational learning theory is also applied to reinforcement learning, where the
agent must learn to take actions in an environment to maximize cumulative rewards.
The theory can provide insights into sample complexity and the efficiency of
reinforcement learning algorithms.

Challenges and Limitations

1. Real-World Complexity:

o While CLT provides valuable insights into the theoretical limits of learning algorithms,
real-world problems are often much more complex than the assumptions made in
theory (e.g., noisy data, non-stationary environments, unstructured data). This
means that practical machine learning might not always align perfectly with
theoretical guarantees.

2. Computational Efficiency:

o Many of the algorithms analyzed in CLT are not always computationally efficient in
practice. For example, finding an optimal hypothesis might require exponential time
in some cases, even though a theoretical algorithm might guarantee PAC learning.

3. Assumptions about Data:

o CLT often assumes that the data follows a specific distribution (like the IID
assumption), but in real-world applications, data might be noisy, unstructured, or
non-independent, making theoretical results harder to apply directly.

Conclusion

Computational Learning Theory (CLT) plays a crucial role in understanding the theoretical
foundations of machine learning. It provides valuable insights into the complexity of learning
algorithms, the sample complexity required to achieve good generalization, and the efficiency of
algorithms in terms of computation and data. Concepts such as PAC learning, VC dimension, and the
bias-variance tradeoff are central to CLT and help researchers and practitioners understand the
limitations and capabilities of machine learning models.

Although CLT provides a robust framework, real-world machine learning often deals with problems
that go beyond its assumptions. Nevertheless, the insights from CLT continue to influence the design
and analysis of learning algorithms across many domains.