Machine learning

Machine learning is a growing technology which enables computers to learn

automatically from past data. Machine learning uses various algorithms for building
mathematical models and making predictions using historical data or information.
Currently, it is being used for various tasks such as image recognition, speech
recognition, email filtering, Facebook auto-tagging, recommender system, and
many more.

This machine learning tutorial gives you an introduction to machine learning along with
the wide range of machine learning techniques such as Supervised, Unsupervised,
and Reinforcement learning. You will learn about regression and classification models,
clustering methods, hidden Markov models, and various sequential models.

Features of Machine Learning:

o Machine learning uses data to detect various patterns in a given dataset.
o It can learn from past data and improve automatically.
o It is a data-driven technology.
o Machine learning is much similar to data mining as it also deals with the huge
amount of the data.

Classification of Machine Learning

At a broad level, machine learning can be classified into three types:

1. Supervised learning
2. Unsupervised learning
3. Reinforcement learning

1) Supervised Learning
Supervised learning is a type of machine learning method in which we provide sample
labeled data to the machine learning system in order to train it, and on that basis, it
predicts the output.
The system creates a model using labeled data to understand the datasets and learn
about each data, once the training and processing are done then we test the model by
providing a sample data to check whether it is predicting the exact output or not.

The goal of supervised learning is to map input data with the output data. The
supervised learning is based on supervision, and it is the same as when a student learns
things in the supervision of the teacher. The example of supervised learning is spam
filtering.

Supervised learning can be grouped further in two categories of algorithms:

o Classification
o Regression

2) Unsupervised Learning
Unsupervised learning is a learning method in which a machine learns without any
supervision.

The training is provided to the machine with the set of data that has not been labeled,
classified, or categorized, and the algorithm needs to act on that data without any
supervision. The goal of unsupervised learning is to restructure the input data into new
features or a group of objects with similar patterns.

In unsupervised learning, we don't have a predetermined result. The machine tries to

find useful insights from the huge amount of data. It can be further classifieds into two
categories of algorithms:

o Clustering
o Association

3) Reinforcement Learning
Reinforcement learning is a feedback-based learning method, in which a learning agent
gets a reward for each right action and gets a penalty for each wrong action. The agent
learns automatically with these feedbacks and improves its performance. In
reinforcement learning, the agent interacts with the environment and explores it. The
goal of an agent is to get the most reward points, and hence, it improves its
performance.
The robotic dog, which automatically learns the movement of his arms, is an example of
Reinforcement learning.

Bias and Variance in Machine Learning

Machine learning is a branch of Artificial Intelligence, which allows machines to perform
data analysis and make predictions. However, if the machine learning model is not
accurate, it can make predictions errors, and these prediction errors are usually known
as Bias and Variance. In machine learning, these errors will always be present as there is
always a slight difference between the model predictions and actual predictions. The
main aim of ML/data science analysts is to reduce these errors in order to get more
accurate results. In this topic, we are going to discuss bias and variance, Bias-variance
trade-off, Underfitting and Overfitting. But before starting, let's first understand what
errors in Machine learning are?

What is Bias?
In general, a machine learning model analyses the data, find patterns in it and make
predictions. While training, the model learns these patterns in the dataset and applies
them to test data for prediction. While making predictions, a difference occurs
between prediction values made by the model and actual values/expected
values, and this difference is known as bias errors or Errors due to bias. It can be
defined as an inability of machine learning algorithms such as Linear Regression to
capture the true relationship between the data points. Each algorithm begins with some
amount of bias because bias occurs from assumptions in the model, which makes the
target function simple to learn. A model has either:

o Low Bias: A low bias model will make fewer assumptions about the form of the target
function.
o High Bias: A model with a high bias makes more assumptions, and the model becomes
unable to capture the important features of our dataset. A high bias model also cannot
perform well on new data.

Generally, a linear algorithm has a high bias, as it makes them learn fast. The simpler the
algorithm, the higher the bias it has likely to be introduced. Whereas a nonlinear
algorithm often has low bias.

Some examples of machine learning algorithms with low bias are Decision Trees, k-
Nearest Neighbours and Support Vector Machines. At the same time, an algorithm
with high bias is Linear Regression, Linear Discriminant Analysis and Logistic
Regression.

Ways to reduce High Bias:

High bias mainly occurs due to a much simple model. Below are some ways to reduce
the high bias:

o Increase the input features as the model is underfitted.

o Decrease the regularization term.
o Use more complex models, such as including some polynomial features.

What is a Variance Error?

The variance would specify the amount of variation in the prediction if the different
training data was used. In simple words, variance tells that how much a random
variable is different from its expected value. Ideally, a model should not vary too
much from one training dataset to another, which means the algorithm should be good
in understanding the hidden mapping between inputs and output variables. Variance
errors are either of low variance or high variance.

Low variance means there is a small variation in the prediction of the target function
with changes in the training data set. At the same time, High variance shows a large
variation in the prediction of the target function with changes in the training dataset.

A model that shows high variance learns a lot and perform well with the training
dataset, and does not generalize well with the unseen dataset. As a result, such a model
gives good results with the training dataset but shows high error rates on the test
dataset.

Since, with high variance, the model learns too much from the dataset, it leads to
overfitting of the model. A model with high variance has the below problems:

o A high variance model leads to overfitting.

o Increase model complexities.

Usually, nonlinear algorithms have a lot of flexibility to fit the model, have high variance.
Some examples of machine learning algorithms with low variance are, Linear
Regression, Logistic Regression, and Linear discriminant analysis. At the same time,
algorithms with high variance are decision tree, Support Vector Machine, and K-
nearest neighbours.

Ways to Reduce High Variance:

o Reduce the input features or number of parameters as a model is overfitted.
o Do not use a much complex model.
o Increase the training data.
o Increase the Regularization term.

Linear Regression vs Logistic Regression

Linear Regression and Logistic Regression are the two famous Machine Learning
Algorithms which come under supervised learning technique. Since both the algorithms
are of supervised in nature hence these algorithms use labeled dataset to make the
predictions. But the main difference between them is how they are being used. The
Linear Regression is used for solving Regression problems whereas Logistic Regression
is used for solving the Classification problems. The description of both the algorithms is
given below along with difference table.

Linear Regression:

o Linear Regression is one of the most simple Machine learning algorithm that comes
under Supervised Learning technique and used for solving regression problems.
o It is used for predicting the continuous dependent variable with the help of independent
variables.
o The goal of the Linear regression is to find the best fit line that can accurately predict the
output for the continuous dependent variable.
 o If single independent variable is used for prediction then it is called Simple Linear
Regression and if there are more than two independent variables then such regression is
called as Multiple Linear Regression.
o By finding the best fit line, algorithm establish the relationship between dependent
variable and independent variable. And the relationship should be of linear nature.
o The output for Linear regression should only be the continuous values such as price, age,
salary, etc. The relationship between the dependent variable and independent variable
can be shown in below image:

In above image the dependent variable is on Y-axis (salary) and independent variable is
on x-axis(experience). The regression line can be written as:

y= a0+a1x+ ε

Where, a0 and a1 are the coefficients and ε is the error term.

Logistic Regression:

o Logistic regression is one of the most popular Machine learning algorithm that comes
under Supervised Learning techniques.
o It can be used for Classification as well as for Regression problems, but mainly used for
Classification problems.
o Logistic regression is used to predict the categorical dependent variable with the help of
independent variables.
o The output of Logistic Regression problem can be only between the 0 and 1.
o Logistic regression can be used where the probabilities between two classes is required.
Such as whether it will rain today or not, either 0 or 1, true or false etc.
 o Logistic regression is based on the concept of Maximum Likelihood estimation.
According to this estimation, the observed data should be most probable.
o In logistic regression, we pass the weighted sum of inputs through an activation function
that can map values in between 0 and 1. Such activation function is known as sigmoid
function and the curve obtained is called as sigmoid curve or S-curve. Consider the
below image:

o The equation for logistic regression is:

Difference between Linear Regression and Logistic Regression:

Linear Regression Logistic Regression

Linear regression is used to predict the Logistic Regression is used to predict the
continuous dependent variable using a given set categorical dependent variable using a given set
of independent variables. of independent variables.

Linear Regression is used for solving Regression Logistic regression is used for solving
problem. Classification problems.

In Linear regression, we predict the value of In logistic Regression, we predict the values of
continuous variables. categorical variables.
In linear regression, we find the best fit line, by In Logistic Regression, we find the S-curve by
which we can easily predict the output. which we can classify the samples.

Least square estimation method is used for Maximum likelihood estimation method is used
estimation of accuracy. for estimation of accuracy.

The output for Linear Regression must be a The output of Logistic Regression must be a
continuous value, such as price, age, etc. Categorical value such as 0 or 1, Yes or No, etc.

In Linear regression, it is required that relationship In Logistic regression, it is not required to have
between dependent variable and independent the linear relationship between the dependent
variable must be linear. and independent variable.

In linear regression, there may be collinearity In logistic regression, there should not be
between the independent variables. collinearity between the independent variable.

What is Inductive Learning Algorithm?

Inductive Learning Algorithm (ILA) is an iterative and inductive machine learning algorithm
that is used for generating a set of classification rules, which produces rules of the form “IF-
THEN”, for a set of examples, producing rules at each iteration and appending to the set of rules.
There are basically two methods for knowledge extraction firstly from domain experts and then
with machine learning. For a very large amount of data, the domain experts are not very useful
and reliable. So we move towards the machine learning approach for this work. To use machine
learning One method is to replicate the expert’s logic in the form of algorithms but this work is
very tedious, time taking, and expensive. So we move towards the inductive algorithms which
generate the strategy for performing a task and need not instruct separately at each step.
Why you should use Inductive Learning?
The ILA is a new algorithm that was needed even when other reinforcement learnings like ID3
and AQ were available.
The need was due to the pitfalls which were present in the previous algorithms, one of the major
pitfalls was the lack of generalization of rules.
The ID3 and AQ used the decision tree production method which was too specific which were
difficult to analyze and very slow to perform for basic short classification problems.
The decision tree-based algorithm was unable to work for a new problem if some attributes are
missing.
The ILA uses the method of production of a general set of rules instead of decision trees, which
overcomes the above problems
Basic Requirements to Apply Inductive Learning Algorithm
1. List the examples in the form of a table ‘T’ where each row corresponds to an example
and each column contains an attribute value.
2. Create a set of m training examples, each example composed of k attributes and a class
attribute with n possible decisions.
3. Create a rule set, R, having the initial value false.
4. Initially, all rows in the table are unmarked.
Necessary Steps for Implementation
Step 1: divide the table ‘T’ containing m examples into n sub-tables (t1, t2,…..tn). One table for
each possible value of the class attribute. (repeat steps 2-8 for each sub-table)
Step 2: Initialize the attribute combination count ‘ j ‘ = 1.
Step 3: For the sub-table on which work is going on, divide the attribute list into distinct
combinations, each combination with ‘j ‘ distinct attributes.
Step 4: For each combination of attributes, count the number of occurrences of attribute values
that appear under the same combination of attributes in unmarked rows of the sub-table under
consideration, and at the same time, not appears under the same combination of attributes of
other sub-tables. Call the first combination with the maximum number of occurrences the max-
combination ‘ MAX’.
Step 5: If ‘MAX’ == null, increase ‘ j ‘ by 1 and go to Step 3.
Step 6: Mark all rows of the sub-table where working, in which the values of ‘MAX’ appear, as
classified.
Step 7: Add a rule (IF attribute = “XYZ” –> THEN decision is YES/ NO) to R whose left-hand
side will have attribute names of the ‘MAX’ with their values separated by AND, and its right-
hand side contains the decision attribute value associated with the sub-table.
Step 8: If all rows are marked as classified, then move on to process another sub-table and go to
Step 2. Else, go to Step 4. If no sub-tables are available, exit with the set of rules obtained till
then.
Types of Inductive Bias in ML:
The most significant inductive biases in machine learning algorithms are listed here.
Types of Inductive Bias in ML

Maximum conditional independence:

It aims to maximize conditional independence if the hypothesis can be framed within a Bayesian
framework. The Naive Bayes classifier employs this bias.
Minimum cross-validation error:
It picks the hypothesis with the lowest cross-validation error when trying to decide between
them. Despite the fact that cross-validation may appear to be bias-free, the "no free lunch"
theorems demonstrate that cross-validation is in fact biased.
Maximum margin:
When dividing a group of students into two classes, try to make the boundary as wide as
possible. The bias in support vector machines is this. It is assumed that different classes often
have a lot of space between them.
Minimum description length:
When formulating a hypothesis, make an effort to keep the description as brief as possible.
Minimum features:
Unless a feature is supported by solid evidence, it should be removed. The underlying premise of
feature selection algorithms is this.
Nearest neighbors:
In a small neighborhood in feature space, it is reasonable to assume that the majority of the cases
belong to the same class. Assume that a case, for which the class is unknown, belongs to the
same class as the majority in the area. The k-nearest neighbors' algorithm employs this bias.
The underlying premise is that cases that are close to one another typically belong to the same
class.