Introduction to ML

What is Machine Learning?

Machine learning is the field of study that gives capability to machines or

computers to learn without being explicitly programmed.

It is a branch of Artificial Intelligence and computer science that helps build a

model based on training data and make predictions and decisions without being
constantly programmed. Machine Learning is used in various applications such
as email filtering, speech recognition, computer vision, self-driven cars,
Amazon product recommendation, etc.

In the real world, we are surrounded by humans who can learn everything from
their experiences with their learning capability, and we have computers or
machines which work on our instructions. But can a machine also learn from
experiences or past data like a human does? So here comes the role of Machine
Learning.

A subset of artificial intelligence known as machine learning focuses primarily

on the creation of algorithms that enable a computer to independently learn
from data and previous experiences.

Arthur Samuel first used the term "machine learning" in 1959. It could be
summarized as follows:

Without being explicitly programmed, machine learning enables a machine

to automatically learn from data, improve performance from experiences,
and predict things.

Machine learning algorithms create a mathematical model that, without being

explicitly programmed, aids in making predictions or decisions with the
assistance of sample historical data, or training data. For the purpose of
developing predictive models, machine learning brings together statistics and
computer science. Algorithms that learn from historical data are either
constructed or utilized in machine learning. The performance will rise in
proportion to the quantity of information we provide.

A machine can learn if it can gain more data to improve its performance.

PROF.MISS. RANDIVE T. S. 1
How does Machine Learning work

A machine learning system builds prediction models, learns from previous data,
and predicts the output of new data whenever it receives it. The amount of data
helps to build a better model that accurately predicts the output, which in turn
affects the accuracy of the predicted output.

Let's say we have a complex problem in which we need to make predictions.

Instead of writing code, we just need to feed the data to generic algorithms,
which build the logic based on the data and predict the output. Our perspective
on the issue has changed as a result of machine learning. The Machine Learning
algorithm's operation is depicted in the following block diagram:

Features of Machine Learning:

o Machine learning uses data to detect various patterns in a given dataset.

o It can learn from past data and improve automatically.
o It is a data-driven technology.
o Machine learning is much similar to data mining as it also deals with the
huge amount of the data.

Need for Machine Learning

The demand for machine learning is steadily rising. Because it is able to

perform tasks that are too complex for a person to directly implement, machine
learning is required. Humans are constrained by our inability to manually access
vast amounts of data; as a result, we require computer systems, which is where
machine learning comes in to simplify our lives.

PROF.MISS. RANDIVE T. S. 2
By providing them with a large amount of data and allowing them to
automatically explore the data, build models, and predict the required output,
we can train machine learning algorithms. The cost function can be used to
determine the amount of data and the machine learning algorithm's
performance. We can save both time and money by using machine learning.

The significance of AI can be handily perceived by its utilization's cases,

Presently, AI is utilized in self-driving vehicles, digital misrepresentation
identification, face acknowledgment, and companion idea by Facebook, and so
on. Different top organizations, for example, Netflix and Amazon have
constructed AI models that are utilizing an immense measure of information to
examine the client interest and suggest item likewise.

Following are some key points which show the importance of Machine
Learning:

o Rapid increment in the production of data

o Solving complex problems, which are difficult for a human
o Decision making in various sector including finance
o Finding hidden patterns and extracting useful information from data.

Importance of Machine Learning:

o Rapid increment in the production of data

o Solving complex problems, which are difficult for a human
o Decision making in various sector including finance
o Finding hidden patterns and extracting useful information from data.

PROF.MISS. RANDIVE T. S. 3
Types of Machine Learning

Machine learning is the branch of Artificial Intelligence that focuses on

developing models and algorithms that let computers learn from data and
improve from previous experience without being explicitly programmed for
every task. In simple words, ML teaches the systems to think and understand
like humans by learning from the data.
Machine learning is generally a training system to learn from past experiences
and improve performance over time. Machine learning helps to predict massive
amounts of data. It helps to deliver fast and accurate results to get profitable
opportunities.

Types of Machine Learning

There are several types of machine learning, each with special characteristics
and applications. Some of the main types of machine learning algorithms are as
follows:
1. Supervised Machine Learning
2. Unsupervised Machine Learning
3. Semi-Supervised Machine Learning
4. Reinforcement Learning

1. Supervised Machine Learning

Supervised learning is defined as when a model gets trained on a “Labelled
Dataset”. Labelled datasets have both input and output parameters.
In Supervised Learning algorithms learn to map points between inputs and
correct outputs. It has both training and validation datasets labelled.

Supervised Learning

PROF.MISS. RANDIVE T. S. 4
Example: Consider a scenario where you have to build an image classifier to
differentiate between cats and dogs. If you feed the datasets of dogs and cats
labelled images to the algorithm, the machine will learn to classify between a
dog or a cat from these labeled images. When we input new dog or cat images
that it has never seen before, it will use the learned algorithms and predict
whether it is a dog or a cat. This is how supervised learning works, and this is
particularly an image classification.

There are two main categories of supervised learning that are mentioned
below:
 Classification
 Regression

Classification
Classification deals with predicting categorical target variables, which
represent discrete classes or labels. For instance, classifying emails as spam or
not spam, or predicting whether a patient has a high risk of heart disease.
Classification algorithms learn to map the input features to one of the
predefined classes.

Here are some classification algorithms:

 Logistic Regression
 Support Vector Machine
 Random Forest
 Decision Tree
 K-Nearest Neighbors (KNN)
 Naive Bayes

Regression
Regression, on the other hand, deals with predicting continuous target
variables, which represent numerical values. For example, predicting the price
of a house based on its size, location, and amenities, or forecasting the sales of a
product. Regression algorithms learn to map the input features to a continuous
numerical value.

Here are some regression algorithms:

 Linear Regression
 Polynomial Regression
 Ridge Regression
 Lasso Regression
 Decision tree
 Random Forest

Advantages of Supervised Machine Learning

PROF.MISS. RANDIVE T. S. 5
 Supervised Learning models can have high accuracy as they are trained
on labelled data.
 The process of decision-making in supervised learning models is often
interpretable.
 It can often be used in pre-trained models which saves time and resources
when developing new models from scratch.

Disadvantages of Supervised Machine Learning

 It has limitations in knowing patterns and may struggle with unseen or
unexpected patterns that are not present in the training data.
 It can be time-consuming and costly as it relies on labeled data only.
 It may lead to poor generalizations based on new data.

Applications of Supervised Learning

Supervised learning is used in a wide variety of applications, including:
 Image classification: Identify objects, faces, and other features in images.
 Natural language processing: Extract information from text, such as
sentiment, entities, and relationships.
 Speech recognition: Convert spoken language into text.
 Recommendation systems: Make personalized recommendations to users.
 Predictive analytics: Predict outcomes, such as sales, customer churn, and
stock prices.
 Medical diagnosis: Detect diseases and other medical conditions.
 Fraud detection: Identify fraudulent transactions.
 Autonomous vehicles: Recognize and respond to objects in the
environment.
 Email spam detection: Classify emails as spam or not spam.
 Quality control in manufacturing: Inspect products for defects.
 Credit scoring: Assess the risk of a borrower defaulting on a loan.
 Gaming: Recognize characters, analyze player behavior, and create NPCs.
 Customer support: Automate customer support tasks.
 Weather forecasting: Make predictions for temperature, precipitation, and
other meteorological parameters.
 Sports analytics: Analyze player performance, make game predictions, and
optimize strategies.

2. Unsupervised Machine Learning

Unsupervised Learning Unsupervised learning is a type of machine learning
technique in which an algorithm discovers patterns and relationships
using unlabeled data. Unlike supervised learning, unsupervised learning doesn’t
involve providing the algorithm with labeled target outputs. The primary goal of
Unsupervised learning is often to discover hidden patterns, similarities, or
clusters within the data, which can then be used for various purposes, such as
data exploration, visualization, dimensionality reduction, and more.

PROF.MISS. RANDIVE T. S. 6
Unsupervised Learning

Example: Consider that you have a dataset that contains information about the
purchases you made from the shop. Through clustering, the algorithm can group
the same purchasing behavior among you and other customers, which reveals
potential customers without predefined labels. This type of information can help
businesses get target customers as well as identify outliers.

There are two main categories of unsupervised learning that are mentioned
below:
 Clustering
 Association

Clustering
Clustering is the process of grouping data points into clusters based on their
similarity. This technique is useful for identifying patterns and relationships in
data without the need for labeled examples.
Here are some clustering algorithms:
 K-Means Clustering algorithm
 Mean-shift algorithm
 DBSCAN Algorithm
 Principal Component Analysis
 Independent Component Analysis

Association
Association rule learning is a technique for discovering relationships between
items in a dataset. It identifies rules that indicate the presence of one item
implies the presence of another item with a specific probability.
Here are some association rule learning algorithms:
 Apriori Algorithm
 Eclat
 FP-growth Algorithm

PROF.MISS. RANDIVE T. S. 7
Advantages of Unsupervised Machine Learning
 It helps to discover hidden patterns and various relationships between the
data.
 Used for tasks such as customer segmentation, anomaly
detection, and data exploration.
 It does not require labeled data and reduces the effort of data labeling.

Disadvantages of Unsupervised Machine Learning

 Without using labels, it may be difficult to predict the quality of the model’s
output.
 Cluster Interpretability may not be clear and may not have meaningful
interpretations.
 It has techniques such as autoencoders and dimensionality reduction that can
be used to extract meaningful features from raw data.

Applications of Unsupervised Learning

Here are some common applications of unsupervised learning:
 Clustering: Group similar data points into clusters.
 Anomaly detection: Identify outliers or anomalies in data.
 Dimensionality reduction: Reduce the dimensionality of data while
preserving its essential information.
 Recommendation systems: Suggest products, movies, or content to users
based on their historical behavior or preferences.
 Topic modeling: Discover latent topics within a collection of documents.
 Density estimation: Estimate the probability density function of data.
 Image and video compression: Reduce the amount of storage required for
multimedia content.
 Data preprocessing: Help with data preprocessing tasks such as data
cleaning, imputation of missing values, and data scaling.
 Market basket analysis: Discover associations between products.
 Genomic data analysis: Identify patterns or group genes with similar
expression profiles.
 Image segmentation: Segment images into meaningful regions.
 Community detection in social networks: Identify communities or groups
of individuals with similar interests or connections.
 Customer behavior analysis: Uncover patterns and insights for better
marketing and product recommendations.
 Content recommendation: Classify and tag content to make it easier to
recommend similar items to users.
 Exploratory data analysis (EDA): Explore data and gain insights before
defining specific tasks.

3. Semi-Supervised Learning

PROF.MISS. RANDIVE T. S. 8
Semi-Supervised learning is a machine learning algorithm that works between
the supervised and unsupervised learning so it uses both labelled and
unlabelled data. It’s particularly useful when obtaining labeled data is costly,
time-consuming, or resource-intensive. This approach is useful when the dataset
is expensive and time-consuming. Semi-supervised learning is chosen when
labeled data requires skills and relevant resources in order to train or learn from
it.
We use these techniques when we are dealing with data that is a little bit labeled
and the rest large portion of it is unlabeled. We can use the unsupervised
techniques to predict labels and then feed these labels to supervised techniques.
This technique is mostly applicable in the case of image data sets where usually
all images are not labeled.

Semi-Supervised Learning

Example: Consider that we are building a language translation model, having

labeled translations for every sentence pair can be resources intensive. It allows
the models to learn from labeled and unlabeled sentence pairs, making them
more accurate. This technique has led to significant improvements in the quality
of machine translation services.

Types of Semi-Supervised Learning Methods

There are a number of different semi-supervised learning methods each with its
own characteristics. Some of the most common ones include:
 Graph-based semi-supervised learning: This approach uses a graph to
represent the relationships between the data points. The graph is then used to
propagate labels from the labeled data points to the unlabeled data points.
 Label propagation: This approach iteratively propagates labels from the
labeled data points to the unlabeled data points, based on the similarities
between the data points.

PROF.MISS. RANDIVE T. S. 9
 Co-training: This approach trains two different machine learning models on
different subsets of the unlabeled data. The two models are then used to label
each other’s predictions.
 Self-training: This approach trains a machine learning model on the labeled
data and then uses the model to predict labels for the unlabeled data. The
model is then retrained on the labeled data and the predicted labels for the
unlabeled data.
 Generative adversarial networks (GANs): GANs are a type of deep
learning algorithm that can be used to generate synthetic data. GANs can be
used to generate unlabeled data for semi-supervised learning by training two
neural networks, a generator and a discriminator.

Advantages of Semi- Supervised Machine Learning

 It leads to better generalization as compared to supervised learning, as it
takes both labeled and unlabeled data.
 Can be applied to a wide range of data.

Disadvantages of Semi- Supervised Machine Learning

 Semi-supervised methods can be more complex to implement compared to
other approaches.
 It still requires some labeled data that might not always be available or easy
to obtain.
 The unlabeled data can impact the model performance accordingly.

Applications of Semi-Supervised Learning

Here are some common applications of semi-supervised learning:
 Image Classification and Object Recognition: Improve the accuracy of
models by combining a small set of labeled images with a larger set of
unlabeled images.
 Natural Language Processing (NLP): Enhance the performance of
language models and classifiers by combining a small set of labeled text data
with a vast amount of unlabeled text.
 Speech Recognition: Improve the accuracy of speech recognition by
leveraging a limited amount of transcribed speech data and a more extensive
set of unlabeled audio.
 Recommendation Systems: Improve the accuracy of personalized
recommendations by supplementing a sparse set of user-item interactions
(labeled data) with a wealth of unlabeled user behavior data.
 Healthcare and Medical Imaging: Enhance medical image analysis by
utilizing a small set of labeled medical images alongside a larger set of
unlabeled images.

PROF.MISS. RANDIVE T. S. 10
4. Reinforcement Machine Learning
Reinforcement machine learning algorithm is a learning method that interacts
with the environment by producing actions and discovering errors. Trial, error,
and delay are the most relevant characteristics of reinforcement learning. In
this technique, the model keeps on increasing its performance using Reward
Feedback to learn the behavior or pattern. These algorithms are specific to a
particular problem e.g. Google Self Driving car, AlphaGo where a bot competes
with humans and even itself to get better and better performers in Go Game.
Each time we feed in data, they learn and add the data to their knowledge which
is training data. So, the more it learns the better it gets trained and hence
experienced.

Here are some of most common reinforcement learning algorithms:

 Q-learning: Q-learning is a model-free RL algorithm that learns a Q-
function, which maps states to actions. The Q-function estimates the
expected reward of taking a particular action in a given state.
 SARSA (State-Action-Reward-State-Action): SARSA is another model-
free RL algorithm that learns a Q-function. However, unlike Q-learning,
SARSA updates the Q-function for the action that was actually taken, rather
than the optimal action.
 Deep Q-learning: Deep Q-learning is a combination of Q-learning and deep
learning. Deep Q-learning uses a neural network to represent the Q-function,
which allows it to learn complex relationships between states and actions.

Reinforcement Machine Learning

Example: Consider that you are training an AI agent to play a game like chess.
The agent explores different moves and receives positive or negative feedback
based on the outcome. Reinforcement Learning also finds applications in which
they learn to perform tasks by interacting with their surroundings.

PROF.MISS. RANDIVE T. S. 11
Types of Reinforcement Machine Learning
There are two main types of reinforcement learning:
Positive reinforcement
 Rewards the agent for taking a desired action.
 Encourages the agent to repeat the behavior.
 Examples: Giving a treat to a dog for sitting, providing a point in a game for
a correct answer.
Negative reinforcement
 Removes an undesirable stimulus to encourage a desired behavior.
 Discourages the agent from repeating the behavior.
 Examples: Turning off a loud buzzer when a lever is pressed, avoiding a
penalty by completing a task.

Advantages of Reinforcement Machine Learning

 It has autonomous decision-making that is well-suited for tasks and that can
learn to make a sequence of decisions, like robotics and game-playing.
 This technique is preferred to achieve long-term results that are very difficult
to achieve.
 It is used to solve a complex problems that cannot be solved by conventional
techniques.

Disadvantages of Reinforcement Machine Learning

 Training Reinforcement Learning agents can be computationally expensive
and time-consuming.
 Reinforcement learning is not preferable to solving simple problems.
 It needs a lot of data and a lot of computation, which makes it impractical
and costly.

Applications of Reinforcement Machine Learning

Here are some applications of reinforcement learning:
 Game Playing: RL can teach agents to play games, even complex ones.
 Robotics: RL can teach robots to perform tasks autonomously.
 Autonomous Vehicles: RL can help self-driving cars navigate and make
decisions.
 Recommendation Systems: RL can enhance recommendation algorithms by
learning user preferences.
 Healthcare: RL can be used to optimize treatment plans and drug discovery.
 Natural Language Processing (NLP): RL can be used in dialogue systems
and chatbots.
 Finance and Trading: RL can be used for algorithmic trading.
 Supply Chain and Inventory Management: RL can be used to optimize
supply chain operations.
 Energy Management: RL can be used to optimize energy consumption.

PROF.MISS. RANDIVE T. S. 12
 Game AI: RL can be used to create more intelligent and adaptive NPCs in
video games.
 Adaptive Personal Assistants: RL can be used to improve personal
assistants.
 Virtual Reality (VR) and Augmented Reality (AR): RL can be used to
create immersive and interactive experiences.
 Industrial Control: RL can be used to optimize industrial processes.
 Education: RL can be used to create adaptive learning systems.
 Agriculture: RL can be used to optimize agricultural operations.

PROF.MISS. RANDIVE T. S. 13
Difference between Supervised & Unsupervised Learning
Supervised Learning Unsupervised Learning
Supervised learning algorithms are Unsupervised learning algorithms are
trained using labeled data. trained using unlabeled data.
Supervised learning model takes direct Unsupervised learning model does not
feedback to check if it is predicting take any feedback.
correct output or not.
Supervised learning model predicts the Unsupervised learning model finds the
output. hidden patterns in data.
In supervised learning, input data is In unsupervised learning, only input
provided to the model along with the data is provided to the model.
output.
The goal of supervised learning is to The goal of unsupervised learning is to
train the model so that it can predict the find the hidden patterns and useful
output when it is given new data. insights from the unknown dataset.
Supervised learning needs supervision Unsupervised learning does not need
to train the model. any supervision to train the model.
Supervised learning can be categorized Unsupervised Learning can be
in Classification and Regression proble classified
ms. in Clustering and Associations proble
ms.
Supervised learning can be used for Unsupervised learning can be used for
those cases where we know the input as those cases where we have only input
well as corresponding outputs. data and no corresponding output data.
Supervised learning model produces an Unsupervised learning model may
accurate result. give less accurate result as compared
to supervised learning.
Supervised learning is not close to true Unsupervised learning is more close to
Artificial intelligence as in this, we first the true Artificial Intelligence as it
train the model for each data, and then learns similarly as a child learns daily
only it can predict the correct output. routine things by his experiences.
It includes various algorithms such as It includes various algorithms such as
Linear Regression, Logistic Regression, Clustering, KNN, and Apriori
Support Vector Machine, Multi-class algorithm.
Classification, Decision tree, Bayesian
Logic, etc.

PROF.MISS. RANDIVE T. S. 14
 Applications of Machine learning

Machine learning is a buzzword for today's technology, and it is growing very

rapidly day by day. We are using machine learning in our daily life even
without knowing it such as Google Maps, Google assistant, Alexa, etc. Below
are some most trending real-world applications of Machine Learning:

1. Image Recognition:

Image recognition is one of the most common applications of machine learning.

It is used to identify objects, persons, places, digital images, etc. The popular
use case of image recognition and face detection is, Automatic friend tagging
suggestion:

Facebook provides us a feature of auto friend tagging suggestion. Whenever we

upload a photo with our Facebook friends, then we automatically get a tagging
suggestion with name, and the technology behind this is machine learning's face
detection and recognition algorithm.

It is based on the Facebook project named "Deep Face," which is responsible

for face recognition and person identification in the picture.

2. Speech Recognition

While using Google, we get an option of "Search by voice," it comes under

speech recognition, and it's a popular application of machine learning.

PROF.MISS. RANDIVE T. S. 15
Speech recognition is a process of converting voice instructions into text, and it
is also known as "Speech to text", or "Computer speech recognition." At
present, machine learning algorithms are widely used by various applications of
speech recognition. Google assistant, Siri, Cortana, and Alexa are using
speech recognition technology to follow the voice instructions.

3. Traffic prediction:

If we want to visit a new place, we take help of Google Maps, which shows us
the correct path with the shortest route and predicts the traffic conditions.

It predicts the traffic conditions such as whether traffic is cleared, slow-moving,

or heavily congested with the help of two ways:

o Real Time location of the vehicle form Google Map app and sensors
o Average time has taken on past days at the same time.

Everyone who is using Google Map is helping this app to make it better. It takes
information from the user and sends back to its database to improve the
performance.

4. Product recommendations:

Machine learning is widely used by various e-commerce and entertainment

companies such as Amazon, Netflix, etc., for product recommendation to the
user. Whenever we search for some product on Amazon, then we started getting
an advertisement for the same product while internet surfing on the same
browser and this is because of machine learning.

Google understands the user interest using various machine learning algorithms
and suggests the product as per customer interest.

As similar, when we use Netflix, we find some recommendations for

entertainment series, movies, etc., and this is also done with the help of machine
learning.

5. Self-driving cars:

One of the most exciting applications of machine learning is self-driving cars.

Machine learning plays a significant role in self-driving cars. Tesla, the most
popular car manufacturing company is working on self-driving car. It is using
unsupervised learning method to train the car models to detect people and
objects while driving.

PROF.MISS. RANDIVE T. S. 16
 6. Email Spam and Malware Filtering:

Whenever we receive a new email, it is filtered automatically as important,

normal, and spam. We always receive an important mail in our inbox with the
important symbol and spam emails in our spam box, and the technology behind
this is Machine learning. Below are some spam filters used by Gmail:

o Content Filter
o Header filter
o General blacklists filter
o Rules-based filters
o Permission filters

Some machine learning algorithms such as Multi-Layer Perceptron, Decision

tree, and Naïve Bayes classifier are used for email spam filtering and malware
detection.

7. Virtual Personal Assistant:

We have various virtual personal assistants such as Google

assistant, Alexa, Cortana, Siri. As the name suggests, they help us in finding
the information using our voice instruction. These assistants can help us in
various ways just by our voice instructions such as Play music, call someone,
Open an email, Scheduling an appointment, etc.

These virtual assistants use machine learning algorithms as an important part.

These assistant record our voice instructions, send it over the server on a cloud,
and decode it using ML algorithms and act accordingly.

8. Online Fraud Detection:

Machine learning is making our online transaction safe and secure by detecting
fraud transaction. Whenever we perform some online transaction, there may be
various ways that a fraudulent transaction can take place such as fake
accounts, fake ids, and steal money in the middle of a transaction. So to detect
this, Feed Forward Neural network helps us by checking whether it is a
genuine transaction or a fraud transaction.

For each genuine transaction, the output is converted into some hash values, and
these values become the input for the next round. For each genuine transaction,
there is a specific pattern which gets change for the fraud transaction hence, it
detects it and makes our online transactions more secure.

PROF.MISS. RANDIVE T. S. 17
 9. Stock Market trading:

Machine learning is widely used in stock market trading. In the stock market,
there is always a risk of up and downs in shares, so for this machine
learning's long short term memory neural network is used for the prediction
of stock market trends.

10. Medical Diagnosis:

In medical science, machine learning is used for diseases diagnoses. With this,
medical technology is growing very fast and able to build 3D models that can
predict the exact position of lesions in the brain. It helps in finding brain tumors
and other brain-related diseases easily.

11. Automatic Language Translation:

Nowadays, if we visit a new place and we are not aware of the language then it
is not a problem at all, as for this also machine learning helps us by converting
the text into our known languages. Google's GNMT (Google Neural Machine
Translation) provide this feature, which is a Neural Machine Learning that
translates the text into our familiar language, and it called as automatic
translation.

The technology behind the automatic translation is a sequence to sequence

learning algorithm, which is used with image recognition and translates the text
from one language to another language.

PROF.MISS. RANDIVE T. S. 18
 General steps to follow in a Machine Learning Problem

Machine learning is a method of data analysis that automates analytical model

building. In simple terms, machine learning is “making a machine learn”.
Machine learning is a new field that combines many traditional disciplines. It is
a subset of AI.
What is ML pipeline?
 ML pipeline expresses the workflow by providing a systematic way on how
to proceed with the machine learning model.
 ML pipelines automate the process of machine learning and following the
pipeline makes the process of making ML models systematic and easy.

Here is the diagrammatic view of the ML pipeline:

MACHINE LEARNING PIPELINE

 The Machine Learning pipeline starts with data collection and integration.
After data is collected analysis and visualization of data is done. Further, the
most crucial step feature selection and engineering is performed then the
model is trained. After that model, evaluation is done and our model
becomes ready for prediction!
 To understand the pipeline well, consider building an ML model for a
company’s customer care service. Consider a company XYZ, as an online
book shop, that delivers books and kindle to its customer and this company
wants an improved customer care service. It wants that if a customer calls
the helpline for any kind of issue, say for replacement of books, complaint of
kindle purchased or some other services. The company wants to ensure that
the customer’s call gets directed to the right service person in minimum time
and that the process should be smooth. To build a model for customer care
services of the company, we will use the ML pipeline for the systematic
development of the model.

PROF.MISS. RANDIVE T. S. 19
1. Data Collection and integration:
 The first step of the ML pipeline involves the collection of data and
integration of data.
 Data collected acts as an input to the model (data preparation phase)
 Inputs are called features.
 Data collected in the case of our considered example involves a lot of data.
The collected data should answer the following questions- What is past
customer history? What were the past orders? Is the customer a prime
member of our bookstore? Does the customer own a kindle? Has the
customer made any previous complaints? What was the most number of
complaints?
 The more the data is, more the better our model becomes.
 Once the data is collected we need to integrate and prepare the data.
 Integration of data means placing all related data together.
 Then data preparation phase starts in which we manually and critically
explore the data.
 The data preparation phase tells the developer that is the data matching the
expectations. Is there enough info to make an accurate prediction? Is the data
consistent?

2. Exploratory Data Analysis and Visualisation:

 Once the data is prepared developer needs to visualize the data to have a
better understanding of relationships within the dataset.
 When we get to see data, we can notice the unseen patterns that we may not
have noticed in the first phase.
 It helps developers easily identify missing data and outliers.
 Data visualization can be done by plotting histograms, scatter plots, etc.
 After visualization is done data is analyzed so that developer can decide
what ML technique he may use.
 In the considered example case unsupervised learning may be used to
analyze customer purchasing habits.

3. Feature Selection and Engineering:

 Feature selection means selecting what features the developer wants to use
within the model.
 Features should be selected so that a minimum correlation exists between
them and a maximum correlation exists between the selected features and
output.
 Feature engineering is the process to manipulate the original data into new
and potential data that has a lot many features within it.
 In simple words Feature engineering is converting raw data into useful data
or getting the maximum out of the original data.

PROF.MISS. RANDIVE T. S. 20
 Feature engineering is arguably the most crucial and time-consuming step of
the ML pipeline.
 Feature selection and engineering answers questions – Are these features
going to make any sense in our prediction?
 It deals with the accuracy and precision of data.

4. Model Training:
 After the first three steps are done completely we enter the model training
phase.
 It is the first step officially when the developer gets to train the model on
basis of data.
 To train the model, data is split into three parts- Training data, validation
data, and test data.
 Around 70%-80% of data goes into the training data set which is used in
training the model.
 Validation data is also known as development set or dev set and is used to
avoid overfitting or underfitting situations i.e. enabling hyperparameter
tuning.
 Hyperparameter tuning is a technique used to combat overfitting and
underfitting.
 Validation data is used during model evaluation.
 Around 10%-15% of data is used as validation data.
 Rest 10%-15% of data goes into the test data set. Test data set is used for
testing after the model preparation.
 It is crucial to randomize data sets while splitting the data to get an accurate
model.
 Data can be randomized using Scikit learn in python.

5. Model Evaluation:
 After the model training, validation, or development data is used to evaluate
the model.
 To get the most accurate predictions to test data may be used for further
model evaluation.
 A confusion matrix is created after model evaluation to calculate accuracy
and precision numerically.
 After model evaluation, our model enters the final stage that is prediction.

6. Prediction:
 In the prediction phase developer deploys the model.
 After model deployment, it becomes ready to make predictions.
 Predictions are made on training data and test data to have a better
understanding of the build model.

PROF.MISS. RANDIVE T. S. 21
Challenges or Issues in Machine Learning

"Machine Learning" is one of the most popular technology among all data
scientists and machine learning enthusiasts. It is the most effective Artificial
Intelligence technology that helps create automated learning systems to take
future decisions without being constantly programmed. Machine Learning
offers great opportunities, but some issues need to be solved.

1. Inadequate Training Data

The major issue that comes while using machine learning algorithms is the lack
of quality as well as quantity of data. Although data plays a vital role in the
processing of machine learning algorithms, many data scientists claim that
inadequate data, noisy data, and unclean data are extremely exhausting the
machine learning algorithms. For example, a simple task requires thousands of
sample data, and an advanced task such as speech or image recognition needs
millions of sample data examples. Further, data quality is also important for the
algorithms to work ideally, but the absence of data quality is also found in
Machine Learning applications. Data quality can be affected by some factors as
follows:

o Noisy Data- It is responsible for an inaccurate prediction that affects the

decision as well as accuracy in classification tasks.
o Incorrect data- It is also responsible for faulty programming and results
obtained in machine learning models. Hence, incorrect data may affect
the accuracy of the results also.
o Generalizing of output data- Sometimes, it is also found that
generalizing output data becomes complex, which results in
comparatively poor future actions.

2. Poor quality of data

As we have discussed above, data plays a significant role in machine learning,

and it must be of good quality as well. Noisy data, incomplete data, inaccurate
data, and unclean data lead to less accuracy in classification and low-quality
results. Hence, data quality can also be considered as a major common problem
while processing machine learning algorithms.

PROF.MISS. RANDIVE T. S. 22
 3. Non-representative training data

To make sure our training model is generalized well or not, we have to ensure
that sample training data must be representative of new cases that we need to
generalize. The training data must cover all cases that are already occurred as
well as occurring.

Further, if we are using non-representative training data in the model, it results

in less accurate predictions. A machine learning model is said to be ideal if it
predicts well for generalized cases and provides accurate decisions. If there is
less training data, then there will be a sampling noise in the model, called the
non-representative training set. It won't be accurate in predictions. To overcome
this, it will be biased against one class or a group.

Hence, we should use representative data in training to protect against being

biased and make accurate predictions without any drift.

4. Overfitting and Underfitting

Overfitting:

Overfitting is one of the most common issues faced by Machine Learning

engineers and data scientists. Whenever a machine learning model is trained
with a huge amount of data, it starts capturing noise and inaccurate data into the
training data set. It negatively affects the performance of the model. Let's
understand with a simple example where we have a few training data sets such
as 1000 mangoes, 1000 apples, 1000 bananas, and 5000 papayas. Then there is
a considerable probability of identification of an apple as papaya because we
have a massive amount of biased data in the training data set; hence prediction
got negatively affected. The main reason behind overfitting is using non-linear
methods used in machine learning algorithms as they build non-realistic data
models. We can overcome overfitting by using linear and parametric algorithms
in the machine learning models.

Methods to reduce overfitting:

o Increase training data in a dataset.

o Reduce model complexity by simplifying the model by selecting one with
fewer parameters
o Ridge Regularization and Lasso Regularization
o Early stopping during the training phase
o Reduce the noise
o Reduce the number of attributes in training data.

PROF.MISS. RANDIVE T. S. 23
 o Constraining the model.

Underfitting:

Underfitting is just the opposite of overfitting. Whenever a machine learning

model is trained with fewer amounts of data, and as a result, it provides
incomplete and inaccurate data and destroys the accuracy of the machine
learning model.

Underfitting occurs when our model is too simple to understand the base
structure of the data, just like an undersized pant. This generally happens when
we have limited data into the data set, and we try to build a linear model with
non-linear data. In such scenarios, the complexity of the model destroys, and
rules of the machine learning model become too easy to be applied on this data
set, and the model starts doing wrong predictions as well.

Methods to reduce Underfitting:

o Increase model complexity

o Remove noise from the data
o Trained on increased and better features
o Reduce the constraints
o Increase the number of epochs to get better results.

5. Monitoring and maintenance

As we know that generalized output data is mandatory for any machine learning
model; hence, regular monitoring and maintenance become compulsory for the
same. Different results for different actions require data change; hence editing
of codes as well as resources for monitoring them also become necessary.

6. Getting bad recommendations

A machine learning model operates under a specific context which results in

bad recommendations and concept drift in the model. Let's understand with an
example where at a specific time customer is looking for some gadgets, but now
customer requirement changed over time but still machine learning model
showing same recommendations to the customer while customer expectation
has been changed. This incident is called a Data Drift. It generally occurs when
new data is introduced or interpretation of data changes. However, we can
overcome this by regularly updating and monitoring data according to the
expectations.

PROF.MISS. RANDIVE T. S. 24
 7. Lack of skilled resources

Although Machine Learning and Artificial Intelligence are continuously

growing in the market, still these industries are fresher in comparison to others.
The absence of skilled resources in the form of manpower is also an issue.
Hence, we need manpower having in-depth knowledge of mathematics, science,
and technologies for developing and managing scientific substances for
machine learning.

8. Customer Segmentation

Customer segmentation is also an important issue while developing a machine

learning algorithm. To identify the customers who paid for the
recommendations shown by the model and who don't even check them. Hence,
an algorithm is necessary to recognize the customer behavior and trigger a
relevant recommendation for the user based on past experience.

9. Process Complexity of Machine Learning

The machine learning process is very complex, which is also another major
issue faced by machine learning engineers and data scientists. However,
Machine Learning and Artificial Intelligence are very new technologies but are
still in an experimental phase and continuously being changing over time. There
is the majority of hits and trial experiments; hence the probability of error is
higher than expected. Further, it also includes analyzing the data, removing data
bias, training data, applying complex mathematical calculations, etc., making
the procedure more complicated and quite tedious.

10. Data Bias

Data Biasing is also found a big challenge in Machine Learning. These errors
exist when certain elements of the dataset are heavily weighted or need more
importance than others. Biased data leads to inaccurate results, skewed
outcomes, and other analytical errors. However, we can resolve this error by
determining where data is actually biased in the dataset. Further, take necessary
steps to reduce it.

Methods to remove Data Bias:

o Research more for customer segmentation.

o Be aware of your general use cases and potential outliers.
o Combine inputs from multiple sources to ensure data diversity.
o Include bias testing in the development process.

PROF.MISS. RANDIVE T. S. 25
 o Analyze data regularly and keep tracking errors to resolve them easily.
o Review the collected and annotated data.
o Use multi-pass annotation such as sentiment analysis, content
moderation, and intent recognition.

Machine learning Life cycle

Machine learning has given the computer systems the abilities to automatically
learn without being explicitly programmed. But how does a machine learning
system work? So, it can be described using the life cycle of machine learning.
Machine learning life cycle is a cyclic process to build an efficient machine
learning project. The main purpose of the life cycle is to find a solution to the
problem or project.

Machine learning life cycle involves seven major steps, which are given below:

o Gathering Data
o Data preparation
o Data Wrangling
o Analyse Data
o Train the model
o Test the model
o Deployment

PROF.MISS. RANDIVE T. S. 26
The most important thing in the complete process is to understand the problem
and to know the purpose of the problem. Therefore, before starting the life
cycle, we need to understand the problem because the good result depends on
the better understanding of the problem.

In the complete life cycle process, to solve a problem, we create a machine

learning system called "model", and this model is created by providing
"training". But to train a model, we need data, hence, life cycle starts by
collecting data.

1. Gathering Data:

Data Gathering is the first step of the machine learning life cycle. The goal of
this step is to identify and obtain all data-related problems.

In this step, we need to identify the different data sources, as data can be
collected from various sources such as files, database, internet, or mobile
devices. It is one of the most important steps of the life cycle. The quantity and
quality of the collected data will determine the efficiency of the output. The
more will be the data, the more accurate will be the prediction.

This step includes the below tasks:

o Identify various data sources

o Collect data
o Integrate the data obtained from different sources

By performing the above task, we get a coherent set of data, also called as
a dataset. It will be used in further steps.

2. Data preparation

After collecting the data, we need to prepare it for further steps. Data
preparation is a step where we put our data into a suitable place and prepare it to
use in our machine learning training.

In this step, first, we put all data together, and then randomize the ordering of
data.

This step can be further divided into two processes:

PROF.MISS. RANDIVE T. S. 27
 o Data exploration:
It is used to understand the nature of data that we have to work with. We
need to understand the characteristics, format, and quality of data.
A better understanding of data leads to an effective outcome. In this, we
find Correlations, general trends, and outliers.
o Data pre-processing:
Now the next step is preprocessing of data for its analysis.

3. Data Wrangling

Data wrangling is the process of cleaning and converting raw data into a useable
format. It is the process of cleaning the data, selecting the variable to use, and
transforming the data in a proper format to make it more suitable for analysis in
the next step. It is one of the most important steps of the complete process.
Cleaning of data is required to address the quality issues.

It is not necessary that data we have collected is always of our use as some of
the data may not be useful. In real-world applications, collected data may have
various issues, including:

o Missing Values
o Duplicate data
o Invalid data
o Noise

So, we use various filtering techniques to clean the data.

It is mandatory to detect and remove the above issues because it can negatively
affect the quality of the outcome.

4. Data Analysis

Now the cleaned and prepared data is passed on to the analysis step. This step
involves:

o Selection of analytical techniques

o Building models
o Review the result
The aim of this step is to build a machine learning model to analyze the data
using various analytical techniques and review the outcome. It starts with the
determination of the type of the problems, where we select the machine learning

PROF.MISS. RANDIVE T. S. 28
techniques such as Classification, Regression, Cluster analysis, Association,
etc. then build the model using prepared data, and evaluate the model.

Hence, in this step, we take the data and use machine learning algorithms to
build the model.

5. Train Model

Now the next step is to train the model, in this step we train our model to
improve its performance for better outcome of the problem.

We use datasets to train the model using various machine learning algorithms.
Training a model is required so that it can understand the various patterns, rules,
and, features.

6. Test Model

Once our machine learning model has been trained on a given dataset, then we
test the model. In this step, we check for the accuracy of our model by
providing a test dataset to it.

Testing the model determines the percentage accuracy of the model as per the
requirement of project or problem.

7. Deployment

The last step of machine learning life cycle is deployment, where we deploy the
model in the real-world system.

If the above-prepared model is producing an accurate result as per our

requirement with acceptable speed, then we deploy the model in the real
system. But before deploying the project, we will check whether it is improving
its performance using available data or not. The deployment phase is similar to
making the final report for a project.

PROF.MISS. RANDIVE T. S. 29