Scribd
Upload
30 views14 pages

Loss Functions in Machine Learning

The document discusses loss functions in machine learning, highlighting their role in measuring the difference between model predictions and actual values. It covers types of loss functions, such as Mean Squared Error and Mean Absolute Error, and factors influencing their selection. Additionally, it addresses the credit assignment problem in neural networks and techniques for network pruning to optimize model performance.

Uploaded by

Stu udy
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
30 views14 pages

Loss Functions in Machine Learning

The document discusses loss functions in machine learning, highlighting their role in measuring the difference between model predictions and actual values. It covers types of loss functions, such as Mean Squared Error and Mean Absolute Error, and factors influencing their selection. Additionally, it addresses the credit assignment problem in neural networks and techniques for network pruning to optimize model performance.

Uploaded by

Stu udy
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 14

Lo ss Functions in

Machine
Learning
A loss function is a key component of machine learning
models,
acting as a mathematical measure of the difference
between the
model's predictions and the actual target values. This
presentation
investigate into the purpose, types, and application of loss
functions in both machine learning and deep learning.

DY Patil International Universityb1y Dr. Dipika Pradhan


Why Use a Lo ss
Function?
Feedback Optimization
The loss function acts as feedback for the model, It guides the optimization algorithm to adjust
the
providing a quantitative measure of its
performance. It model's parameters in the direction that
reduces the
tells the model how well or poorly it is
performing. error, leading to improved predictions.
Types of Lo ss Functions:
Regression
Mean Squared Error Mean Absolute Error
(MSE)
Penalizes large errors more heavily, making it (MAE)
Treats all errors equally, making it less
sensitive sensitive to
to outliers. Commonly used for predicting outliers than MSE. Useful when the
continuous distribution of
values. errors is skewed or contains outliers.
Choosing the Right Lo
ss
Function
1 Problem Type 2 Data
Different tasks, such
The presence of
Distribution
as
outliers
regression or can influence the
classification, require
choice
different loss
functions. of loss function, as
some
functions are more
sensitive to extreme
values.
3 Sensitivity to
The desired sensitivity to errors, whether large errors
Errors
should
be penalized more heavily or all errors should be
treated
equally, is crucial in selecting the appropriate loss
function.
Hessian matrix
The Hessian matrix is a square matrix of second-order partial derivatives
of a scalar-valued function. It describes the local curvature of a function
and is widely used in optimization, machine learning, and physics.
The credit assignment problem in neural networks refers to the challenge of
determining which neurons (or layers) are responsible for a given success or failure
in the network’s output. This is crucial for effectively updating the network's
weights during training.

Types of Credit Assignment Problems

Structural Credit Assignment – Identifying which neuron or synapse contributed


most to the error.
Temporal Credit Assignment – Assigning credit (or blame) for decisions made
based on past inputs, especially in sequential data.
Solutions to the Credit Assignment Problem
Backpropagation (BP) – The most widely used algorithm, which calculates
gradients using the chain rule and updates weights accordingly.
Reinforcement Learning (RL) Methods – Techniques like policy gradients or Q-
learning address credit assignment over long time horizons.
Attention Mechanisms – Used in deep learning to focus on relevant input features
dynamically.

Hebbian Learning – Based on "neurons that fire together, wire together,"


assigning credit based on activity correlation.
Evolutionary Algorithms – Assign fitness scores to entire models or subnetworks
rather than individual neurons.
Network Pruning Techniques in Neural Networks

Network pruning is a technique used to reduce the size of a neural network by removing
redundant or less important parameters (weights, neurons, or layers). This helps in
reducing computational cost and improving inference speed while maintaining accuracy.
1. Weight Pruning (Unstructured Pruning)
Removes individual weights based on their magnitude.
Common approach: Magnitude-Based Pruning (eliminate weights below a threshold).
Requires sparse matrix operations for efficient implementation.
2. Neuron Pruning (Structured Pruning)
•Removes entire neurons, filters, or layers.
•More hardware-friendly as it leads to structured reductions in computation.
3. Filter Pruning (Channel Pruning)
•Eliminates entire filters in CNNs.
4. Bayesian Pruning
•Uses Bayesian inference to determine the importance of network parameters and prune
accordingly.
Naïve Bayes Classifier
Understanding the
Code
Input Loss Output
The code takes two inputs: Functions
The functions The program prints the
y_true mean_squared_error, calculated
(the actual values) and y_pred(the mean_absolute_error, and
values for MSE, MAE, and BCE,
model's predictions).
binary_cross_entropy providing a quantitative
calculate the assessment of the model's
corresponding losses
performance.
based on the
input data.

You might also like