Scribd
Upload
8 views64 pages

Gradient Descend

Uploaded by

bn23mer2r15
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
8 views64 pages

Gradient Descend

Uploaded by

bn23mer2r15
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 64

Introduction to Gradient Descent Algorithm

The gradient descent algorithm is an optimization algorithm used to minimize an objective


function, commonly called the cost or loss function in machine learning.

This cost function is a mathematical function used to calculate loss or error which is the difference between the
model’s predicted value and the actual value in the real world for a given input data set.

While training a model, the ask is to find a high-performance model (model with a minimal loss).

To find the best model, we need to minimize the cost function. And, minimizing the cost function is
about finding the most optimal values of parameters or weights vector that give the minimal loss for
any given input data set. The algorithm used to minimize the cost function is called the gradient
descent algorithm.
How gradient descent works
How gradient descent works
The term “gradient descent” is derived from two key concepts: “gradient” and “descent.”

1.Gradient:

In mathematics and physics, the gradient of a function at any point is a multi-dimensional vector that
points in the direction of the steepest increase of that function.
The magnitude of the gradient vector at a point corresponds to the rate of greatest increase in the function’s
value.

In the context of a cost function in machine learning, the gradient indicates the direction in which the cost
function rises most rapidly.
2. Descent:

The word “descent” refers to the method’s objective of moving downwards to find the minima of the
function.
Since the gradient points toward the greatest increase, the opposite direction, in which the algorithm moves, will
lead to the steepest decrease.

By iteratively moving in the direction opposite to the gradient (hence “descending”), the algorithm finds the
point where the cost or loss function attains its minimum value.
The plot visualizes the concept of gradient descent on a simple quadratic function f(x)=x2.

The red dots represent the steps taken by the gradient descent algorithm starting from an initial
point (here, x=9) and moving towards the minimum of the function at x=0.

Each step is determined by the gradient (slope) of the function at that point, and the algorithm
iteratively moves in the direction that reduces the function’s value (i.e., descending).

The red dashed line connects these steps, illustrating the path taken by the algorithm as it seeks
the function’s minimum.

This exemplifies how gradient descent navigates the function’s landscape to find the point of
lowest value.
For the function f (x) = (x − 3)2 + 4, find the x-value that produces
the lowest point of that function. Let’s use gradient descent to do
it.
Performing gradient descent for a linear regression
Performing gradient descent for a linear regression
Performing gradient descent for a linear regression
Performing gradient descent for a linear regression
Performing gradient descent for a linear regression
Performing gradient descent for a linear regression
Performing gradient descent for a linear regression
Performing gradient descent for a linear regression
Performing gradient descent for a linear regression
Performing gradient descent for a linear regression
Gradient Descent for Linear Regression Using SymPy
Performing gradient descent for a linear regression
Solving linear regression using SymP
Plotting the loss function for linear regression

You might also like