Scribd
Upload
6 views4 pages

Chat GPT

Uploaded by

kk
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
6 views4 pages

Chat GPT

Uploaded by

kk
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

ChatGPT

details about adagrad rmsprop SGD GD optimizer


technique . exam perspective

Here’s a comprehensive breakdown of SGD, GD, AdaGrad, and RMSProp optimization


techniques with an emphasis on their exam relevance and details to remember:

1. Gradient Descent (GD)


Concept: Iteratively minimizes the cost function by calculating the gradient (slope) of the
cost function w.r.t model parameters.

Update Rule:

θ = θ − η ⋅ ∇J(θ)

Where:

θ: Parameters
η : Learning rate
∇J(θ): Gradient of the cost function.
Variants:

Batch GD: Uses the entire dataset for each update. Slow for large datasets.

Stochastic GD (SGD): Updates parameters using a single data point. Faster but
noisy.

Mini-batch GD: Updates parameters using small batches. Combines benefits of


batch and SGD.

Pros:

Conceptually simple.

Cons:

Fixed learning rate may lead to suboptimal convergence.

Sensitive to the shape of the cost surface.

1/4
2. Stochastic Gradient Descent (SGD)
Concept: A variant of GD where updates are made after evaluating a single data point.

Update Rule:

θ = θ − η ⋅ ∇J(θ; x(i) )

Where x(i) is a random data point from the training set.

Pros:

Faster updates.

Helps escape shallow minima or saddle points due to noise.

Cons:

High variance in updates can make convergence challenging.

Requires tuning learning rate carefully.

3. AdaGrad (Adaptive Gradient Algorithm)


Concept: Adjusts learning rate for each parameter based on the frequency of its
updates. Parameters with large gradients get smaller learning rates, and parameters
with small gradients get larger learning rates.

Update Rule:

gt = gt−1 + (∇J(θt ))2


​ ​ ​

η
θt+1 = θt − ⋅ ∇J(θt )
gt + ϵ
​ ​ ​ ​

​ ​

Where:

gt : Accumulated sum of squared gradients.


​

ϵ: Small value to prevent division by zero.


Pros:

Good for sparse data.

2/4
Automatically adjusts learning rate.

Cons:

Accumulated gradients can lead to very small updates over time (learning rate
decay).

4. RMSProp (Root Mean Square Propagation)


Concept: Fixes the decay issue of AdaGrad by introducing an exponentially decaying
moving average of squared gradients.

Update Rule:

gt = βgt−1 + (1 − β)(∇J(θt ))2


​ ​ ​

η
θt+1 = θt − ⋅ ∇J(θt )
gt + ϵ
​ ​ ​ ​

​ ​

Where:

β : Decay rate (commonly 0.9).


ϵ: Small value to avoid division by zero.
Pros:

Suitable for non-stationary objectives.

Handles noisy gradients well.

Cons:

Requires careful tuning of β and learning rate.

Exam Tips
Comparison: Be ready to compare techniques (e.g., AdaGrad adjusts rates based on past
gradients; RMSProp adds a moving average to counteract decay).

Applications:

Use GD for theoretical understanding and when you can afford full-batch updates.

3/4
Use SGD for large datasets.

Use AdaGrad for sparse features like text data.

Use RMSProp for RNNs or when the learning rate needs to adapt dynamically.

Mathematical Details: Memorize update rules and key parameters (e.g., η, β, ϵ) for
quick recall.

Advantages/Disadvantages: Highlight at least one for each method.

Practical Scenarios: Know which optimizer to use based on data size and feature
sparsity.

Quick Table for Revision


Optimizer Key Idea Pros Cons

GD Minimizes cost by moving in Simple, theoretical Slow for large datasets.


gradient direction. clarity.

SGD Updates on single data Fast, helps escape Noisy convergence.


points. shallow minima.

AdaGrad Adjusts learning rates based Good for sparse data. Learning rate decays too
on gradient history. fast.

RMSProp Combines moving average Handles non-stationary Requires


and adaptive rates. objectives well. hyperparameter tuning.

Would you like detailed examples for any of these optimizers?

4/4

You might also like