Scribd
Upload
62 views3 pages

20-Delta Rule-02-09-2024

Uploaded by

gupta
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
62 views3 pages

20-Delta Rule-02-09-2024

Uploaded by

gupta
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 3

The Delta Rule, also known as the LMS (Least Mean Squares) algorithm or Widrow-Hoff

learning rule, is a gradient descent method used in machine learning to train linear classifiers
and also as part of the backpropagation algorithm in neural networks. It is a rule for updating
the weights of the neurons in artificial neural networks based on the error made by the
network on a data example.

The Delta Rule can be described as follows for a single output unit:

1. Error Term (δ): Calculate the difference between the


predicted output (y^) and the actual target output (y) for a
given training example. This error is often calculated as:

δ=y−y^
2. Weight Update: Adjust each weight (w) in proportion to
the error term and the input value (x) for that weight. The
update is typically done using the following equation:

wnew=wold+α⋅δ⋅x

where α is the learning rate, a hyperparameter that controls


how much we adjust the weights with respect to the loss
gradient. The lower the value, the slower we travel along the
downward slope.

While the Delta Rule can be used for single-layer networks, for multilayer networks (like the
one in the backpropagation example), a generalization of the Delta Rule is applied, which
takes into account the derivative of the activation function used in the neurons (sigmoid in the
example).
The learning rate (α) is a crucial hyperparameter in the training process of neural networks
and affects the convergence of the backpropagation algorithm:

 If the learning rate is too high, the algorithm might


overshoot the optimal solution, leading to divergence or
unstable training.
 If the learning rate is too low, the training process will be
very slow, as the algorithm makes very small adjustments to
the weights. This could also lead to the algorithm getting
stuck in local minima.

Selecting an appropriate learning rate is dependent on the specific problem and often requires
empirical tuning or techniques like learning rate schedules (where the learning rate changes
over time), or adaptive learning rate methods like Adam, RMSprop, or AdaGrad, which
adjust the learning rate during training based on the weights.
The Least Mean Squares (LMS) algorithm is an adaptive filter algorithm, designed to mimic
a desired filter by finding the filter coefficients that relate to producing the least mean squares
of the error signal (difference between the desired and the actual signal). It's a type of
adaptive filter known for its simplicity and robustness in the presence of noisy signals.
The LMS algorithm uses the method of steepest descent and updates the filter's weights in a
way that the mean square error between the desired and actual output signals is minimized.
It's widely used in signal processing for adaptive filtering, noise cancellation, and echo
cancellation.

The LMS algorithm is iterative and continues to process data and update weights until the
weights converge to a stable solution that minimizes the error, or until a predefined number of
iterations is reached.
The simplicity of the LMS algorithm makes it attractive for many applications, but it can be
slow to converge and sensitive to the choice of the learning rate μ. If μ is too small, the
algorithm will converge slowly, while a μ that is too large may cause the algorithm to become
unstable and diverge.
import numpy as np
class LMSFilter:
def __init__(self, num_taps, learning_rate):
self.num_taps = num_taps
self.learning_rate = learning_rate
self.weights = np.zeros(num_taps)
def filter(self, input_vector):
return np.dot(self.weights, input_vector)
def update(self, desired_output, input_vector):
actual_output = self.filter(input_vector)
error = desired_output - actual_output
self.weights += self.learning_rate * error * input_vector
return error

# Example usage:
num_taps = 4 # Number of filter coefficients
mu = 0.01 # Learning rate
lms_filter = LMSFilter(num_taps, mu)

# Dummy data
desired_output = np.array([1, 2, 3, 4, 5])
input_signal = np.array([1, 2, 3, 4, 5])

for i in range(len(desired_output)):
# Assuming input_signal is a sliding window over your signal
input_vector = input_signal[i:i+num_taps]
if len(input_vector) == num_taps:
error = lms_filter.update(desired_output[i], input_vector)
print(f'Error: {error}')

You might also like