Module 2
Module 2
Introduction
Definition: An artificial neural network (ANN) consists of interconnected processing units (artificial
neurons) designed to perform tasks such as pattern recognition. While understanding the structure of a
neural network is essential, it's equally important to comprehend the rules that govern changes in
activation values and connection weights. These rules are encapsulated in the activation and synaptic
dynamics equations, which dictate the network's behaviour and learning process.
Key Concepts:
• Processing Units (Neurons): The basic elements that receive inputs, process them, and produce
outputs.
• Connections (Weights): Links between neurons that carry signals and can be adjusted during
learning.
• Activation Values: The output signal of a neuron, determined by its inputs and weights.
• Activation Dynamics: The rules that dictate changes in activation values over time.
• Synaptic Dynamics: The rules that dictate changes in connection weights over time.
Activation State
Definition: In a neural network with N processing units, the activation state at any given moment is
defined by the set of activation values of all neurons.
• The activation state changes over time, creating a trajectory in the activation state space.
• The trajectory leads to a solution state, reflecting the dynamics of the network.
• This Activation State trajectory is determined by the activation dynamics equations, which
specify the network's state at the next instant based on its current state.
Example: Consider a simple feedforward neural network designed to recognize handwritten digits. Each
neuron's activation value represents its response to a part of the input image. As the network processes
the image, the activation state changes, leading to the final recognition of the digit. The path from initial
activations to the final recognition state is the trajectory in the activation state space.
Weight State
Definition: The weight state of a network at any given moment is defined by the set of connection
weight values.
• For a given input, the network adjusts its weights to learn the input pattern.
• The weight state changes over time, creating a trajectory in the weight state space.
• This Weight State trajectory is determined by the synaptic dynamics, which dictate how
weights change in response to training.
Example: In the same handwritten digit recognition network, during training, the weights are adjusted
based on the errors in recognition. The changes in these weights form the trajectory in the weight state
space, leading to the network's ability to accurately recognize digits over time.
• Defined by the network's steady activation state in response to a specific input pattern.
• Changes with different input patterns.
• Reflects the network's immediate response to inputs.
Example: In our digit recognition network, short term memory is the network's current activation pattern
when presented with an image of a digit. Long term memory is the stable set of weights that the network
has learned through training to recognize different digits.
Understanding
1. Data Preparation
What Happens:
• You start with a dataset, which might be images, text, or any other type of data. This data is
divided into training, validation, and test sets.
Role of Dynamics:
• At this stage, activation and synaptic dynamics are not directly involved. This is about preparing
the data for the network to use.
What Happens:
• You design the architecture of the neural network, specifying the number of layers, neurons per
layer, activation functions, etc.
Role of Dynamics:
• Activation dynamics and synaptic dynamics are not yet active. This is about setting up the
network’s structure.
3. Training Phase
What Happens:
• Forward Pass: During training, you feed the training data into the network. Each input goes
through the layers of neurons, and the network produces an output (prediction).
o Activation Dynamics:
▪ As each input is processed, the neurons in the network compute their activations
based on the current weights and biases. This involves applying activation
functions to the weighted sums of inputs.
▪ Example: In an image classification task, as an image is processed, neurons in
the network rapidly adjust their activation values based on the input features, such
as edges or textures.
• Loss Calculation: After the forward pass, you compute the loss, which is a measure of how far
the network’s prediction is from the actual target value.
• Backward Pass: The network then performs backpropagation to update the weights based on
the loss. This involves calculating gradients and adjusting weights to minimize the error.
o Synaptic Dynamics:
▪ During backpropagation, synaptic dynamics are at work. The weights and biases
in the network are updated based on the gradients computed from the loss
function.
▪ Example: If the network predicts the wrong label, the weights are adjusted to
reduce the prediction error for similar future inputs. This process involves gradual
changes to weights over multiple iterations or epochs.
Role of Dynamics:
• Activation Dynamics: Active during the forward pass, where neurons quickly adjust their
activations in response to the input data.
• Synaptic Dynamics: Active during the backward pass, where weights are adjusted based on the
error to improve future predictions.
4. Validation Phase
What Happens:
• You use the validation set to tune hyperparameters and check how well the network performs on
unseen data. This helps to avoid overfitting and ensure generalization.
Role of Dynamics:
• Activation Dynamics: When feeding validation data into the network, the activation dynamics
still apply as the network processes inputs in real-time.
• Synaptic Dynamics: The weights are not updated during this phase; it’s about evaluating the
model’s performance.
5. Testing Phase
What Happens:
• After training and validation, you test the network on a separate test set to evaluate its
performance and generalization ability.
Role of Dynamics:
• Activation Dynamics: When test data is fed into the network, activation dynamics come into
play as the network computes predictions based on fixed weights.
• Synaptic Dynamics: Not active during testing, as the weights are not updated.
Summary
1. Training Phase:
o Activation Dynamics: Involves real-time changes in neuron activations during the
forward pass.
o Synaptic Dynamics: Involves gradual updates to weights during the backward pass.
2. Validation and Testing Phases:
o Activation Dynamics: Continues to play a role in processing inputs and generating
predictions.
o Synaptic Dynamics: Not active; weights are fixed and not updated.
In essence, activation dynamics are about how neurons respond and activate in real-time as data flows
through the network, while synaptic dynamics are about how the network learns and updates its weights
during training.
Activation Dynamics
Activation dynamics relate to the fluctuations at the neuronal level in a biological neural network.
In a neural network, think of each neuron (or brain cell) as a small unit that receives signals, processes
them, and then sends out its own signal. Activation dynamics is about how the signal inside a neuron
change over time as it receives inputs.
• Scenario: A security system uses a neural network to recognize objects in real-time video feeds.
• Activation Dynamics: The network's activation values change rapidly as new video frames are
processed every millisecond.
• Synaptic Dynamics: The network's weights are periodically updated based on feedback from
incorrect recognitions, but these updates are less frequent than the frame processing.
• Autonomous Model: The network uses a fixed learning rate during its initial deployment phase,
making it an autonomous system.
• Scenario: A smartphone's keyboard predicts the next word a user will type based on previous
inputs.
• Activation Dynamics: The neural network quickly adjusts its activation values based on the
immediate context of typed words (milliseconds).
• Synaptic Dynamics: The weights are updated as the network learns from the user's typing
patterns over days or weeks (seconds to longer periods).
• Nonautonomous Model: The learning rate may adapt over time as the system becomes more
familiar with the user's language style, making it a nonautonomous system.
Here, we will explore the models for activation and synaptic dynamics, the requirements for learning
laws, various categories of learning, issues of stability and convergence, and methods for neural network
recall. By understanding these concepts, we can better design and implement effective neural networks
for various applications.
Activation dynamics is described by the first derivative of the activation value of a neuron. For the ith
neuron, it is expressed as:
Here:
Let's consider a network of N interconnected processing units, where the variables and constants of each
unit are shown in Figure 2.1.
• In neural networks, the activation value of a neuron is often related to the membrane potential
in biological neurons. The membrane potential is the electrical charge difference across the
neuron's membrane.
• The output function f(.) determines how the neuron's output signal is generated based on its
membrane potential. In other words, it takes the neuron's activation (membrane potential) and
produces the final signal that the neuron sends out.
• The output function f(.) usually has a specific range within which it can produce values. This is
known as being bounded. For instance, in many models, the output is constrained between a
minimum and maximum value, like 0 and 1.
• The function f(.) is often a non-decreasing function of the activation value. This means that as
the activation value increases, the output signal either stays the same or increases, but never
decreases.
• The activation value of a neuron theoretically can vary widely, meaning it could have a very
large range. In reality, due to physical limitations (like the neuron’s ability to carry electrical
current), the membrane potential (the actual charge difference across the neuron's membrane)
cannot vary infinitely. It’s constrained to a specific range.
• This constraint means that the neuron’s activation value (and thus its output) is limited to a
smaller, practical range. For example, instead of the activation value being able to go from
negative infinity to positive infinity, it might only vary between -70 mV and -50 mV.
• The range of values that external inputs can have can be very large. For example, when you’re
looking at an object, the amount of light reflecting off it can vary greatly from very dim to very
bright.
• Sensory inputs, like vision or touch, can produce signals with a wide range of intensities. The
brightness of light, for instance, can vary from almost dark to very bright. We typically don’t
have control over the range of these input values. For instance, you can’t control how bright or
dim the light is when you’re looking at something; it’s just a factor of the environment.
Imagine you're looking at a red apple on a table. The apple's color and brightness are affected by the
lighting conditions:
1. Dim Light:
o In a dimly lit room, the light reflecting off the apple is very faint. Your eyes receive weak
signals about the apple’s color and brightness. The sensory input (light intensity) is very
low.
2. Bright Light:
o Now, imagine the same apple under bright sunlight. The light reflecting off the apple is
intense, and your eyes receive much stronger signals about its color and brightness. The
sensory input (light intensity) is very high.
• No Threshold: In this figure, the output function of a neuron doesn’t have a threshold. This
means that the neuron's output is directly proportional to its activation value without any
minimum requirement to start producing a signal. For instance, a linear function with no
threshold would continue to increase as the activation value increases.
• Saturation: This figure shows how the output function behaves when the input values are very
large. Saturation occurs when the output levels off and doesn’t increase further despite an
increase in input. For example, a sigmoid function becomes flat at its maximum and minimum
values, indicating that very high or very low inputs produce outputs close to these extremes.
• High Threshold: This figure demonstrates that if a neuron has a high threshold, it won’t respond
to small input values. The neuron will only produce an output when the input exceeds this
threshold. If the input is below this threshold, the neuron's output remains unchanged. This
creates a noise-saturation dilemma: balancing between being sensitive to small changes
(avoiding noise) and not saturating at large values.
• Problem: Neurons or processing units face a challenge due to their limited operating range. They
need to handle a vast range of input values (from tiny sensory signals to large signals). The
dilemma is how to make a neuron or processing unit responsive to small input variations without
saturating (i.e., becoming unresponsive to large inputs) or being overwhelmed by noise.
• Excitatory Components: Inputs that increase the neuron’s activation value. Think of these as
signals that stimulate or activate the neuron.
• Inhibitory Components: Inputs that decrease the neuron’s activation value. These signals
dampen or inhibit the neuron’s response.
• Deterministic vs. Stochastic: Inputs and outputs can be deterministic (predictable and
consistent) or stochastic (random and varying).
• Crisp vs. Fuzzy: Inputs and outputs can be crisp (clear and precise) or fuzzy (vague and
imprecise).
• Discrete vs. Continuous: Inputs and outputs can be discrete (distinct, separate values) or
continuous (smooth, flowing values).
• Scenario: Imagine a thermostat that adjusts the temperature of a room with no minimum limit.
Even the smallest temperature change is registered and adjusted. For instance, if the room’s
temperature drops just a tiny fraction, the thermostat immediately tries to heat up the room.
• Problem: If the thermostat is too sensitive, it might overreact to minor fluctuations, making the
room temperature uncomfortable by constantly adjusting. Similarly, a neuron without a threshold
responds to even the smallest changes in input, which can lead to over-sensitivity and erratic
responses to insignificant inputs.
• Scenario: Consider a speaker with a volume control that reaches a maximum level. When the
volume is increased to its maximum, the sound stops getting louder and only distorts further.
• Problem: Beyond a certain point, increasing the volume doesn’t improve the sound quality but
makes it worse. This is like neuron saturation, where the output level caps and additional
increases in input don’t lead to a greater response, limiting the neuron's ability to handle very
large inputs effectively.
• Scenario: A smoke detector can be set to different sensitivity levels. If the sensitivity is set too
high, it might trigger for minor smoke like from a burnt toast. However, if the sensitivity is set
too low, it might not react to small amounts of smoke, only reacting to significant smoke, like
from a large fire.
• Problem: When the sensitivity is too low, the smoke detector ignores small but potentially
important signs of smoke, potentially failing to alert you in case of a minor but dangerous
situation. Similarly, a neuron with a high threshold only responds to large inputs, missing out on
smaller yet potentially significant signals.
In developing models for activation dynamics, it is necessary to consider the known behaviour from
biological neuronal dynamics studies. However, the models must also be tractable for analysis to
examine the global behaviour of a network consisting of many interconnecting processing units. In
particular, the model should be such that it is possible to study the behaviour of the equilibrium states of
the network to determine whether the network is globally and structurally stable.
1. Range and Physical Constraints: Accurately modeling the bounded activation values.
2. Non-linearity and Saturation: Developing non-linear output functions.
3. Noise-Saturation Dilemma: Balancing sensitivity and saturation.
4. Dynamic Range of Inputs: Normalizing large input ranges.
5. Differential Equations: Capturing the rate of change in activation values.
6. Input Variability: Handling various inputs.
Additive Activation Models
• Purpose: These models describe how the activation value (cell membrane potential) of a neuron
change over time.
• Focus: The first derivative of the activation value, dxi(t)/dt, is used to model the rate of change
of the activation.
Resting Potential: The steady state value is xi(∞)= 0. Here, Ai represents the passive decay rate (related
to membrane conductance).