0% found this document useful (0 votes)
5 views14 pages

Module 2

Uploaded by

maheshmkvb92
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
5 views14 pages

Module 2

Uploaded by

maheshmkvb92
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

MODULE 2

Activation and synaptic dynamics - I

Introduction

Definition: An artificial neural network (ANN) consists of interconnected processing units (artificial
neurons) designed to perform tasks such as pattern recognition. While understanding the structure of a
neural network is essential, it's equally important to comprehend the rules that govern changes in
activation values and connection weights. These rules are encapsulated in the activation and synaptic
dynamics equations, which dictate the network's behaviour and learning process.

Key Concepts:

• Processing Units (Neurons): The basic elements that receive inputs, process them, and produce
outputs.
• Connections (Weights): Links between neurons that carry signals and can be adjusted during
learning.
• Activation Values: The output signal of a neuron, determined by its inputs and weights.
• Activation Dynamics: The rules that dictate changes in activation values over time.
• Synaptic Dynamics: The rules that dictate changes in connection weights over time.

Activation State

Definition: In a neural network with N processing units, the activation state at any given moment is
defined by the set of activation values of all neurons.

Trajectory in Activation State Space:

• The activation state changes over time, creating a trajectory in the activation state space.
• The trajectory leads to a solution state, reflecting the dynamics of the network.
• This Activation State trajectory is determined by the activation dynamics equations, which
specify the network's state at the next instant based on its current state.

Example: Consider a simple feedforward neural network designed to recognize handwritten digits. Each
neuron's activation value represents its response to a part of the input image. As the network processes
the image, the activation state changes, leading to the final recognition of the digit. The path from initial
activations to the final recognition state is the trajectory in the activation state space.
Weight State

Definition: The weight state of a network at any given moment is defined by the set of connection
weight values.

Trajectory in Weight State Space:

• For a given input, the network adjusts its weights to learn the input pattern.
• The weight state changes over time, creating a trajectory in the weight state space.
• This Weight State trajectory is determined by the synaptic dynamics, which dictate how
weights change in response to training.

Example: In the same handwritten digit recognition network, during training, the weights are adjusted
based on the errors in recognition. The changes in these weights form the trajectory in the weight state
space, leading to the network's ability to accurately recognize digits over time.

Short Term and Long-Term Memory

Short Term Memory:

• Defined by the network's steady activation state in response to a specific input pattern.
• Changes with different input patterns.
• Reflects the network's immediate response to inputs.

Long Term Memory:

• Defined by the network's steady weight state after training.


• Remains stable unless the network undergoes further training.
• Represents the network's learned knowledge.

Example: In our digit recognition network, short term memory is the network's current activation pattern
when presented with an image of a digit. Long term memory is the stable set of weights that the network
has learned through training to recognize different digits.

Practical Use Cases

Use Case 1: Predictive Maintenance

• Scenario: Predicting machinery failures in a manufacturing plant.


• Implementation: A neural network is trained with historical data of machinery performance and
failure instances. The network learns patterns in the data (long term memory). For real-time
monitoring, the network processes current performance data to predict potential failures (short
term memory), allowing for timely maintenance.

Use Case 2: Customer Behaviour Analysis

• Scenario: Analysing customer purchasing patterns to recommend products.


• Implementation: A neural network is trained on past purchase data, learning the correlation
between different products (long term memory). When a customer browses or purchases an item,
the network updates its activation state to recommend related products (short term memory).

Understanding

1. Data Preparation

What Happens:

• You start with a dataset, which might be images, text, or any other type of data. This data is
divided into training, validation, and test sets.

Role of Dynamics:

• At this stage, activation and synaptic dynamics are not directly involved. This is about preparing
the data for the network to use.

2. Model Architecture Design

What Happens:

• You design the architecture of the neural network, specifying the number of layers, neurons per
layer, activation functions, etc.

Role of Dynamics:

• Activation dynamics and synaptic dynamics are not yet active. This is about setting up the
network’s structure.

3. Training Phase

What Happens:

• Forward Pass: During training, you feed the training data into the network. Each input goes
through the layers of neurons, and the network produces an output (prediction).
o Activation Dynamics:
▪ As each input is processed, the neurons in the network compute their activations
based on the current weights and biases. This involves applying activation
functions to the weighted sums of inputs.
▪ Example: In an image classification task, as an image is processed, neurons in
the network rapidly adjust their activation values based on the input features, such
as edges or textures.
• Loss Calculation: After the forward pass, you compute the loss, which is a measure of how far
the network’s prediction is from the actual target value.
• Backward Pass: The network then performs backpropagation to update the weights based on
the loss. This involves calculating gradients and adjusting weights to minimize the error.
o Synaptic Dynamics:
▪ During backpropagation, synaptic dynamics are at work. The weights and biases
in the network are updated based on the gradients computed from the loss
function.
▪ Example: If the network predicts the wrong label, the weights are adjusted to
reduce the prediction error for similar future inputs. This process involves gradual
changes to weights over multiple iterations or epochs.

Role of Dynamics:

• Activation Dynamics: Active during the forward pass, where neurons quickly adjust their
activations in response to the input data.
• Synaptic Dynamics: Active during the backward pass, where weights are adjusted based on the
error to improve future predictions.

4. Validation Phase

What Happens:

• You use the validation set to tune hyperparameters and check how well the network performs on
unseen data. This helps to avoid overfitting and ensure generalization.

Role of Dynamics:

• Activation Dynamics: When feeding validation data into the network, the activation dynamics
still apply as the network processes inputs in real-time.
• Synaptic Dynamics: The weights are not updated during this phase; it’s about evaluating the
model’s performance.
5. Testing Phase

What Happens:

• After training and validation, you test the network on a separate test set to evaluate its
performance and generalization ability.

Role of Dynamics:

• Activation Dynamics: When test data is fed into the network, activation dynamics come into
play as the network computes predictions based on fixed weights.
• Synaptic Dynamics: Not active during testing, as the weights are not updated.

Summary

1. Training Phase:
o Activation Dynamics: Involves real-time changes in neuron activations during the
forward pass.
o Synaptic Dynamics: Involves gradual updates to weights during the backward pass.
2. Validation and Testing Phases:
o Activation Dynamics: Continues to play a role in processing inputs and generating
predictions.
o Synaptic Dynamics: Not active; weights are fixed and not updated.

In essence, activation dynamics are about how neurons respond and activate in real-time as data flows
through the network, while synaptic dynamics are about how the network learns and updates its weights
during training.

Activation Dynamics

Activation dynamics relate to the fluctuations at the neuronal level in a biological neural network.
In a neural network, think of each neuron (or brain cell) as a small unit that receives signals, processes
them, and then sends out its own signal. Activation dynamics is about how the signal inside a neuron
change over time as it receives inputs.

Typically, these fluctuations occur at intervals of a few milliseconds. Therefore, neuronal-level


dynamics are significantly faster than synaptic-level dynamics, where significant changes in synaptic
weights occur over seconds. During activation dynamics, synaptic weights can be considered constants
because they do not change significantly.
The objective of this chapter is to discuss models for activation and synaptic dynamics. First, we will
focus on activation dynamics. It's important to distinguish two situations:

1. Models of Biological Neural Networks:


o These models represent our understanding and observed behaviour of biological neural
networks.
o They aim to capture the knowledge of biological processes and can be complex, involving
first and higher-order derivatives of activations and weights, along with several nonlinear
interactions.
o Example: A detailed model of a human brain's neuron interactions to study cognitive
functions.
2. Neural Network Models for Artificial Networks:
o These models provide a representation for the dynamics of an artificial network, inspired
by biological neural networks.
o They are mathematical models designed to analyse the gross characteristics of an
artificial network.
o Typically described by the first-order time derivative of the activation state (for activation
dynamics) and the weight state (for synaptic dynamics).

Types of Dynamical Systems:

• Nonautonomous Dynamical Systems:


o If the expression for the first derivative contains the time parameter explicitly, the system
is nonautonomous.
o Example: A neural network model where the learning rate changes over time.
• Autonomous Dynamical Systems:
o If the expression for the first derivative does not contain the time parameter explicitly,
the system is autonomous, making it easier to analyse.
o Example: A neural network with a constant learning rate during training.

Practical Examples and Use Cases

Example 1: Real-Time Object Recognition in Video Feeds

• Scenario: A security system uses a neural network to recognize objects in real-time video feeds.
• Activation Dynamics: The network's activation values change rapidly as new video frames are
processed every millisecond.
• Synaptic Dynamics: The network's weights are periodically updated based on feedback from
incorrect recognitions, but these updates are less frequent than the frame processing.
• Autonomous Model: The network uses a fixed learning rate during its initial deployment phase,
making it an autonomous system.

Use Case: Predictive Text Input

• Scenario: A smartphone's keyboard predicts the next word a user will type based on previous
inputs.
• Activation Dynamics: The neural network quickly adjusts its activation values based on the
immediate context of typed words (milliseconds).
• Synaptic Dynamics: The weights are updated as the network learns from the user's typing
patterns over days or weeks (seconds to longer periods).
• Nonautonomous Model: The learning rate may adapt over time as the system becomes more
familiar with the user's language style, making it a nonautonomous system.

Here, we will explore the models for activation and synaptic dynamics, the requirements for learning
laws, various categories of learning, issues of stability and convergence, and methods for neural network
recall. By understanding these concepts, we can better design and implement effective neural networks
for various applications.

Activation Dynamics Models

Issues in the Development of Activation Dynamics Models

Activation dynamics is described by the first derivative of the activation value of a neuron. For the ith
neuron, it is expressed as:

Here:

• xi: Activation value of the i-th neuron.


• xi\dot: First derivative of xi with respect to time (t), representing the rate of change of activation.
• where h(.) is a function of the activation state and synaptic weights of the network.

Let's consider a network of N interconnected processing units, where the variables and constants of each
unit are shown in Figure 2.1.
• In neural networks, the activation value of a neuron is often related to the membrane potential
in biological neurons. The membrane potential is the electrical charge difference across the
neuron's membrane.

• The output function f(.) determines how the neuron's output signal is generated based on its
membrane potential. In other words, it takes the neuron's activation (membrane potential) and
produces the final signal that the neuron sends out.

• The output function f(.) usually has a specific range within which it can produce values. This is
known as being bounded. For instance, in many models, the output is constrained between a
minimum and maximum value, like 0 and 1.

• The function f(.) is often a non-decreasing function of the activation value. This means that as
the activation value increases, the output signal either stays the same or increases, but never
decreases.

• The activation value of a neuron theoretically can vary widely, meaning it could have a very
large range. In reality, due to physical limitations (like the neuron’s ability to carry electrical
current), the membrane potential (the actual charge difference across the neuron's membrane)
cannot vary infinitely. It’s constrained to a specific range.

• This constraint means that the neuron’s activation value (and thus its output) is limited to a
smaller, practical range. For example, instead of the activation value being able to go from
negative infinity to positive infinity, it might only vary between -70 mV and -50 mV.

• The range of values that external inputs can have can be very large. For example, when you’re
looking at an object, the amount of light reflecting off it can vary greatly from very dim to very
bright.
• Sensory inputs, like vision or touch, can produce signals with a wide range of intensities. The
brightness of light, for instance, can vary from almost dark to very bright. We typically don’t
have control over the range of these input values. For instance, you can’t control how bright or
dim the light is when you’re looking at something; it’s just a factor of the environment.

Example: Vision in Different Lighting Conditions

Imagine you're looking at a red apple on a table. The apple's color and brightness are affected by the
lighting conditions:

1. Dim Light:
o In a dimly lit room, the light reflecting off the apple is very faint. Your eyes receive weak
signals about the apple’s color and brightness. The sensory input (light intensity) is very
low.
2. Bright Light:
o Now, imagine the same apple under bright sunlight. The light reflecting off the apple is
intense, and your eyes receive much stronger signals about its color and brightness. The
sensory input (light intensity) is very high.

• Large Dynamic Range:


o The difference between the faint light in the dim room and the intense light in bright
sunlight represents a large dynamic range of inputs. The apple can appear very
differently depending on the lighting conditions, which means the range of light intensity
your eyes detect can vary significantly.

• Processing Unit’s Challenge:


o A processing unit (like a neuron or an artificial neural network) that receives these inputs
needs to handle this large range of values. If the unit’s operating range is limited, it might
have trouble accurately processing or distinguishing between very faint and very bright
signals.
• Adaptation:
o To manage this, processing units often have mechanisms to adjust or normalize these
inputs so they can be processed effectively. For example, in vision systems, there might
be algorithms to adjust for different lighting conditions, ensuring that the apple is
recognized consistently regardless of how bright or dim the light is.
Figure 2.2a: Output Function with No Threshold

• No Threshold: In this figure, the output function of a neuron doesn’t have a threshold. This
means that the neuron's output is directly proportional to its activation value without any
minimum requirement to start producing a signal. For instance, a linear function with no
threshold would continue to increase as the activation value increases.

Figure 2.2b: Saturation for Large Input Values

• Saturation: This figure shows how the output function behaves when the input values are very
large. Saturation occurs when the output levels off and doesn’t increase further despite an
increase in input. For example, a sigmoid function becomes flat at its maximum and minimum
values, indicating that very high or very low inputs produce outputs close to these extremes.

Figure 2.2c: Insensitivity to Small Input Values Due to High Threshold

• High Threshold: This figure demonstrates that if a neuron has a high threshold, it won’t respond
to small input values. The neuron will only produce an output when the input exceeds this
threshold. If the input is below this threshold, the neuron's output remains unchanged. This
creates a noise-saturation dilemma: balancing between being sensitive to small changes
(avoiding noise) and not saturating at large values.

The Noise-Saturation Dilemma

• Problem: Neurons or processing units face a challenge due to their limited operating range. They
need to handle a vast range of input values (from tiny sensory signals to large signals). The
dilemma is how to make a neuron or processing unit responsive to small input variations without
saturating (i.e., becoming unresponsive to large inputs) or being overwhelmed by noise.

Types of Inputs and Outputs:

• Excitatory Components: Inputs that increase the neuron’s activation value. Think of these as
signals that stimulate or activate the neuron.
• Inhibitory Components: Inputs that decrease the neuron’s activation value. These signals
dampen or inhibit the neuron’s response.
• Deterministic vs. Stochastic: Inputs and outputs can be deterministic (predictable and
consistent) or stochastic (random and varying).
• Crisp vs. Fuzzy: Inputs and outputs can be crisp (clear and precise) or fuzzy (vague and
imprecise).
• Discrete vs. Continuous: Inputs and outputs can be discrete (distinct, separate values) or
continuous (smooth, flowing values).

Figure 2.2a: Output Function with No Threshold

Example: Thermostat with No Minimum Temperature Setting

• Scenario: Imagine a thermostat that adjusts the temperature of a room with no minimum limit.
Even the smallest temperature change is registered and adjusted. For instance, if the room’s
temperature drops just a tiny fraction, the thermostat immediately tries to heat up the room.
• Problem: If the thermostat is too sensitive, it might overreact to minor fluctuations, making the
room temperature uncomfortable by constantly adjusting. Similarly, a neuron without a threshold
responds to even the smallest changes in input, which can lead to over-sensitivity and erratic
responses to insignificant inputs.

Figure 2.2b: Saturation for Large Input Values

Example: Volume Control on a Speaker

• Scenario: Consider a speaker with a volume control that reaches a maximum level. When the
volume is increased to its maximum, the sound stops getting louder and only distorts further.
• Problem: Beyond a certain point, increasing the volume doesn’t improve the sound quality but
makes it worse. This is like neuron saturation, where the output level caps and additional
increases in input don’t lead to a greater response, limiting the neuron's ability to handle very
large inputs effectively.

Figure 2.2c: Insensitivity to Small Input Values Due to High Threshold

Example: Smoke Detector Sensitivity

• Scenario: A smoke detector can be set to different sensitivity levels. If the sensitivity is set too
high, it might trigger for minor smoke like from a burnt toast. However, if the sensitivity is set
too low, it might not react to small amounts of smoke, only reacting to significant smoke, like
from a large fire.
• Problem: When the sensitivity is too low, the smoke detector ignores small but potentially
important signs of smoke, potentially failing to alert you in case of a minor but dangerous
situation. Similarly, a neuron with a high threshold only responds to large inputs, missing out on
smaller yet potentially significant signals.

In developing models for activation dynamics, it is necessary to consider the known behaviour from
biological neuronal dynamics studies. However, the models must also be tractable for analysis to
examine the global behaviour of a network consisting of many interconnecting processing units. In
particular, the model should be such that it is possible to study the behaviour of the equilibrium states of
the network to determine whether the network is globally and structurally stable.

Issues in Activation Dynamics Models:

1. Range and Physical Constraints: Accurately modeling the bounded activation values.
2. Non-linearity and Saturation: Developing non-linear output functions.
3. Noise-Saturation Dilemma: Balancing sensitivity and saturation.
4. Dynamic Range of Inputs: Normalizing large input ranges.
5. Differential Equations: Capturing the rate of change in activation values.
6. Input Variability: Handling various inputs.
Additive Activation Models

Introduction to Activation Models

• Purpose: These models describe how the activation value (cell membrane potential) of a neuron
change over time.
• Focus: The first derivative of the activation value, dxi(t)/dt, is used to model the rate of change
of the activation.

Passive Decay Model

• Equation: For passive decay, the activation dynamics can be described by

Solution: The solution to this differential equation is:

Resting Potential: The steady state value is xi(∞)= 0. Here, Ai represents the passive decay rate (related
to membrane conductance).

Passive Decay with Capacitance

• Equation: Introducing membrane capacitance Ci (assuming Ci=1)


Steady State: The steady state activation is xi(∞)=Pi =. Here, Pi represents a constant resting potential
or input.

You might also like