Unit II Convolutional Neural Network

Introduction to CNN, Convolution Operation, Parameter Sharing, Equivariant

Representation, Pooling, Variants of the Basic Convolution Function, The basic Architecture
of CNN, Popular CNN Architecture – AlexNet.

Introduction to CNN

A Convolutional Neural Network (CNN) is a specialized type of deep learning

model designed for processing and analyzing grid-like data, such as images,
video frames, and even sequential data like speech and text. CNNs are
particularly well-suited for tasks involving pattern recognition and feature
extraction from visual data. They have revolutionized the field of computer
vision and achieved state-of-the-art performance in various image-related
tasks.

Key Components and Concepts:

​ Convolutional Layer:
● The cornerstone of CNNs is the convolutional layer. It performs
convolution operations on input data using learnable filters (also
called kernels). These filters slide over the input to extract local
patterns and features.
● Convolution involves element-wise multiplication of the filter
with a local region of the input and then summing the results.
This process captures spatial hierarchies.
​ Pooling Layer:
● After convolutional layers, pooling layers are employed for
downsampling and dimensionality reduction. Pooling
aggregates information from neighboring regions, reducing the
spatial dimensions.
● Common pooling techniques include max pooling (selecting the
maximum value in a region) and average pooling (taking the
average value).
​ Activation Functions:
● Activation functions like ReLU (Rectified Linear Unit) introduce
non-linearity into the network, allowing it to capture complex
relationships in the data.
● ReLU outputs the input directly if it's positive, otherwise outputs
zero.
​ Weight Sharing and Local Connectivity:
 ● CNNs leverage weight sharing, where the same set of filters is
applied across different spatial locations in the input. This
allows the network to learn local features irrespective of their
position.
● This local connectivity reduces the number of parameters and
enhances the model's ability to capture patterns across the
entire input.
​ Hierarchy of Features:
● CNNs learn features hierarchically. Lower layers capture simple
features like edges, corners, and textures, while deeper layers
learn more complex features like object parts and shapes.
● This hierarchy enables the network to understand the context
and composition of objects.
​ Fully Connected Layers:
● Typically, one or more fully connected layers follow the
convolutional and pooling layers to make predictions based on
the extracted features.
● These layers integrate the learned features and produce the final
output.
​ Training and Backpropagation:
● CNNs are trained using backpropagation and optimization
algorithms. Gradients are computed to update the network's
parameters (weights and biases) iteratively.
● The loss function measures the discrepancy between predicted
and actual values, and the network learns to minimize this loss.

Advantages of CNNs:

1. Local Invariance: CNNs are inherently translation invariant, meaning

they can recognize features regardless of their position in the image.
2. Parameter Efficiency: Weight sharing and local connectivity lead to
fewer parameters, making CNNs more efficient than fully connected
networks for grid-like data.
3. Feature Hierarchy: CNNs capture hierarchical features, allowing them
to learn and represent complex structures.
4. State-of-the-Art Performance: CNNs achieve outstanding
performance in various computer vision tasks, often surpassing
human-level performance.

Applications of CNNs:

1. Image Classification: Identifying the main object in an image.

 2. Object Detection: Detecting and localizing multiple objects within an
image.
3. Semantic Segmentation: Assigning a class label to each pixel in an
image.
4. Face Recognition: Identifying individuals in images or videos.
5. Medical Imaging: Analyzing medical images for diagnosis and
treatment.
6. Autonomous Vehicles: Object detection, lane detection, and more.

Convolution Operation

The convolution operation is a fundamental mathematical operation used in

various fields, including signal processing and image analysis. In the context
of deep learning, it plays a pivotal role in Convolutional Neural Networks
(CNNs) for extracting features from input data, particularly grid-like data such
as images.

1. Definition: Convolution is a mathematical operation that combines two

functions to create a third function. In the context of CNNs, it involves sliding a
filter (also called a kernel) over the input data and computing the dot product
between the filter's weights and the overlapping region of the input. The result
of this dot product forms a single element in the output, known as a feature
map.

2. Local Receptive Field: During each step of the sliding process, the filter
covers a small region of the input, known as the local receptive field. This
receptive field captures local patterns and features. By convolving the filter
over the entire input, the network can detect various features across the data.

3. Filter and Feature Map: The filter is a small matrix of learnable weights. It
represents a pattern or feature that the network aims to detect. As the filter
slides over the input, its weights are element-wise multiplied with the
corresponding input values within the receptive field. The results are summed
to produce a single value in the feature map.

4. Convolution Process: Here's a step-by-step breakdown of the convolution

process:

● Place the filter at the top-left corner of the input.

● Perform element-wise multiplication between the filter and the input
values within the receptive field.
 ● Sum up the results of the element-wise multiplications to calculate the
value for the first element in the feature map.
● Slide the filter to the right by a certain stride, overlapping with the next
receptive field.
● Repeat the multiplication and summation process to calculate the next
feature map element.
● Continue sliding the filter over the entire input, calculating the
corresponding feature map elements.
● The resulting feature map highlights regions in the input that match the
pattern represented by the filter.

5. Strides and Padding: The stride defines the step size by which the filter
moves as it slides over the input. A larger stride results in smaller output
dimensions. Padding can be added to the input to ensure that the filter covers
the edges and corners adequately. Common padding methods include "same"
padding (output size equals input size) and "valid" padding (no padding,
output size is reduced).

6. Stacking Feature Maps: In CNNs, multiple filters are used to capture

different features from the input. Each filter produces a separate feature map.
The collection of these feature maps forms the output of a convolutional layer.
Each feature map highlights a specific aspect of the input's patterns.

7. Activation Function: Typically, an activation function, such as the Rectified

Linear Unit (ReLU), is applied element-wise to the feature map. This
introduces non-linearity, allowing the network to capture complex relationships
between features.

8. Advantages of Convolution:

● Parameter Sharing: The same filter weights are shared across different
spatial locations, reducing the number of parameters and making
learning more efficient.
● Local Patterns: Convolution focuses on local patterns, allowing CNNs
to capture features regardless of their position in the input.
● Feature Hierarchy: By stacking multiple convolutional layers, CNNs can
learn hierarchical features, from simple edges to complex objects.

9. Applications: Convolution operations are used extensively in CNNs for

tasks like image classification, object detection, image segmentation, and
more. They allow the network to automatically learn relevant features from
raw pixel data.
Parameter Sharing

Introduction to Parameter Sharing: Parameter sharing is a fundamental

concept embedded in Convolutional Neural Networks (CNNs), a class of deep
learning models designed for tasks involving grid-like data, such as images
and audio spectrograms. Unlike traditional fully connected neural networks,
where each parameter is unique to a specific connection, CNNs employ the
practice of using the same set of weights (parameters) across different spatial
locations of the input data.

Motivation for Parameter Sharing: The motivation behind parameter sharing

lies in the inherent characteristics of visual data. In images, certain features,
such as edges, textures, and patterns, carry essential information regardless
of their position within the image. For instance, the ability to detect a
horizontal edge is relevant whether it's located at the top, middle, or bottom of
an image. Parameter sharing leverages this property to enhance the
network's learning efficiency and its ability to generalize patterns.

How Parameter Sharing Works: In the context of CNNs, parameter sharing

is most prominent in the convolutional layers—a central component of these
networks. Consider a convolutional filter, which is a matrix of learnable
weights. As the filter slides over the input data, it performs a convolution
operation. During this operation, the same set of weights is applied across
different spatial locations of the input. This shared set of weights is used to
calculate the dot product between the filter and the overlapping region of the
input, generating a single value in the output feature map.

Advantages of Parameter Sharing:

1. Reduced Parameters: The most notable advantage of parameter

sharing is the drastic reduction in the number of parameters. In
contrast to fully connected networks, where each neuron is connected
to every neuron in the previous layer, CNNs exploit parameter sharing
to use a limited set of weights, resulting in a significant reduction in
memory usage, computational complexity, and the risk of overfitting.
2. Translation Invariance: Parameter sharing leads to a desirable
property called translation invariance. This means that the network can
recognize the same pattern or feature regardless of its position within
the input. This property is crucial in tasks like image recognition, where
the position of an object can vary.
3. Effective Feature Learning: By sharing parameters, CNNs efficiently
capture local features and patterns. These shared parameters act as
filters that are applied across different regions of the input, enabling the
network to capture relevant information consistently.
 4. Data Efficiency: Parameter sharing allows the network to learn
general features that apply to various regions of the input. As a result,
the network becomes more data-efficient, as it doesn't need to learn
specialized parameters for every spatial location.

Implementation in Convolutional Layers: In a convolutional layer, each filter

comprises a set of shared weights. These weights are learned during the
training process. As the filter slides over the input, it applies the same weights
to different local regions, capturing relevant features in a consistent manner.

Difference from Fully Connected Layers: In traditional fully connected layers,

each neuron is connected to every neuron in the previous layer, resulting in a
high number of parameters. In contrast, convolutional layers utilize parameter
sharing to significantly reduce the number of parameters, which is particularly
advantageous in deep networks.

Applications and Significance: Parameter sharing is not just a technical detail;

it's a fundamental reason behind the success of CNNs in image-related tasks.
From image classification to object detection and image segmentation, CNNs
leverage parameter sharing to automatically learn hierarchical features and
patterns from raw pixel data, providing state-of-the-art performance.

Equivariant Representation

Introduction to Equivariant Representation: Equivariant representation is a

crucial concept in deep learning that addresses how neural networks respond
to transformations in the input data. It pertains to maintaining meaningful
relationships between features under certain transformations, making it
particularly relevant for tasks involving structured data like images, signals,
and sequences.

Understanding Equivariance: To understand equivariance, it's essential to

differentiate between two concepts: invariance and equivariance.

● Transformation Invariance: A network is invariant to a transformation if

its output remains the same regardless of that transformation applied
to the input. For instance, in image recognition, a translation-invariant
network would identify an object in an image regardless of its position.
● Equivariance: A network is equivariant if the relationship between
features in the output changes in a predictable manner when the input
undergoes a specific transformation. For example, a
 rotation-equivariant network would recognize rotated versions of an
object and represent them with corresponding rotations in the output.

Importance of Equivariant Representation: Equivariant representation is

particularly important for structured data because real-world data often
undergoes various transformations due to factors like viewpoint changes,
noise, or deformations. By ensuring that a network's response corresponds
predictably to transformations in the input, equivariant networks can capture
meaningful patterns and relationships, leading to more robust and
interpretable models.

Examples to Illustrate Equivariant Representation:

1. Image Data - Rotation Equivariance: Imagine a CNN trained to

recognize handwritten digits. If the network is rotation-equivariant,
when a digit image is rotated, the corresponding output of the network
will also exhibit the same rotation. This is important because the
orientation of the digit should not affect the network's ability to
recognize it.
2. Audio Data - Time Equivariance: In speech recognition, a
time-equivariant network ensures that phoneme relationships are
preserved even when the timing of phonemes in the input audio
changes. This allows the network to recognize phonemes even if they
are spoken faster or slower.

Benefits of Equivariant Representation:

1. Enhanced Robustness: Equivariant networks are often more robust to

variations and perturbations in the input. This is especially important in
scenarios where data can undergo different transformations due to
real-world factors.
2. Efficient Learning: Equivariant representation can enable the network
to learn more efficiently. Rather than learning separate representations
for each transformed version of the input, the network can generalize
across transformations.
3. Interpretability: Equivariant networks can provide insights into how the
network processes and understands the input data. The
transformations observed in the output can reveal the network's
understanding of the underlying structures.
 Implementing Equivariant Representation:

1. Architectures: Specific network architectures are designed to achieve

equivariant behavior. For example, Convolutional Neural Networks
(CNNs) exhibit equivariance with respect to translations due to their
shared weight filters.
2. Transformations: Equivariance is typically achieved by ensuring that
the network's layers respond to transformations in a consistent
manner. This might involve designing specific layers to capture specific
types of equivariance.

Challenges and Implementations:

1. Complexity: Designing equivariant networks can be complex, especially

for certain transformations. Researchers explore various architectures
and techniques to achieve equivariant behavior.
2. Architectures: Convolutional Neural Networks (CNNs) and other
specialized architectures are often designed to exhibit equivariant
behavior with respect to specific transformations.

Applications of Equivariant Representation:

1. Computer Vision: Equivariant CNNs are used in image analysis tasks

like object detection, image segmentation, and image generation to
capture spatial relationships.
2. Speech Processing: Equivariant networks play a role in speech
recognition, where preserving phoneme relationships under time
variations is vital.

Pooling
● Pooling layers are commonly inserted between successive
convolutional layers. We want to follow convolutional layers with
pooling layers to progressively reduce the spatial size (width and
height) of the data representation. Pooling layers reduce the data
representation progressively over the network and help control
overfitting. The pooling layer operates independently on every depth
slice of the input.
● The pooling layer uses the max() operation to resize the input data
spatially (width, height). This operation is referred to as max pooling.
With a 2 × 2 filter size, the max() operation is taking the largest of four
numbers in the filter area. This operation does not affect the depth
dimension.
● Pooling layers use filters to perform the downsampling process on the
input volume. These layers perform downsampling operations along
the spatial dimension of the input data. This means that if the input
image were 32 pixels wide by 32 pixels tall, the output image would be
smaller in width and height (e.g., 16 pixels wide by 16 pixels tall).
● The most common setup for a pooling layer is to apply 2 × 2 filters with
a stride of 2. This will downsample each depth slice in the input volume
 by a factor of two on the spatial dimensions (width and height). This
downsampling operation will result in 75 percent of the activations
being discarded.
● Pooling layers do not have parameters for the layer but do have
additional hyperparameters. This layer does not involve parameters,
because it computes a fixed function of the input volume. It is not
common to use zero-padding for pooling layers.
(chatgpt)

Introduction to Pooling: Pooling is a fundamental operation used in

Convolutional Neural Networks (CNNs) and other deep learning architectures
to downsample and reduce the spatial dimensions of feature maps. It plays a
crucial role in extracting essential information from data while reducing
computation and the risk of overfitting.

Purpose of Pooling: Pooling serves several purposes within deep learning:

1. Dimensionality Reduction: Pooling reduces the size of feature maps,

making them computationally more efficient to process in subsequent
layers. This aids in managing computational resources and speeding
up training.
2. Translation Invariance: Pooling helps to achieve a degree of translation
invariance by summarizing local patterns and features. This is
particularly important for tasks like object recognition where the
position of the object is not as relevant as its presence.
3. Feature Generalization: Pooling captures the most salient features
within a local region, allowing the network to focus on the most
relevant information while ignoring less important details.

Types of Pooling:

1. Max Pooling:
a. Max pooling extracts the maximum value within each pooling
window. It effectively retains the most activated feature in the
region.
b. Max pooling is robust to noise and minor variations in the data.
c. It emphasizes dominant features and helps the network learn
patterns regardless of their precise location.
2. Average Pooling:
a. Average pooling calculates the average value within each
pooling window.
b. It's less sensitive to outliers and emphasizes overall trends in
the data.
 3. Global Average Pooling (GAP):
a. GAP takes the average of all values in the feature map, reducing
it to a single value per feature channel.
b. It serves as a form of regularization by encouraging the network
to focus on the most important features.

Pooling Process:

1. Pooling Window:
a. A pooling window (also known as the pooling kernel) is a
fixed-size window that slides over the input feature map.
b. It defines the local region from which information will be
summarized.
2. Pooling Operation:
a. For each pooling window position, the pooling operation (max or
average) is applied to the values within the window.
b. This operation produces a downsampled output value for that
region.
3. Stride:
a. The stride determines the step size at which the pooling window
moves over the input.
b. Larger strides result in more aggressive downsampling.
4. Padding:
a. Padding can be added around the input to control the output
dimensions after pooling.
b. It ensures that spatial dimensions are preserved or adjusted as
needed.

Benefits and Considerations:

1. Reduction in Spatial Dimensions: Pooling reduces the spatial

dimensions of the feature maps, which can help reduce computation
and memory usage in subsequent layers.
2. Translation Invariance: Pooling helps the network become partially
invariant to translations, making it more robust to slight variations in
the object's position.
3. Loss of Information: While pooling is beneficial for downsampling and
generalization, it does lead to a loss of spatial information. The degree
of loss depends on the pooling operation and window size.

Use Cases:
 1. Image Classification: Pooling is widely used in image classification
Image Segmentation:tasks to extract key features from images.
2. Object Detection: Pooling helps extract features that are robust to
changes in object position and size.
3. In segmentation tasks, pooling can be used to reduce spatial
dimensions while preserving key information for segmentation masks.

Variants of the Basic Convolution Function

(book)

The basic Architecture of CNN

Introduction: Convolutional Neural Networks (CNNs) are a class of deep

learning models specifically designed for processing structured grid data,
such as images. They leverage the spatial relationships present in the data,
making them highly effective for tasks like image classification, object
detection, and segmentation.

1. Input Layer: The input layer of a CNN receives the raw data, which is usually
an image or a set of images. Images are represented as multi-dimensional
arrays of pixel values, where each pixel's intensity or color information is
encoded.

2. Convolutional Layer: Convolutional layers are the fundamental building

blocks of CNNs. They apply multiple filters or kernels to the input data. Each
filter is responsible for detecting specific features like edges, textures, or more
complex patterns. Convolution involves element-wise multiplication between
the filter and a local patch of the input followed by summation. This process
generates a feature map that highlights the presence of that specific feature
in the input.

3. Activation Function: After convolution, an activation function is applied

element-wise to each value in the feature map. Common activation functions
include ReLU (Rectified Linear Unit), which replaces negative values with zeros
and introduces non-linearity into the network. Activation functions enable the
network to learn complex relationships between features.

4. Pooling Layer: Pooling layers downsample the feature maps by selecting a

representative value from each local region. Max pooling selects the
maximum value, while average pooling calculates the average value. Pooling
helps make the network partially invariant to small translations in the input,
reduces the computational burden, and helps control overfitting.

5. Fully Connected Layer: After several convolutional and pooling layers, fully
connected layers are employed for high-level feature capture. These layers
resemble traditional neural network layers. Each neuron is connected to all
neurons in the previous and subsequent layers. Fully connected layers can
learn complex relationships but come with a large number of parameters.

6. Flatten Layer: Before entering the fully connected layers, the feature maps
are flattened into a one-dimensional vector. This transformation is necessary
as fully connected layers require fixed-size inputs, whereas feature maps are
two-dimensional and spatially organized.

7. Output Layer: The output layer produces the final predictions or

classifications based on the learned features. In classification tasks, the
number of neurons in this layer corresponds to the number of classes. Each
neuron's activation indicates the likelihood of the input belonging to the
corresponding class.

8. Softmax Activation: In classification tasks, the softmax activation function

is often applied to the output layer. It converts the raw class scores into a
probability distribution. Each class probability represents the likelihood of the
input belonging to that class. The class with the highest probability is
considered the predicted class.

9. Backpropagation and Optimization: CNNs are trained using

backpropagation, where the difference between predicted and actual outputs
(loss) is calculated and propagated backward through the layers. Optimization
algorithms like gradient descent update the model's parameters to minimize
the loss function and improve predictions.

10. Architecture Design Considerations:

● Depth and Width: Increasing depth (more layers) and width (more
neurons per layer) enhances the network's capacity to capture complex
features but requires more computational resources.
● Padding: Padding can be applied to maintain spatial dimensions after
convolutions (same padding) or reduce dimensions (valid padding).
● Strides: Strides control how the convolutional kernels move over the
input, affecting output size and feature extraction.
 11. Data Augmentation and Regularization: To prevent overfitting and enhance
generalization, data augmentation involves creating variations of the training
data. Regularization techniques like dropout (randomly disabling neurons) and
L2 regularization (penalizing large weights) are used.

12. Hyperparameter Tuning: Hyperparameters like learning rate, batch size,

and filter sizes need careful tuning to achieve optimal performance. Learning
rate controls gradient descent's step size, batch size affects parameter
updates' efficiency, and filter sizes determine the receptive field.

Popular CNN Architecture – AlexNet.

1. Introduction to AlexNet:

● AlexNet is a pioneering convolutional neural network architecture

designed for image classification tasks.
● It gained widespread recognition for winning the ImageNet Large Scale
Visual Recognition Challenge (ILSVRC) in 2012.
● AlexNet's victory demonstrated the effectiveness of deep learning
models in handling large-scale image datasets.

2. Input Layer:

● AlexNet's input layer receives images as input, with each image having
a fixed size of 227x227 pixels.
● The images consist of three color channels (RGB), representing red,
green, and blue color intensities.

3. Convolutional Layers:

● AlexNet comprises five convolutional layers that serve as feature

extractors.
● The first convolutional layer uses a large 11x11 filter with a stride of 4,
enabling the extraction of low-level features such as edges.
● Subsequent convolutional layers employ smaller filters (3x3 and 5x5)
with a stride of 1 for capturing more complex patterns.
● Applying convolutional filters across the input data helps the network
learn hierarchical features from raw pixel information.

4. Activation Functions:
 ● ReLU (Rectified Linear Unit) activation functions are employed after
each convolutional layer.
● ReLU introduces non-linearity by setting negative values to zero and
passing positive values unchanged.
● This non-linearity enables the network to learn complex relationships in
the data.

5. Max Pooling Layers:

● AlexNet utilizes max pooling layers after the first and second
convolutional layers.
● A 3x3 window is moved with a stride of 2, resulting in downsampling
and enhancing translation invariance.
● Max pooling selects the maximum value within each local region,
reducing the spatial dimensions while preserving key features.

6. Local Response Normalization (LRN):

● Local Response Normalization (LRN) is applied after the first and

second convolutional layers.
● LRN normalizes the activations within a local neighborhood, enhancing
the contrast between features and reducing sensitivity to variations in
illumination.

7. Fully Connected Layers:

● AlexNet is characterized by three fully connected layers at the end.

● The first two fully connected layers consist of 4096 neurons each,
capturing high-level abstractions from the features learned in previous
layers.
● To prevent overfitting, a dropout layer with a dropout rate of 0.5 is
introduced after these fully connected layers.
● The final fully connected layer has a number of neurons corresponding
to the classes in the classification task.

8. Softmax Activation:

● The output layer follows the final fully connected layer and utilizes the
softmax activation function.
● Softmax transforms the raw class scores into a probability distribution,
assigning a probability to each class.
● The class with the highest probability is predicted as the class label.
9. Training Details:

1. Data Augmentation:

● AlexNet employs data augmentation techniques during training,

including random cropping and horizontal flipping of images. This
increases the effective size of the training dataset and improves model
generalization.

2. Regularization:

● Dropout layers are used in the fully connected layers to prevent

overfitting.

3. Optimization:

● AlexNet uses the stochastic gradient descent (SGD) optimization

algorithm with momentum.

4. Learning Rate:

● A learning rate of 0.01 is used initially, and it is reduced as the training

progresses.

10. Data Augmentation and Regularization:

● During training, AlexNet employs data augmentation techniques, such

as random cropping and horizontal flipping.
● Data augmentation increases the effective size of the training dataset,
leading to improved generalization.
● Dropout layers are introduced after the first two fully connected layers
to mitigate overfitting. Dropout randomly deactivates neurons during
training.

11. Optimization and Learning Rate: - AlexNet uses the stochastic gradient
descent (SGD) optimization algorithm with momentum to update model
parameters. - Momentum accelerates convergence by adding a fraction of the
previous parameter update to the current update. - The initial learning rate
(e.g., 0.01) is gradually reduced during training to achieve better convergence.