21CS743

Model Question Paper*1/2 with effect from 2021(CBCS Scheme)

US
N 7th Semester B.E. Degree Examination Deep
Learning

TIME: 03 Hours Max. Marks: 100

Note: 01. Answer any FIVE full questions, choosing at least ONE question from each MODULE.

Module *1 Mar
DOWNLOAD COs ks
Q.01 a Explain the historical trends in deep learning. 1 10
 The historical trends in deep learning can be categorized into several key phases.
First, the concept of deep learning dates back to the 1940s to 1960s, when it was
referred to as cybernetics. This early phase focused on understanding how systems
could mimic biological learning processes.
In the 1980s to 1990s, the field was known as connectionism, which emphasized the
use of neural networks to model cognitive processes. During this time, researchers
began to explore how these networks could learn from data, although the technology
was not widely adopted due to limitations in computational power and available data.
The third wave of deep learning began around 2006, marking a resurgence in interest
and application, officially termed "deep learning." This resurgence was driven by
several factors, including the exponential growth of available training data,
advancements in computer hardware (especially the advent of general*purpose
GPUs), and improvements in software infrastructure for distributed computing.
Throughout its history, deep learning has experienced fluctuations in popularity.
Despite being successfully applied in commercial settings since the 1990s, it was
often viewed as more of an art than a technology, requiring specialized knowledge to
achieve good performance. However, as the amount of training data has increased,
the skill required to effectively use deep learning algorithms has decreased, making it
more accessible.
Moreover, the size and complexity of deep learning models have grown significantly
over time. Early models could only process very small images, while modern object
recognition networks can handle high*resolution images without the need for
cropping. This evolution reflects the increasing accuracy and capability of deep
learning systems in tackling more complex applications.
b Define machine learning. Explain different types of ML algorithms. 1 10
Machine learning (ML) is a subset of artificial intelligence (AI) that focuses on the
development of algorithms that allow computers to learn from and make predictions
or decisions based on data. The core idea is that instead of being explicitly
programmed to perform a task, a machine learning model is trained on a dataset,
allowing it to identify patterns and improve its performance over time as it is
exposed to more data.
2. Types of Machine Learning Algorithms :
* Supervised Learning : This type involves training a model on a labeled dataset,
where the input data is paired with the correct output. The model learns to map
inputs to outputs. Common algorithms include:
* Linear Regression : Used for predicting continuous values.
* Logistic Regression : Used for binary classification tasks.
* Support Vector Machines (SVM) : Effective for classification tasks by finding
the hyperplane that best separates classes.
* Decision Trees : A flowchart*like structure used for classification and
regression tasks.
* Unsupervised Learning : In this approach, the model is trained on data without
labeled responses. The goal is to identify patterns or groupings within the data.
Common algorithms include:
* K*Means Clustering : Groups data into k distinct clusters based on feature
similarity.
* Hierarchical Clustering : Builds a tree of clusters based on the distance
between data points.
* Principal Component Analysis (PCA) : Reduces the dimensionality of data
while preserving variance.
* Reinforcement Learning : This type of learning involves training an agent to
make decisions by taking actions in an environment to maximize cumulative reward.
The agent learns from the consequences of its actions rather than from a labeled
dataset. Key concepts include:
* Agent : The learner or decision*maker.
* Environment : The context in which the agent operates.
 * Reward Signal : Feedback from the environment based on the agent's actions.
3. Deep Learning : A specialized form of machine learning that uses neural
networks with many layers (deep networks) to model complex patterns in large
datasets. It is particularly effective in tasks such as image and speech recognition.
4. Applications of Machine Learning : Machine learning algorithms are widely
used in various fields, including:
* Healthcare : Predicting patient outcomes and diagnosing diseases.
* Finance : Fraud detection and algorithmic trading.
* Marketing : Customer segmentation and recommendation systems.
5. Challenges in Machine Learning : Some challenges include overfitting,
underfitting, and the need for large amounts of labeled data for supervised learning.
Additionally, ensuring the model generalizes well to unseen data is crucial.
OR
Q.02 a Explain in detail about the supervised learning approach by taking suitable ex. 1 10
Supervised learning is a fundamental approach in machine learning where an
algorithm learns to map inputs to outputs based on a labeled training dataset. In this
context, the training dataset consists of input*output pairs, where the input is
typically a feature vector, and the output is the corresponding label or value that we
want the model to predict.
2. Training Process : The algorithm learns from the training data by adjusting its
parameters to minimize the difference between its predictions and the actual labels.
This process is often achieved through techniques like gradient descent.
3. Types of Problems : Supervised learning can be used for two main types of
problems:
* Classification : The output is a category or class label. For example, an email
can be classified as "spam" or "not spam" based on its content.
* Regression : The output is a continuous value. For instance, predicting the price
of a house based on features like size, location, and number of bedrooms.
4. Common Algorithms : Some widely used supervised learning algorithms
include:
* Linear Regression : Used for regression tasks, it predicts a continuous output
by fitting a linear equation to the data.
* Logistic Regression : Despite its name, it is used for binary classification tasks,
predicting the probability that an instance belongs to a particular class.
* Support Vector Machines (SVM) : A powerful classification technique that
finds the hyperplane that best separates different classes in the feature space.
5. Example of Classification : Consider a supervised learning model designed to
identify whether an image contains a cat or a dog. The training dataset would consist
of images labeled as "cat" or "dog." The model learns to recognize patterns and
features that distinguish the two classes.
6. Example of Regression : A practical example of regression is predicting the
future sales of a product based on historical sales data. The model would be trained
on past sales figures (input) and the corresponding sales amounts (output) to forecast
future sales.
7. Evaluation Metrics : The performance of supervised learning models is
evaluated using metrics such as accuracy, precision, recall, and F1*score for
classification tasks, and mean squared error (MSE) or R*squared for regression
tasks.
8. Challenges : Supervised learning requires a large amount of labeled data, which
can be expensive and time*consuming to obtain. Additionally, the model may overfit
the training data, leading to poor generalization on unseen data.
9. Applications : Supervised learning is widely used in various fields, including:
* Healthcare : Predicting disease outcomes based on patient data.
* Finance : Credit scoring and fraud detection.
 * Marketing : Customer segmentation and targeted advertising.
b Write a note on support vector machine and PCA. 1 10
Support Vector Machine (SVM)
1. Definition : SVM is a supervised learning algorithm used for classification and
regression tasks. It aims to find the optimal hyperplane that separates different
classes in the feature space.
2. Mechanism : The SVM model is driven by a linear function represented as
where \( w \) is the weight vector, \( x \) is the input feature vector,
and \( b \) is the bias term.
3. Class Prediction : Unlike logistic regression, SVM does not provide
probabilities. Instead, it predicts class identities based on the sign of the linear
function:
* Positive class if \( w^T x + b > 0 \)
* Negative class if \( w^T x + b < 0 \)
4. Support Vectors : The data points that are closest to the hyperplane and
influence its position are called support vectors. These points are critical for the
model's performance.
5. Kernel Trick : SVM can utilize the kernel trick, which allows it to operate in a
higher*dimensional space without explicitly transforming the data. This is
particularly useful for non*linear classification problems.
6. Limitations : SVMs can be computationally expensive, especially with large
datasets, and may struggle to generalize well with certain kernel choices.
Principal Component Analysis (PCA)
1. Definition : PCA is an unsupervised learning algorithm used for dimensionality
reduction. It transforms the data into a new coordinate system where the greatest
variance lies along the first coordinate (principal component).
2. Mathematical Foundation : PCA finds a linear transformation \( z = x^T W \)
where \( W \) consists of the eigenvectors of the covariance matrix of the data. The
goal is to ensure that the variance of \( z \) is diagonal.
3. Variance Maximization : The principal components are derived from the
eigenvectors of the covariance matrix \( X^T X \), aligning the principal axes of
variance with the new representation space.
4. Data Decorrelation : PCA transforms the data into a representation where the
elements are mutually uncorrelated, which is a significant property for many
machine learning tasks.
5. Dimensionality Reduction : By projecting the original data onto a
lower*dimensional space, PCA preserves as much information as possible, measured
by the least*squares reconstruction error.
6. Applications : PCA is widely used in exploratory data analysis, noise reduction,
and as a preprocessing step for other machine learning algorithms.

Module*2
DOWNLOAD

Q. 03 a Explain the working of deep forward networks. 2 10

Deep feedforward networks, also known as feedforward neural networks or
multilayer perceptrons (MLPs), are fundamental models in deep learning. Their
primary goal is to approximate a function y = f ∗ (x) that maps input data \( x \) to an
output \( y \). This mapping is expressed y = f (x; θ), where (θ) represents the
parameters of the model that are learned during training.
 Structure and Functioning
1. Architecture : A deep feedforward network consists of an input layer, one or
more hidden layers, and an output layer. The depth of the network is determined by
the number of hidden layers it contains. Each layer is made up of units (or neurons)
that perform computations.
2. Information Flow : In a feedforward network, information flows in one
direction—from the input layer through the hidden layers to the output layer. There
are no feedback connections, meaning the output of the network does not loop back
into itself. This unidirectional flow is what distinguishes feedforward networks from
recurrent neural networks (RNNs), which do have feedback connections.
3. Activation Functions : Each neuron in the hidden layers applies an activation
function to its input, which introduces non*linearity into the model. Common
activation functions include the sigmoid, hyperbolic tangent (tanh), and rectified
linear unit (ReLU). The choice of activation function can significantly affect the
network's ability to learn complex patterns.
4. Training Process : The training of a deep feedforward network involves
adjusting the parameters \( \theta \) to minimize the difference between the predicted
output and the actual target output \( y \). This is typically done using a method
called backpropagation, which computes gradients of a loss function with respect to
the parameters. The loss function quantifies how well the network's predictions
match the actual data.
5. Cost Functions : The choice of cost function is crucial as it influences the
model's statistical properties. Common cost functions include mean squared error for
regression tasks and cross*entropy loss for classification tasks. The total cost may
also include a regularization term to prevent overfitting.
6. Optimization : Various optimization algorithms, such as stochastic gradient
descent (SGD) and its variants (like Adam), are used to update the parameters during
training. The optimizer determines how the parameters are adjusted based on the
computed gradients.
Importance in Machine Learning
Deep feedforward networks are foundational to many applications in machine
learning, including image recognition, natural language processing, and more. They
are particularly effective in learning complex functions due to their ability to model
hierarchical representations of data. The introduction of hidden layers allows these
networks to capture intricate patterns and relationships within the input data.
b What is regularization? How does regularization help in reducing overfitting. 2 10
Regularization is a crucial concept in machine learning that refers to a set of
techniques designed to improve a model's generalization ability, which is its
performance on unseen data, rather than just the training data. The primary purpose of
regularization is to combat overfitting, a common issue where a model learns the
training data too well, including its noise and outliers, resulting in poor performance on
new inputs.
There are several forms of regularization, each with its own approach to mitigating
overfitting:
1. Penalty Terms : One of the most common methods involves adding a penalty term
to the model's objective function. This penalty can be based on the norms of the model
parameters. For example:
* L1 Regularization (Lasso) : This adds the absolute values of the coefficients as a
penalty, encouraging sparsity in the model parameters.
* L2 Regularization (Ridge) : This adds the squared values of the coefficients as a
penalty, which tends to shrink the parameter values towards zero but does not
necessarily produce sparse solutions.
2. Early Stopping : This technique involves monitoring the model's performance on a
validation set during training and stopping the training process once performance starts
 to degrade. This prevents the model from fitting the training data too closely, thus
promoting better generalization.
3. Ensemble Methods : Techniques like bagging (bootstrap aggregating) combine
multiple models to reduce generalization error. By training several different models
and averaging their predictions, the ensemble can mitigate the risk of overfitting that
might occur with any single model.
4. Data Augmentation : This involves artificially increasing the size of the training
dataset by creating modified versions of the existing data. This can help the model
learn more robust features and reduce overfitting.
5. Dropout : In neural networks, dropout is a regularization technique where random
neurons are "dropped out" during training. This prevents the model from becoming
overly reliant on any single neuron, promoting a more distributed representation of the
data.
OR
Q.04 a Explain briefly about gradient descent algorithm. 2 10
The gradient descent algorithm is a fundamental optimization technique used in
machine learning and deep learning to minimize a cost function, which measures
how well a model's predictions match the actual data. Here's a brief overview:
1. Objective : The primary goal of gradient descent is to find the optimal
parameters (weights) for a model that minimize the cost function. This is crucial for
improving the model's accuracy.
2. Mechanism : The algorithm works by iteratively updating the model parameters
in the opposite direction of the gradient of the cost function with respect to those
parameters. The gradient indicates the direction of the steepest ascent, so moving in
the opposite direction helps to reduce the cost.
3. Learning Rate : A key component of gradient descent is the learning rate, a
hyperparameter that determines the size of the steps taken towards the minimum. If
the learning rate is too high, the algorithm may overshoot the minimum; if it's too
low, convergence can be very slow.
4. Types of Gradient Descent :
* Batch Gradient Descent : Uses the entire dataset to compute the gradient, which
can be computationally expensive for large datasets.
* Stochastic Gradient Descent (SGD) : Updates the parameters using only one
training example at a time, which can lead to faster convergence but introduces more
noise in the updates.
* Mini*batch Gradient Descent : A compromise between batch and stochastic
methods, it uses a small random subset of the data to compute the gradient, balancing
efficiency and stability.
5. Convergence : The algorithm continues to update the parameters until the
changes in the cost function are negligible, indicating that a minimum has been
reached. However, in practice, especially with non*convex functions like those often
encountered in neural networks, gradient descent may converge to a local minimum
rather than the global minimum.
6. Applications : Gradient descent is widely used in training various machine
learning models, including linear regression, logistic regression, and neural
networks, making it a cornerstone of modern machine learning practices.
b Discuss the working of Backpropogation. 2 10
Backpropagation is a fundamental algorithm used in training artificial neural
networks, particularly in deep feedforward networks. It is essential for optimizing the
weights of the network by minimizing the cost function, which measures the
difference between the predicted output and the actual target values. Here’s a
detailed discussion of how backpropagation works:
 1. Forward Propagation : The process begins with forward propagation, where the
input data is passed through the network layer by layer. Each neuron in the network
applies a weighted sum of its inputs, followed by a non*linear activation function
(like ReLU or sigmoid), to produce an output. This continues until the final layer
produces the predicted output \(\hat{y}\).
2. Cost Function Calculation : Once the output is generated, the next step is to
calculate the cost (or loss) using a cost function \(J(\theta)\). Common cost functions
include mean squared error for regression tasks and cross*entropy loss for
classification tasks. The cost function quantifies how well the model's predictions
match the actual target values.
3. Backward Propagation : After calculating the cost, backpropagation begins. This
involves two main steps:
* Gradient Calculation : The algorithm computes the gradient of the cost function
with respect to each weight in the network. This is done using the chain rule of
calculus, which allows the algorithm to propagate the error backward through the
network. The gradients indicate how much each weight should be adjusted to reduce
the cost.
* Weight Update : Once the gradients are computed, the weights are updated
using an optimization algorithm, typically stochastic gradient descent (SGD). The
weights are adjusted in the opposite direction of the gradient to minimize the cost
function. The update rule can be expressed as:
4. Efficiency through Dynamic Programming : Backpropagation employs a
technique similar to dynamic programming to avoid redundant calculations. By
storing intermediate results (like the outputs of neurons during forward propagation),
the algorithm efficiently computes gradients without recalculating the same values
multiple times.
5. Handling Complex Networks : While the basic idea of backpropagation is
straightforward, implementing it in practice can be complex due to the need to
support operations that return multiple outputs or manage memory consumption
effectively. Modern frameworks handle these complexities, allowing for efficient
computation even in deep networks.
6. Iterative Process : The forward and backward propagation steps are repeated for
many iterations (epochs) over the training dataset until the model converges,
meaning the cost function reaches a minimum value, and the model's predictions
become sufficiently accurate.
Module*3
DOWNLOAD
Q. 05 a Explain Empirical risk minimization. 3 10
Empirical Risk Minimization (ERM) is a fundamental concept in machine learning
that focuses on minimizing the expected loss of a model based on a given training
dataset. The idea is to find a model that performs well on the training data, which is
assumed to be representative of the underlying distribution of the data.
1. Definition : ERM aims to minimize the empirical risk, which is the average loss
calculated over the training samples. The empirical risk is defined as:

where \(L\) is the loss function, \(f\) is the model parameterized by \(\theta\), \(x_i\)
are the input features, \(y_i\) are the true labels, and \(m\) is the number of training
examples.
2. Loss Function : The choice of loss function \(L\) is crucial as it directly influences
the optimization process. Common loss functions include mean squared error for
regression tasks and cross*entropy loss for classification tasks. The goal is to select a
loss function that aligns well with the specific problem being solved.
 3. Optimization Process : The optimization process involves adjusting the model
parameters \(\theta\) to minimize the empirical risk. This is typically done using
gradient descent or its variants, where the gradients of the loss function with respect to
the parameters are computed and used to update the parameters iteratively.
4. Generalization : While minimizing empirical risk is essential, it is also important
to ensure that the model generalizes well to unseen data. Overfitting can occur if the
model learns the noise in the training data rather than the underlying patterns.
Techniques such as regularization, cross*validation, and early stopping are often
employed to mitigate overfitting.
5. Theoretical Foundation : The theoretical foundation of ERM is rooted in statistical
learning theory, which provides insights into how well a model trained on a finite
sample can be expected to perform on the entire population. The goal is to ensure that
the empirical risk converges to the true risk as the sample size increases.
6. Limitations : One limitation of ERM is that it relies heavily on the assumption that
the training data is representative of the true distribution. If the training data is biased
or not sufficiently diverse, the model may not perform well on new, unseen data.
b Explain the challenges occur in neural network optimization in detail. 3 10
Neural network optimization presents several significant challenges that can
complicate the training process. Here are some of the key challenges in detail:
1. Non*Convexity : Unlike traditional optimization problems that often deal with
convex functions, neural networks typically involve non*convex cost functions. This
means that the optimization landscape can contain multiple local minima, saddle
points, and flat regions, making it difficult for optimization algorithms to converge to a
global minimum. Practitioners often find that many local minima have similar cost
function values, which can lead to good generalization even if they are not the absolute
lowest.
2. Ill*Conditioning : The Hessian matrix, which describes the curvature of the cost
function, can be ill*conditioned. This means that the eigenvalues of the Hessian can
vary widely, leading to slow convergence rates in certain directions of the parameter
space. Ill*conditioning can make it challenging for optimization algorithms to
effectively navigate the parameter space, especially when the landscape is steep in
some directions and flat in others.
3. Inexact Gradients : Most optimization algorithms assume access to exact gradients
or Hessians. However, in practice, we often deal with noisy or biased estimates of
these quantities. This inaccuracy can lead to suboptimal updates and hinder the
convergence of the optimization process. Techniques like mini*batch gradient descent
can help mitigate this issue, but they introduce their own challenges, such as variance
in the gradient estimates.
4. Long*Term Dependencies : In recurrent neural networks (RNNs), the
optimization process can be complicated by long*term dependencies. As the depth of
the computational graph increases, the gradients can either vanish (become too small)
or explode (become too large), making it difficult to learn long*range dependencies in
the data. This is particularly problematic in tasks involving sequences, such as
language modeling or time series prediction.
5. Choice of Optimization Algorithm : Selecting the right optimization algorithm is
crucial. While algorithms like AdaGrad and RMSProp adapt the learning rate based on
past gradients, they may not perform uniformly well across all types of neural
networks. For instance, AdaGrad can lead to a premature decrease in the effective
learning rate, which can stall training. RMSProp modifies this by using an
exponentially weighted moving average of past gradients, but it still requires careful
tuning.
6. Initialization Strategies : The choice of weight initialization can significantly
impact the training process. Poor initialization can lead to slow convergence or cause
the optimization to get stuck in local minima. Techniques such as Xavier or He
initialization are designed to address this issue by setting the initial weights based on
 the number of input and output units in a layer.
7. Computational Bottlenecks : As the number of training examples increases,
computational bottlenecks can arise, affecting the generalization error. Efficiently
managing memory and computational resources becomes critical, especially when
training large models on extensive datasets.
8. Regularization Techniques : To prevent overfitting, regularization techniques such
as dropout, L1/L2 regularization, and early stopping are often employed. However,
these techniques introduce additional hyperparameters that need to be tuned, adding
complexity to the optimization process.
OR
Q. 06 a Explain AdaGrad and write an algorithm of AdaGrad. 3 10
AdaGrad, which stands for Adaptive Gradient Algorithm, is a popular optimization
algorithm used in training machine learning models, particularly deep learning
models. The primary feature of AdaGrad is its ability to adaptively adjust the
learning rates of model parameters based on the historical gradients. This means that
each parameter has its own learning rate that is scaled inversely proportional to the
square root of the sum of the squares of all historical gradients for that parameter.
Key Features of AdaGrad:
1. Adaptive Learning Rates : Each parameter's learning rate is adjusted based on
the accumulated past gradients, allowing for more significant updates for parameters
with smaller gradients and smaller updates for those with larger gradients.
2. Effective for Sparse Data : AdaGrad is particularly beneficial for problems with
sparse data, as it can adaptively adjust the learning rates for parameters that are
infrequently updated.
3. Rapid Convergence : The algorithm converges quickly when applied to convex
functions. However, it may lead to premature convergence in non*convex
optimization problems due to the accumulation of squared gradients.
Limitations:
* Decreasing Learning Rate : One of the main drawbacks of AdaGrad is that the
learning rate can decrease too much over time, which may slow down the training
process, especially in non*convex optimization scenarios.
AdaGrad Algorithm:
Explanation of the Algorithm:
1. Initialization : Start with an initial set of parameters and an accumulated squared
gradient variable set to zero.
2. Gradient Calculation : For each minibatch, compute the gradient of the loss
function with respect to the parameters.
3. Accumulation : Update the accumulated squared gradients by adding the square
of the current gradient.
4. Learning Rate Adjustment : Calculate the update for the parameters using the
adaptive learning rate, which is scaled by the accumulated squared gradients.
5. Parameter Update : Update the parameters accordingly.
 6. Repeat : Continue this process until the stopping criterion is met.

This algorithm allows for efficient training of deep learning models by adapting the
learning rates based on the historical performance of each parameter, making it a
valuable tool in the optimization toolkit for machine learning practitioners.
b Explain Adam algorithm in detail. 3 10
The Adam algorithm, short for Adaptive Moment Estimation, is a popular
optimization algorithm used in training deep learning models. It was introduced by
D.P. Kingma and J.B. Ba in 2014 and has since become one of the go*to methods for
many practitioners in the field of deep learning. Here’s a detailed explanation of the
Adam algorithm, which can help you understand its workings and advantages.
 Key Features of Adam Algorithm:
1. Adaptive Learning Rates : Adam combines the advantages of two other extensions
of stochastic gradient descent (SGD): AdaGrad and RMSProp. It adapts the learning
rate for each parameter individually, which helps in dealing with sparse gradients and
varying data distributions.
2. Momentum : Adam incorporates momentum by maintaining a moving average of
the gradients. This helps to smooth out the updates and can lead to faster convergence.
The momentum term is calculated as an exponentially weighted average of past
gradients.
3. Bias Correction : Since the moving averages of the gradients (first moment) and
the squared gradients (second moment) are initialized to zero, they can be biased
towards zero, especially during the initial steps of training. Adam includes a
bias*correction mechanism to counteract this effect, ensuring that the estimates are
more accurate.
Advantages of Adam:
* Efficiency : Adam is computationally efficient and requires little memory
overhead, making it suitable for large datasets and high*dimensional parameter spaces.
* Robustness : It performs well in practice across a wide range of problems and is
less sensitive to the choice of hyperparameters compared to other optimization
algorithms.
* Convergence : The combination of adaptive learning rates and momentum helps in
achieving faster convergence, especially in complex models.
Module*4
DOWNLOAD
Q. 07 a Explain the components of CNN layer. 4 10
Convolutional Neural Networks (CNNs) are a specialized type of neural network
primarily used for processing structured grid data, such as images. The architecture
of a CNN typically consists of several key components, each playing a crucial role in
feature extraction and classification.

main components of a CNN layer:

1. Convolutional Layer : This is the core building block of a CNN. It applies a set
of filters (or kernels) to the input data. Each filter slides (or convolves) across the
input image, performing element*wise multiplication and summing the results to
produce a feature map. This process helps in detecting various features such as
edges, textures, and patterns. The convolution operation reduces the spatial
dimensions of the input, which enhances computational efficiency.
2. Activation Function : After the convolution operation, an activation function is
applied to introduce non*linearity into the model. The most commonly used
activation function in CNNs is the Rectified Linear Unit (ReLU), which replaces all
negative values in the feature map with zero. This helps the network learn complex
patterns.
3. Pooling Layer : Pooling layers are used to down*sample the feature maps,
reducing their dimensionality while retaining the most important information. The
most common type of pooling is max pooling, which takes the maximum value from
a defined window (e.g., 2x2) of the feature map. This not only reduces the
computational load but also helps in making the model invariant to small translations
in the input.
4. Fully Connected Layer : After several convolutional and pooling layers, the
high*level reasoning in the neural network is performed by fully connected layers. In
this layer, every neuron is connected to every neuron in the previous layer. The
output from the last pooling layer is flattened into a one*dimensional vector and fed
 into the fully connected layer, which ultimately produces the final output, such as
class scores in classification tasks.
5. Dropout Layer : To prevent overfitting, dropout layers may be included in the
architecture. During training, dropout randomly sets a fraction of the input units to
zero, which helps the model generalize better by preventing it from becoming too
reliant on any particular feature.
6. Normalization Layers : Techniques like Batch Normalization can be applied to
stabilize and accelerate the training process. This layer normalizes the output of a
previous activation layer by adjusting and scaling the activations.
7. Output Layer : The final layer of the CNN is typically a softmax layer for
multi*class classification tasks. It converts the raw output scores from the fully
connected layer into probabilities, allowing the model to predict the class of the input
image.
b Explain Pooling with network representation. 4 10
Pooling is a crucial operation in convolutional neural networks (CNNs) that helps to
reduce the spatial dimensions of the input feature maps while retaining the most
important information. This process not only decreases the computational load but
also helps to prevent overfitting by providing a form of translation invariance. Let's
break down the concept of pooling with a network representation.
1. Max Pooling : This operation selects the maximum value from a defined
window (or region) of the feature map. For example, if we have a 2x2 window, max
pooling will take the highest value from each 2x2 block of the feature map and create
a new, smaller feature map.
2. Average Pooling : In contrast to max pooling, average pooling computes the
average value of the elements in the defined window. This can help in smoothing the
feature maps.
3. Global Average Pooling : This technique takes the average of all values in the
feature map, resulting in a single value per feature map. This is often used in the final
layers of a CNN before the classification layer.
Network Representation
To illustrate pooling in a network representation, consider a simple CNN
architecture:
1. Input Layer : An image of size 32x32 pixels.
2. Convolutional Layer : A convolutional layer with several filters (e.g., 16 filters
of size 3x3) is applied, resulting in a feature map of size 30x30 (assuming no
padding).
3. Pooling Layer : A max pooling layer with a 2x2 window and a stride of 2 is
applied. This reduces the size of the feature map from 30x30 to 15x15, as it takes the
maximum value from each 2x2 block.
Example Representation
* Input Feature Map (30x30) :
```
[ 1, 2, 3, 4, ..., 30 ]
[ 1, 2, 3, 4, ..., 30 ]
[ ... ]
[ 1, 2, 3, 4, ..., 30 ]
```
* Max Pooling Operation (2x2) :
```
[ 1, 2 ] *> 2
[ 3, 4 ] *> 4
```
 * Output Feature Map (15x15) :
```
[ 2, 4, ..., 30 ]
[ 2, 4, ..., 30 ]
[ ... ]
[ 2, 4, ..., 30 ]
```
Benefits of Pooling
1. Dimensionality Reduction : Pooling significantly reduces the number of
parameters and computations in the network, which helps in speeding up the training
process.
2. Translation Invariance : By summarizing the features in a local region, pooling
helps the network to become invariant to small translations in the input image.
3. Prevention of Overfitting : By reducing the complexity of the model, pooling
can help mitigate overfitting, especially when the training dataset is small.

OR
Q. 08 a Explain the variants of the CNN model. 4 10
Convolutional Neural Networks (CNNs) have several variants that enhance their
performance and adaptability for different tasks. Here are some notable variants:
1. LeNet*5 : One of the earliest CNN architectures, designed for handwritten digit
recognition. It consists of two convolutional layers followed by subsampling layers,
and it uses a fully connected layer at the end. LeNet*5 laid the groundwork for future
CNN designs.
2. AlexNet : This architecture significantly advanced the field of deep learning by
winning the ImageNet competition in 2012. AlexNet introduced deeper networks
with more convolutional layers, ReLU activation functions, and dropout for
regularization. It also utilized data augmentation techniques to improve
generalization.
3. VGGNet : Known for its simplicity and depth, VGGNet uses small 3x3
convolutional filters stacked on top of each other, which allows for a deeper
architecture while maintaining a manageable number of parameters. VGGNet has
been influential in demonstrating the benefits of depth in CNNs.
4. GoogLeNet (Inception) : This model introduced the Inception module, which
allows for multiple filter sizes to be applied simultaneously at each layer. This
architecture is efficient in terms of computation and memory, as it captures various
features at different scales.
5. ResNet (Residual Networks) : ResNet introduced the concept of skip
connections, which allow gradients to flow through the network more effectively
during training. This architecture enables the construction of very deep networks
(hundreds of layers) without suffering from vanishing gradients.
6. DenseNet : Similar to ResNet, DenseNet connects each layer to every other layer
in a feed*forward fashion. This dense connectivity pattern improves feature
propagation and reduces the number of parameters, making it efficient for training.
7. MobileNet : Designed for mobile and embedded vision applications, MobileNet
uses depthwise separable convolutions to reduce the model size and computational
cost while maintaining accuracy. This makes it suitable for real*time applications on
devices with limited resources.
8. EfficientNet : This model scales up the network width, depth, and resolution in a
balanced way, achieving state*of*the*art accuracy with fewer parameters.
EfficientNet uses a compound scaling method to optimize performance across
various tasks.
9. U*Net : Primarily used for biomedical image segmentation, U*Net features a
 symmetric architecture with an encoder*decoder structure. It combines
high*resolution features from the encoder with upsampled features in the decoder,
allowing for precise localization.
10. Faster R*CNN : This variant integrates region proposal networks (RPN) with
CNNs for object detection tasks. It improves the speed and accuracy of detecting
objects in images by sharing convolutional features between the RPN and the
detection network.
b Explain structured output with neural network. 4 10
Structured output in neural networks refers to the ability of a model to predict
multiple interdependent outputs simultaneously, rather than treating each output as
an independent prediction. This is particularly useful in tasks where the outputs are
related or have a specific structure, such as in natural language processing, image
segmentation, or multi*label classification.

2. Applications : Common applications of structured output include:

* Natural Language Processing (NLP) : Tasks like part*of*speech tagging or
named entity recognition, where the output is a sequence of labels corresponding to
the input sequence of words.
* Computer Vision : Image segmentation, where each pixel in an image is
classified into different categories, resulting in a structured map of the image.
* Graph*based Predictions : Predicting relationships in social networks or
molecular structures, where the output is a graph.
3. Modeling Techniques : To handle structured outputs, various techniques can be
employed:
* Conditional Random Fields (CRFs) : Often used in conjunction with neural
networks to model the dependencies between output variables.
* Recurrent Neural Networks (RNNs) : Particularly useful for sequence
prediction tasks, where the output is a sequence of labels.
* Convolutional Neural Networks (CNNs) : Used for tasks like image
segmentation, where the output is a spatial map of labels.
4. Loss Functions : The choice of loss function is crucial in structured output tasks.
Traditional loss functions like mean squared error may not be suitable. Instead,
specialized loss functions that account for the structure of the output, such as
structured SVM loss or CRF loss, are often used.
5. Training : Training structured output models can be more complex than standard
models due to the interdependencies between outputs. Techniques like joint training,
where the model learns to predict all outputs simultaneously, can be beneficial.
6. Evaluation Metrics : Evaluating structured outputs requires metrics that consider
the structure of the output. For instance, in image segmentation, metrics like
Intersection over Union (IoU) are used to assess the accuracy of pixel*wise
predictions.
Module*5
DOWNLOAD
Q. 09 a Explain how the recurrunt neural network (RNN) processes data sequences. 5 10
Recurrent Neural Networks (RNNs) are a specialized type of neural network
designed to process sequential data, which means they can handle inputs that come in
the form of sequences, such as time series data, sentences, or any other ordered data.
Here’s a detailed explanation of how RNNs work:
1. Sequential Processing : Unlike traditional feedforward neural networks that
process inputs independently, RNNs maintain a hidden state that captures
information about previous inputs in the sequence. This allows them to consider the
context of the entire sequence when making predictions.
2. Parameter Sharing : RNNs utilize a technique called parameter sharing, where
the same weights are applied across different time steps. This means that the model
can learn patterns in the data without needing separate parameters for each position
in the sequence, making it more efficient.
3. Hidden State : At each time step, the RNN takes the current input and the
previous hidden state to compute the new hidden state. This is typically done using a
function that combines the input and the previous hidden state, allowing the network
to retain information over time. The hidden state acts as a summary of the
information processed so far.
4. Variable Length Sequences : RNNs can handle sequences of varying lengths,
which is particularly useful in applications like natural language processing where
sentences can differ in length. The architecture allows the network to process each
element of the sequence one at a time, updating the hidden state accordingly.
5. Bidirectional RNNs : To enhance the model's ability to understand context,
bidirectional RNNs were developed. These networks process the input sequence in
both forward and backward directions, allowing the model to capture information
from both past and future states, which is especially beneficial in tasks like speech
recognition and handwriting recognition.
6. Gated RNNs : More advanced versions of RNNs, such as Long Short*Term
Memory (LSTM) networks and Gated Recurrent Units (GRUs), introduce gating
mechanisms. These gates control the flow of information, allowing the network to
retain or forget information over longer sequences, effectively addressing the
vanishing gradient problem that can occur in standard RNNs.
7. Applications : RNNs are widely used in various applications, including language
modeling, machine translation, speech recognition, and time series prediction. Their
ability to learn from sequences makes them particularly powerful for tasks where
context and order are crucial.
b Discuss about Bidirectional RNNs. 5 10
Bidirectional Recurrent Neural Networks (RNNs) are a powerful extension of
traditional RNNs designed to enhance the model's ability to capture context from both
past and future inputs in sequence data. Unlike standard RNNs, which process data in
a single direction (usually from past to present), bidirectional RNNs consist of two
separate RNNs: one that processes the input sequence from the beginning to the end
(forward RNN) and another that processes it from the end to the beginning (backward
RNN). This dual processing allows the model to have a more comprehensive
understanding of the context surrounding each input element.
2. Architecture : A typical Bidirectional RNN consists of two separate RNNs: one
that processes the input sequence from the beginning to the end (forward RNN) and
another that processes it from the end to the beginning (backward RNN). The outputs
from both RNNs are then combined to make predictions.
3. State Representation : At each time step \( t \), the output of the Bidirectional RNN
can leverage information from both the past (via the forward RNN) and the future (via
the backward RNN). This allows for a more comprehensive understanding of the input
sequence.
4. Applications : Bidirectional RNNs have been successfully applied in various
 fields, including:
* Speech Recognition : They help in understanding phonemes by considering the
context of upcoming phonemes.
* Handwriting Recognition : They improve the accuracy of recognizing characters
by utilizing both preceding and succeeding characters.
* Natural Language Processing : They are used in tasks like sentiment analysis and
machine translation, where context from both directions is crucial.
5. Advantages :
* Contextual Awareness : By processing sequences in both directions,
Bidirectional RNNs can capture dependencies that might be missed by unidirectional
models.
* Improved Performance : They often outperform standard RNNs in tasks where
context is important, leading to better accuracy in predictions.
6. Training : Bidirectional RNNs are typically trained using backpropagation through
time (BPTT), which involves calculating gradients for both the forward and backward
passes. This can be computationally intensive but is essential for learning effective
representations.
7. Limitations :
* Increased Complexity : The architecture is more complex than standard RNNs,
which can lead to longer training times and higher resource requirements.
* Memory Usage : They require more memory due to the need to store the states of
both RNNs.
8. Gated Variants : Many modern implementations of Bidirectional RNNs use gated
architectures, such as Long Short*Term Memory (LSTM) networks or Gated
Recurrent Units (GRUs), to mitigate issues like vanishing gradients and to better
manage long*range dependencies.
9. Use in Sequence*to*Sequence Models : Bidirectional RNNs are often used in
encoder*decoder architectures for tasks like machine translation, where the encoder
processes the input sequence and the decoder generates the output sequence based on
the context provided by the encoder.
OR
Q. 10 a Explain LSTM working principle along with equations. 5 10
The Long Short*Term Memory (LSTM) network is a type of recurrent neural
network (RNN) designed to effectively learn and remember over long sequences of
data. The key innovation of LSTMs is their ability to maintain long*term
dependencies, which is achieved through a unique architecture that includes memory
cells and gating mechanisms.
Working Principle of LSTM
1. Memory Cells : LSTMs have memory cells that can maintain information over
long periods. Each memory cell has a state that can be updated, read, or reset based
on the input and the previous state.
2. Gates : LSTMs utilize three types of gates to control the flow of information:
* Input Gate : This gate determines how much of the new information from the
current input should be added to the memory cell. It uses a sigmoid activation
function to output values between 0 and 1, where 0 means "ignore" and 1 means
"fully retain."
* Forget Gate : This gate decides what information should be discarded from the
memory cell. Similar to the input gate, it uses a sigmoid function to output values
that control the retention of previous information.
* Output Gate : This gate controls what information from the memory cell should
be output to the next layer. It also uses a sigmoid function to determine the output.
Equations
The LSTM's operations can be described mathematically with the following
equations:
b Write a note on Speech Recognition and NLP. 5 10
Speech Recognition and Natural Language Processing (NLP) are two interconnected
fields that focus on enabling computers to understand and process human language.
Speech Recognition is the technology that converts spoken language into text. The
process involves mapping an acoustic signal, which contains a spoken utterance, into a
sequence of words that the speaker intended. This is typically done using a sequence of
acoustic input vectors, which are generated by splitting audio into short frames (usually
around 20 milliseconds). Most speech recognition systems preprocess these inputs using
specialized features, although some modern deep learning systems can learn features
directly from raw audio data. The goal is to create a function that computes the most
probable linguistic sequence given the acoustic input. Over the years, advancements in
deep learning have significantly improved the accuracy of speech recognition systems,
particularly with the introduction of neural networks that replace traditional Gaussian
Mixture Models (GMMs) for associating acoustic features with phonemes.
1. Definition : Technology that converts spoken language into text by identifying and
processing acoustic signals.
2. Process :
* Preprocessing : Involves cleaning and preparing audio signals for analysis.
* Feature Extraction : Identifying key characteristics of the audio signal to aid in
recognition.
* Modeling : Traditionally used Gaussian Mixture Models (GMM) and Hidden
Markov Models (HMM) for predicting word sequences.
3. Advancements :
 * Since the late 2000s, deep learning techniques have transformed speech recognition.
* Neural networks have replaced GMMs, leading to improved accuracy.
4. Key Techniques :
* Restricted Boltzmann Machines (RBMs) : Used for unsupervised pretraining,
enhancing recognition rates.
* Benchmark Performance : Significant improvements noted on datasets like TIMIT,
reducing phoneme error rates from 26% to 20.7%.
5. Applications : Used in virtual assistants, transcription services, and voice*controlled
devices.

Natural Language Processing (NLP), on the other hand, involves the interaction between
computers and human languages. It encompasses a variety of applications, including
machine translation, where a sentence in one language is converted into an equivalent
sentence in another language. NLP relies on language models that define probability
distributions over sequences of words, characters, or bytes. Traditional models, like
n*grams, have been enhanced by neural language models (NLMs), which use distributed
representations of words to overcome the curse of dimensionality. This allows the model
to recognize similarities between words while still treating them as distinct entities. For
instance, if "dog" and "cat" share many attributes in their representations, the model can
leverage this similarity to improve predictions in sentences containing either word.
1. Definition : The field that focuses on the interaction between computers and human
languages, enabling machines to understand and generate human language.
2. Applications :
* Machine Translation : Converting text from one language to another.
* Sentiment Analysis : Determining the emotional tone behind a series of words.
* Text Summarization : Creating concise summaries of larger texts.
3. Language Models :
* N*grams : Sequences of n tokens used to predict the next token based on previous
ones.
* Neural Networks : Employed to learn contextual embeddings, capturing semantic
relationships between words.
4. Challenges :
* Ambiguity in language and the need for large datasets for effective model training.
* Handling diverse linguistic structures and variations.
5. Integration with Deep Learning : Enhances performance in tasks like parsing,
part*of*speech tagging, and overall language understanding.

Bloom’s Taxonomy Level: Indicate as L1, L2, L3, L4, etc. It is also desirable to indicate the COs and POs to
be attained by every bit of questions.

Page 01 of 01