Generative AI Assignment 01

Malaika Hussain
March 2025

1 Question 1: Implementing Rosenblatt’s Per-

ceptron from Scratch
1.1 Objective
This task involves implementing a single-layer perceptron model from scratch
to understand forward propagation, backward propagation, and weight updates.
The model is evaluated on a synthetic dataset.

1.2 Dataset Generation

A synthetic dataset with 500 samples and two features was generated. The data
points were assigned binary labels based on a non-linear decision boundary. The
dataset was split into 80% training and 20% testing.

1.3 Perceptron Model

The perceptron consists of: - A weight matrix initialized with zeros. - A bias
term. - A step function as the activation function. - Forward pass to compute
the weighted sum of inputs. - Backward pass to update weights using the
perceptron learning rule.
Training was performed over multiple epochs, adjusting weights to minimize
classification errors.

1.4 Training Performance

During training, the number of misclassified samples was recorded at each epoch
to observe convergence.

1.5 Decision Boundary and Evaluation

After training, the perceptron’s decision boundary was visualized. The test data
was overlaid to analyze classification accuracy.
The perceptron achieved a test accuracy of 72%, indicating reasonable clas-
sification performance given the non-linearity of the dataset.

1
 Figure 1: Visualization of the generated dataset

Figure 2: Error rate reduction over training epochs

2
Figure 3: Trained perceptron decision boundary with test data

Figure 4: Final test results and model performance

3
2 Question 2: Implementing Convolution from
Scratch
2.1 Objective
The purpose of this assignment is to develop a deeper understanding of convo-
lution operations by manually implementing them without using built-in deep
learning libraries. This will help analyze how different kernels influence image
processing.

2.2 Implementation Details

2.3 Generalized Convolution Function
A convolution function was implemented that takes a grayscale image and ap-
plies a user-defined kernel. The function includes the following parameters:
• Input image: The grayscale image to be processed.
• Kernel: User-defined kernel (defaults to a random kernel if none is pro-
vided).

• Kernel size: Size of the kernel matrix.

• Stride: Step size for sliding the kernel.
• Padding: Can be ”valid” (no padding) or ”same” (zero-padding to main-
tain size).

• Mode: Option to perform either convolution or correlation.

The function manually implements the convolution operation using nested
loops, ensuring that no deep learning libraries are used.

2.4 Application of Different Kernels

The convolution function was applied to a grayscale image using various kernels
to achieve different effects:
• Edge Detection: A Sobel filter was used to detect edges in the image.

• Blurring: A Gaussian blur kernel was applied to smoothen the image.

• Sharpening: A sharpening kernel was applied to enhance image details.

4
2.5 Comparison of Convolution vs. Correlation
To compare convolution and correlation, two different kernels were used:
• A symmetric kernel:
0.250.250.250.25

• A non-symmetric kernel:

121000 − 1 − 2 − 1

The results were analyzed to observe how convolution and correlation yield
different outcomes.

2.6 Results and Analysis

2.7 Visualization of Results
• The original grayscale image was displayed.
• Output images for each applied kernel were visualized.
• A side-by-side comparison was conducted between manually implemented
convolution and NumPy-based convolution.

• The differences between convolution and correlation results were analyzed.

2.8 Effect of Different Kernels

• Edge Detection: The Sobel filter effectively highlighted edges in the
image by detecting intensity changes.
• Blurring: The Gaussian blur kernel smoothed the image by reducing
high-frequency components.
• Sharpening: The sharpening kernel enhanced details by amplifying high-
frequency components.

2.9 Impact of Kernel Size, Stride, and Padding

• Kernel Size: Larger kernels tend to blur the image more, while smaller
kernels maintain finer details.
• Stride: Increasing the stride reduces the output size and can lead to
information loss.
• Padding: Using ”same” padding preserves the original image size, whereas
”valid” padding reduces the output size.

5
2.10 Observations on Convolution vs. Correlation
• Convolution flips the kernel before applying it, while correlation does not.
• The difference in results is more pronounced when using asymmetric ker-
nels.

• Edge detection kernels, such as Sobel, yield different results when used in
convolution versus correlation.

2.11 Application of Multiple Kernels

Applying multiple kernels sequentially can enhance feature extraction. For ex-
ample:
• First applying an edge detection kernel, followed by a sharpening kernel,
enhances detected edges.

• Combining a blur filter followed by an edge detection filter reduces noise

before extracting edges.
This approach is often used in image preprocessing for deep learning models.

2.12 Conclusion
Through manual implementation, we have gained insights into how convolution
operations function and how different kernel types affect image processing. The
analysis of stride, padding, and kernel size provided a better understanding of
convolution’s impact on image transformation.

2.13 Images and Results

6
Figure 5: Original image before processing.

7
 Figure 6: Converted grayscale image.

Figure 7: Effect of applying multiple kernels sequentially.

8
 Figure 8: Edge detection using the Sobel filter.

Figure 9: Effect of padding on convolution output.

Figure 10: Effect of different stride values on convolution output.

9
Figure 11: Demonstration of different kernel applications.

10
3 Question 3: Implementation of a Convolu-
tional Neural Network for CIFAR-10
3.1 Introduction
The objective of this report is to implement a Convolutional Neural Network
(CNN) for image classification using the CIFAR-10 dataset. The study includes
dataset preparation, CNN model implementation, evaluation, data augmenta-
tion, and an ablation study to analyze hyperparameter effects.

3.2 Dataset
The CIFAR-10 dataset consists of 60,000 32x32 RGB images belonging to 10
classes. The dataset is split into 50,000 training images and 10,000 testing
images.

3.3 Dataset Preparation

• Load the dataset from Hugging Face.
• Normalize pixel values between 0 and 1.
• Convert labels into one-hot encoded format.

• Split dataset into training (80%) and testing (20%) subsets.

3.4 CNN Model Architecture

The Convolutional Neural Network (CNN) designed for CIFAR-10 classification
follows a hierarchical structure to extract spatial features from images effectively.
The architecture consists of multiple layers, each playing a crucial role in the
learning process.

3.5 Convolutional Layers

The network begins with convolutional layers that apply a set of learnable filters
to the input image. These filters help capture local patterns such as edges,
textures, and shapes. Each convolutional layer is followed by a Rectified Linear
Unit (ReLU) activation function, introducing non-linearity into the model and
enhancing its ability to learn complex features.

3.6 Batch Normalization

Batch normalization is applied after each convolutional layer to standardize
activations, stabilize training, and accelerate convergence. This reduces internal
covariate shifts and helps improve generalization.

11
3.7 Pooling Layers
Pooling layers are introduced after groups of convolutional layers to reduce the
spatial dimensions of feature maps. Max pooling is employed to retain the most
significant features while reducing computational complexity. This helps make
the model more robust to small translations and distortions in images.

3.8 Dropout Layers

To prevent overfitting, dropout layers are included after pooling layers. Dropout
randomly disables a fraction of neurons during training, forcing the network to
learn more generalized patterns rather than memorizing specific examples.

3.9 Fully Connected Layers

Following the feature extraction process, the output from the convolutional lay-
ers is flattened into a one-dimensional vector and passed through fully connected
layers. These layers learn complex representations and patterns in the data. The
final fully connected layer outputs class probabilities using a softmax activation
function.

3.10 Optimization and Loss Function

The model is optimized using the Adam optimizer, which combines momentum
and adaptive learning rates for efficient training. Categorical cross-entropy is
used as the loss function since CIFAR-10 is a multi-class classification problem.
This architecture ensures that the CNN captures both low-level and high-
level features, leading to effective classification of CIFAR-10 images.

3.11 Evaluation and Comparison of Model Performance

3.12 Performance Metrics
The CNN models are evaluated using the following metrics:

• Accuracy
• Precision
• Recall
• F1-Score

• Confusion Matrix

12
3.13 Comparison of Models
Two models were trained and evaluated:
• Model without Data Augmentation
• Model with Data Augmentation (flipping, rotation, shifting)

Model Accuracy Precision Recall F1-Score

Without Augmentation 0.8587 0.8604 0.8587 0.8589
With Augmentation 0.8507 0.8564 0.8507 0.8506

Table 1: Performance Metrics Comparison of CNN Models

3.14 Confusion Matrix Visualization

For both models, confusion matrices were generated to analyze classification
performance. The misclassified categories were identified and examined.

3.15 Loss and Accuracy Curves

Training and validation loss/accuracy curves were plotted to analyze conver-
gence and detect overfitting or underfitting.

3.16 Feature Map Visualization

Feature maps were extracted and visualized from different layers to analyze how
the model learns representations. Early layers capture edges, while deeper layers
capture high-level features.

3.17 Ablation Study: Hyperparameter Impact

An ablation study was conducted by modifying hyperparameters:
• Learning Rate: Tested values: 0.001, 0.01, 0.1.
• Batch Size: Compared batch sizes: 16, 32, 64.
• Number of Convolutional Filters: Evaluated 16, 32, and 64 filters.
• Number of Layers: Compared architectures with 3, 5, and 7 convolu-
tional layers.

3.18 Conclusion
This report presented the implementation of a CNN for CIFAR-10 classification.
The effect of data augmentation and hyperparameters on model performance
was analyzed. Results show that data augmentation improves generalization,
while hyperparameters significantly impact accuracy.

13
3.19 Images and Results

Figure 12: Confusion Matrix with Augmentation

14
Figure 13: Confusion Matrix with Augmentation

Figure 14: Loss Accuracy Curves

15
4 Question 4: Implementation of a Vanilla RNN
for Next-Word Prediction
4.1 Introduction
The objective of this project is to implement a Vanilla Recurrent Neural Network
(RNN) trained on a Shakespeare text dataset for next-word prediction. Instead
of using pre-trained embeddings (e.g., Word2Vec or GloVe), the model will learn
its own embeddings using a trainable embedding layer.

4.2 Dataset
The dataset used for training is the publicly available Shakespeare text dataset
from Hugging Face. The dataset consists of a large corpus of Shakespeare’s
works, and preprocessing involves tokenization and vocabulary creation.

4.3 Implementation Steps

4.4 Data Preprocessing
• Load the Shakespeare dataset from Hugging Face.
• Tokenize the text and create a vocabulary.
• Convert words into integer sequences.
• Split the dataset into training (80

4.5 Vanilla RNN Model

• Implement a custom RNN cell without LSTMs or GRUs.
• Utilize a trainable Embedding Layer to learn word representations.

• Process word sequences and predict the next word.

• Use Cross-Entropy Loss and the Adam optimizer for training.

4.6 Training the Model

• Train the model using Backpropagation Through Time (BPTT).
• Monitor training and validation loss across epochs.
• Save the trained model for text generation.

16
4.7 Evaluation and Performance Metrics
4.8 Generated Text Sequences
To test the model, a seed phrase such as “To be or not to” is provided, and the
model generates the next words iteratively.

4.9 Performance Metrics

The model is evaluated using the following metrics:
• Perplexity: Measures the uncertainty of the model.
• Word-level accuracy: Measures how often the predicted words match
the expected words.
• Loss curve visualization: Helps analyze model convergence.

4.10 Experimental Results

The trained model achieved the following results on the test dataset:

• Test Loss: 6.0157

• Test Perplexity: 409.8084
• Test Accuracy: 9.37

4.11 Ablation Study: Impact of Pretrained Word Embed-

dings
To analyze the impact of using pretrained word embeddings, the model is trained
separately using:
• Randomly initialized embeddings.
• Learned embeddings from the trainable embedding layer.

4.12 Conclusion
This report presents the implementation of a Vanilla RNN for next-word pre-
diction using a Shakespeare text dataset. The model successfully learns word
embeddings and generates text sequences. An ablation study is conducted to
compare learned embeddings with random embeddings, showing the effect of
word representation on model performance. The test results indicate that the
model struggles with long-term dependencies, leading to high perplexity and
low accuracy.

4.13 Images and Results

17
Figure 15: Pretrained Embeddings Confusion matrix.

Figure 16: Pretrained Embeddings Learning curves.

18
5 Question 5: Hyperparameter Search for CNN
and RNN
5.1 Introduction
Hyperparameter tuning is crucial for optimizing deep learning models. In this
task, we perform a random search over a predefined set of hyperparameters for
both Convolutional Neural Networks (CNNs) and Recurrent Neural Networks
(RNNs). The best configurations are selected based on validation accuracy, and
the models are evaluated on the test dataset.

5.2 Hyperparameter Search

The following hyperparameters were considered in the search:
• Learning rate
• Number of layers
• Number of neurons (for RNN) or filters (for CNN)
• Batch size
• Optimizer (Adam, SGD, RMSprop)
• Activation functions (ReLU, Tanh, Sigmoid)
• Dropout rate
• Kernel size (for CNN)
• Stride (for CNN)
• Weight initialization method (Xavier, He Normal)
RandomizedSearchCV from Scikit-Learn or a custom random sampling ap-
proach was used for hyperparameter tuning. Multiple models were trained, and
the best-performing configuration was selected based on validation accuracy.

5.3 Best Hyperparameter Configurations

The optimal hyperparameters for both CNN and RNN models are as follows:

5.4 Test Results

The best hyperparameter configurations were evaluated on the test dataset. The
results are summarized below:

5.5 Model Evaluation Metrics

The classification report for the best RNN and CNN models is presented below:

19
 Hyperparameter Best Value
Weight Initialization Glorot Uniform
Stride 2
Optimizer SGD
Number of Units (RNN) 128
Number of Layers 2
Number of Filters (CNN) 128
Learning Rate 0.0001
Kernel Size (CNN) 3
Dropout 0.5
Batch Size 32
Activation Function Sigmoid

Table 2: Best Hyperparameter Configurations for CNN and RNN

Model Test Accuracy

Best RNN 92.70%
Best CNN 92.50%

Table 3: Test Accuracy for Best CNN and RNN Models

5.6 RNN Model Performance

Class Precision Recall F1-Score Support

0 0.85 0.93 0.89 310
1 0.97 0.93 0.95 690
Accuracy 0.93
Macro Avg 0.91 0.93 0.92 1000
Weighted Avg 0.93 0.93 0.93 1000

Table 4: RNN Model Evaluation

5.7 CNN Model Performance

20
 Class Precision Recall F1-Score Support
0 0.85 0.91 0.88 310
1 0.96 0.93 0.94 690
Accuracy 0.93
Macro Avg 0.91 0.92 0.91 1000
Weighted Avg 0.93 0.93 0.93 1000

Table 5: CNN Model Evaluation

5.8 Conclusion
The hyperparameter search yielded optimal configurations for both CNN and
RNN models. The RNN model achieved slightly better test accuracy (92.70%)
than the CNN model (92.50%). Both models demonstrated high classification
performance, confirming the effectiveness of hyperparameter tuning in improv-
ing model performance.

21