Machine Unlearning

Reddy Gaurav

Advisor: Dr. Kaustuv Nag

Department of Computer Science and Engineering

Indian Institute of Information Technology Guwahati

This dissertation is submitted for the degree of

Master of Technology

IIIT Guwahati November 2024

 Declaration

I hereby declare that except where specific reference is made to the work of others,
the contents of this dissertation are original and have not been submitted in whole
or in part for consideration for any other degree or qualification in this or any
other university. This dissertation is my own work and contains nothing that is the
outcome of work done in collaboration with others except as specified in the text
and Acknowledgements.

Reddy Gaurav
Roll: 2101167,
Department of Computer Science and Engineering,
Indian Institute of Information Technology Guwahati.
 Acknowledgements

I acknowledge Dr. Kaustuv Nag’s invaluable guidance and support throughout

this project. My gratitude for their mentorship extends to all the Department of
Computer Science and Engineering faculty at the Indian Institute of Information
Technology Guwahati. I also thank my friends for my cherished time during this
degree. My appreciation goes out to my parents and family members for their
unwavering encouragement and support throughout all my studies.
 Abstract

This project focuses on implementing machine unlearning using a structured ap-

proach that combines adding noise to the model and subsequent impairment and
repair phases. This method enables selective unlearning of data while maintaining
overall model integrity on retained data. The unlearning methodology could be
valuable for privacy preservation, compliance with data regulations, or targeted
data modification without full retraining.
Table of Contents

1 Introduction 1
1.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 3
2.1 Dataset Pre processing . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Transformation Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2.1 Transform Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2.2 Data Augmentation . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 Model Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.4 Model Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.5 Training Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.5.1 Dataset Preparation . . . . . . . . . . . . . . . . . . . . . . . . 5
2.6 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.7 Training the Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.7.1 Training Hyperparameters . . . . . . . . . . . . . . . . . . . . 5
2.7.2 Training Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.7.3 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Evaluation 7
3.1 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.2 Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.2.1 Baseline Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.2.2 Post-Impair Step Evaluation . . . . . . . . . . . . . . . . . . . . 7
3.2.3 Post-Repair Step Evaluation . . . . . . . . . . . . . . . . . . . . 8
3.3 Analysis and Observations . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.3.1 Forgetting Effectiveness . . . . . . . . . . . . . . . . . . . . . . 8
3.3.2 Retention Robustness . . . . . . . . . . . . . . . . . . . . . . . 9
3.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Chapter 1

Introduction

Machine unlearning is an emerging field in machine learning that aims to selectively

"forget" specific data within a trained model, allowing it to comply with data privacy
regulations or correct targeted data without needing a complete retraining. This
project explores a structured machine unlearning approach, utilizing noise addition,
impairment, and repair phases to ensure that specific data points are removed
from the model’s knowledge base while preserving the accuracy on remaining
data. By training the model with an intentional injection of noise, the model can
"forget" certain information, thereby achieving a state of unlearning for privacy or
compliance purposes.
The methodology includes an initial model training and evaluation, followed by
an impairment step where noise is introduced to degrade the model’s dependency
on unlearned data. The repair step then helps refine the model’s performance
on retained data, ensuring it remains effective after the unlearning process. This
approach offers a practical and efficient way to handle unlearning in machine
learning applications, making it highly relevant for scenarios requiring adherence
to strict privacy policies, rapid corrections, or adjustments to model knowledge
without the cost and time of full retraining.

1.1 Objective
The objective of this project is to develop and implement an efficient machine
unlearning framework that allows a trained machine learning model to selectively
"forget" specific data while preserving its performance on the remaining dataset.
This approach seeks to address the growing need for privacy compliance, such as

1
 1.1 Objective

adhering to data deletion requests mandated by regulations like GDPR. Using noise
injection and subsequent impairment and repair steps, the project aims to create a
systematic method that can modify model knowledge without requiring complete
retraining. This method aims to offer a practical, resource-efficient solution for
organizations needing to maintain model compliance and adaptability in response
to dynamic data privacy requirements.

2
Chapter 2

This section discusses prior work in neural network-based classification tasks, specif-
ically using fully connected layers for image recognition on datasets like MNIST. It
highlights PyTorch’s contributions to simplifying deep learning implementations.

2.1 Dataset Pre processing

Discusses the MNIST dataset, detailing how it contains handwritten digits (0–9)
and the pre-processing steps applied, including normalization to improve learning
performance.

2.2 Transformation Steps

2.2.1 Transform Pipeline
The transforms.Compose pipeline for the MNIST dataset is defined to ensure proper
preprocessing of the data before feeding it into the neural network. The pipeline
includes the following steps:

• Conversion to Tensor: The raw image data is converted to a PyTorch tensor

using transforms.ToTensor(). This step scales pixel values from the range
[0, 255] to [0, 1] for numerical stability during training.

• Normalization: The tensor is normalized using transforms.Normalize(mean,

std), where the mean and standard deviation are calculated across the MNIST
dataset. This ensures that the input data has a zero mean and unit variance,
accelerating convergence during training.

3
 2.3 Model Architecture

2.2.2 Data Augmentation

Data augmentation techniques are applied (if applicable) to artificially increase the
size and diversity of the dataset, thereby improving the model’s generalization. For
MNIST, the following augmentations were considered:

• Random Rotation: Images are randomly rotated within a small range (e.g.,
±10◦ ). This helps the model generalize to digit orientations not seen in the
original dataset.

• Random Crop or Padding: Padding is added around the digits, followed by

random cropping to simulate shifts in digit placement within the image.

These transformations were chosen to enhance the model’s robustness to varia-

tions commonly encountered in handwritten digit recognition tasks, such as rota-
tions and positional shifts. The use of normalization ensures consistent input scaling,
which is critical for effective gradient descent optimization.

2.3 Model Architecture

This section outlines the architecture of the neural network.

Layer Definition

The model includes three fully connected layers: fc1, fc2, and fc3. Each layer
transforms input dimensions to the next, ensuring appropriate intermediate repre-
sentations.

Activation Functions

ReLU activations are applied to hidden layers for non-linear transformations, while
the final output uses softmax for classification probabilities.

2.4 Model Parameters

Lists the trainable parameters in the network, detailing the role of weights and biases
within the network’s layers.

4
 2.5 Training Process

2.5 Training Process

2.5.1 Dataset Preparation
The CIFAR-10 dataset is utilized, which contains 10 classes of images such as dog,
cat, airplane, and truck. The dataset is split into training and test sets as follows:

• Training Set: Images from the train folder are transformed using the follow-
ing techniques:

– ToTensor: Converts images into tensors.

– Normalize: Applies mean and standard deviation normalization based
on the CIFAR-10 dataset statistics.

• Test Set: Similar transformations are applied for consistency in data represen-
tation.

2.6 Training

2.7 Training the Model

The ResNet-18 architecture is utilized for the classification task. The details of the
training process are as follows:

2.7.1 Training Hyperparameters

• Batch size: 256

• Optimizer: Adam

• Maximum learning rate: 0.01

• Gradient clipping: 0.1

• Weight decay: 1 × 10−4

5
 2.7 Training the Model

2.7.2 Training Strategy

• A one-cycle learning rate policy is implemented to ensure robust convergence.

• The model is trained for 40 epochs, with the learning rate dynamically adjusted
based on validation loss using ReduceLROnPlateau.

2.7.3 Performance
• Training loss decreased significantly across epochs, indicating effective learn-
ing.

• Validation accuracy plateaued at approximately 77.17% by the end of training.

6
Chapter 3

Evaluation

3.1 Evaluation Metrics

The following metrics are used:

• Accuracy: Measures the proportion of correct predictions.

• Loss: Quantifies the difference between the predicted and actual labels using
cross-entropy loss.

3.2 Evaluation Results

3.2.1 Baseline Evaluation
Before implementing the unlearning pipeline, the ResNet-18 model was trained on
the full CIFAR-10 dataset. Key results:

• Overall Accuracy: 77.17%.

• Validation Loss: 1.396.

These results indicate strong baseline performance, forming the benchmark for
subsequent evaluations.

3.2.2 Post-Impair Step Evaluation

The impair step retrains the model using noisy data for the forgotten classes (classes
0 and 2) and clean data for the retained classes. Results are as follows:

7
 3.3 Analysis and Observations

• Forgotten Classes:

– Accuracy: 63.47%.
– Loss: 9.182.

• Retained Classes:

– Accuracy: 68.18%.
– Loss: 0.930.

The impaired model shows a significant reduction in performance for the forgotten
classes, indicating partial unlearning. The retained classes maintain stable accuracy,
demonstrating minimal negative impact.

3.2.3 Post-Repair Step Evaluation

The repair step fine-tunes the model using only the retained class samples to enhance
robustness. Results are:

• Forgotten Classes:

– Accuracy: 0.0%.
– Loss: 10.907.

• Retained Classes:

– Accuracy: 70.95%.
– Loss: 0.827.

The forgotten classes are completely unlearned, achieving the desired goal of 0%
accuracy. The retained class performance improves slightly after repair, with a
decrease in loss and an increase in accuracy.

3.3 Analysis and Observations

3.3.1 Forgetting Effectiveness
The unlearning process successfully removes the model’s knowledge of the forgotten
classes:

8
 3.4 Conclusion

• Accuracy for forgotten classes dropped from 63.47% (post-impair) to 0.0%

(post-repair).

• High loss for forgotten classes (10.907) post-repair indicates the model’s inabil-
ity to predict these classes.

3.3.2 Retention Robustness

The retention objective is well-maintained:

• Retained class accuracy improved from 68.18% (post-impair) to 70.95% (post-

repair).

• Reduced loss for retained classes (from 0.930 to 0.827) shows enhanced model
stability.

3.4 Conclusion
The evaluation demonstrates the effectiveness of the machine unlearning pipeline:

• The impair step partially removes knowledge of the forgotten classes with
minimal impact on the retained classes.

• The repair step ensures complete forgetting of the specified classes while
improving robustness and performance on the retained classes.

The approach is efficient and achieves the unlearning objectives, making it suitable
for applications requiring privacy or adaptability in machine learning models.