Autoencoders and GAN Lab Programs - Algorithms and

Explanations
Program 1: Undercomplete Autoencoder

Algorithm:
1. Load and normalize MNIST dataset (28×28 grayscale images)
2. Build encoder: Conv2D(32) → MaxPool → Conv2D(16) → MaxPool

3. Build decoder: Conv2D(16) → UpSample → Conv2D(32) → UpSample → Conv2D(1)

4. Compile model with Adam optimizer and binary crossentropy loss

5. Train model where input = output (self-supervised learning)

6. Generate reconstructed images and visualize results

Important Modules:
tensorflow.keras : Deep learning framework

layers.Conv2D : Convolutional layers for feature extraction

layers.MaxPooling2D/UpSampling2D : Dimensionality reduction/expansion

matplotlib.pyplot : Visualization

Approach/Logic:
Undercomplete means the hidden layer has fewer neurons than input, forcing the network to learn
compressed representations. The encoder compresses 28×28=784 pixels into a smaller latent space, then
the decoder reconstructs the original image. This learns essential features by removing noise and
redundancy.

Program 2: Denoising Autoencoder

Algorithm:
1. Load and flatten MNIST data to 784-dimensional vectors
2. Create encoder: Dense(32) with ReLU activation

3. Create decoder: Dense(784) with sigmoid activation

4. Train autoencoder to reconstruct clean images from clean inputs

5. Test reconstruction quality on test set

Important Modules:
tensorflow.keras : Neural network framework
 layers.Dense : Fully connected layers

Input : Input layer specification

Model : Model creation and compilation

Approach/Logic:
Unlike regular autoencoders, denoising autoencoders are trained to reconstruct clean data from
corrupted inputs. The 32-dimensional bottleneck forces the network to learn robust features that can
handle noise. The reconstruction loss encourages learning meaningful representations rather than just
memorizing inputs.

Program 3: Variational Autoencoder (VAE)

Algorithm:
1. Define encoder that outputs mean (μ) and log-variance (σ²) of latent distribution

2. Implement reparameterization trick: z = μ + σ × ε (where ε ~ N(0,1))

3. Build decoder that reconstructs from sampled latent vector z

4. Define VAE loss = Reconstruction Loss + KL Divergence Loss

5. Train on MNIST and visualize latent space and generated samples

Important Modules:
keras.layers.Lambda : Custom sampling layer

keras.backend.K : Backend operations for random sampling

scipy.stats.norm : Normal distribution for latent space sampling

keras.objectives : Loss function utilities

Approach/Logic:
VAEs are probabilistic autoencoders. Instead of deterministic encoding, they learn probability
distributions in latent space. The reparameterization trick allows backpropagation through stochastic
nodes. KL divergence loss ensures latent space follows standard normal distribution, enabling smooth
interpolation and generation of new samples.

Program 4: Restricted Boltzmann Machine (RBM)

Algorithm:
1. Load MovieLens dataset and convert to binary ratings (like/dislike)

2. Initialize weights W, visible bias b, hidden bias a randomly

 3. For each training epoch:
Positive phase: v₀ → h₀ (sample hidden given visible)

Negative phase: h₀ → v₁ → h₁ (reconstruct visible, then hidden)

Update weights: ΔW = η(v₀h₀ᵀ - v₁h₁ᵀ)

4. Evaluate reconstruction error on test set

Important Modules:
torch : PyTorch deep learning framework

torch.mm : Matrix multiplication

torch.sigmoid : Sigmoid activation function

torch.bernoulli : Binary sampling from probabilities

Approach/Logic:
RBMs are energy-based generative models. They learn probability distributions over visible units
through hidden units. Contrastive Divergence training approximates maximum likelihood by comparing
positive (data) and negative (model) statistics. The reconstruction error indicates how well the model
captures data patterns.

Program 5: Stacked RBM

Algorithm:
1. Load digits dataset and normalize to [0,1]

2. Create pipeline: RBM₁(64 components) → RBM₂(32 components) → LogisticRegression

3. Train first RBM on raw data to learn first-level features

4. Train second RBM on first RBM's hidden representations

5. Train classifier on final hidden representations

6. Evaluate classification performance

Important Modules:
sklearn.neural_network.BernoulliRBM : RBM implementation

sklearn.pipeline.Pipeline : Sequential model chaining

sklearn.linear_model.LogisticRegression : Final classifier

sklearn.metrics.classification_report : Performance evaluation

Approach/Logic:
Deep learning through layer-wise pretraining. Each RBM learns increasingly abstract features: RBM₁
learns edge-like features, RBM₂ learns part-like features. This hierarchical feature learning was crucial
before modern deep learning techniques, providing better initialization than random weights.

Program 6: Deep Belief Network (DBN)

Algorithm:
1. Load MNIST dataset with standard preprocessing

2. Build feed-forward network: 784 → 256 → 128 → 10 neurons

3. Use ReLU activations in hidden layers, no activation in output

4. Train with CrossEntropyLoss and Adam optimizer

5. Evaluate accuracy on test set and visualize predictions

Important Modules:
torch.nn : Neural network modules

torch.nn.functional.F : Activation functions

torchvision.datasets : Standard datasets

torch.utils.data.DataLoader : Batch processing

Approach/Logic:
DBNs combine generative pretraining with discriminative fine-tuning. Though this implementation
shows the discriminative part, true DBNs first pretrain each layer as an RBM, then fine-tune the entire
network. This creates robust feature hierarchies for classification tasks.

Program 7: Deep Boltzmann Machine (DBM)

Algorithm:
1. Initialize two weight matrices W₁(visible↔hidden₁), W₂(hidden₁↔hidden₂)

2. Upward pass: v₀ → h₁ → h₂ (sample each layer)

3. Downward pass: h₂ → h₁ → v₁ (reconstruct backwards)

4. Update weights using contrastive divergence-like rule

5. Display updated parameters

Important Modules:
numpy : Numerical computing

np.random.binomial : Binary sampling

 np.outer : Outer product for weight updates

np.dot : Matrix multiplication

Approach/Logic:
DBMs are undirected graphical models with multiple hidden layers. Unlike stacked RBMs, all layers
interact bidirectionally during inference. The mean-field approximation is used for inference, making
training more complex but potentially more powerful for learning hierarchical representations.

Program 8: Convolutional Autoencoder for Image Compression

Algorithm:
1. Load MNIST and reshape for convolutional processing

2. Encoder: Conv2D(32) → MaxPool → Conv2D(16) → MaxPool

3. Decoder: Conv2D(16) → UpSample → Conv2D(32) → UpSample → Conv2D(1)

4. Train with binary crossentropy loss

5. Calculate Mean Squared Error between original and compressed images

Important Modules:
keras.Sequential : Sequential model building

keras.layers.Conv2D : Convolutional layers

keras.layers.MaxPooling2D/UpSampling2D : Spatial operations

sklearn.metrics.mean_squared_error : Compression quality metric

Approach/Logic:
Spatial compression using convolutions. Max pooling reduces spatial dimensions while preserving
important features. The bottleneck layer represents compressed data. Upsampling reconstructs spatial
dimensions. This is more efficient than fully-connected autoencoders for images as it preserves spatial
relationships.

Program 9: Generative AI using CNN

Algorithm:
1. Load and normalize MNIST dataset

2. Build CNN autoencoder: Conv2D(16)→MaxPool→Conv2D(8)→MaxPool (encoder)

3. Decoder: Conv2D(8)→UpSample→Conv2D(16)→UpSample→Conv2D(1)

4. Train for 5 epochs with binary crossentropy

 5. Generate reconstructions and compare with originals

Important Modules:
tensorflow.keras.models.Model : Functional API model

tensorflow.keras.layers : Various layer types

numpy.reshape : Data preprocessing

matplotlib.pyplot : Visualization

Approach/Logic:
CNN-based generation through reconstruction. The model learns to generate images by
reconstructing training examples. The convolutional structure captures spatial hierarchies: early layers
learn edges, deeper layers learn textures and shapes. This forms the basis for more complex generative
models.

Program 10: Generative Adversarial Networks (GANs)

Algorithm:
1. Create Generator: Linear(100→256→784) with Tanh output
2. Create Discriminator: Linear(784→256→1) with Sigmoid output

3. For each epoch:

Train Discriminator: maximize D(real) and minimize D(fake)
Train Generator: maximize D(G(z)) to fool discriminator

4. Generate sample images from random noise

Important Modules:
torch.nn.Sequential : Sequential model building

torch.optim.Adam : Adam optimizer

torch.nn.BCELoss : Binary cross-entropy loss

torch.randn : Random noise generation

Approach/Logic:
Adversarial training between two networks. Generator learns to create realistic data from noise, while
Discriminator learns to distinguish real from fake data. This creates a minimax game: G tries to minimize
log(1-D(G(z))) while D tries to maximize log(D(x)) + log(1-D(G(z))). The competition drives both networks
to improve, eventually generating high-quality samples.
Key Concepts Summary:
1. Autoencoders: Self-supervised learning through reconstruction

2. VAEs: Probabilistic generation with regularized latent space

3. RBMs: Energy-based unsupervised learning

4. DBNs/DBMs: Deep hierarchical feature learning

5. GANs: Adversarial training for realistic generation

Each approach solves different aspects of representation learning and generation, from compression to
probabilistic modeling to adversarial training.