GENERATIVE AI AND LARGE LANGUAGE MODELS

SCSB4014

UNIT – 4

IMAGE GENERATION

UNIT 4: IMAGE GENERATION

Auto Encoders - Latent Variable Model - Variational Inference - Evidence Lower Bound - VAE architecture
and workflow - VAE application - Introduction to Adversarial Networks - GANs Architecture and Workflow -
Case Studies on: GANs - DCGAN, StyleGAN, Applications - Deepfakes, Art Generation.
AI IMAGE GENERATION

AI image generators utilize trained artificial neural networks to create images from scratch.
These generators have the capacity to create original, realistic visuals based on textual input
provided in natural language. What makes them particularly remarkable is their ability to fuse
styles, concepts, and attributes to fabricate artistic and contextually relevant imagery. This is
made possible through Generative AI, a subset of artificial intelligence focused on content
creation.
AI image generators are trained on an extensive amount of data, which comprises large
datasets of images. Through the training process, the algorithms learn different aspects and
characteristics of the images within the datasets. As a result, they become capable of
generating new images that bear similarities in style and content to those found in the training
data.
There is a wide variety of AI image generators, each with its own unique capabilities. Notable
among these are the neural style transfer technique, which enables the imposition of one
image's style onto another; Generative Adversarial Networks (GANs), which employ a duo of
neural networks to train to produce realistic images that resemble the ones in the training
dataset; and diffusion models, which generate images through a process that simulates the
diffusion of particles, progressively transforming noise into structured images.

The technologies behind AI image generation

AI image generators understand text prompts using a process that translates textual data into a
machine-friendly language — numerical representations or embeddings. This conversion is
initiated by a Natural Language Processing (NLP) model, such as the Contrastive Language-
Image Pre-training (CLIP) model used in diffusion models like DALL-E.
Visit our other posts to learn how prompt engineering works and why the prompt engineer's
role has become so important lately.
This mechanism transforms the input text into high-dimensional vectors that capture the
semantic meaning and context of the text. Each coordinate on the vectors represents a distinct
attribute of the input text.
Consider an example where a user inputs the text prompt "a red apple on a tree" to an image
generator. The NLP model encodes this text into a numerical format that captures the various
elements — "red," "apple," and "tree" — and the relationship between them. This numerical
representation acts as a navigational map for the AI image generator.
During the image creation process, this map is exploited to explore the extensive potentialities
of the final image. It serves as a rulebook that guides the AI on the components to incorporate
into the image and how they should interact. In the given scenario, the generator would create
an image with a red apple and a tree, positioning the apple on the tree, not next to it or
beneath it.
This smart transformation from text to numerical representation, and eventually to images,
enables AI image generators to interpret and visually represent text prompts.

Generative AI model for image synthesis

Artificial intelligence has made great strides in the area of content generation. From translating
straightforward text instructions into images and videos to creating poetic illustrations and
even 3D animation, there is no limit to AI’s capabilities, especially in terms of image synthesis.
And with tools like Midjourney and DALL-E, the process of image synthesis has become simpler
and more efficient than ever before. But what makes these tools so capable? The power of
generative AI! Generative AI models for image synthesis are becoming increasingly important
for both individual content creators and businesses. These models use complex algorithms to
generate new images that are similar to the input data they are trained on. Generative AI
models for image synthesis can quickly create high-quality, realistic images, which is difficult or
impossible to achieve through traditional means. In fields such as art and design, generative AI
models are being used to create stunning new artworks and designs that push the boundaries
of creativity. In medicine, generative AI models for image synthesis are used to generate
synthetic medical images for diagnostic and training purposes, allowing doctors to understand
complex medical conditions better and improve patient outcomes. In addition, generative AI
models for image synthesis are also being used to create more realistic and immersive virtual
environments for entertainment and gaming applications. In fact, the ability to generate high-
quality, realistic images using generative AI models is causing new possibilities for innovation
and creativity to emerge across industries.

Understanding image synthesis and its importance

Generative models are a type of artificial intelligence that can create new images that are
similar to the ones they were trained on. This technique is known as image synthesis, and it is
achieved through the use of deep learning algorithms that learn patterns and features from a
large database of photographs. These models are capable of correcting any missing, blurred or
misleading visual elements in the images, resulting in stunning, realistic and high-quality
images.
Generative AI models can even make low-quality pictures appear to have been taken by an
expert by increasing their clarity and level of detail. Additionally, AI can merge existing portraits
or extract features from any image to create synthetic human faces that look like real people.
The value of generative AI in image synthesis lies in its ability to generate new, original images
that have never been seen before. This has significant implications for various industries,
including creative, product design, marketing, and scientific fields, where it can be used to
create lifelike models of human anatomy and diseases.
The most commonly used generative models in image synthesis include variational
autoencoder (VAE), autoregressive models, and generative adversarial networks (GANs).

Types of generative AI models for image synthesis

Images may be synthesized using a variety of generative AI models, each of which has its own
advantages and disadvantages. Here, we will discuss some of the most popular generative AI
model types used for picture synthesis.
Diffusion Models

Diffusion models are named so because they function inspired by thermodynamic diffusion, the
process by which a drop of food coloring spreads in water to eventually create a uniform color.

Fig: Diffusion Model

Diffusion models use this principle with images, by first taking an image and diffusing it, altering
the pixels until the image gradually becomes TV static. Through this diffusion process, the
diffusion model is learning how to reverse the diffusion process, that is, taking a noisy image
and diffusing it backward to create images. We can think of the forward-diffusion process of
diffusing clear images into static as the training process. Reverse diffusion can be thought of as
the act of generating new images with static noise.
 Fig: Forward/ReverseDiffusion

The key idea is that it is easy for computers to generate TV static, and use the randomness of
that generated static to create new images each time. The randomness of the noise is also why
diffusion models generate different images each time even if the same prompts are used.

AUTO ENCODERS

Auto encoders are unsupervised neural networks used to learn efficient representations of data
by encoding it into a lower-dimensional latent space and then decoding it back to its original
form. This compression-decompression process allows the model to capture essential features
of the data.
Autoencoders are a specialized class of algorithms that can learn efficient representations of
input data with no need for labels. It is a class of artificial neural networks designed
for unsupervised learning. Learning to compress and effectively represent input data without
specific labels is the essential principle of an automatic decoder. This is accomplished using a
two-fold structure that consists of an encoder and a decoder. The encoder transforms the input
data into a reduced-dimensional representation, which is often referred to as “latent space” or
“encoding”. From that representation, a decoder rebuilds the initial input. For the network to
gain meaningful patterns in data, a process of encoding and decoding facilitates the definition
of essential features.

At its core, an autoencoder is a type of neural network designed for unsupervised learning.
Comprising an encoder and a decoder, an autoencoder is tasked with learning a compressed
representation, or encoding, of input data. The encoder maps the input data to a lower-
dimensional representation, and the decoder reconstructs the original data from this
representation. In the context of generative AI, autoencoders take on a creative role by
generating new data samples that share similar characteristics with the training set.

Key Components and Principles:

Encoder: The encoder component of an autoencoder compresses input data into a latent space,
capturing its essential features. The design of the encoder influences the quality and
expressiveness of the generative model.

Latent Space: The latent space represents a compact, lower-dimensional representation of the
input data. This space serves as the foundation for generating new samples during the
generative process.

Decoder: The decoder reconstructs data from the latent space, attempting to reproduce the
input as faithfully as possible. The quality of the decoder is crucial for generating realistic and
high-quality samples.
Applications of Autoencoders in Generative AI:

Image Synthesis:

Autoencoders are extensively used for generating realistic images. By training on a dataset of
images, the autoencoder learns to represent visual features in the latent space, allowing for the
generation of new, visually coherent images.

Anomaly Detection:

Autoencoders excel in anomaly detection tasks. During training, the model learns to reconstruct
normal data, and when presented with anomalous samples, the reconstruction error is typically
higher, making it a valuable tool for detecting outliers.

Data Augmentation:

In the realm of machine learning, autoencoders contribute to data augmentation. By generating

synthetic samples from the learned latent space, the model aids in diversifying training
datasets, improving model generalization.

Style Transfer:

Autoencoders can be employed for style transfer tasks, where the model learns the stylistic
features of one set of images and applies them to another. This enables the creation of unique
and artistic visual compositions.
Architecture:

Fig: Architecture
Consists of two main parts: the encoder, which compresses the input into a latent space, and
the decoder, which reconstructs the input from the latent representation.

Structure:

Encoder:

Reduces input dimensions, compressing the data into a latent space representation. The
encoder learns a mapping from the input to a smaller dimension, allowing data to be
represented more compactly.
Decoder:

Expands the latent representation back to the input's original dimensions, reconstructing the
data as accurately as possible.

Latent Variable Model

Latent variable models introduce hidden variables that capture underlying patterns within the
observed data. This modeling enables us to generate or reconstruct data by sampling from
these hidden variables.

Latent Space:

A lower-dimensional space representing compressed features of the data. In image generation,

latent spaces allow the model to maintain essential visual information for reconstructing or
synthesizing images.

A face image might be represented by latent variables capturing features such as age, gender,
and facial expression. These features are not directly observed but inferred by the model.

Some of the most common hyperparameters that can be tuned when optimizing the
Autoencoder are:

1. The number of layers for the Encoder and Decoder neural networks
2. The number of nodes for each of these layers
3. The loss function to use for the optimization process (e.g., binary cross-entropy or mean
squared error)
4. The size of the latent space (the smaller, the higher the compression, acting, therefore
as a regularization mechanism)

Types of Autoencoders

There are diverse types of autoencoders and analyze the advantages and disadvantages
associated with different variation:
Denoising Autoencoder

Denoising autoencoder works on a partially corrupted input and trains to recover the original
undistorted image. As mentioned above, this method is an effective way to constrain the
network from simply copying the input and thus learn the underlying structure and important
features of the data.

Advantages

This type of autoencoder can extract important features and reduce the noise or the useless
features.

Denoising autoencoders can be used as a form of data augmentation, the restored images can
be used as augmented data thus generating additional training samples.

Disadvantages

Selecting the right type and level of noise to introduce can be challenging and may require
domain knowledge.

Denoising process can result into loss of some information that is needed from the original
input. This loss can impact accuracy of the output.

Sparse Autoencoder

This type of autoencoder typically contains more hidden units than the input but only a few are
allowed to be active at once. This property is called the sparsity of the network. The sparsity of
the network can be controlled by either manually zeroing the required hidden units, tuning the
activation functions or by adding a loss term to the cost function.

Advantages

The sparsity constraint in sparse autoencoders helps in filtering out noise and irrelevant
features during the encoding process.
These autoencoders often learn important and meaningful features due to their emphasis on
sparse activations.

Disadvantages

The choice of hyperparameters play a significant role in the performance of this autoencoder.
Different inputs should result in the activation of different nodes of the network.

The application of sparsity constraint increases computational complexity.

Variational Autoencoder

Variational autoencoder makes strong assumptions about the distribution of latent variables
and uses the Stochastic Gradient Variational Bayes estimator in the training process. It assumes
that the data is generated by a Directed Graphical Model and tries to learn an approximation

to to the conditional property where and 0 are the parameters of the

encoder and the decoder respectively.

Advantages

Variational Autoencoders are used to generate new data points that resemble the original
training data. These samples are learned from the latent space.

Variational Autoencoder is probabilistic framework that is used to learn a compressed

representation of the data that captures its underlying structure and variations, so it is useful in
detecting anomalies and data exploration.

Disadvantages

Variational Autoencoder use approximations to estimate the true distribution of the latent
variables. This approximation introduces some level of error, which can affect the quality of
generated samples.

The generated samples may only cover a limited subset of the true data distribution. This can
result in a lack of diversity in generated samples.
Convolutional Autoencoder

Convolutional autoencoders are a type of autoencoder that use convolutional neural networks
(CNNs) as their building blocks. The encoder consists of multiple layers that take a image or a
grid as input and pass it through different convolution layers thus forming a compressed
representation of the input. The decoder is the mirror image of the encoder it deconvolves the
compressed representation and tries to reconstruct the original image.

Advantages

Convolutional autoencoder can compress high-dimensional image data into a lower-

dimensional data. This improves storage efficiency and transmission of image data.

Convolutional autoencoder can reconstruct missing parts of an image. It can also handle images
with slight variations in object position or orientation.

Disadvantages

These autoencoder are prone to overfitting. Proper regularization techniques should be used to
tackle this issue.

Compression of data can cause data loss which can result in reconstruction of a lower quality
image.

LATENT VARIABLE MODELS (LVMS)

LVM have become an indispensable tool. Essentially, these are statistical models that are used
for unobserved or “latent” variables. The main objective of LVMs is to elucidate relationships
between multiple observable variables by introducing these unobserved, latent variables into
the model

Key properties of Latent Variable Models:

 Predictive Power: Latent Variable Models have predictive potential by incorporating

unobserved variables that can explain variations in the observed variables.
  Dimensionality Reduction: LVMs are particularly useful in scenarios with high-
dimensional data. They work by suppressing irrelevant dimensions and concentrating on
key latent factors, facilitating interpretation and reducing computational complexity.
 Robustness to Noise: Latent Variable Models are inherently robust to noise since they
consider unobserved variables that may explain noise in the observed data.
 Expressive Power: By including latent variables, these models can capture complex, non-
linear relationships among variables, thereby increasing the expressivity of the model.
 Understanding Hidden Processes: LVMs can reveal the unseen processes or entities that
may be contributing to particular pattern in data.

Advantages of Latent Variable Models

Organizations and researchers in numerous domains, including economics, psychology, social
sciences, and machine learning, value Latent Variable Models due to several inherent
advantages:

 Efficiency in Unraveling Complex Data Relationships: LVMs serve as powerful tools when
it comes to unraveling the intricacies hidden in high-dimensional data. They succinctly
simplify complexity and make it interpretable.
 Understanding Unobservable Processes: Latent variable models are effectively used to
introspect and understand hidden layers and processes in data that are not directly
measurable or observed, thereby unraveling previously unseen patterns.
 Robustness: LVMs are robust against outliers, since they leverage unobserved latent
variables that can absorb the effect of these outliers.
 Boosting Predictive Accuracy: By incorporating latent variables, predictive models can
gain a significant boost in accuracy by exploiting the hidden relationships within the
data.
 Highly Versatile: LVMs could be used for a wide range of tasks in machine learning, such
as classification, regression, clustering, dimensionality reduction, among others.
Limitations of Latent Variable Models
Despite these many advantages, users should also be aware of several limitations of Latent
Variable Models:

 Assumptions: Like all models, LVMs make certain assumptions about the distribution of
data, which may not always hold true. Violating these assumptions could lead to
potential inaccuracies.
 Difficulty in Interpretation: While these models are powerful, they can sometimes be
difficult to interpret, particularly when it comes to understanding the role and nature of
the latent variables.
 Complexity: LVMs can be relatively complex to implement, especially when compared to
simpler methods that do not incorporate latent variables. The complexity can increase
dramatically with the number of latent variables and their interactions.
 Overfitting: Like any machine learning models, LVMs are prone to overfitting especially
in the high dimensional settings where the number of parameters could be much larger
than the number of samples.
 Computation and Scalability: For large-scale and high-dimensional datasets, estimating
the parameters of LVMs can be computationally intensive and may pose scalability
issues
VARIATIONAL INFERENCE

Variational inference is a technique to approximate complex probability distributions by

optimizing a simpler distribution. It is particularly useful when dealing with high-dimensional
data, where exact inference is computationally challenging.

Process:

A distribution is assumed over latent variables.

The model learns an approximation to this distribution.

Applications:

Primarily used in Bayesian neural networks and other models requiring probabilistic
representations, making it easier to estimate probabilities without performing exact
calculations.

EVIDENCE LOWER BOUND (ELBO)

ELBO is a lower bound to the log likelihood of observing data, optimized during variational
inference. This function balances two objectives: accurately reconstructing data and
regularizing the latent space for a well-organized distribution.

Role in VAEs:

ELBO ensures the model learns to both compress data efficiently and generate realistic outputs.
By maximizing the ELBO, VAEs learn a distribution close to the actual data distribution, useful in
generating new, realistic data.
VAE ARCHITECTURE AND WORKFLOW:

Variational Autoencoders (VAEs)

Fig: VAE Architecture

ARCHITECTURE AND WORKFLOW

Encoder:
Maps input data to a probability distribution in the latent space, allowing the model to sample
different variations.

Latent Space:
VAEs use a probabilistic latent space, capturing a distribution of possible representations rather
than a single point.
Decoder:

Reconstructs data from samples in the latent space, which allows VAEs to generate realistic
images by decoding random samples.

Image Generation:

By sampling different points in the latent space, VAEs can generate a variety of realistic images
similar to the training data.

Interpolation Between Images:

VAEs can blend images by linearly interpolating between latent vectors, useful in visual
morphing or style transfer.

Data Compression:

Reduces large datasets to smaller, manageable representations while retaining essential

information.

VAE APPLICATION

One of the fundamental models used in generative AI is the Variational Autoencoder or VAE. By
employing an encoder-decoder architecture, VAEs capture the essence of input data by
compressing it into a lower-dimensional latent space. From this latent space, the decoder
generates new samples that resemble the original data.

VAEs have found applications in image generation, text synthesis, and more, allowing machines
to create novel content that captivates and inspires.
 Fig: Adversarial Network

Introduction to Adversarial Networks

Adversarial networks, specifically GANs, consist of two networks: a generator that creates
synthetic data, and a discriminator that evaluates the authenticity of data. They work in a
competitive setup, where the generator tries to fool the discriminator with realistic data
samples.

The adversarial nature of GANs makes the generator progressively better at creating realistic
data, while the discriminator sharpens its ability to detect fakes. This push-pull dynamic drives
both networks to improve, resulting in high-quality generated data.
GENERATIVE ADVERSARIAL NETWORKS (GANS):

Fig: GAN Architecture

A class of models that generate new data similar to the training data by learning from feedback.

A generative adversarial network (GAN) is a type of AI model. The architecture of a GAN

consists of two separate neural networks that are pitted against each other in a game-like
scenario. The first network, known as the generator network, tries to create fake data that
looks real.

The second network, known as the discriminator network, is typically a convolutional neural
network (CNN) that tries to distinguish between data generated by the GAN (fake data) and real
data. The network learns to classify these examples correctly, and this information is used to
adjust the generator network to create more realistic data that is indistinguishable from real
data, as determined by the discriminator network.
The image below visualizes the concept of how GANs work: Generative adversarial nets are
trained until they reach a point when they both cannot improve because the generative
distribution (green) is equal to the data generating distribution (dotted line). And the
discriminator is unable to differentiate between the two distributions (dashed blue line shows
the discriminative distribution).

Fig: The key mechanics of GANs

Choosing the right dataset for the model

Generative AI models rely heavily on the dataset they are trained on to generate high-quality,
diverse images. To achieve this, the dataset should be large enough to represent the richness
and variety of the target picture domain, ensuring that the generative model can learn from a
wide range of examples. For example, if the goal is to create medical images, the dataset should
contain a diverse range of medical photos capturing various illnesses, organs, and imaging
modalities.
In addition to size and diversity, the dataset should also be properly labeled to ensure that the
generative model learns the correct semantic properties of the photos. This means that each
image in the dataset should be accurately labeled, indicating the object or scene depicted in the
picture. Both manual and automated labeling methods can be used for this purpose.
Finally, the quality of the dataset is also important. It should be free of errors, artifacts, and
biases to ensure that the generative model learns accurate and unbiased representations of the
picture domain. For instance, if the dataset has biases towards certain objects or features, the
generative model may learn to replicate these biases in the generated images.
Selecting the right dataset is critical for the success of generative AI models for image synthesis.
A suitable dataset should be large, diverse, properly labeled, and of high quality to ensure that
the generative model can learn accurate and unbiased representations of the target picture
domain.

Preparing data for training

Preparing data for training a generative AI model used for image synthesis involves collecting
the data, preprocessing it, augmenting it, normalizing it, and splitting it into training, validation,
and testing sets. Each step is crucial in ensuring that the model can learn the patterns and
features of the data correctly, leading to more accurate image synthesis.
There are several phases involved in getting data ready for generative AI model training so that
the model can accurately learn the patterns and properties of the data.
Data collection: This is the initial stage in gathering the data needed to train a generative AI
model for picture synthesis. The model’s performance may be significantly impacted by the
type and volume of data gathered. The data may be gathered from a variety of places, including
web databases, stock picture archives, and commissioned photo or video projects.
Data preprocessing: Preprocessing involves a series of operations performed on the raw data to
make it usable and understandable by the model. In the context of image data, preprocessing
typically involves cleaning, resizing, and formatting the images to a standard that the model can
work with.
Data augmentation: It involves making various transformations to the original dataset to
artificially create additional examples for training the model. It can help expand the range of
the data used to train the model. This can be especially important when working with a limited
dataset, as it allows the model to learn from a greater variety of examples, which can improve
its ability to generalize to new, unseen examples. Data augmentation can help prevent
overfitting, a common problem in machine learning. Overfitting occurs when a model becomes
too specialized to the training data, to the point that it performs poorly on new, unseen data.
Data normalization: Data normalization, entails scaling the pixel values to a predetermined
range, often between 0 and 1. Normalization helps to avoid overfitting by ensuring that the
model can learn the patterns and characteristics of the data more quickly.
Dividing the data: Training, validation, and testing sets are created from the data. The validation
set is used to fine-tune the model’s hyperparameters, the testing set is used to assess the
model’s performance, and the training set is used to train the model. Depending on the size of
the dataset, the splitting ratio can change, but a typical split is 70% training, 15% validation, and
15% testing.

Building a generative AI model using GANs (Generative Adversarial Networks)

Creating a generative AI model for image synthesis using GAN entails carefully gathering and
preprocessing the data, defining the architecture of the generator and discriminator networks,
training the GAN model, tracking the training process, and assessing the performance of the
trained model.
Here are the steps discussed in detail:
1. Gather and prepare the data: The data must be cleaned, labeled, and preprocessed to
ensure that it is suitable for the model’s training.
2. Define the architecture of the generator and discriminator networks: The generator network
creates images using a random noise vector as input, while the discriminator network tries
to differentiate between the generated images and the real images from the dataset.
3. Train the GAN model: The generator and discriminator networks are trained concurrently,
with the generator attempting to deceive the discriminator by producing realistic images
and the discriminator attempting to accurately differentiate between the generated and real
images.
4. Monitor the training process: Keep an eye on the produced images and the loss functions of
both networks to ensure that the generator and discriminator networks are settling on a
stable solution. Tweaking the hyperparameters can help to improve the results.
5. Test the trained GAN model: Use a different testing set to evaluate the performance of the
trained GAN model by creating new images and comparing them to the real images in the
testing set. Compute several metrics to evaluate the model’s performance.
6. Fine-tune the model: Adjust the model’s architecture or hyperparameters, or retrain it on
new data to improve its performance.
7. Deploy the model: Once the model has been trained and fine-tuned, it can be used to
generate images for a variety of applications.
Creating a GAN model for image synthesis requires careful attention to data preparation, model
architecture, training, testing, fine-tuning, and deployment to ensure that the model can
generate high-quality and realistic images.

Fig: GAN model

Generating new images with the model

As discussed earlier, a GAN model consists of two networks: the generator and the
discriminator. The generator network takes a random noise vector as input and generates an
image that is intended to look like a real image. The discriminator network’s task is to
determine whether an image is real or fake, i.e., generated by the generator network.
During training, the generator network produces fake images, and the discriminator network
tries to distinguish between the real and fake images. The generator network learns to produce
better fake images by adjusting its parameters to fool the discriminator network. This process
continues until the generator network produces images that are indistinguishable from real
images.
Once the GAN model is trained, new images can be generated by providing a random noise
vector to the generator network. By adjusting the noise input, interpolating between two
images, or applying style transfer, the generator network can be fine-tuned to produce images
in a particular style.
However, it’s important to note that the GAN model’s capacity to produce high-quality images
may be limited. Therefore, it’s crucial to assess the produced images’ quality using various
metrics, such as visual inspection or automated evaluation metrics. If the quality of the
generated images is not satisfactory, the GAN model can be adjusted, or more training data can
be provided to improve the outcomes.
To ensure that the produced images look realistic and of excellent quality, post-processing
methods like picture filtering, color correction, or contrast adjustment can be used. The images
generated using the GAN model can be used for various applications, such as art, fashion,
design, and entertainment.

Applications of generative AI models for image synthesis

There are several uses for generative AI models, especially GANs, in picture synthesis. The
following are some of the main applications of generative AI models for picture synthesis:
Art and design: New works of art and design, such as paintings, sculptures, and even furniture,
may be produced using generative AI models. For instance, artists can create new patterns,
textures, or colour schemes for their artwork using GANs.
Gaming: Realistic gaming assets, such as people, locations, or items, can be created using GANs.
This can improve the aesthetic appeal of games and provide gamers with a more engaging
experience.
Fashion: Custom clothing, accessory, or shoe designs can be created with generative AI models
for image synthesis. For apparel designers and retailers, this may open up fresh creative
opportunities.
Animation and film: GANs may be used to create animation, visual effects, or even whole
scenes for movies and cartoons. By doing this, developing high-quality visual material may be
done faster and cheaper.
X-rays, MRIs, and CT scans are just a few examples of the kinds of medical pictures that may be
produced with GANs. This can help with medical research, treatment planning, and diagnosis.
GANs may also be used in photography to create high-quality photos from low-resolution ones.
This can improve the quality of pictures shot using cheap cameras or mobile devices.

THE DIFFERENCE BETWEEN CNN VS. GAN

Both Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs) are
deep learning architectures. GANs are generative models that can generate new examples from
a given training set, while convolutional neural networks (CNN) are primarily used for
classification and recognition tasks.

While a single CNN can also be used as a generative model if that set it up to be a Variational
Autoencoder (VAE), CNNs are powerful tools for discriminative learning and are particularly
suitable for classifying images in computer vision.

Discriminative Models vs. Generative Models

The discriminative model is a machine learning algorithm used to distinguish between different
categories of data, for example, for image classification and object detection. A generative
modeling algorithm, on the other hand, is used to generate new data that is similar to the data
that was used to train the model.
One of the key differences between generative and discriminative models is that a generative
model can generate new examples, while a discriminative model can classify data. Another
difference is that a generative model is typically more complex than a discriminative model.

This is because a generative model needs to learn the underlying probability distribution of the
data, while a discriminative model only needs to learn the mapping between inputs and
outputs.

GANS ARCHITECTURE AND WORKFLOW

Components:
Generator:
Creates synthetic data by sampling from the latent space, attempting to resemble the real data.

Discriminator:

Acts as a classifier to distinguish between real and synthetic data.

Workflow:
The generator produces a synthetic image based on latent space samples.
The discriminator evaluates whether the image is real or synthetic.

Feedback from the discriminator is used to update both networks. The generator learns to
produce more convincing images, while the discriminator becomes better at identifying fakes.

Loss Function:

The generator aims to maximize the probability of the discriminator incorrectly classifying fake
images as real, while the discriminator tries to maximize its correct classifications.
TYPES OF GANS

Vanilla GAN:
This is the simplest type of GAN. Here, the Generator and the Discriminator are simple a
basic multi-layer perceptrons. In vanilla GAN, the algorithm is really simple, it tries to optimize
the mathematical equation using stochastic gradient descent.
Conditional GAN (CGAN): CGAN can be described as a deep learning method in which some
conditional parameters are put into place.
1. In CGAN, an additional parameter ‘y’ is added to the Generator for generating the
corresponding data.
2. Labels are also put into the input to the Discriminator in order for the Discriminator to
help distinguish the real data from the fake generated data.

Deep Convolutional GAN (DCGAN): DCGAN is one of the most popular and also the most
successful implementations of GAN. It is composed of ConvNets in place of multi-layer
perceptrons.
1. The ConvNets are implemented without max pooling, which is in fact replaced by
convolutional stride.
2. Also, the layers are not fully connected.

Laplacian Pyramid GAN (LAPGAN): The Laplacian pyramid is a linear invertible image
representation consisting of a set of band-pass images, spaced an octave apart, plus a low-
frequency residual.
1. This approach uses multiple numbers of Generator and Discriminator networks and
different levels of the Laplacian Pyramid.
2. This approach is mainly used because it produces very high-quality images. The image is
down-sampled at first at each layer of the pyramid and then it is again up-scaled at each
layer in a backward pass where the image acquires some noise from the Conditional
GAN at these layers until it reaches its original size.
Super Resolution GAN (SRGAN):
SRGAN as the name suggests is a way of designing a GAN in which a deep neural network is
used along with an adversarial network in order to produce higher-resolution images. This type
of GAN is particularly useful in optimally up-scaling native low-resolution images to enhance
their details minimizing errors while doing so.

Architecture of GANs
A Generative Adversarial Network (GAN) is composed of two primary parts, which are the
Generator and the Discriminator.

Generator Model
A key element responsible for creating fresh, accurate data in a Generative Adversarial Network
(GAN) is the generator model. The generator takes random noise as input and converts it into
complex data samples, such text or images. It is commonly depicted as a deep neural network.
The training data’s underlying distribution is captured by layers of learnable parameters in its
design through training. The generator adjusts its output to produce samples that closely mimic
real data as it is being trained by using back propagation to fine-tune its parameters.
The generator’s ability to generate high-quality, varied samples that can fool the discriminator
is what makes it successful.

Generator Loss
The objective of the generator in a GAN is to produce synthetic samples that are realistic
enough to fool the discriminator. The generator achieves this by minimizing its loss
function JGJG. The loss is minimized when the log probability is maximized, i.e., when the
discriminator is highly likely to classify the generated samples as real. The following equation is
given below:

JG=−1mΣi=1mlogD(G(zi))JG=−m1Σi=1mlogD(G(zi))
Where,
1. JGJG measure how well the generator is fooling the discriminator.
2. log D(G(zi))D(G(zi))represents log probability of the discriminator being correct for
generated samples.
3. The generator aims to minimize this loss, encouraging the production of samples that
the discriminator classifies as real (logD(G(zi))(logD(G(zi)), close to 1.

Discriminator Model

An artificial neural network called a discriminator model is used in Generative Adversarial

Networks (GANs) to differentiate between generated and actual input. By evaluating input
samples and allocating probability of authenticity, the discriminator functions as a binary
classifier.
Over time, the discriminator learns to differentiate between genuine data from the dataset and
artificial samples created by the generator. This allows it to progressively hone its parameters
and increase its level of proficiency.

Convolutional layers or pertinent structures for other modalities are usually used in its
architecture when dealing with picture data. Maximizing the discriminator’s capacity to
accurately identify generated samples as fraudulent and real samples as authentic is the aim of
the adversarial training procedure. The discriminator grows increasingly discriminating as a
result of the generator and discriminator’s interaction, which helps the GAN produce extremely
realistic-looking synthetic data overall.

Discriminator Loss

The discriminator reduces the negative log likelihood of correctly classifying both produced and
real samples. This loss incentivizes the discriminator to accurately categorize generated
samples as fake and real samples with the following equation:

JD=−1mΣi=1mlogD(xi)–1mΣi=1mlog(1–D(G(zi))JD=−m1Σi=1mlogD(xi)–m1Σi=1mlog(1–D(G(zi))
 1. JDJD assesses the discriminator’s ability to discern between produced and actual
samples.
2. The log likelihood that the discriminator will accurately categorize real data is
represented by logD(xi)logD(xi).
3. The log chance that the discriminator would correctly categorize generated samples as
fake is represented by log (1−D(G(zi)))log (1−D(G(zi))).
4. The discriminator aims to reduce this loss by accurately identifying artificial and real
samples.

MinMax Loss

In a Generative Adversarial Network (GAN), the minimax loss formula is provided by:
minGmaxD(G,D)=[Ex∼pdata[logD(x)]+Ez∼pz(z)[log(1–D(g(z)))]minGmaxD(G,D)=[Ex∼pdata
[logD(x)]+Ez∼pz(z)[log(1–D(g(z)))]

Where,
1. G is generator network and is D is the discriminator network
2. Actual data samples obtained from the true data distribution pdata(x)pdata(x) are
represented by x.
3. Random noise sampled from a previous distribution pz(z)pz(z)(usually a normal or
uniform distribution) is represented by z.
4. D(x) represents the discriminator’s likelihood of correctly identifying actual data as real.
5. D(G(z)) is the likelihood that the discriminator will identify generated data coming from
the generator as authentic.
 Fig: GAN Structure

Generative Adversarial Network Frameworks

Several frameworks provide tools and libraries for implementing and training GANs, including:

TensorFlow:

TensorFlow is an open-source machine learning framework developed by Google. It provides

various tools and libraries for implementing and training GANs, including the tf.keras.layers. We
can use the GAN layer to build a GAN model in just a few lines of code.

PyTorch:

PyTorch is an open-source machine learning framework developed by Facebook. It provides

tools and libraries for implementing and training GANs, including the torch.nn.Module class,
which we can use to build custom GAN models.

Keras:

Keras is an open-source deep learning library that provides a high-level API for building and
training deep learning models. It includes a GAN class that can quickly build and train GANs.
Chainer:

Chainer is an open-source deep-learning framework developed by Preferred Networks. It

provides tools and libraries for implementing and training GANs, including the
chainer.links.model.Generator and chainer.links.model.Discriminator classes can be used to
GANLab:

build custom GAN models.

GANLab is a web-based tool that allows users to experiment with GANs in a visual, interactive
environment. It provides a simple, drag-and-drop interface for building and training GANs
without the need to write any code.

DCGAN (Deep Convolutional GAN):

1. Uses convolutional layers for enhanced stability and better feature extraction.
2. Widely used in image generation, especially for producing high-quality, realistic images.

StyleGAN:

1. A GAN architecture that allows control over image features (e.g., style, details).
2. Applications include facial image generation, providing photorealistic images with
customizable styles.

APPLICATIONS OF GENERATIVE ADVERSARIAL NETWORKS

GANs can be used for a variety of AI tasks, such as machine learning-based image generation,
video generation, and text generation (for example, in natural language processing, NLP). The
major benefit of generative adversarial networks is that they can be used to create new data
instances where data collection is difficult or impossible.

Hence, GANs have been successfully applied in various practical applications in image synthesis
and computer vision.
GENERATING IMAGES FROM SCRATCH

Image generation is the process of creating new images from scratch. This is often done by first
training a GAN to learn the distribution of a dataset, and then generating new images from
random noise vectors. GANs can be applied to generate realistic images of people, animals, and
other objects. This can be used for things like creating realistic-looking advertising visuals or
adding new content to video games.

In Healthcare, GANs have been shown to be very effective in generating images for medical
image analysis. In particular, GANs have been used to create realistic images of organs for
surgical planning or simulation training. For example, generated samples of tumors can be used
for diagnosis and treatment planning.

Fig: Application of GAN in medical imaging

Generating 3D from 2D

Another application is to use GANs to create 3D images from 2D ones. This can be used to
create more realistic-looking 3D models or add new depth and realism to existing images.
Create art with AI

GANs have been used to generate art that replicates the styles of famous artists. In one study, a
generative adversarial network was trained to generate portraits in the style of Rembrandt
(style transfer). The portraits generated by the GAN were indistinguishable from genuine
Rembrandt portraits.

Check out our article about other generative models, such as the popular DALLE-2, which uses a
version of GTP-3 to generate ultra-realistic images from text.

Fig: GAN examples of Monet-style visualizations

Face generation

GANs have also been used to generate realistic-looking images of faces, so-called deepfakes. In
a research project, a GAN was trained on a dataset of celebrity faces and was able to generate
new, realistic-looking faces that resembled the celebrities in the training dataset.

Medical image processing

Generative Adversarial Networks (GANs) are widely used in medical image processing for data
augmentation due to their excellent image-generation capabilities. Using GANs for image
augmentation in existing medical image datasets can significantly increase the sample size of
training sets for AI medical image diagnosis and treatment models.

To a certain extent, it alleviates the limited sample size of medical images due to inherent
limitations such as imaging cost, labeling cost, and patient privacy. Read more about computer
vision in healthcare.

Improve image quality

Other applications for GANs include image super-resolution, where a low-resolution image is
upscaled to a higher resolution. A generative adversarial network can be used to remove
artifacts from images or to improve the resolution of images.

Additionally, GANs can be used to colorize black-and-white images or to add new details to an
image.

Social media fake bots

GAN has also been used to create fake news articles and reviews and to generate text
conversations that seem realistic. Using a GAN, a bot can be trained to generate data such as
realistic tweets that are more likely to fool other users into thinking they are real.

This could be used for several purposes, such as creating fake accounts that are used to spread
disinformation or promote a certain agenda. GANs could also be used to create believable
automated replies to tweets, which is used for automated customer service on Twitter or
Facebook.

Different Generative Adversarial Network variants

Conditional GAN (CGAN)

Recently, conditional GANs (cGAN) have received significant attention in the field of image
generation and text-to-image synthesis. A conditional generative adversarial network (CGAN) is
a supervised learning technique that involves using both labeled and unlabeled data to train a
generative adversarial network. The aim is to improve the accuracy of predictions by the model.

The ability of conditional GANs to learn from both annotated and un-annotated data is
beneficial because it can reduce the amount of labeled data required to train the model. In
addition, Conditional GANs can also handle data that is not linearly separable.

There are some drawbacks of cGAN as well. One limitation is that the model can only generate
examples that are similar to the training data. This means that the model is not able to
generalize to new data.

In addition, cGAN can be sensitive to changes in the training data. This can lead to model
overfitting and poor performance on test data.

Adversarial Autoencoder (AAE)

An adversarial autoencoder is an autoencoder that uses an adversarial network to regularize

the latent space of the autoencoder. The adversarial network is used to encourage the latent
space to have desired properties, such as being Gaussian or having a uniform distribution.

The autoencoder part of the network is trained to reconstruct the input, while the adversarial
network is trained to distinguish between the latent code produced by the autoencoder and a
sample from the desired distribution.
This setup can be thought of as a game between the autoencoder and adversarial network,
where the autoencoder is trying to fool the adversarial network by producing latent codes that
match the desired distribution, and the adversarial network is trying to learn to distinguish
between the codes produced by the autoencoder and the samples from the desired
distribution.

Dual GAN (DGAN)

A variant of GAN where two networks are trained in parallel with two sets of unlabeled images
as input, one network for generating images and the other for discriminating between
generated images and real images.

DualGAN simultaneously learns two reliable image translators from one domain to the other
and hence can be used for a broad range of image-to-image translation tasks.

Stack GAN (StackGAN)

A variation of GAN where multiple generators are stacked together to produce a more realistic
image. Stacked GANs form a network capable of generating high-resolution images.

Cycle GAN (CycleGAN)

A CycleGAN is a technique to translate from one image domain to another for automatic image-
to-image translation models, without requiring paired data samples.

Superresolution GAN (SRGAN)

A GAN that can generate high-resolution images from low-resolution inputs. Super-resolution
GANs apply a deep network in combination with an adversary network to increase the
resolution of input data.

Deep convolutional GAN (DCGAN)

A GAN that uses deep convolutional neural networks in the generator and discriminator. The
GAN consists entirely of convolution-deconvolution layers (Fully convolutional networks).
Research indicates that images generated using the DCGAN model architecture were
significantly better (less noisy).

Wasserstein GAN (WGAN)

A GAN that minimizes the Wasserstein-1 distance between the real and generated
distributions. The Wasserstein distance is a metric for the distance between two probability
distributions.

Energy-based GAN (EBGAN)

A GAN uses an energy function to measure the similarity between real and generated images.
The energy function is used to define a loss function that is minimized during training.

Mode regularized GAN (MRGAN)

A GAN variation that uses a mode regularizer to encourage the generator to generate images
from all modes of the data distribution. The mode regularizer is a penalty function that
encourages the generator to generate images that are close to the modes of the data
distribution.

Fig: The CycleGAN architecture.

APPLICATIONS OF GANS

Deepfakes:

GANs can generate realistic videos where faces are altered to appear like different people.
Deepfakes involve altering faces in video footage using GAN-generated images, creating
realistic but altered videos.

A deepfake refers to manipulated media, particularly videos, images, or audio, generated using
deep learning algorithms, typically Generative Adversarial Networks (GANs). The term
"deepfake" is derived from "deep learning" and "fake," which points to the use of AI to create
hyper-realistic, yet fabricated, representations of people, often in videos where they say or do
things they never actually did.

Deepfakes use GANs in a specific way: one neural network (the generator) creates fake images
or videos, while another (the discriminator) evaluates their authenticity. Over time, the
generator improves based on feedback from the discriminator, resulting in increasingly
convincing content.

How Deepfakes Work:

Training Phase:

GANs are trained on large datasets of facial images and videos of a target person. These
datasets include various angles, lighting conditions, facial expressions, and more, which allow
the model to capture the subtle details of a person's face.

Generator vs. Discriminator:

Generator:

The generator creates synthetic videos or images. It can alter an existing video, replace faces, or
generate entirely new ones.
Discriminator:

The discriminator's job is to determine whether the generated video or image is real or fake.
Over time, through this feedback loop, the generator becomes more adept at producing
realistic fake content.

Result:

The resulting deepfake video or image looks remarkably realistic, with the face convincingly
swapped, expressions mimicked, or speech synced, making it challenging for humans to
distinguish between real and fake content.

Applications of Deepfakes

Entertainment and Film:

De-Aging and Resurrection of Actors:

In the film industry, deepfakes are used to create realistic CGI effects. For example, actors can
be digitally "de-aged" to portray their younger selves, or deceased actors can be digitally
resurrected, as seen in movies like Star Wars: Rogue One (where Peter Cushing’s character was
brought back) and The Irishman (where Robert De Niro’s character was de-aged).

Misinformation and Fake News:

Deepfakes are increasingly used to spread disinformation. Political figures, journalists, and
celebrities can be shown saying or doing things they never actually did. These fake videos can
be used to manipulate public opinion or tarnish reputations. In 2018, deepfake videos of
politicians and world leaders spread across social media, prompting concerns over security and
trust in the media.
Personalization in Marketing:

Companies can use deepfakes to create personalized content for advertising. For example, an
ad might show an influencer or celebrity endorsing a product in a personalized video message
that addresses the viewer directly, enhancing engagement.

Virtual Conferencing and Social Media:

In virtual meetings, deepfake technology can be used to create avatars or alter people's
appearances in real-time. This is especially relevant in platforms like Zoom, where people might
use deepfake filters to change their facial appearance or replace themselves with avatars
during video calls.

Cybersecurity:

Voice Synthesis: Deepfake technology can also manipulate audio, not just video. For example,
AI can mimic a person's voice to commit fraud or identity theft, a phenomenon known as voice
deepfakes. Cybercriminals can impersonate CEOs or other authority figures to issue fraudulent
commands.

Virtual Reality (VR) and Gaming:

Character Modeling: Deepfake technology can be applied in VR or gaming to create hyper-

realistic avatars or to insert a player's face into a game environment.

Ethical and Legal Concerns

1. Deepfakes raise serious ethical and legal issues, especially concerning privacy, consent,
and authenticity
2. Consent: Creating deepfakes without the consent of the people depicted can violate
their personal privacy and intellectual property rights.
3. Misinformation: Deepfakes can be used to deceive viewers into believing fabricated
events or statements, leading to the spread of false information.
 4. Reputation Damage: Deepfakes can be weaponized to create fake scandals, ruining an
individual’s career or reputation.
5. To combat these issues, researchers are working on deepfake detection tools, but the
arms race between creating deepfakes and detecting them continues to evolve.

Art Generation:

1. GANs have become popular in art for generating unique, creative artworks. By training
on various art styles, GANs can produce novel images that blend artistic styles or mimic
traditional art, often used in digital art, advertising, and entertainment.
2. Art generation with GANs refers to the process of using AI to create new pieces of
artwork, either by learning from the styles of existing artists or creating entirely novel
artistic expressions. In this context, GANs are employed to learn the intricacies of
different art styles—such as painting, sculpture, or digital art—and then generate new
pieces that emulate or combine these styles.
3. The most famous instance of AI-generated art is "Edmond de Belamy", a portrait
created by the Paris-based collective Obvious using a GAN. This piece was sold at
auction for over $432,000, sparking widespread discussion about the role of AI in the art
world.

Applications of GANs in Art Generation

Training GANs on Artworks:

The GAN is trained on a large dataset of artworks from specific genres, artists, or time periods.
The dataset can include anything from classical paintings to modern art, allowing the generator
to learn patterns, brushstrokes, compositions, and color schemes.
Digital Art Creation:

1. GANs are used to generate completely new and original pieces of art, often blending or
remixing different artistic traditions or genres. These AI-generated artworks can range
from abstract expressionism to photorealistic portraits.
2. For example, the Artbreeder platform allows users to combine and manipulate portraits,
landscapes, and abstract art in real-time. By adjusting "genes," users can create unique
art pieces that blend features from various sources.

Art Restoration and Reconstruction:

1. GANs can be used to digitally restore old paintings or artwork that have been damaged
over time. By analyzing patterns in the remaining portions of a piece, GANs can
reconstruct missing sections in a way that looks authentic and coherent.
2. Additionally, GANs can be used to create versions of works in styles that have been lost
to history. For example, scholars could generate new paintings in the style of artists who
didn't leave behind many surviving works.

Generator vs. Discriminator:

1. Just like in deepfakes, the GAN's generator creates images, while the discriminator
evaluates whether the generated image fits the intended style. This feedback loop
refines the generator, improving its ability to create realistic and aesthetically appealing
artworks.

Creative Control:

1. Artists can guide the process by selecting the source material or setting parameters for
the GAN, such as the style, color palette, or composition. This allows artists to
collaborate with the AI in new and exciting ways, often combining human creativity with
machine learning's ability to explore vast possibilities quickly.
Style Transfer:

1. Style transfer is one of the most popular applications of GANs in art. It involves taking
the style of one artwork and applying it to another image or photograph, creating a
hybrid piece. For instance, a photo can be transformed into a painting resembling Van
Gogh’s Starry Night or Picasso’s Cubism style.
2. Many online platforms, such as DeepArt or Prisma, allow users to upload their own
images and apply famous artistic styles to them using AI.

Advertising and Branding:

Companies use GANs to create unique artwork for marketing campaigns, where they can
generate custom graphics that are aligned with a brand’s aesthetic. AI-generated art can also
be used to design logos, banners, and other marketing materials, allowing for rapid prototyping
and design iteration.

Fashion and Design:

GANs are being used in the fashion industry to generate new clothing designs. By training on
datasets of past collections, GANs can create novel pieces that blend traditional designs with
futuristic concepts. Some fashion brands even use GANs to produce virtual fashion shows.

Generative Music and Album Covers:

1. GANs are not limited to visual art. They can also be used to generate music that fits
specific genres or styles. Artists in the music industry are experimenting with AI-
generated album covers and promotional artwork.
2. Similarly, GANs can be used to create generative visuals for music videos or
accompanying graphic art, adding an extra layer of creativity to the project.
Personalized Art:

GANs can also be used to create personalized art for individuals. By inputting certain
preferences or style choices, users can generate artworks that are tailored to their tastes,
whether they prefer minimalist design or vibrant, abstract works.
Challenges and Criticisms
1. While AI-generated art is increasingly popular, it has faced criticism, especially regarding
its originality and the role of the artist. The questions of authorship, authenticity, and
copyright are central issues, as AI often learns from pre-existing works, raising concerns
about intellectual property rights.
2. Moreover, some critics argue that art generated by AI lacks the emotional depth,
cultural context, or personal experience that human artists bring to their work. Others,
however, view the collaboration between humans and machines as a new frontier in
creativity, broadening the potential for art in unexpected ways.