AI WORKPLACE FOUNDATIONS

INTRODUCTION TO

Generative AI
DAY 2, MODULE 1
Agenda

01 First Insights in Generative AI

02 The Surge of Generative AI

03 Historical Evolution of Generative AI
04 What is a Generative Adversarial Network?
05 GAN Applications
06 Other GenAI Applications
07 Known Risks of Generative AI
First Insights

Generative AI is an AI paradigm that allows models to create new

output such as text, images, videos, audio, or other types of data.

These systems utilise generative deep learning architecture such as

variational encoders, generative adversarial networks (GAN) and
autoregressive models like transformer architectures.

Top Generative AI use cases:

• Natural language generation
• Image generation and translation
• Video generation
• Audio generation
The Surge of Generative AI

Generative AI has been around for a while and was actively

being utilised by the research and developer community. Early
models and applications required any of the following:

• Understanding specialized tools (e.g., PyTorch, HuggingFace

transformers, GitHub Copilot, etc.)
• Writing applications via Python or other languages.
• Submitting data through APIs (e.g., Vertex AI, OpenAI API)
• Using a playground for conversations (e.g., OpenAI
playground).

The arrival of OpenAI’s ChatGPT in late 2022 changed the

landscape as generative AI in the form of conversational
chatbots gained wide adoption.
Historical Evolution of Generative AI

Generative deep learning is the bedrock of Generative AI. It’s a

subfield of machine learning that focuses on creating models that
can generate new data similar to a given dataset.

1 2

Deep Learning Renaissance

Early Foundations
• Deep Belief Networks (2006):
• Perceptrons (1950s-60s): An early
Generative models comprising
type of artificial neural network,
multiple layers of stochastic, latent
introduced by Rosenblatt.
variables. Introduced: Hinton et al.
• Backpropagation (1980s): The
• Autoencoders: Neural networks
rediscovery of backpropagation
trained to reconstruct their input,
algorithm by Rumelhart, Hinton,
by learning a compressed
and Williams allowed for more
representation (encoding) of the
efficient training of multi-layer
data. Variational autoencoders
neural networks.
introduced in 2013.
Historical Evolution of Generative AI

Generative deep learning is the bedrock of Generative AI. It’s a

subfield of machine learning that focuses on creating models that
can generate new data similar to a given dataset.

3 4

Generative Adversarial Network Attention Mechanisms

• Ian Goodfellow et al (2014): GANs • Vaswani et al. (2017):Transformer
consist of 2 networks, a generator architecture was introduced, to
and a discriminator, simultaneously replace RNNs with self-attention
trained through an adversarial mechanisms, enabling parallel
process. One creates data, and the processing of data and capturing
other evaluates it against real data long-range dependencies.
• Further Work: Conditional GANs • Transformers: Backbone of
for controlled data generation. modern language models, able to
CycleGANs and StyleGANs for handle large-scale data and model
high-quality image gen. complex language patterns.
Historical Evolution of Generative AI

Modern large language models have become the most common

examples of Generative AI, given their excellent capability at
human-like natural language interactions.

5 6

Pretrained Language Models Finetuning & Transfer Learning

• BERT (2018): Bidirectional Encoder • Finetuning pretrained models on
Representations from Transformers, specific tasks became a standard
by Google, pre-trained on large approach, enabling the transfer of
corpus using a MLM objective. knowledge from large, generic
models to specialised applications.
• OpenAI (2018): Introduced GPT
(Generative Pretrained Transformer) • Transfer learning allowed models
which took a generative approach, like BERT and GPT to excel across
pretraining on a vast corpus to a wide range of tasks, from
generate coherent & contextually translation and summarization to
relevant language. question answering and dialogue.
What is a GAN?

A generative adversarial network (GAN) is a model where 2

networks (a generator + a discriminator) are trained in opposition
to each other.

• Generator: It’s goal is to produce data that is hard to

distinguish from real data. It takes random noise as input and
transforms it into synthetic data (e.g., images, audio, text). The
generator network learns to generate data that mimics the
real data distribution as closely as possible.

• Discriminator: It’s goal is to distinguish between real data

and the generated data. It takes an input (either real or
generated data) and returns a probability indicating whether
the input is real or fake. The discriminator network learns to
become better at identifying real versus fake data.
What is a GAN?

GANs undergo an adversarial training process:

• The generator and discriminator are trained simultaneously in a zero-sum game. The generator tries to fool the
discriminator by producing more realistic data, while the discriminator tries to become better at detecting fake data.

• The training process involves alternating updates to

the generator and discriminator networks. The
generator is optimized to maximize the probability
of the discriminator incorrectly labeling generated
data as real, while the discriminator is optimized to
correctly label real versus fake data
GAN Applications

• Image synthesis – creating realistic images

• Image completion – generating missing parts of an image,
background fill, etc.
• Text-to-image generation
• Image-to-image translation – e.g., converting sketches to photos
• Data augmentation and color restoration
• Super-resolution – enhancing image resolution
• 3D shape construction and generation
• Neural style transfer – video generation using reference video
• Deepfakes – manipulating videos to swap faces
Other GenAI Applications

• Text generation and summarisation: Content creation,

content aggregation, language translation, virtual
assistants and chatbots, automatic report generation.
• Audio generation: GAN-based text-to-speech, music
generation, speech-to-speech conversion
• Code generation and assistance: Generating entire
functions and code snippets, code completion and
suggestions, interactive coding copilots, automated testing.
• Generative search: Through retrieval augmented
generation over documents, databases, search engines,
etc., generative models can be augmented with additional
data to streamline their results and improve their
performance.
Known Risks of Generative AI

• Lack of transparency: Can be unpredictable in their responses,

and people using such applications may not understand how
they operate. This lack of transparency makes generative AI
applications a bit troublesome to work with.

• Biased responses: When using generative AI applications,

especially businesses, it’s important to have controls or policies
in place to detect biased responses to ensure their appropriate
usage.

• Inaccuracies: Generative AI models are not necessarily

designed to always produce factual responses. They often
produce fabricated and inaccurate responses. Hence, one must
assess responses for appropriateness, usefulness, and accuracy
before relying on them or distributing the information.
Known Risks of Generative AI

• Privacy concerns: Generative models have been found to

inadvertently memorize and recreate content around private
or sensitive information from their training data. This is a major
concern for enterprises implementing generative AI
applications, as it can compromise individual privacy.

• Intellectual property: Generative AI models can also be

trained on proprietary or copyrighted data. If so, there could
be ethical and legal concerns regarding the ownership and
usage of the generated content.
Reading

• Aritra Ghosh (Linkedin): The Evolution of Generative AI

• Dattatray G. Takale (ResearchGate): Advancements and
Applications of Generative Artificial Intelligence
• GeekForGeek: Generative Adversarial Network (GAN)
• Leeway Hertz: Generative adversarial networks (GANs)
– A deep dive into the architecture and training process
• Google Machine Learning: Overview of GAN Structure
• Pixelplex: Generative AI Market Map: From History and
State to Trends and Applications
 AI WORKPLACE FOUNDATIONS

INTRODUCTION TO

Generative AI
DAY 2, MODULE 1