bhumareddy@deloitte.

com
[email protected] Session 1

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 In this session, we will discuss:
• Introduction to Generative AI
• Overview of the LLM Ecosystem
[email protected] Agenda • Understanding LLMs and their Cost Structures
[email protected]
• A Workflow for Enterprise LLM Applications
• Prompt Engineering Fundamentals
• Hands-on Implementation of Prompt Engineering
Techniques

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Introduction to Generative AI & the LLM
[email protected]

Ecosystem
[email protected]

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Generative AI - An Introduction
Design and implementation of computer programs that can reason,
Artificial Intelligence learn and act in complex environments.

Superset of

Design of algorithms that enable computer systems to

Machine Learning efficiently learn from data.

[email protected] Superset of
[email protected]
A subset of ML that uses neural network algorithms for
Deep Learning predictive applications.

Superset of

Algorithms that generate original content, such as images,

Generative AI text, voice, or music, by learning patterns from a dataset.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Generative AI - An Introduction

Generative AI

Large Language Large Multimodal

Models (LLMs) Models (LMMs)
[email protected]
[email protected]
Trained using language Trained using a mixture of
modeling, that is, predicting images, video and text.
the next word in a sentence. LMMs demonstrate nuanced
LLMs have excellent understanding of multimodal
reasoning capabilities and inputs and can be used for
can be used for several several mixed-input tasks
natural language tasks (e.g., (e.g., video summarization)
summarization).

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 LLM Applications

Applications

Classification
Question-Answering
Promp
[email protected] t LLM Summarization
[email protected]
Code Generation
Generative AI Creatives Generation
Resource

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 LLM Applications - Success Stories
Generative AI has enabled document Q&A at scale

Question

[email protected]
Document
[email protected] Dennis Walsh, Goldman Sachs Asset Management
Store

Answer

Generative AI is reducing the human effort

required in synthesizing information from
documents

Ernst & Young

This file is meant for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 LLM Applications - Shortcomings
● Vulnerability to injection attacks
● Data privacy concerns
● Intellectual property concerns on LLM outputs
● Lack of explainability

[email protected]
[email protected]

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Accessing Large Language Models (LLMs)
LLMs (both paid and open source) can be accessed either through public cloud providers or LLM
vendors.

Google: Gemini
Open AI: GPT-3.5, GPT-4
Paid Anthropic: Claude
Cohere: Command

[email protected]
[email protected]
LLMs

Open Meta AI: Llama

Source Mistral AI: Mistral, Mixtral

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Accessing Large Language Models (LLMs)
Open source/open weight LLMs can also be accessed using self-hosted company servers.

Efficient
Hugging Face
implementations of
Access LLama Cpp
open models for
Unsloth
specific target devices.
[email protected]
[email protected]
LLMs
Open Source
Open Weight
Efficient inference
vLLM abstractions
Servers Ollama compatible with Open
AI APIs

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Comparing LLM Providers - Features

Feature Azure AWS GCP

Exclusive model access GPT Claude Gemini Pro

(Open AI) (Anthropic) (Google)

Application ecosystem integration (e.g., High Medium Low

LangChain,
[email protected])
[email protected]

Stability & reliability of APIs High Medium High

Customizability for production systems High Medium High

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Comparing LLM Providers - Cost Structures

Cost comparison per 1000 tokens

Model Type Azure AWS GCP

Input Output Input Output Input Output

Value-for-money,
production grade (e.g.,
[email protected]
[email protected] $0.0005 $0.0015 $0.0008 $0.0024 $0.000125 $0.000375
GPT3.5, Claude Instant,
Gemini 1.5 Flash)

Advanced Reasoning
Models (e.g., GPT4, $0.01 $0.03 $0.015 $0.075 $0.00125 $0.00375
Claude, Gemini 1.5 Pro)

Fine-tuned LLM variants $0.0005 $0.0015

Note* - The cost of OpenAI services is based on the number of tokens.

This file is meant for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Costing for LLM Projects
Request Response

Prompt Tokens Completion Tokens

× Cost/1k Tokens × Cost/1k Tokens

[email protected]
[email protected]
Total Token
Cost
∕ Total Requests

Avg Cost/Prediction
Monthly
Deployment Cost
# Daily Predictions ×

30 Days
This file is meant for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 LLMs for Enterprise Applications
[email protected]
[email protected]

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 What are Large Language Models (LLMs)?
LLMs are trained using language modeling, that is, predicting the next word in a sequence.
They do so by assigning probabilities to a fixed vocabulary.
Vocabulary

positive p = .03

[email protected]
negative p = .00001
The movie is a visually stunning, action-packed, and
[email protected]
emotionally resonant thrill ride that will leave you on the
edge of the seat from the beginning to end. Overall, the
experience was magical.
…

magical p = .83

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 What are Large Language Models (LLMs)?
LLMs are trained using language modeling, that is, predicting the next word in a sequence.
They do so by assigning probabilities to a fixed vocabulary.
Vocabulary

positive p = .03

[email protected]
negative p = .00001
The movie is a visually stunning, action-packed, and
[email protected]
emotionally resonant thrill ride that will leave you on the
edge of the seat from the beginning to end. Overall, the
experience was magical.
…

magical p = .83

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 What are Large Language Models (LLMs)?
LLMs are trained using language modeling, that is, predicting the next word in a sequence.
They do so by assigning probabilities to a fixed vocabulary.
Vocabulary

positive p = .03

[email protected]
negative p = .00001
The movie is a visually stunning, action-packed, and
[email protected]
emotionally resonant thrill ride that will leave you on the
edge of the seat from the beginning to end. Overall, the
experience was magical.
…
match

magical p = .83

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 What are Large Language Models (LLMs)?
During inference, the LLM predicts the next word in the input sequence.
Input word = prompt
The

Output, word-by-word
positive
The movie was awesome. Overall, the experience was positive.
[email protected]
negative
…
The movie was awesome. Overall, the experience was positive.
[email protected]

The movie was awesome. Overall, the experience was positive. movie

The movie was awesome. Overall, the experience was positive. magical
The movie was awesome. Overall, the experience was positive.
Vocabulary
The movie was awesome. Overall, the experience was positive.

⁞ This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 What are Large Language Models (LLMs)?
Over the last 2 years, LLMs (e.g., Open AI GPT) have evolved to be AI assistants

GPT (117M parameters) InstructGPT

First model to be trained in Instruction-tuned models
a “generative” mode by understand human inputs as
masking portions of input instructions; path to ChatGPT
text from left-to-right is paved
[email protected]
[email protected]

2019 2020 2023

2018 2022
GPT-2 (1.5B parameters) GPT-3 (175B parameters) GPT-4 (1T parameters?)
The era of prompting begins. Large scale foundation models are Era of completely closed
Models are relatively small, born. Prompting is shown to models begins; API
open-source and fine-tuning induce
This file is meant for personal userobust performance on
by [email protected] only. access only
is possible natural in
Sharing or publishing the contents language
part or fulltasks
is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Business Problems Solved by LLMs - A Taxonomy

Text Sentence

Fixed set of
classes Label Positive
Context + Context +
[email protected] Classification Text Question
[email protected]
(Text) → (Label)

Conversatio Contextualized Text Answer

Text
n
Augmented Generation
(Context + Text) → (Text)
Open-ended Text Summary

Generation
This file is meant for personal use by [email protected] only.
(Text) → (Text)
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 A Workflow for Enterprise LLM Applications
Container Approvals,
Dev Workspace
Registry Guardrails

Data Sources

Package prompts for

Build and manage prompts deployment Deployment
Data Lake Environment

[email protected]
[email protected]
Staging Production
Database environment environment

Logging & Monitoring

Event Logs
Infrastructure &
resource metrics

Collect live data & annotations

Performance metrics
(Accuracy, F1 score,
ThisRatings)
BERTScore, LLM file is meant
for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Prompt Engineering Fundamentals
Designing specific instructions that enable LLMs to accomplish business tasks

[email protected]
[email protected]

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Accessing LLMs using Azure Open AI
Azure provides controlled, fast access to a host of Open AI LLMs.

[email protected]
[email protected]

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Accessing LLMs using Azure Open AI
Azure provides controlled, fast access to a host of Open AI LLMs.

Copy of base model +

Model dedicated hosting + rate limit Deployment
(tokens per minute)
[email protected]
[email protected]

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Understanding Tokens
A token refers to a segment or piece of text, such as a word, punctuation, or other
meaningful element, into which input text is divided for processing by the model.

The movie was awesome.

[The,movie,was,awesome,.,Overall,,,the
Overall, the experience
,experience,was,positive,.]
was positive.

[email protected]
[email protected]
Tokenize LLM Model
r

Parses sentences Predicts the next

as tokens token

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Accessing LLMs using Azure Open AI APIs
The Azure Open AI Playground enables iterative development for prompt engineering, that is,
designing specific instructions for LLMs to accomplish a task.

[email protected]
[email protected]

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Accessing LLMs using Azure Open AI APIs
The Azure Open AI Playground enables iterative development for prompt engineering, that is,
designing specific instructions for LLMs to accomplish a task.

[email protected] Azure Open

[email protected] AI API

Prompt
API Key
format
Playground enables quick iterations on
prompts. Once an effective prompt is
discovered, it is translated to code for efficient
API access

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Azure Open AI API Prompt Format
Azure Open AI APIs are compatible with the Open AI APIs and have the following three components.

System Message
Clear instructions explaining the task that the LLM should accomplish.
These instructions are agnostic to the user input and appended to the
user input with higher priority. Can also be used to prime LLM behavior

[email protected]
User Message
LLM API Prediction
[email protected]

Specific instructions from the user describing the task that needs to
be accomplished.

Assistant Message
Not needed for single-turn conversations. Can be used to showcase
expected completions in multi-turn conversations.

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Prompt Engineering Fundamentals
Prompt = Specific set of instructions sent to a LLM to accomplish a task
Engineering = Iteratively deriving a specific prompt for the task

Length of output Max output length

Prompt
More temperature = More
[email protected]
[email protected] Temperature Parameters Structure
randomness in response

More Top P = More

tokens selected for Top P
completion

More FP = Less chance of Frequency

tokens repeating Penalty (FP)

This file is meant for personal use by [email protected] only.

[Playground Demo]
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Prompt Engineering Fundamentals
Understanding temperature
Temperature = 0

The overall The overall experience was positive Repeated

experience was execution
The overall experience was positive
produces the
The overall experience was positive same results

[email protected] LLM
[email protected]
Temperature = 1

The overall experience was positive Repeated

positive negative magical execution can
The overall experience was magical
produce different
0.6 0.1 0.3
The overall experience was negative results

This file is meant for personal use by [email protected] only.

[Playground Demo]
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Prompt Engineering Hands-on

- Zero-shot, Few-shot
- Chain-of-Thought
- Rephrase & Respond
- Self-Consistency
- LLM-as-a-judge
- Tree-of-thought
[email protected]
[email protected]

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
 Summary
Trained on
language modeling

Foundation
Models

Train

[email protected]
[email protected]
LLMs Templates System Message,
Few Shot Examples

Infer Prompts

Max Length,
Parameters
Temperature
This file is meant for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.