Meta Releases Prompt Engineering Guide
Meta Releases Prompt Engineering Guide
You can find the full prompt engineering guide from Meta here.
This interactive guide covers prompt engineering & best practices with Llama 2.
Introduction
Why now?
Vaswani et al. (2017) introduced the world to transformer neural networks (originally for
machine translation). Transformers ushered an era of generative AI with diffusion models for
image creation and large language models (LLMs) as programmable deep learning networks.
Llama Models
In 2023, Meta introduced the Llama language models (Llama Chat, Code Llama, Llama Guard).
These are general purpose, state-of-the-art LLMs.
Llama 2 models come in 7 billion, 13 billion, and 70 billion parameter sizes. Smaller models are
cheaper to deploy and run (see: deployment and performance); larger models are more capable.
Llama 2
1. llama-2-7b - base pretrained 7 billion parameter model
2. llama-2-13b - base pretrained 13 billion parameter model
3. llama-2-70b - base pretrained 70 billion parameter model
4. llama-2-7b-chat - chat fine-tuned 7 billion parameter model
5. llama-2-13b-chat - chat fine-tuned 13 billion parameter model
6. llama-2-70b-chat - chat fine-tuned 70 billion parameter model (flagship)
Code Llama is a code-focused LLM built on top of Llama 2 also available in various sizes and
finetunes:
Code Llama
Getting an LLM
Large language models are deployed and accessed in a variety of ways, including:
1. Self-hosting: Using local hardware to run inference. Ex. running Llama 2 on your
Macbook Pro using llama.cpp.
o Best for privacy/security or if you already have a GPU.
2. Cloud hosting: Using a cloud provider to deploy an instance that hosts a specific model.
Ex. running Llama 2 on cloud providers like AWS, Azure, GCP, and others.
o Best for customizing models and their runtime (ex. fine-tuning a model for your
use case).
3. Hosted API: Call LLMs directly via an API. There are many companies that provide
Llama 2 inference APIs including AWS Bedrock, Replicate, Anyscale, Together and
others.
o Easiest option overall.
Hosted APIs
Hosted APIs are the easiest way to get started. We'll use them here. There are usually two main
endpoints:
Tokens
LLMs process inputs and outputs in chunks called tokens. Think of these, roughly, as words –
each model will have its own tokenization scheme. For example, this sentence...
...is tokenized into ["our", "dest", "iny", "is", "written", "in", "the", "stars"]
for Llama 2.
Tokens matter most when you consider API pricing and internal behavior (ex. hyperparameters).
Each model has a maximum context length that your prompt cannot exceed. That's 4096 tokens
for Llama 2 and 100K for Code Llama.
Notebook Setup
The following APIs will be used to call LLMs throughout the guide. As an example, we'll call
Llama 2 chat using Replicate and use LangChain to easily set up a chat completion API.
LLAMA2_70B_CHAT = "meta/llama-2-70b-
chat:2d19859030ff705a87c746f7e96eea03aefb71f166725aee39692f1476566d48"
LLAMA2_13B_CHAT = "meta/llama-2-13b-
chat:f4e2de70d66816a838a89eeeb621910adffb0dd0baba3976c96980970978018d"
# We'll default to the smaller 13B model for speed; change to LLAMA2_70B_CHAT
for more advanced (but slower) generations
DEFAULT_MODEL = LLAMA2_13B_CHAT
def completion(
prompt: str,
model: str = DEFAULT_MODEL,
temperature: float = 0.6,
top_p: float = 0.9,
) -> str:
llm = Replicate(
model=model,
model_kwargs={"temperature": temperature,"top_p": top_p,
"max_new_tokens": 1000}
)
return llm(prompt)
def chat_completion(
messages: List[Dict],
model = DEFAULT_MODEL,
temperature: float = 0.6,
top_p: float = 0.9,
) -> str:
history = ChatMessageHistory()
for message in messages:
if message["role"] == "user":
history.add_user_message(message["content"])
elif message["role"] == "assistant":
history.add_ai_message(message["content"])
else:
raise Exception("Unknown role")
return completion(
get_buffer_string(
history.messages,
human_prefix="USER",
ai_prefix="ASSISTANT",
),
model,
temperature,
top_p,
)
Completion APIs
Llama 2 models tend to be wordy and explain their rationale. Later we'll explore how to manage
the response length.
Messages with the system role are used to provide core instruction to the LLM by
developers.
Messages with the user role are typically human-provided messages.
Messages with the assistant role are typically generated by the LLM.
response = chat_completion(messages=[
user("My favorite color is blue."),
assistant("That's great to hear!"),
user("What is my favorite color?"),
])
print(response)
# "Sure, I can help you with that! Your favorite color is blue."
LLM Hyperparameters
These APIs also take parameters which influence the creativity and determinism of your output.
At each step, LLMs generate a list of most likely tokens and their respective probabilities. The
least likely tokens are "cut" from the list (based on top_p), and then a token is randomly selected
from the remaining candidates (temperature).
In other words: top_p controls the breadth of vocabulary in a generation and temperature
controls the randomness within that vocabulary. A temperature of ~0 produces almost
deterministic results.
print_tuned_completion(0.01, 0.01)
print_tuned_completion(0.01, 0.01)
# These two generations are highly likely to be the same
print_tuned_completion(1.0, 1.0)
print_tuned_completion(1.0, 1.0)
# These two generations are highly likely to be different
Prompting Techniques
Explicit Instructions
You can think about giving explicit instructions as using rules and restrictions to how Llama 2
responds to your prompt.
Stylization
o Explain this to me like a topic on a children's educational
network show teaching elementary students.
o I'm a software engineer using large language models for
summarization. Summarize the following text in under 250 words:
o Give your answer like an old timey private investigator hunting
down a case step by step.
Formatting
o Use bullet points.
o Return as a JSON object.
o Use less technical terms and help me apply it in my work in
communications.
Restrictions
o Only use academic papers.
o Never give sources older than 2020.
o If you don't know the answer, say that you don't know.
Here's an example of giving explicit instructions to give more specific results by limiting the
responses to recently created sources.
A shot is an example or demonstration of what type of prompt and response you expect from a
large language model. This term originates from training computer vision models on
photographs, where one shot was one example or instance that the model used to classify an
image (Fei-Fei et al. (2006)).
Zero-Shot Prompting
Large language models like Llama 2 are unique because they are capable of following
instructions and producing responses without having previously seen an example of a task.
Prompting without examples is called "zero-shot prompting".
Let's try using Llama 2 as a sentiment detector. You may notice that output format varies - we
can improve this with better prompting.
complete_and_print("Text: This was the best movie I've ever seen! \n The
sentiment of the text is: ")
# Returns positive sentiment
Few-Shot Prompting
Adding specific examples of your desired output generally results in more accurate, consistent
output. This technique is called "few-shot prompting".
In this example, the generated response follows our desired format that offers a more nuanced
sentiment classifer that gives a positive, neutral, and negative response confidence percentage.
See also: Zhao et al. (2021), Liu et al. (2021), Su et al. (2022), Rubin et al. (2022).
def sentiment(text):
response = chat_completion(messages=[
user("You are a sentiment classifier. For each message, give the
percentage of positive/netural/negative."),
user("I liked it"),
assistant("70% positive 30% neutral 0% negative"),
user("It could be better"),
assistant("0% positive 50% neutral 50% negative"),
user("It's fine"),
assistant("25% positive 50% neutral 25% negative"),
user(text),
])
return response
def print_sentiment(text):
print(f'INPUT: {text}')
print(sentiment(text))
Role Prompting
Llama 2 will often give more consistent responses when given a role (Kong et al. (2023)). Roles
give context to the LLM on what type of answers are desired.
Let's use Llama 2 to create a more focused, technical response for a question around the pros and
cons of using PyTorch.
Chain-of-Thought
Simply adding a phrase encouraging step-by-step thinking "significantly improves the ability of
large language models to perform complex reasoning" (Wei et al. (2022)). This technique is
called "CoT" or "Chain-of-Thought" prompting:
Self-Consistency
LLMs are probablistic, so even with Chain-of-Thought, a single generation might produce
incorrect results. Self-Consistency (Wang et al. (2022)) introduces enhanced accuracy by
selecting the most frequent answer from multiple generations (at the cost of higher compute):
import re
from statistics import mode
def gen_answer():
response = completion(
"John found that the average of 15 numbers is 40."
"If 10 is added to each number then the mean of the numbers is?"
"Report the answer surrounded by three backticks, for example:
```123```",
model = LLAMA2_70B_CHAT
)
match = re.search(r'```(\d+)```', response)
if match is None:
return None
return match.group(1)
answers = [gen_answer() for i in range(5)]
print(
f"Answers: {answers}\n",
f"Final answer: {mode(answers)}",
)
Retrieval-Augmented Generation
You'll probably want to use factual knowledge in your application. You can extract common
facts from today's large models out-of-the-box (i.e. using just the model weights):
However, more specific facts, or private information, cannot be reliably retrieved. The model
will either declare it does not know or hallucinate an incorrect answer:
This could be as simple as a lookup table or as sophisticated as a vector database containing all
of your company's knowledge:
MENLO_PARK_TEMPS = {
"2023-12-11": "52 degrees Fahrenheit",
"2023-12-12": "51 degrees Fahrenheit",
"2023-12-13": "51 degrees Fahrenheit",
}
def ask_for_temperature(day):
temp_on_day = MENLO_PARK_TEMPS.get(day) or "unknown temperature"
prompt_with_rag(
f"The temperature in Menlo Park was {temp_on_day} on {day}'", #
Retrieved fact
f"What is the temperature in Menlo Park on {day}?", # User question
)
ask_for_temperature("2023-12-12")
# "Sure! The temperature in Menlo Park on 2023-12-12 was 51 degrees
Fahrenheit."
ask_for_temperature("2023-07-18")
# "I'm not able to provide the temperature in Menlo Park on 2023-07-18 as the
information provided states that the temperature was unknown."
complete_and_print("""
Calculate the answer to the following math problem:
Gao et al. (2022) introduced the concept of "Program-aided Language Models" (PAL). While
LLMs are bad at arithmetic, they're great for code generation. PAL leverages this fact by
instructing the LLM to write code to solve calculation tasks.
complete_and_print(
"""
# Python code to calculate: ((-5 + 93 * 4 - 0) * (4^4 + -7 + 0 * 5))
""",
model="meta/codellama-
34b:67942fd0f55b66da802218a19a8f0e1d73095473674061a6ea19f2dc8c053152"
)
# The following code was generated by Code Llama 34B:
num1 = (-5 + 93 * 4 - 0)
num2 = (4**4 + -7 + 0 * 5)
answer = num1 * num2
print(answer)
Limiting Extraneous Tokens
A common struggle is getting output without extraneous tokens (ex. "Sure! Here's more
information on...").
Check out this improvement that combines a role, rules and restrictions, explicit instructions, and
an example:
complete_and_print(
"Give me the zip code for Menlo Park in JSON format with the field
'zip_code'",
model = LLAMA2_70B_CHAT,
)
# Likely returns the JSON and also "Sure! Here's the JSON..."
complete_and_print(
"""
You are a robot that only outputs JSON.
You reply in JSON format with the field 'zip_code'.
Example question: What is the zip code of the Empire State Building?
Example answer: {'zip_code': 10118}
Now here is my question: What is the zip code of Menlo Park?
""",
model = LLAMA2_70B_CHAT,
)
# "{'zip_code': 94025}"
Additional References
PromptingGuide.ai
LearnPrompting.org
Lil'Log Prompt Engineering Guide