Scribd
Upload
6 views15 pages

Reflection Prompting Making LLM Think

The document outlines the Reflection framework for enhancing language-based agents through linguistic feedback, emphasizing its components such as Actor, Evaluator, and Self-Reflection. It discusses the advantages and limitations of using Reflection, including its suitability for trial and error scenarios and the challenges of hallucination in LLMs. Additionally, it highlights the evolving nature of agents that combine LLMs with capabilities for task planning, tool use, memory management, reasoning, and learning.

Uploaded by

pathakpritee20
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
6 views15 pages

Reflection Prompting Making LLM Think

The document outlines the Reflection framework for enhancing language-based agents through linguistic feedback, emphasizing its components such as Actor, Evaluator, and Self-Reflection. It discusses the advantages and limitations of using Reflection, including its suitability for trial and error scenarios and the challenges of hallucination in LLMs. Additionally, it highlights the evolving nature of agents that combine LLMs with capabilities for task planning, tool use, memory management, reasoning, and learning.

Uploaded by

pathakpritee20
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 15

Reflection Prompting

Making LLM Think


Logistics
● One week from now - you will have an exam
● It will be one hour - multiple choice questions
● Whatever covered until last week will be covered

● Projects - Next Wednesday is the deadline for picking your project topics and
teams
Recall
● We learned about ReAct framework and saw an example of how to make an
LLM reason - a bit unsuccessfully
● ReAct => Reason + Action
● Reasoning involves, LLM “thinking” through results of its actions
● Action involves calling tools that can take action on LLM’s direction with LLM
specified inputs and produce results
● We keep repeating until LLM “thinks” it has found the “answer” to user query
● This is one of the basic frameworks for building LLM Agent
Reflection
Like ReAct framework, we can also use Reflection framework.

“Reflection is a framework to reinforce language-based agents through


linguistic feedback” - Prompt Engineering Guide

Per the Paper Shinn et al. (2023) - "Reflexion is a new paradigm for ‘verbal‘
reinforcement that parameterizes a policy as an agent’s memory encoding paired
with a choice of LLM parameters."
Reflection Diagram

Actor: Takes an Action.


Environment: Provides feedback as
reward or observation
Trajectory: series of feedback and
action for a shorter duration are
stored as Short Term Memory.
Self-Reflection: LLM that processes
the external feedback and internal
feedback - evaluator’s evaluation of
the Trajectory of the conversation.
Experience: text from
Self-Reflection to store for future use
as Long Term Memory.
Reflection - Actor
Actor:
● The Actor takes an action in an environment and receives an observation
which results in a trajectory.
● Chain-of-Thought (CoT) and ReAct are used as Actor modalities.
○ These are prompting methods used to make the LLM “think”
● A memory component is also added to provide additional context to the agent
○ To provide more context in the prompt for the LLM to decide next action
○ Feedback from environment - user or another LLM is added as short term memory of the
conversation / transaction
○ We also provide memory of prior actions and their reflections to Actor as input as long term
memory
Reflection - Evaluator
Evaluator:

● Judges Actor’s actions based on external feedback and recent


action-feedback pairs (short term memory)
● This is a combination of LLM judgements or rules set that is applied to the
input
● The goal is to create internal feedback for the reflection LM to process the
external feedback with
Reflection - Self-Reflection
Self-Reflection LLM:

● Generates text to reinforce decision for the Actor in self-improvements


● This provides valuable feedback for the actor and is stored in long term
memory for future actions
● To generate this feedback, the model takes input, external rewards to Actor’s
actions, the evaluator’s judgement of this action-feedback pair and collections
of past such pairs.
Reflection - When To Use
Reflection is best when used for

● Trial and Error is acceptable


● Other ML techniques like Reinforcement Learning (RL) is not feasible
○ Training data is too expensive
○ Not enough experts
● Detailed feedback is useful
● Explainability / Traceability of decisions is important
○ Medical diagnosis
○ Learning from previous unrelated runs is important
Reflection - When NOT To Use
Reflection LLM -

● It is an LLM - It can “Hallucinate”...


● When your reflection based teacher is hallucinating -
○ You learn to hallucinate MORE!!

Long Term Memory -

● Is complicated to store and retrieve


Reflection - What is it good for?
● Answers follow a linear path - one decision leads to the next one
● Time is not an issue in answering
○ The back and forth takes time
● Decision needs reasoning
● Programming assignment - you can make LLMs write your assignment :-D …
○ IF you learn how to prompt them!!
Agent
Exciting and rapidly evolving area in artificial intelligence

● At their core, these agents are systems that combine large language models
(LLMs) with additional capabilities to perform tasks or solve problems more
effectively than a standard LLM alone
● "agent" is a system that can perceive its environment, make decisions, and
take actions to achieve specific goals. It's like a digital entity with a degree of
autonomy.
Agent
Key Features:
● Task Planning: They can break down complex tasks into smaller,
manageable steps.
● Tool Use: They can utilize various digital tools or APIs to gather information
or perform actions.
● Memory and Context Management: They can maintain context over longer
interactions and "remember" important information.
● Reasoning: They can apply logical reasoning to solve problems or make
decisions.
● Learning and Adaptation: Some advanced agents can learn from their
interactions and improve over time.
References
ReAct Prompting: https://www.promptingguide.ai/techniques/react

Reflection Prompting: https://www.promptingguide.ai/techniques/reflexion

LLM Reasoning: https://www.promptingguide.ai/research/llm-reasoning


Agent
To be continued…

You might also like