Building LLM applications with LangChain

Lance Martin
Software Engineer, LangChain
@RLanceMartin
LangChain makes it as easy as possible to develop LLM-powered applications.
 Platform (LangSmith) and Library (LangChain)

Use-cases RAG Chatbots Summarize Extraction SQL Agents ...

(Applications)

Building Blocks Document Document Storage +

Prompt LLM ...
Loaders Transformers Embeddings
(Integrations, chains)

Platform
(Debug, Test, Monitor)

LangSmith Blog
Building Blocks
 Document Loaders: > 140 Integrations Document
Loaders

Unstructured Structured

Public

Private / Company Data

.pdf, .txt, .json., ,md, … Datastores
Proprietary

Integrations hub
Text splitters Document
Transformers

Playground
Beyond basic splitting: Context-aware splitting, Function calling (Doctran) Document
Transformers

Document Document Embeddings +

Loaders Transformers Storage

Code:
Def foo
Def foo(...): for section in sections:
for section in sections: sect = section.find("head")
sect = section.find("head")

Markdown:
Introduction
# Introduction Notion templates are effective . . .
Notion templates are effective . . .

PDF:
Abstract
Abstract
The dominant sequence The dominant sequence . . .
transduction models . . .

Context-aware splitting, Doctran

 > 40 vectorstore integrations, > 30 embeddings Embeddings
Document +
Storage
Transformers

Document Loaders
(> 140 Integrations)

Vector Storage
(> 40 Integrations)
Document
Transformers
(e.g., Text Splitters, OAI functions)

Embeddings
(> 30 Integrations)
 Hosted or Private vectorstore + embeddings Embeddings
Document +
Storage
Transformers

Hosted Private (on device)

Embeddings

Vector Storage

GPT4All embeddings
 > 60 LLM integrations LLM

Document Loaders
(> 140 Integrations)

Vector Storage
(> 40 Integrations)
Document
Transformers LLMs
(e.g., Text Splitters, OAI functions)
(> 60 Integrations)

Embeddings
(> 30 Integrations)
 LLM landscape LLM

OpenAI Anthropic Llama-2 (SOTA OSS)

Context Window (tokens) 4k - 32k 100k 4k

Performance GPT-4 SOTA (best overall) Claude-2 getting closer to GPT4 70b on par w/ GPT-3.5-turbo*

Cost $0.06 / 1k token (input) 4-5x cheaper than GPT-4-32K Free

*Llama2-70b on par w/ GPT-3.5-turbo on language, but lags on coding

Math

Language
Code
 OSS models can run on device (private) LLM

Llama2-13b running ~50 tok / sec (Mac M2 max, 32gb)

Ollama, Llama.cpp
Integrations Hub

Integrations hub
Use Cases
 RAG: Load working memory w/ retrieved information relevant to a task RAG

Document Loading Splitting Storage Retrieval Output

URLs

PDFs
Relevant
Prompt
Splits
Documents Splits LLM Answer

Database Query
Question

Use case documentation

 Pick desired level of abstraction RAG

Abstraction / Simplicity

VectorstoreIndexCreator Answer

RetrievalQA Answer

Relevant
Splits Answer
Load_QA_chain

Use case documentation

 LangSmith trace for RetrievalQA chain RAG

Trace
Prompt

Retrieved docs

Question

Response

LangSmith trace
 Chat: Persist conversation history Chatbots

Retrieval (Optional)

Storage Retrieved Chunks

Question Prompt LLM Answer

Memory

Use case documentation

LangSmith trace for LLMChain w/ chat model + memory Chatbots

Prompt

Chat history

Response

LangSmith trace, also works with retrieval

 Summarization: Summarize a corpus of text Summarize

Final
Stuff document in context window Summary
Fits in LLM
context window
Prompt
Extract final summary LLM
from input list

Document Loader

Sample from clusters

Embed-and-cluster
Summarize each cluster

Prompt Cluster
Summarize themes in LLM Summaries
the group of docs

Does not fit in LLM

context window Final
Summarize chunks (map)
Summary
Distill into summary (reduce)

Prompt Prompt
Summarize themes in LLM Summary Extract final summary LLM
the group of docs from input list

Use case documentation

 Case-study: Apply to thousands of questions asked about LangChain docs Summarize

Summarized themes from LangChain questions using different methods and LLMs

User Questions
(Thousands)

Blog Post
 Extraction: Getting structured output from LLMs Extraction

Input

Alex is 5 feet tall. Claudia is 1 feet taller Alex and jumps higher
than him. Claudia is a brunette and Alex is blonde.

Schema (tell LLM schema we want)

{'name': 'Alex', 'height': 5, 'hair_color':
schema = { LLM 'blonde'}
"properties": {
"name": {"type": "string"}, Function call {'name': 'Claudia', 'height': 6,
"height": {"type": "integer"}, 'hair_color': 'brunette'}
"hair_color": {"type": "string"},
},}

Function (tell the LLM function)

"name": "information_extraction",
"description": "Extracts information from the passage.",
"parameters": {schema}

Use case documentation

LangSmith trace for LLMChain w/ function call + output parsing Extraction

Prompt

Response

Output from function call JSON parser

LangSmith Trace
 Text-to-SQL SQL

Query

Question LLM LLM Answer

Optional: SQL Agent

Use Case Documentation

LangSmith trace for text-to-sql SQL

Prompt

CREATE TABLE description for each table and

and three example rows in a SELECT statement

Response

LangSmith trace, Paper

 Agents

Chat Chains

Yes
ConversationalRetrievalChain
Agents

Access to memory

API / Function Chains

No

Basic LLM APIChain

No Yes
Access to tools
 Large agent ecosystem (will focus on ReAct as one example) Agents

Short-term Long-term
(Buffers) (> 40 vectorstores)

Action
Tools
(> 60 tools + toolkits)
Memory
Agents Simulation
(> 15 agent types)

Autonomous
Tools Agent Plan

LLMs
(> 60 integrations)

Action
 Agents

Action-Observation (Tool Use)

Yes
Say-Can ReAct

No

Standard Prompting Chain-of-thought*

No Yes
Multi-Step Reasoning
*Condition LLM to show its work
 LangSmith trace for SQL ReAct agent Agents

Prompt

Tool / action
Observation

Uses tool at
next step
Response

(Chain-of-thought) reasoning

LangSmith trace
 Case-study on reliability: Web researcher started an agent, retriever was better Agents

Document
Document Loader Document Retrieval + QA
Transformation

HTML
pages
Query 1
Vector Retrieved
Storage Chunks
Research Query 2 LLM Answer
Question LLM

Query N

Blog Post
Case-study on reliability: Web researcher started an agent, retriever was better Agents

Hosted streamlit app

Tooling
 LangSmith: Tracing and evaluation

Inspect Traces View, correct bad feedback, save to dataset

Collect app feedback

Correct

Save to dataset

Cookbook, Docs
 LangSmith: Tracing and evaluation

Collect eval set

Collect app feedback Fix & Re-evaluate

Re-deploy

Cookbook, Docs
Questions