2024/10/27 上午11:55 Building a Multi-LLM Chatbot with Langchain: OpenAI and Ollama | by Gayani Parameswaran | Medium
Open in app
38
Search
Get unlimited access to the best of Medium for less than $1/week. Become a member
Building a Multi-LLM Chatbot with Langchain:
OpenAI and Ollama
Gayani Parameswaran · Follow
4 min read · May 27, 2024
Listen Share More
Introduction
C hatbots have become ubiquitous, offering a convenient and interactive way to
access information and complete tasks. This project explores building a
chatbot that leverages the power of both paid and open-source large language
models (LLMs) through the Langchain framework. We’ll utilize OpenAI’s powerful
API for access to a commercially available LLM and Ollama, a local runtime
environment for running open-source LLMs.
Project Overview
T his project demonstrates how to create a chatbot that can interact with users
through a text interface. Users can pose questions, and the chatbot will
https://medium.com/@gayani.parameswaran/building-a-multi-llm-chatbot-with-langchain-openai-and-ollama-e4836b5160f7 1/13
2024/10/27 上午11:55 Building a Multi-LLM Chatbot with Langchain: OpenAI and Ollama | by Gayani Parameswaran | Medium
respond using the capabilities of either the OpenAI LLM or the Ollama-based open-
source LLM (Llama2 in this example).
Key Components
Langchain: An open-source framework that simplifies LLM integration and
streamlines chatbot development.
Langchain Core: Provides foundational components for building LLM pipelines.
Langchain OpenAI: Facilitates access and interaction with OpenAI’s LLM API.
Langchain Community: Offers integrations with various third-party tools and
open-source LLMs.
Streamlit: A Python library for building user interfaces for data apps.
Ollama: An open-source runtime environment for running LLMs locally on your
machine.
OpenAI API: Provides access to a commercially available LLM with high
performance.
Llama2: A powerful open-source LLM available through Ollama.
Project Flow
1. Environment Setup:
Create a virtual environment and install required libraries (langchain_core,
langchain_openai, langchain, python-dotenv, streamlit, langchain_community).
Set up a .env file to store API keys for Langchain and OpenAI.
LANGCHAIN_API_KEY = < YOUR_LANGCHAIN_API_KEY _HERE>
OPENAI_API_KEY = <YOUR_OPENAI_API_KEY _HERE>
LANGCHAIN_PROJECT = <YOUR_LANGCHAIN_PROJECT _HERE>
2. Chatbot with OpenAI LLM:
Define a prompt template that establishes the conversation flow between the
system and the user.
https://medium.com/@gayani.parameswaran/building-a-multi-llm-chatbot-with-langchain-openai-and-ollama-e4836b5160f7 2/13
2024/10/27 上午11:55 Building a Multi-LLM Chatbot with Langchain: OpenAI and Ollama | by Gayani Parameswaran | Medium
Utilize ChatOpenAI from langchain_openai to interact with the OpenAI LLM
API.
Employ StrOutputParser from langchain_core to handle the LLM response as a
string.
Build a Langchain pipeline by chaining the prompt template, ChatOpenAI
instance, and output parser.
Integrate Streamlit to create a user interface where users can input questions.
Upon receiving user input, trigger the Langchain pipeline to generate a response
using the OpenAI LLM.
3. Chatbot with Ollama LLM:
Download the desired open-source LLM (Llama2 in this example) using Ollama’s
command-line interface.
Use Ollama from langchain_community to interact with the locally running
LLM.
Create a separate Langchain pipeline using the prompt template, Ollama
instance with the Llama2 model, and output parser.
Integrate Streamlit to create a user interface where users can input questions.
Upon receiving user input, trigger the Langchain pipeline to generate a response
using the OpenAI LLM.
Within the Streamlit app, allow users to select between the OpenAI and Ollama-
based chatbot options.
Code Implementation
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
import streamlit as st
import os
from dotenv import load_dotenv
load_dotenv()
# Langchain tracking and API key environment variables
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = os.getenv("LANGCHAIN_API_KEY")
https://medium.com/@gayani.parameswaran/building-a-multi-llm-chatbot-with-langchain-openai-and-ollama-e4836b5160f7 3/13
2024/10/27 上午11:55 Building a Multi-LLM Chatbot with Langchain: OpenAI and Ollama | by Gayani Parameswaran | Medium
# Prompt template defining conversation flow
prompt = ChatPromptTemplate.from_messages([
("system", "You are a helpful assistant. Please respond to the user queries
("user", "Question: {question}")
])
# Streamlit app title
st.title('Langchain Chatbot Demo')
# User input field
input_text = st.text_input("Search the topic you want")
def get_response(llm_choice):
# Define output parser
output_parser = StrOutputParser()
# Choose LLM based on input
if llm_choice == "OpenAI":
# Access OpenAI LLM through Langchain OpenAI
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-3.5-turbo")
else:
# Access Ollama-based LLM (Llama2 in this example)
from langchain_community.llms import Ollama
llm = Ollama(model="llama2")
# Build Langchain pipeline
chain = prompt | llm | output_parser
# Generate response if user input exists
if input_text:
return chain.invoke({'question': input_text})
else:
return None
# Select LLM option (can be extended for more choices)
llm_options = ["OpenAI", "Ollama (Llama2)"]
selected_llm = st.selectbox("Choose LLM", llm_options)
# Generate response based on selection
response = get_response(selected_llm)
# Display response if available
if response:
st.write(f"**Response from {selected_llm}:**")
st.write(response)
Outcome
https://medium.com/@gayani.parameswaran/building-a-multi-llm-chatbot-with-langchain-openai-and-ollama-e4836b5160f7 4/13
2024/10/27 上午11:55 Building a Multi-LLM Chatbot with Langchain: OpenAI and Ollama | by Gayani Parameswaran | Medium
Benefits & Consideration
Flexibility: The ability to switch between paid and open-source LLMs offers
cost-effectiveness and access to cutting-edge models.
Cost: Utilizing OpenAI’s LLM API incurs costs based on the token size of each
request. For frequent chatbot interactions, these costs can accumulate.
Performance: OpenAI provides high-performance LLMs, while Ollama offers a
free, open-source alternative with potentially higher latency.
Monitoring: Langchain facilitates tracking LLM interactions through its
Langsmith dashboard. Project will be created with the specific name
<LANGCHAIN_PROJECT > inside langsmith as we mentioned in the .env file.
https://medium.com/@gayani.parameswaran/building-a-multi-llm-chatbot-with-langchain-openai-and-ollama-e4836b5160f7 5/13
2024/10/27 上午11:55 Building a Multi-LLM Chatbot with Langchain: OpenAI and Ollama | by Gayani Parameswaran | Medium
Monitoring the project with langsmith
Conclusion
T his project demonstrates the power of Langchain in building a versatile
chatbot that leverages both paid and open-source LLMs. By combining
Streamlit for the user interface and Ollama for local LLM execution, we create a
cost-effective and customizable chatbot framework.
Ollama Opanai Langsmith Llama 2 Langchain
https://medium.com/@gayani.parameswaran/building-a-multi-llm-chatbot-with-langchain-openai-and-ollama-e4836b5160f7 6/13
2024/10/27 上午11:55 Building a Multi-LLM Chatbot with Langchain: OpenAI and Ollama | by Gayani Parameswaran | Medium
Follow
Written by Gayani Parameswaran
46 Followers
Machine Learning Engineer | Passionate about Generative AI
More from Gayani Parameswaran
Gayani Parameswaran
Advanced Q&A Chatbot with Chain and Retriever using Langchain
Langchain is a library designed specifically for building workflows around LLMs, and it can be a
useful tool for creating RAG-based…
May 27 3
https://medium.com/@gayani.parameswaran/building-a-multi-llm-chatbot-with-langchain-openai-and-ollama-e4836b5160f7 7/13
2024/10/27 上午11:55 Building a Multi-LLM Chatbot with Langchain: OpenAI and Ollama | by Gayani Parameswaran | Medium
Gayani Parameswaran
Building a Versatile LLM API with Langchain, LangServe, and FastAPI
Introduction
May 27 16
Gayani Parameswaran
Voice based Chatbot using OpenAI models
In the era of AI and Machine Learning, chatbots have become a crucial part of our digital
experience. They are everywhere, from customer…
https://medium.com/@gayani.parameswaran/building-a-multi-llm-chatbot-with-langchain-openai-and-ollama-e4836b5160f7 8/13
2024/10/27 上午11:55 Building a Multi-LLM Chatbot with Langchain: OpenAI and Ollama | by Gayani Parameswaran | Medium
May 1 8
Gayani Parameswaran
Retrieval-Augmented Pipeline (RAG) with Langchain
How to query from different data sources and get results quickly
May 27 3 1
See all from Gayani Parameswaran
Recommended from Medium
https://medium.com/@gayani.parameswaran/building-a-multi-llm-chatbot-with-langchain-openai-and-ollama-e4836b5160f7 9/13
2024/10/27 上午11:55 Building a Multi-LLM Chatbot with Langchain: OpenAI and Ollama | by Gayani Parameswaran | Medium
Devvrat Rana
Understanding LangChain Agents: A Beginner’s Guide to How LangChain
Agents Work
Introduction:
Jun 2 66
Ashish Malhotra
PromptTemplate and ChatPromptTemplate Explained
You can get the definition of Prompt over the internet but net net, Prompt is a basic building
block of a #Langchain code. It provides a…
https://medium.com/@gayani.parameswaran/building-a-multi-llm-chatbot-with-langchain-openai-and-ollama-e4836b5160f7 10/13
2024/10/27 上午11:55 Building a Multi-LLM Chatbot with Langchain: OpenAI and Ollama | by Gayani Parameswaran | Medium
Jul 23 53
Lists
Natural Language Processing
1777 stories · 1376 saves
Staff Picks
751 stories · 1399 saves
Coldstart Coder
Fine Tuning Llama 3.2 11B for Question Answering
Large Language Models are pretty awesome, but because they are trained to be generic and
broadly focused, they sometimes don’t perform…
Oct 14 31
https://medium.com/@gayani.parameswaran/building-a-multi-llm-chatbot-with-langchain-openai-and-ollama-e4836b5160f7 11/13
2024/10/27 上午11:55 Building a Multi-LLM Chatbot with Langchain: OpenAI and Ollama | by Gayani Parameswaran | Medium
Gayani Parameswaran
Building a Powerful Document Q&A ChatBot with Llama3, LangChain, and
Groq API
Introduction
May 28 6
Vishal Rajput in AIGuys
The Prompt Report: Prompt Engineering Techniques
Prompting Techniques Survey
https://medium.com/@gayani.parameswaran/building-a-multi-llm-chatbot-with-langchain-openai-and-ollama-e4836b5160f7 12/13
2024/10/27 上午11:55 Building a Multi-LLM Chatbot with Langchain: OpenAI and Ollama | by Gayani Parameswaran | Medium
Oct 14 904 9
Boqiang & Henry
Run OpenAI Whisper Locally: Step-by-Step Guide
How to Run OpenAI Whisper Locally
Aug 9 145
See more recommendations
https://medium.com/@gayani.parameswaran/building-a-multi-llm-chatbot-with-langchain-openai-and-ollama-e4836b5160f7 13/13
