Machine Translated by Google

Overview of the RAG system

1. Purpose

Required for the RAG system to operate

ÿ Installing Python ÿ How to

create a Python environment
ÿ How to install a package
ÿ Packages required for RAG
ÿ Read the skeleton code of the RAG system and understand its basic operation.

2. Building a Python operating environment

2.1 Installing Python Install

Python on your PC.
(1) Search for Python on the Microsoft store site. (2)
Download the latest version of Python (probably python 3.12).
(3) Double-click the downloaded setup file.

2.2 Installing packages with Pip

(1) Pip is a Python package installer that installs required packages before running Python.

(2) It is a good idea to put together the installation of the required packages in a shell

script. For example, if the PowerShell executable file is named “RAGenv.ps1”, the
contents will be as follows:

<#
.SYNOPSIS
RAG environment setup script

.DESCRIPTION

This script installs the Python libraries required for developing the Retrieval-
Augmented Generation (RAG) system.

.NOTES
Author: Kazuo Hashimoto
Created: March 26, 2024
Version: 1.0
#>
Machine Translated by Google

# Installing libraries # 0. General

mschine learning tools # 1. langchain: Installs the

LangChain library itself. # 2. openai: Required for using the OpenAI API. # 3.
chromadb: Required for using the Chroma vector store. # 4. tiktoken:
OpenAI's tokenizer, used for text splitting. # 5. pypdf: Required for reading PDF files. #

6. python-dotenv: Used to read environment variables from .env files.

pip install pip setuptools pip install numpy

pandas matplotlib seaborn scikit-learn scikit-learn-intelex

pip install langchain pip install

openai pip install chromadb
pip install tiktoken pip install
pypdf pip install python-dotenv

3. Setting up your Python environment

3.1 How to use environment variables

(1) If file paths or personal OPENAI_API_KEYs are written directly into a program, the portability of the program

will be significantly reduced. Therefore, for values that are highly system-dependent, we recommend that

you define them as environment variables.

There is a way.

from dotenv import load_dotenv # Load

environment variables from a .env fileload_dotenv ()

(2) Place the .env file in the execution directory. .env is <environment variable> = <value>

It is written in the form

.env example

OPENAI_API_KEY = "Your OpenAI_API_KEY"

Machine Translated by Google

ANTHROPIC_API_KEY = "Your Anthropic_API_KEY"

# Specify the FAQ data file

FAQ_FILE_PATH = 'Data¥FAQ_converted.xlsx'
VECTORSTORE_UPDATED_TIME='20240718_215338'
VECTORSTORE_UPDATED='vetorestore1.bin'

(3) In your program, call it like this:

openai_api_key = os.getenv("OPENAI_API_KEY")

3.2 Specifying required modules using the Import statement

At the beginning of your program, call the modules you need. The format is

import <module#1> <module#2> …

import os

import logging from

dotenv import load_dotenv from

langchain.document_loaders import PyPDFLoader, TextLoader

from langchain.text_splitter import

RecursiveCharacterTextSplitter from
langchain.vectorstores import Chroma from
langchain.embeddings import OpenAIEmbeddings from
langchain.chains import ConversationalRetrievalChain from langchain.memory
import ConversationBufferMemory from langchain.prompts import
PromptTemplate from langchain.chat_models import
ChatOpenAI import pandas as from langchain.schema
import Document import
hashlib import json

4. Document loader use case

4.1 Programs to convert text files to Excel files
(1) Input text file: FAQ.txt
Machine Translated by Google

# Source: https://www.jrce.co.jp/medical/recept/index.html
# ORCA 1 - 7

1. "What is ORCA?" , "ORCA is a Japanese medical information system provided by the Japan Medical Association.
This is a medical receipt computer system for medical facilities.
It is used by 17,000 medical institutions and is the standard system in the age of electronic receipts.

"

2. "What are the main features of ORCA?" , The main features of ORCA are:
1. Always up-to-date (the latest programs are available via the Internet)
2. Reduction in implementation and update costs (hardware only
3. Relatively easy to implement (replace existing receipts with the new system)
(Data transfer is possible)

3. "What kind of medical institutions would you recommend ORCA to?" "The following types of medical institutions" ,
Recommended for medical institutions: New openings and lease expirations, introduction of new receipt systems

Medical institutions considering the above Medical institutions that are dissatisfied with the manufacturer-led medical billing system

Improving the efficiency of medical administration at medical institutions that support the Japan Medical Association's philosophy of IT in the medical field

"A cost-conscious medical institution that aims to

4. "What types of ORCA does JRCE provide?" We , "JRCE is 2

provide two types of ORCA: 1. In-hospital server version ORCA
(ORCAPROJECT): Type in which the server is installed within the hospital 2. Cloud version ORCA

(ORCACLOUD): ORCA has been put on the cloud, and is now even safer and more convenient.

Ip"

5. When introducing ORCA, is it possible to transfer data from existing receipt computers?
, Yes, it is possible. You can get basic patient information and disease name from the currently used receipt computer.

However, depending on the manufacturer, data transfer may not be possible.

Please contact us for more information as it may not be possible to do so.

6. "How long does it take to learn how to use ORCA?" "General ,

Generally, it is said that you can learn to use it after just two sessions.
In addition, JMA IT certified instructors are on hand to help ensure smooth business
operations, and call support is also available.
Machine Translated by Google

7. "How will ORCA be updated and how will changes in medical fees be handled?
"
, "You can get the latest programs and drug masters via the Internet.
The changes are reflected automatically with a simple click of a button on the screen.
Similarly, medical fee revision programs can be updated with the click of a button.
There is no cost to you."

# Source:
https://www.jrce.co.jp/medical/product/onpremiseorca.htm
l
# In-hospital server version ORCA 8 - 15

,
8. "What is the in-hospital server version of ORCA?" "The in-hospital server version of ORCA is

A receipt computer system that installs a server within a medical institution.

External backup is standard and can be customized to suit your PC environment.
It will be suggested for the system configuration you have chosen.

9. "What is the ORCA system configuration recommended by , "J.R.C.E.

JRCE?" recommends a two-unit ORCA configuration.
This means that even if one machine breaks down, the other can continue to operate independently, ensuring business continuity.

It will be protected."

10. How is data security ensured in a single ORCA configuration?

It is , In the case of a single ORCA configuration, data is backed up after each medical treatment.

recommended that you save the "Installation" to a USB memory stick.

A technician will visit you and perform recovery work from backup data.

(2) Output Excel file: FAQ.xlsx

FAQ Question Answer

_id

"ORCA is a medical information platform provided by the Japan Medical Association for the medical field in Japan.

This is a receipt computer system.

What is ORCA? Currently, it is used in approximately 17,000 medical institutions, and electronic receipts are

1 mosquito?" It is growing into the standard system for the computing era.
Machine Translated by Google

"The main features of ORCA are as follows: 1. It can always be kept up

to date (the latest programs and masters can be obtained via the Internet).

2. It reduces the cost of introduction and updating (only the hardware needs

Main features of ORCA to be replaced). 3. It is relatively easy to introduce (data can be transferred

2 What is it?" from existing receipt computers).

"Recommended for the following types of medical institutions: Medical institutions

considering the introduction of a new medical receipt system due to new openings or lease

"What types of medical expiration Medical institutions that are dissatisfied with manufacturer-driven medical receipt

institutions would you systems Cost-conscious medical institutions that agree with the Japan Medical Association's

3 recommend ORCA for?" philosophy of IT in the medical field and aim to streamline medical institution administration."

JRCE offers two types of ORCA: 1. In-hospital server version ORCA

(ORCAPROJECT): A type where a server is installed within the hospital.

"What types of ORCA 2. Cloud version ORCA (ORCACLOUD): A type where ORCA is put on

does JRCE offer?" the cloud, and safety and convenience are further improved.

When introducing ORCA,

is it possible to transfer "Yes, it is possible. Patient basic information and disease name information

data from an existing can be transferred from the medical receipt computer currently in use.

receipt computer? However, depending on the manufacturer, data transfer may not be
"
5 possible, so please contact us for details." "It is

"How long does it take generally said that you can learn to use it after just two operating

to learn how to use ORCA? instructions. In addition, JMA IT certified instructors will support smooth

It takes time business operations, and call support is also available." "You can obtain

6 mosquito?" the latest programs

and master data such as medicines via the Internet, and they will be

"How will you respond automatically reflected by simply clicking a button on the screen. Similarly,

to ORCA updates and medical fee revision programs can be updated by clicking a button, and

medical fee revisions? there is no additional cost." "The in-hospital server version of ORCA is a

7 Will it be done?" medical receipt computer

system that is installed on a server within the medical institution. External

"Hospital Server Version backup is standard, and a system configuration that suits the customer's

What is ORCA? PC environment will be proposed."

8 mosquito?"

"What is JRCE's JRCE recommends a two-machine ORCA configuration. By storing data

recommended system on two computers, even if one machine breaks down, the other can

9 configuration for ORCA? continue to operate independently, ensuring business continuity.

Machine Translated by Google

"
Masu."

"How is data security "If you have a single ORCA system, we recommend that you back

ensured in a single up your data after each consultation and save it to a USB

ORCA configuration?" memory stick. If you have a computer problem, a technician will
10 visit you and perform recovery work from the backup data."

(3) Conversion program:

FAQConv.py This script reads a FAQ text file written in a specific format,
analyzes its contents, and converts it into an Excel file in the format
of FAQ_id, Question_id, Answer_id.

Main
functions: 1. Reading FAQ text files 2.
Analyzing FAQ content (extracting FAQ_id, Question_id, Answer_id) 3.
Converting the analyzed data to a
DataFrame 4. Outputting the DataFrame to an Excel file

import pandas as pd

def parse_faq(file_path): data =

[] with
open(file_path, 'r', encoding='utf-8') as file:
for line in file:
line = line.strip() if
line.startswith('#') or not line:
continue
faq_id, question, answer = line.split(',', 2)

data.append({ 'FAQ_id': faq_id.strip(),

'Question': question.strip(), 'Answer':
answer.strip()
})
return pd.DataFrame(data)
Machine Translated by Google

# Path to the FAQ.txt file

faq_file_path = 'RAG/Data/FAQ.txt'

# Parse the FAQ into a DataFrame df

= parse_faq(faq_file_path)

# Save the DataFrame as an Excel file

output_file_path = 'RAG/Data/FAQ_converted.xlsx'
df.to_excel(output_file_path, index=False)

print(f"FAQ converted to Excel file: {output_file_path}")

4.2 Importing Excel files

(1) Read the Excel file created in the previous section 4.1 as a Pandas data frame.
Stores (<Question>, <Answer>) pairs as Document objects in LangChain.

(2) If the document is long, split it into chunks of an appropriate length (e.g. 1000 characters)
using the split_documents function. However, in the following case, it is unlikely that the
strings in the Question and Answer columns will exceed 1000 characters.
Processing using split_documents is not necessary and will degrade performance.

def load_faq(file_path): # Load

Excel file df =
pd.read_excel(file_path)

# Convert the DataFrame to a list of LangChain Document objects

documents = []
for _, row in df.iterrows(): content
= f"Question: {row['Question']}¥nAnswer:
{row['Answer']}"
metadata = {'FAQ_id': row['FAQ_id']} doc =
Document(page_content=content,
metadata=metadata)
documents.append(doc)
Machine Translated by Google

return documents

4.3 Loading PDF files

(1) Here, "information necessary to answer questions from users", Service Manual, related
Assume that the laws governing the use of electronic money, etc., are given as documents in PDF format.

(2) The data needs to be modified to fit the format.

(3) If the document is long, it needs to be split into chunks by split_documents. (4) However, if it is
properly split, skip the split_documents process.

def load_service_manual(file_path):
loader = PyPDFLoader(file_path) return
loader.load()

4.4 Loading from a URL

from langchain.document_loaders import WebBaseLoader

html_loader = WebBaseLoader(“https://xxx.xxx”) html_doc

= html_loader.load_and_split()

5. Creating a Vector Store

5.1 Splitting Long Texts
(1) If the text is long , Divide into chunks of 1000 characters .

Chunk_size=1000 is a commonly used setting.

(2) In order to prevent the loss of context information due to the division,
Define chunk_overlap. Here, we split the chunk with 20% overlap, but you need to
find the optimal value for the target document.

# Split text def

split_documents(documents):
text_splitter =
RecursiveCharacterTextSplitter(chunk_size=1000,
chunk_overlap=200)
return text_splitter.split_documents(documents)
Machine Translated by Google

5.2 Construction of Vector Store using ChromaDB

ÿ The following URL shows how to create a VectoreStore using ChromaDB.
URL: https://github.com/pixegami/langchain-rag-tutorial.git
ÿ There is also a lecture video available at the following URL:
RAG + Langchain Python Project: Easy AI/Chat For Your Docs
https://www.youtube.com/watch?v=tcqEUSNCn8I
Based on this information, please do the following:

(1) The ReadMe.md file explains how to run this program.

Run it and make sure it works.
## Install dependencies

3. Do the following before installing the dependencies found in

`requirements.txt` file because of current challenges installing `onnxruntime`
through `pip install onnxruntime`.

- For MacOS users, a workaround is to first install `onnxruntime`

dependency for `chromadb` using:

```python
conda install onnxruntime -c conda-forge
```

See this
[thread](https://github.com/ nvironme/onnxruntime/issues/11037) for nvironmen help if needed.

- For Windows users, follow the guide [here]

(https://github.com/bycloudai/InstallVSBuildToolsWindows?t ab=readme-ov-file) to
install the Microsoft C++ Build Tools. Be sure to follow through to the last step to set
the nvironment variable path.

2. Now run this command to install dependenies in the `requirements.txt`

file.
Machine Translated by Google

```python
pip install -r requirements.txt
```

3. Install markdown dependencies with:

```python
pip install “unstructured[md]”
```

##Create database

Create the Chroma DB.

```python
python create_database.py
```

## Query the database

Query the Chroma DB.

```python
python query_data.py “How does Alice meet the Mad Hatter?”
```

> You'll also need to set up an OpenAI account (and set the OpenAI key in your
environment variable) for this to work.

Here is a step-by-step tutorial video: [RAG+Langchain Python Project: Easy AI/Chat

For Your Docs](https://
www.youtube.com/watch?v=tcqEUSNCn8I&ab_channel=pix egami).
Machine Translated by Google

(2) create_database.py
ÿ This is a basic program to create a VoctoreStore using ChromaDB.

Run it and understand what the program is about.

(3) query_data.py
ÿ This is a program that queries the database created above. Run it in your own environment and

Understand the content of the gram.

ÿ In this program, the question is passed as an argument when the program runs.

Modify it so that it accepts input from the user and returns an answer.

(4) Testing the behavior using different data

ÿ Enter a Japanese document as input data and check the operation.

(5) Compare_embeddings.py

In compare_embeddings.py, the vector distance between words is calculated as follows, but please modify
it to calculate the vector distance between sentences.

# Compare vector of two words

evaluator = load_evaluator(“pairwise_embedding_distance”) words =
(“apple”, “iphone”) x =
evaluator.evaluate_string_pairs(prediction=words[0],
prediction_b=words[1])
print(f”Comparing ({words[0]}, {words[1]}): {x}”)

6 RAG System

RAG.Sample.py is the skeleton code for the RAG system. 6.1 Each

function has a basic error handling mechanism built in, using the try <code> except <Error-handling-

code> syntax.

The Error Log format is defined below. #

Logging
configuration logging.basicConfig(level=logging.INFO, format='%(asctime)s
- %(levelname)s - %(message)s')

6.2 Checking for file updates

A hash value is used to check whether a file has been updated.

The relevant functions are:

Machine Translated by Google

Def get_file_hash(file_path):
"Compute the MD5 hash of a file"
hash_md5 = hashlib.md5 ()
with open(file_path, "rb") as f:
for chunk in iter(lambda: f.read(4096), b””):
hash_md5.update(chunk)
return hash_md5.hexdigest()

def load_file_hashes(hash_file_path): "Load file

hash history" if os.path.exists(hash_file_path):

with open(hash_file_path, 'r') as f: return

json.load(f)
return {}

def save_file_hashes(hash_file_path, file_hashes): """Save file

hash history""" with open(hash_file_path,
'w') as f:
json.dump(file_hashes, f)

# Define file paths as constants

FAQ_FILE = 'faq.xlsx'
INQUIRY_HISTORY_FILE = 'inquiry_history.txt'
SERVICE_MANUAL_FILE = 'service_manual.pdf'

FILE_PATHS = [FAQ_FILE, INQUIRY_HISTORY_FILE, SERVICE_MANUAL_FILE]

def check_file_exists(file_paths): "Check for

file existence" for file_path in
file_paths: if not
os.path.exists(file_path): raise
FileNotFoundError(f)
{file_path}”)
Machine Translated by Google

6.3 Document loader

Assuming you want to load data in Excel, text, or PDF format,
Defines a function.
(1) The columns to be imported
from the Excel file are assumed to be 'FAQ_id', 'Question', and 'Answer'. You will
need to check them according to your data.
def load_faq(file_path):
try:
# Read the Excel file df =
pd.read_excel(file_path)

# Ensure required columns exist

required_columns = ['FAQ_id', 'Question', 'Answer'] if not all(col in
df.columns for col in required_columns): raise ValueError(f"FAQ file requires
columns
{required_columns} are not included.")

# Convert the DataFrame to a list of LangChain Document objects documents = [] for

_, row in df.iterrows(): content

= f"Question: {row['Question']}\nAnswer: {row['Answer']}" metadata
= {'FAQ_id':
row['FAQ_id']} doc = Document(page_content=content,

metadata=metadata)
documents.append(doc)

logging.info(f"{len(documents)} FAQs loaded.") return documents

except Exception as e:
logging.error(f" Failed to load FAQ: {e}") return []

(2) Text file

We are assuming your CRM data is a text file.
Machine Translated by Google

# Function to load SalesForce CRM

# Need to modify according to CRM data format
def load_inquiry_history(file_path):
try:
loader = TextLoader(file_path) return
loader.load()
except Exception as e:
logging.error(f"Failed to load query history: {e}") return []

(3) PDF file

Service manuals, related laws, etc. are assumed to be documents in PDF format.
Specify the

# Service manual, related laws, etc. # We

assume that these are documents in PDF format for
now, but # they need to be modified to match
the data format. def load_service_manual(file_path):
try:
loader = PyPDFLoader(file_path) return
loader.load()
except Exception as e:
logging.error(f"Failed to load service manual:
{e}")
return []

6.4 VectorStore

The process flow of Manage_vector_store is as follows: 1. Check

for updates to external data:

ÿ Get the last update time of external data (e.g. FAQs) ÿ Compare
with the previous update time stored in an environment variable.
2. Branching based on whether updates exist:

a) If there is an update:

ÿ Create a new VectorStore.

ÿ Read external data and add it to VectorStore.
ÿ Save the new VectorStore to a file.
ÿ Set the new update time in the environment variables.
Machine Translated by Google

ÿ Returns a new VectorStore.

b) If there are no updates:

ÿ Load an existing VectorStore from a file.

ÿ Returns the loaded VectorStore.

3. Error handling:

ÿ Dealing with errors that occur while reading/writing files or processing external data

Masu.

def manage_vector_store(documents, vector_store_path,

openai_api_key, file_paths):
try:
embeddings =
OpenAIEmbeddings(openai_api_key=openai_api_key)
hash_file_path = os.path.join(vector_store_path,
'file_hashes.json')

# Load previous file hashes

previous_hashes = load_file_hashes(hash_file_path) current_hashes
= {file_path: get_file_hash(file_path) for
file_path in file_paths

# Check for file updates

updated_files =
[ file_path for file_path in file_paths if
current_hashes.get(file_path) !=
previous_hashes.get(file_path) ]

if os.path.exists(vector_store_path) and not

updated_files:
logging.info("No updates in file, loading existing VectorStore")

vector_store =
Chroma(persist_directory=vector_store_path,
embedding_function=embeddings)
else:
Machine Translated by Google

if updated_files:
logging.info(f"The following files have been updated,
Recreate the VectorStore: {', '.join(updated_files)}")
else:
logging.info(" Create a new VectorStore") vector_store
= Chroma.from_documents(documents, embeddings,
persist_directory=vector_store_path)

# Save the new hashes

save_file_hashes(hash_file_path, current_hashes)

vector_store.persist()
return vector_store
except Exception as e:
logging.error(f" An error occurred while managing VectorStore:
{e}")
raise

6.5 Main Processing

Now define the main process using the functions we have defined so far.

(1) VECTOR_STORE_PATH must be set to the directory where the script file is located.
It seems to be pre-defined, but it might be better to define a fixed path.

# Define the vector store path as a

constant VECTOR_STORE_PATH = os.path.join(os.path.dirname(__file__),
'vector_store')

(2) Splitting text data

faq_data is already split. inquery_history is probably split as well. If the service
manual is also split by item, split_documents processing is not necessary.
Consideration is required according to the data.
combined_data = faq_data + inquiry_history + service_manual split_data =
split_documents(combined_data)

(3) Error handling and error message confirmation

Machine Translated by Google

If there is a better format definition from the standpoint of usability and debugging, we will improve it.

# Logging
configuration logging.basicConfig(level=logging.INFO, format='%(asctime)s - %
(levelname)s - %(message)s')

(4) Definition of PROMPT

ÿ The following templates will be defined:

ÿ It is necessary to check how it works when actual data is entered.
ÿ In that case, if you define a different template, what will happen?
Check.

template = """Please answer the user's most recent question using the conversation history and information
below. Please

answer concisely in easy-to-understand Japanese. Avoid making

uninformed guesses and if you do not have the information, say "That information is not provided."

Please answer "No, I haven't."

Conversation

history: {chat_history}

References:
{context}

Latest Question: {question}

answer:"""

PROMPT = PromptTemplate(
input_variables=["chat_history", "context", "question"],
template=template,

)
(5) Main processing script
# Main
processing def main():
try:
Machine Translated by Google

# Load environment variables from the .env

file load_dotenv()

# Get the OpenAI API key from the environment

variable openai_api_key = os.getenv("OPENAI_API_KEY") if
not openai_api_key: raise
ValueError("OPENAI_API_KEY is not set. Check your .env file.")

# Check file existence

check_file_exists(FILE_PATHS)

faq_data = load_faq(FAQ_FILE)
inquiry_history =
load_inquiry_history(INQUIRY_HISTORY_FILE)
service_manual = load_service_manual(SERVICE_MANUAL_FILE)

# faq_data is already split. inquery_history is probably split as well. # If the service

manual is also
split into sections, there is no need to use split_documents # Needs consideration!
combined_data =
faq_data +
inquiry_history + service_manual split_data = split_documents(combined_data)

# Define the vector store path as a constant

VECTOR_STORE_PATH =
os.path.join(os.path.dirname(__file__), 'vector_store')

# Create the vector store directory if it does not exist. if not

os.path.exists(VECTOR_STORE_PATH):
os.makedirs(VECTOR_STORE_PATH)
logging.info(f"Created a directory for the vector store:
{VECTOR_STORE_PATH}")
Machine Translated by Google

vector_store = manage_vector_store(split_data,
VECTOR_STORE_PATH, openai_api_key, FILE_PATHS)

# Create a custom prompt templatetemplate = "

""Please answer the user's most recent question using the conversation history and
information below. Please
answer concisely in easy-to-understand Japanese. Avoid making
uninformed guesses and if you do not have the information, say "That information is not provided."
Please answer "No, I haven't."

Conversation

history: {chat_history}

References:
{context}

Latest Question: {question}

answer:"""

PROMPT = PromptTemplate(
input_variables=["chat_history", "context", "question"],
template=template,

# Initialize memory
memory =
ConversationBufferMemory(memory_key="chat_history",
return_messages=True)

# Initialize LLM (explicitly pass API key) llm =

ChatOpenAI(temperature=0,
openai_api_key=openai_api_key)

# Create a RAG chain

Machine Translated by Google

qa_chain = ConversationalRetrievalChain.from_llm( llm=llm,

retriever=vector_store.as_retriever(),
memory=memory,
combine_docs_chain_kwargs={"prompt": PROMPT}
)

# Question-answering loop

while True:
question = input("Enter your question (press 'q' to quit)
input): ")
if question.lower() == 'q':
break

try:
result = qa_chain({"question": question}) print("answer:",
result['answer']) except Exception as e:

logging.error(f"An error occurred while answering the question:

{e}")
print("Sorry, an error occurred. Please try again.
Please do so.")

except FileNotFoundError as e:
logging.error(f"File not found: {e}")
except Exception as e:
logging.error(f"An error occurred while running the program: {e}")