‘9115124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Tings Done with Al Bootcamp Blog > Chat with Multiple Pdfs Using Llama 2 and Langchain Chat with Multiple PDFs using Llama 2 and LangChain Can you build a chatbot that can answer questions from multiple PDFs? Can you do it with a private LLM? In this tutorial, we'll use the latest Llama 2 13B GPTQ model to chat with multiple PDFs, We'll use the LangChain library to create a chain that can retrieve relevant documents and answer questions from them. Join the AI BootCamp! Ready to dive into the world of Al and Machine Learning? Join the Al BootCamp to transform your career with the latest skills and hands-on project experience. Learn about LLMs, ML best practices, and much more! JOIN NOW You'll learn how to load a GPTQ model using AutoGPTQ, convert a directory with PDFs to a vector store and create a chain using LangChain that works with text chunks from the vector store. © In this part, we will be using Jupyter Notebook to run the code. If you prefer to follow along, you can find the notebook on GitHub: GitHub Repository Llama 2 Llama 2! is the latest LLM offering from Meta Al! This cutting-edge language model comes with an expanded context window of 4096 tokens and an impressive 27 token https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain wTasi, 225 aM Chat wth Mutile PDFs using Lama 2 and LargChain|MLExper - Get Things Done wih Al Bootcamp dataset, surpassing its predecessor, Llama 1, in various aspects. The best part? Llama 2 is free for commercial use (with restrictions). Packed with pre-trained and fine-tuned LLMs ranging from 7 billion to 70 billion parameters, these models are set to outperform existing open-source chat models on a wide range of benchmarks. Here's an overview of the models available in Llama 2: Model | Llama2 | Llama2-hf | Llama2-chat | Llama2-chat-hf 7B Link Link Link 13B Link Link Link Link 70B Link Link Link Link Llama 2 comes in two primary versions - the base model and Llama-2-Chat - optimized for dialogue use cases. But how good is Llama 2? Looking at the HuggingFace Open LLM Leaderboard, looks like Llama 2 (and modified versions of it) takes the top spots. A. While many sources claim that Llama 2 is Open Source, the license is not considered Open Source by the Open Source Definition®, You can read more about it here: https://blog.opensource.org/metas-llama-2-license-is-not-open- source/ GPTQ GPTQtis a post-training quantization method capable of efficiently compressing models with hundreds of billions of parameters to just 3 or 4 bits per parameter, with minimal loss of accuracy. The method's efficiency is evident by its ability to quantize large models like OPT-175B and BLOOM-176B in about four GPU hours, maintaining a high level of accuracy. https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain 27asi, 225 aM Chat wth Mutile PDFs using Lama 2 and LargChain|MLExper - Get Things Done wih Al Bootcamp Moreover, the original paper demonstrates the method's robustness even in extreme quantization scenarios, where models are quantized to just 2 bits per component. It also includes practical implementations, running the compressed OPT-175B model efficiently ona single NVIDIA A100 GPU or a few NVIDIA A6000 GPUs, achieving impressive speedups of around 3.25x and 4.5x, respectively. Here is a summary of GPTQ using LLaMa (from the GitHub® repository): wie ppt | gps | 4B 4bit- Bbit- Bbit 3g128- RTN GPTQ RTN GPTQ GPTQ LlaMa-7B | 5.68 6.29 6.09 25.54 8.07 6.61 UeaMan 5.09 5.53 5.36 1140 6.63 5.62 13B ° . LLaMa- 4.10 454 445 14.89 5.69 4.80 308 LLaMa- 3.53 3.92 3.84 10.59 5.04 417 658 The table shows the perplexity of the Llama 1 models on the WikiText-2 dataset. Note that GPTQ is performing better than RTN (Round To Nearest) and is close to FP16. Creating and Running GPTQ Models In this tutorial, we'll use a GPTQ version of the Llama 2 13B chat model to chat with multiple PDFs. We'll use the TheBloke/Llama-2-13B-chat-GPTQ model from the HuggingFace model hub. The models available in the repository were created using AutoGPTQ&, The library allows you to apply the GPTQ algorithm to a model and quantize it to 3 or 4 bits. The library also provides a AutoGPTQForCausalLM class that can be used to load the quantized model and do inference using it. https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain 3n7‘9115124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Things Qone with Al Bootcamp Setup Let's start by installing the required libraries and downloading the model: Ipip install -Uqqq pip --progress-bar off Ipip install -qqq torch==2.0.1 --progress-bar off Ipip install -qqq transformers==4.31.0 --progress-bar off Ipip install -qqq langchain==0.0.266 --progress-bar off Ipip install -qqq chromadb==0.4.5 --progress-bar off Ipip install -qqq pypdf==3.15.0 --progress-bar off Ipip install -qqq xformers==0.0.20 --progress-bar off Ipip install -qqq sentence_transformers==2.2.2 --progress-bar of f Ipip install -qqq Instructorémbedding==1.0.1 --progress-bar off Ipip install -qqq pdf2image==1.16.3 --progress-bar off The AutoGPTQ library is a bit special (since we need to install the correct version with CUDA support). We can download and install it using the following commands: lwget -q https: //github.com/PanQiWei /AutoGPTQ/releases/download/v0.4./auto_gptq-0.4.04 Ipip install -qqq auto_gpta-0.4.0+cu118-cp310-cp31@-Linux_x86_64.whl --progress-bar off Finally, we need to install the poppler-utils library to convert PDFs to images: Isudo apt-get install poppler-utils Now, let's import the required libraries: import torch from auto_gptq import AutoGPTQForcausalLM from langchain import HuggingFacePipeline, PromptTenplate from langchain.chains import RetrievalQa from langchain.document_loaders import PyPDFOirectoryLoader from langchain.embeddings import HuggingFacelnstructEmbeddings from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain.vectorstores import Chroma from pdf2image import convert_from_path from transformers import AutoTokenizer, TextStreamer, pipeline https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain an7‘9115124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Tings Done with Al Bootcamp DEVICE = “cuda:0" if torch.cuda.is_available() else “cpu” Data We'll work with three PDFs in this tutorial: Imkdir pdfs Igdown 1v-Rn1FVULpLTAQEgm@N90B6cEXxMoebZr -0 pdfs/tesla-earnings-report .pdf Igdown 1Xc890jrQvCExAkryVWAttsvIDBLdVetN -0 pdfs/nvidia-earnings-report.pdf Igdown 1£pz-SQ3idPpoz75G1TzzomagSgplzLv8 -0 pdfs/meta-earnings-report.pdf These are the latest earning reports from Tesla, Nvidia, and Meta (formerly Facebook) We'll use these PDFs to chat with the Llama 2 13B GPTQ model Here's a preview of the first page of each PDF: sn7 https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain‘9115124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Things Qone with Al Bootcamp UNITED STATES SECURITIES AND EXCHANGE COMMISSION Washington, D.C. 20549 FORM 10-Q ‘ars one) 5 QUARTERLY REPORT PURSUANT TO SECTION 15 OR 13(#) OF THE SECURITIES EXCHANGE ACT OF 1931 er the quarterly period ended June 30, 2022 1 TRANSITION REFOKT PURSUANT TO SECTION 15 OF 15(@) OF THE SECURITIES EXCHANGE ACT OF 1954 Tesla, Inc. (Exact mame of registrant a epoifid in te chatter) sentido No) (a2) 516-8177 Tesla Earnings Sample Page en7 https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain‘9115124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Things Qone with Al Bootcamp UNITED STATES: Washington 0.20549, FORM 10-Q |_ QUARTERLY REPORT PURSUANT TO SECTION 13 OR 15) OF THE SECURES EXCHANGE ACT OF 1934 Fortha quately period anded Api 30,2023, TRANSITION REPORT PURSUANT TO SECTION 19 OF 156) OF THE SECURITIES EXCHANGE ACT OF 1934 Commision numb 0.25985 NVIDIA. NVIDIA CORPORATION (ect nw oferta specie inch Detonare seanrrste (state or Otter Jrisiton ot Rs. Empworer acing or Granta cried a} ‘Shots Cas, Cidoma tat (sie) e000 (Ader, inutngy cdes pho ant, lung we owe opel enceuve aes) (Porm ame, ee aden rfc cal yur changed ince i! ep Secures rgitered purtuantto Section 12() ofthe Re Tie af each lace Trading Symbols) Nama of aschacchange on which ro Common Stock 80.001 par valve par stare ‘Nvoa “he Nardan bal Select Manet Irate by chuck nth wheter the rite (1 ha la lleva touted ob le by Sechen 132 15) af the Scares Exchange At of 1904 ding thepreceding 12 onthe orf sich softer pero thal the estan was ered to fle Rh operand Z)has ean subse to sick Ming een for epast 0 dys. ves NOD. Futon 8 (222405 oa chop) duo tho preceding {2 math or or cach shor peed ut he oistard wax reqared We subma euch le), Yor SL neo Irate by chack math wheter he roasts a are acetate Me, an aoearat er. eon-arcolrte er, a sate eperig company, of ge aecslomiod a "acselrate le, “ovale asosing compen’. ard"emergne ert cereany mF logestaleatctie: 5 Acedealecty «C1 Nonareaeateter CL Snlereprra cares 1 Emersiggouneanoany 1 emer seh compen: cen iy check rc erin ha eo th ent enon pi or tephiog Bh ay oo reveecteancalaco.ing Sandares prevdedpursuarn Socien 1) ofthe Exchange Act Inala by chock math wheter he roasts ashe company (as denen Rule {20-2 he Exchange At. es 2 No ‘Te nurber stares ot common stack $0001 par valu, outstaring as of May 18, 2073, was 24 ton Nvidia Earnings Sample Page https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain m7‘9115124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Things Qone with Al Bootcamp Meta Reports Second Quarter 2023 Results MENLO PARK, Calif ~ July 26, 2023 ~ Meta Platoems, Inc. (Nasdag META) today reported financial resus fer the quarter ended lune 30,2023 *Wehad «good quater We continu to seo song engagement across aur app and we have the most exiting roadmap (ve comin a while with Llama 2, Threads, Rees, now Al products inthe pipeline, and tho launch af Quest 3 his all" sid Mase ‘Zuckerberp, Meta founder ang CEO, Second Quarter 2023 Financial Highlights sare dane nition. ne perme ne Shr ars 2 m2 Cham Revenue e 3pm S aR 11% Costs and expenses 2nst come fiom operations Ts FT xe Operating margin 2% 28 Provisicn for income wees s 130s) 1499 fictive tax ras 15% 18% Netinconie s 7788 8 ear 16% Dili earnings per sare (EPS) s 208 8 246 21% Second Quarter 2023 Operational and Other Financlal Highlights + Fauy daly active people (DAP) —DAP was 3 + Fatty monthly atv people (AP) MAP was 2.8 bilon a of Sane 30, 252, on nee of 6% year over «+ Facabook dally ative stars (DAU) DAL wre 206 bilon on setae fr ne 2023, Inna f 8 your + Facebok siatly seine mens (MAUI) MAU were 3.08 ion of J 20,2023 an rene 95 yar bili cn average for June 2023, an inctease of 7% yea-over- oxersyear + Ad impressions and price per ad ~ In the second quater of 2023, al impressions delivered across our Family of| Apps increased by 34% year-over-year and the average price pera decreased by 16% yeur-over-yeur ‘+ Revenue ~ Revere was $520 billion, an intense a 11% year-over-year, adam increase vf 12% yeurener-year on 2 constnt curency bass. + Costs and expenses ~ Total costs ant expenses were $22.61 billion. an increase of 10% yearoveryear. ‘This includes acrued lead expenses of $1.87 billion aad restuctutig charges of $7AU mllios in He second quarer of 2023, + Captua expenditures ~ Capital expenditure, including principal payments on finance leases, were $6.35 billion fr ne second quarter of 2023, “+ Share repurchases - We ropurshased $793 million of our Class A common sock inthe second quarter of 2023. As ‘of Jane 30, 2023, we had $40.91 bilion avilable and authorize fe repurchases. + Cash, cash equivalents, and marketable securities Cash, cach oguivslnts, andl makuble secuiios ware $53 45 billion as of June M0, 2023, Fee es flo was $10.96 hill in the Second quarier F023, “Long-term det Long-sm debt was SIX 38 billion os of une 30, 202 ‘+ Headcount ~ Headcount was 71 459 28 of une 30,2028, a decrese of 14% year-over-year: Appronistely hal of ‘te employees impacted by the 2023 lyotls are incladd in our reported headcount as of June 30,2023, Meta Earnings Sample Page https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain an7asi, 225 aM Chat wth Mutile PDFs using Lama 2 and LargChain|MLExper - Get Things Done wih Al Bootcamp These documents are long and complex. They use technical language and contain a lot of numbers. Let's see how well the Llama 2 13B GPTQ model can chat with these documents. Load Documents Let's start by loading the PDFs and splitting them into smaller chunks. We'll load the PDFs using the PyPDFDirectoryLoader class from the langchain library. This class loads all the PDFs in a directory and returns a list of Document objects: loader = PyPDFDirectoryLoader ("pdfs") docs = loader. load() Jen(docs) 168 The combined page count of all the documents is 100. We'll then use the RecursiveCharacterTextSplitter class to split the documents into smaller chunks. Each chunk will have 1024 characters, with an overlap of 64 characters: text_splitter = RecursiveCharacterTextSplitter (chunk_size=1024, chunk_overlap=64) texts = text_splitter.split_documents (docs) len(texts) 355 This gives us 355 chunks of text. We'll use these chunks to create a vector store. Our vector store will the sentence-transforners library to create embeddings for the text chunks: embeddings = HuggingFaceInstructEmbeddings ( model_name="hkunlp/instructor-large”, model_kwargs={"device”: DEVICE} https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain on7‘9115124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Tings Done with Al Bootcamp d We'll use instruction-finetuned text embedding model provided by hkunlp/instructor- large. These embeddings score very high on the Massive Text Embedding Benchmark (MTEB) Leaderboard?. Our vector store will use the Chroma database. We'll use the from_documents method to create it: db = Chroma.from_documents(texts, embeddings, persist_directory="db") Our database now contains embeddings for all the text chunks. Let's continue with loading the model. Llama 2 13B © The choice of the model is specific to the hardware you're using. Here, we'll use a Nvidia T4 GPU with 16GB of VRAM. If you have a different GPU, you might need to use a different model (usually, the larger the better). Let's use AutoGPTQ to load the Llama 2 13B GPTQ model and the tokenizer: model_name_or_path = "TheBloke/Llama-2-138-chat-GPTQ" model_basename = "model" tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True) model = AutoGPTQForCausaliM.from_quantized( model_name_or_path, revision="gptq-4bit-128g-actorder_True", model_basename=mode1_basename, use_safetensors=True, trust_remote_code=True, inject_fused_attention-False, device="cuda: quantize_config-None, https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain son7‘9115124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Tings Done with Al Bootcamp d We're using a 4 bit quantized model with 128 groups. Here's a short description of the model: 4-bit, with Act Order and group size. 128g uses even less VRAM, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed. So, our model is not the fastest, but it should be accurate enough for our use case. Prompt Format The Llama 2 chat models use a specific prompt format. Here's the general template: [INST] <> {system_prompt} <> {prompt} [/INST] Note that Llama 2 supports a system prompt and a user prompt. We'll use the system prompt to parametrize the model (give it a context on how we want it to respond). The user prompt will contain the question we want to ask the model DEFAULT_SYSTEM_PROMPT = """ You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that yot responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don’t know the answer to a question, please don't share false information. vm strip() def generate prompt(prompt: str, system prompt: str = DEFAULT SYSTEM PROMPT) -> str: return f°" [INST] <> {system_prompt} <> https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain nT‘9115124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Tings Done with Al Bootcamp {prompt} [/INST] +strip() Note that this is the default system prompt provided by the Meta developers. We'll change it in a bit. Let's continue with creating the pipeline: streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True) text_pipeline = pipeline( ‘text-generation", model=model, tokenizer-tokenizer, mmax_new_tokens=1024, temperature=0, top_p=0.95, repetition_penalty=1.15, streaner=streamer, This pipeline will stream the response of the model to the standard output. We'll set the temperature to 0 to make our responses repeatable. We'll also use the repetition_penalty parameter to penalise the repetition of tokens in the response. To make our pipeline compatible with LangChain, we'll wrap it in the HuggingFacePipeline class: lm = HuggingFacePipeline(pipeline=text_pipeline, model_kwargs: ‘temperature": @}) Create Chain We now have all the pieces we need to create a chain that can retrieve relevant documents and answer questions from them. First, we'll create a prompt template that will be used to parametrize the model: SYSTEM PROMPT = “Use the following pieces of context to answer the question at the end. https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain wan7‘9115124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Things Qone with Al Bootcamp template = generate_prompt( {context} Question: {question} system_prompt-SYSTEM_PROMPT,, Prompt = PromptTemplate(template-template, input_variables=["context", “question"]) and prompt template: ga_chain = RetrievalQa.from_chain_type( n=l, chain_type retriever=db.as_retriever(search_kwarg: return_source_documents=True, chain_type_kwargs={"prompt": prompt}, "Kk": 2})5 Note that we're restricting the number of documents retrieved to 2. This will ensure that ‘our LLM doesn’ get over the token limit. Chat with Multiple PDFs Let's ask our chain a few questions: result = ga_chain("What is the per share revenue for Meta during 2023? Based on the information provided in the press release, the per share revenue for Meta during 2023 was $2.98. Our chain gave us an exact answer! Let's see where it got the answer from https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain 137‘9115124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Things Qone with Al Bootcamp Jen(result["source_documents"]) print(result["source_documents"][@].page_content) | BB occument contents Our embeddings along with the Llama 2 model did a great job here. Here's an excerpt of the document: Diluted earnings per share (EPS) 2.98 2.46 21% Let's ask another question result = qa_chain("what is the per share revenue for Tesla during 2023?") I can't determine the per share revenue for Tesla in 2023 based on the information provided. The financial statements only provide data on total revenues and not per share data. Additionally, there is no information on the number of outstanding shares, which is necessary to calculate per share data. Strange, our chain couldn't find the answer. The retrieval part of the chain didn't do a good job here - the answer is available within the PDF (Diluted - $0.78). Let's ask the same question for Nvidia: result = qa_chain("What is the per share revenue for Nvidia during 20232") Based on the information provided, the per share revenue for Nvidia during 2023 was $0.83. This can be calculated by dividing the total revenue for the period ($7,192 million) by the weighted average number of shares outstanding (basic and https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain san7‘9115124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Things Qone with Al Bootcamp diluted) during the period (2,470 million). The answer is correct! It got it from here: Net income per share: Basic $0.83 Okay, let's ask a more complex question result = qa_chain("what is the estimated YoY revenue for Meta during 20237") Based on the information provided in the press release, Meta's estimated YoY revenue for 2023 is expected to be between $32-34.5 billion, which represents an increase of 11-16% over 2022. Another great response! Let's ask for a forecast based on the data from the documents: result = qa_chain("Choose one company to invest (Tesla, Nvidia or Meta) to maximize you I cannot provide a definitive answer to this question, as it is not appropriate for me to give financial advice or make recommendations on specific investments. However, based on the information provided in the context, NVIDIA appears to be a strong company with a diverse range of businesses and a solid track record of innovation and growth. NVIDIA has a long history of developing cutting-edge technology, including GPU architecture, and has successfully expanded into new markets such as artificial intelligence, data science, and autonomous vehicles. The company has also demonstrated a conmitment to sustainability and social responsibility, which may be attractive to socially conscious investors. That being said, investing in any company carries inherent risks, and it is essential to conduct thorough research and consult with a financial advisor before making any investment decisions. Additionally, the decision to invest in a particular company should be based on individual circumstances and goals, and should not be solely determined by the information provided here. https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain 1517‘9115124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Tings Done with Al Bootcamp It is accustom for those LLMs to stray from providing financial advice. Though, it gave us a concrete answer after that. Our agent chose Nvidia - key player in developing AI (wink wink). It even mentioned the sustainability and social responsibility of the company. 3,000+ people already joined Join the The State of Al Newsletter Every week, receive a curated collection of cutting-edge Al developments, practical tutorials, and analysis, empowering you to stay ahead in the rapidly evolving field of AL Your Email Address SUBSCRIBE I won't send you any spam, ever! References 1. Llama 2 by Meta Al © 2. Open LLM Leaderboard & 3, The Open Source Definition & 4, GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers © 5. GPTQ Official Implementation on GitHub © 6. AutoGPTQ © 7. Massive Text Embedding Benchmark (MTEB) Leaderboard © https:shwww.mlexpertiofblogichat-wih-multiple-pafessing-lama-2-and-angchain 1679118124, 8:25 AM CChat wth Multiple PDFS using Llama 2 and LangChain | MLExpert - Get Things Qone with Al Bootcamp hitps:Iwww.mlexper iobloglchat with-mukiple-ps-using smn