GenAI POC to

Production
Lessons Learned Developing a GraphRAG Production App

March 21, 2024

 Presenters

Alex Gilmore Alexander Fournier Daniel Bukowski

Consulting Engineer Sales Engineer Sales Engineer

2 Neo4j Inc. All rights reserved 2023

 Key Topics
1. Notebooks → Streamlit

2. Streamlit → Self-Hosted

3. Back-End and Data Engineering

4. Data and Data Model

5. Logging

3 Neo4j Inc. All rights reserved 2023

 Key Takeaways

Work toward building a viable app for today and tomorrow

Real-world usage is critical to inform how you evolve your app

Neo4j knowledge graphs are flexible elements of the architecture:

● Accurate, complete grounding

● Logging usage
● Visibility into LLM responses

Build in a logging function to understand app behavior

4 Neo4j Inc. All rights reserved 2023

 GenAI Project Overview

5 Neo4j Inc. All rights reserved 2023

 What is Agent Neo?
● LLM Chat Application to assist
individuals using the Neo4j and the
Graph Data Science Library
● RAG Application grounded on a Neo4j
AuraDS Knowledge Graph containing
text from Neo4j Documentation and
blogs
● Built using LangChain
● POC built using Streamlit
● Production on cloud infrastructure
with a React front-end

6 Neo4j Inc. All rights reserved 2023

 Notebooks → Streamlit

7 Neo4j Inc. All rights reserved 2023

 Notebooks → Streamlit

The initial POC was built across

four Jupyter notebooks.

It worked, but…

Wrangling multiple notebooks

is error prone and time
consuming

Users had to load and run the

notebook to ask a question

8 Neo4j Inc. All rights reserved 2023

 Notebooks → Streamlit
class LLM(BaseModel):
"""
What variables become Interface for interacting with different LLMs.
"""
changeable parameters?
llm_type: str
● Temperature temperature: float = 0.7
llm_instance: ChatVertexAI
● RAG on / off
def _init_llm(self, llm_type: str,
temperature: float):
What architecture makes sense for """
your application? This function initializes an LLM for conversation.

"""
● LLM handler . . .
● Database handler
def get_response(self, question: str,
context: pd.DataFrame | None) -> str:
"""
Get a response from the LLM.
"""
. . .
9 Neo4j Inc. All rights reserved 2023
 Notebooks → Streamlit

Enables quick iteration and controlled

experimentation

Easy deployment to make your app

available to users and share demos

Expanding users helps gain an

understanding of how the app might be
used in production

10 Neo4j Inc. All rights reserved 2023

 Streamlit → Self-Hosted

11 Neo4j Inc. All rights reserved 2023

 Streamlit → Self-Hosted

What Streamlit managed: Streamlit has challenges:

● Application state management
● Deploy Streamlit “off Streamlit”
● Autoscaling
● Access to private / company repos
● Auto deployments from Github
● Custom / company styling
● UI styling / HTML
● Auth
● Frontend components

12 Neo4j Inc. All rights reserved 2023

 Streamlit → Self-Hosted

13 Neo4j Inc. All rights reserved 2023

 Cloud Infrastructure

14 Neo4j Inc. All rights reserved 2023

 Cloud Infrastructure Recipe

Migrated to Google Cloud Managed Services:

● Google Cloud Run to host our backend API & frontend React app
● Google Cloud DNS
● Google Cloud Secrets Manager
● Google Cloud Artifact Registry / Container repository
● Google Cloud Storage

Github

● Github actions to handle auto-deployment

15 Neo4j Inc. All rights reserved 2023

 Streamlit → React

● Migrating from Streamlit to React allowed us to to optimize performance through

the Virtual DOM and a higher degree of fine grained support
● A greater degree of decoupling from the backend and frontend logic
● Migrated frontend code from python to typescript, moving from a dynamically
typed to static typed language helps us catch more 🪲
● Leaned heavily on Neo4j’s Needle design system - a fantastic asset, styling and
component library to make beautiful Neo4j-y UIs
● Packed in a docker image, pushed to google artifact repository and deployed via
Cloud Run

16 Neo4j Inc. All rights reserved 2023

 Streamlit → Fastapi

● Validate requests
and responses
● Perform logging in
the background
● Automatic
documentation
● Stateless

17 Neo4j Inc. All rights reserved 2023

 Fastapi & Pydantic

● Object
representation
● Advanced
validation

18 Neo4j Inc. All rights reserved 2023

 Self-Hosted UI

19 Neo4j Inc. All rights reserved 2023

 Back-End

20 Neo4j Inc. All rights reserved 2023

 Notebooks →Data Engineering Pipelines

21 Neo4j Inc. All rights reserved 2023

 Chunking Strategies

● Our data sources come in a variety of formats: unstructured, semi structured, and
unstructured.
● In anticipation of encoding our documents as embeddings, we chunk/partition the
documents using some logical pivot point

22 Neo4j Inc. All rights reserved 2023

 Chunking Strategies

23 Neo4j Inc. All rights reserved 2023

 Data and Data Model
Evolution

24 Neo4j Inc. All rights reserved 2023

 Initial Grounding Data and Data Model
● 1,150 documents of official Neo4j documentation
● Developer blogs
● Support knowledge base
● Demo Github repos

25 Neo4j Inc. All rights reserved 2023

 Expanded (then contracted) Grounding Data

As the project progressed, we discussed what additional data sources could further
improve results. We decided to include:
● More code from public Neo4j repos
● Transcripts of trainings from the Neo4j YouTube Page

But…
● More is not always better. Instead of ALL YouTube trainings we focused on the
Going Meta series, which focuses on ontologies and GenAI
● Quality over quantity. We also emphasized ensuring we properly ingested code
repos, rather than arbitrary text lengths.

26 Neo4j Inc. All rights reserved 2023

 Logging and Visualizing Conversations
Logging conversations helped us see how users were interacting with Agent Neo and
which grounding text was being used.

Graph of an actual conversation between an Agent Neo user and the ChatGPT-4 LLM.
Context Documents are labeled with their GDS Community.
27 Neo4j Inc. All rights reserved 2023
 Advanced RAG Strategies

Parent-Child: Subset larger text chunks into smaller chunks for initial matching, then
retrieve the larger chunk for response generation.

Topic Summaries: Identify semantically-similar clusters of text in the grounding

database.

Questions: Use our question logging to identify repeat questions and the grounding
data that was retrieved to produce answers. Also use an LLM to generate additional
questions from the grounding data.

28 Neo4j Inc. All rights reserved 2023

 Basic Data Model

29 Neo4j Inc. All rights reserved 2023

 Parent - Child

30 Neo4j Inc. All rights reserved 2023

 Question Matching

31 Neo4j Inc. All rights reserved 2023

 Topic Summaries

32 Neo4j Inc. All rights reserved 2023

 Advanced Graph Model Emerges

Enhanced
Grounding Data

LLM Conversation

33 Neo4j Inc. All rights reserved 2023

 Logging and Evaluation

34 Neo4j Inc. All rights reserved 2023

 Importance of Logging and Visibility
Existing highly-regulated industries (i.e., finance, life sciences, healthcare, etc…)

AI-Specific Regulations like the EU AI Act and potential regulations in the US and
elsewhere

Critical for production environments where the GenAI app responses may have
real-world implications

Be able to understand how the LLM generated answers (good or bad)

Identify areas for performance improvements

35 Neo4j Inc. All rights reserved 2023

 LangSmith
● LangSmith is one of the available
tools to log and analyze GenAI App
performance.
● Developed by the team behind
LangChain, but is platform-agnostic.
● Provides extensive logs for each step
of a query.
● Can analyze using LangSmith UI or
query the logs into a Notebook (ref:
LangSmith Cookbook Repo).

36 Neo4j Inc. All rights reserved 2023

 Latency can be an issue
LLM 1 LLM 2 LLM 3

Distribution of Latency (seconds) by LLM

37 Neo4j Inc. All rights reserved 2023

 Token count varies widely
● Users can specify the count of
grounding documents (from 0 to 10)
so variation is expected
● While models now have much larger
context windows, more grounding
data is not always better
● Efficiency, cost, and latency all need
to be considered when deploying a
GenAI app

38 Neo4j Inc. All rights reserved 2023

 Potential LangSmith Data as a Graph

39 Neo4j Inc. All rights reserved 2023

 Conclusion

40 Neo4j Inc. All rights reserved 2023

 Key Takeaways

Work toward building a viable app for today and tomorrow

Real-world usage is critical to inform how you evolve your app

Neo4j knowledge graphs are flexible elements of the architecture:

● Accurate, complete grounding

● Logging usage
● Visibility into LLM responses

Build in a logging function to understand app behavior

41 Neo4j Inc. All rights reserved 2023

 Resources
Neo4j GenAI Ecosystem Page:
https://neo4j.com/labs/genai-ecosystem/

Needle Starter Kit:

https://neo4j.com/labs/neo4j-needle-starterkit/

Neo4j Developer Blog:

https://neo4j.com/developer-blog/

Going Meta YouTube Series:

https://www.youtube.com/@neo4j

42 Neo4j Inc. All rights reserved 2023

 Thank you for attending!

43 Neo4j Inc. All rights reserved 2023