Dynamic Multi-Agent Orchestration and Retrieval For Multi-Source Question-Answer Systems Using Large Language Models
Dynamic Multi-Agent Orchestration and Retrieval For Multi-Source Question-Answer Systems Using Large Language Models
6, December2024
1 Introduction
In recent years, the rapid evolution of Large Language Models (LLMs) has led to sig-
nificant advancements in the fields of information retrieval and question-answer (Q&A)
systems. These advanced models have proven capable of understanding and generating
human-like text, offering new possibilities for retrieving precise and contextually relevant
information from diverse sources. However, despite these advancements, challenges remain
when integrating data from heterogeneous sources—such as unstructured text documents,
structured databases, and real-time APIs—into a single system. Traditional systems often
struggle to handle the complexity of retrieving and correlating information across different
formats, leading to issues with accuracy and relevance in responses. This gap underscores
the need for more sophisticated techniques that can dynamically orchestrate and retrieve
information from multiple sources, while maintaining the high accuracy and contextual
awareness that LLMs offer.
In many industries, professionals are required to navigate vast volumes of text-based
documents while simultaneously accessing structured data from databases or other sys-
tems. This process is not only labor-intensive but also time-consuming, as locating specific
pieces of information and correlating them across different sources can be very difficult.
For instance, in domains like Contract Management, retrieving relevant details from both
contract documents and database records can often require manually searching through
hundreds of pages and cross-referencing these with structured metadata—an arduous and
error-prone task.
Bibhu Dash et al: NLAICSE, NLAII, IOTSEC, AIMDS 2024 DOI:10.5121/ijci.2024.130602
pp. 11-30, 2024. IJCI – 2024
International Journal on Cybernetics & Informatics (IJCI) Vol.13, No.6, December2024
To address these challenges, we propose a dynamic multi-agent orchestration and re-
trieval methodology aimed at improving the accuracy of multi-source Q&A systems using
Large Language Models. By combining advanced retrieval-augmented generation (RAG),
text-to-SQL techniques, and dynamic prompt engineering, we enable the system to handle
complex queries across heterogeneous data sources, improving response precision without
the need to retrain the model. At the heart of this approach lies an agent-based architec-
ture that dynamically orchestrates different retrieval strategies based on the nature of the
user query, ensuring optimal data retrieval from multiple sources.
In this paper, we evaluate our approach using the domain of contracts, incorporat-
ing qualitative feedback from users who tested the system. Contract Management systems
often involve retrieving specific data from contract documents (e.g., penalties, SLAs, dead-
lines) as well as structured data from databases. While existing systems can handle basic
information retrieval tasks, they typically struggle when required to provide detailed an-
swers that integrate information from multiple sources. Our proposed system leverages
specialized agents—such as SQL agents, RAG agents, and router agents—to route and
execute the queries to the most appropriate source, thereby offering more comprehensive
and context-aware responses.
Additionally, we introduce dynamic prompt engineering, which adapts the prompt’ in-
structions in real-time, based on the context of the query, the type of data being retrieved,
and the user’s input. This ensures that the language model’s responses are accurate, con-
textual, and optimized for each query’s specific requirements, whether it’s retrieving in-
formation from a structured database or extracting text from an unstructured document.
The paper is organized as follows: Section 2 provides technical background on agents
orchestration and retrieval techniques for LLMs, like RAG, text-to-SQL, and Prompt En-
gineering. Section 3 discusses our methodology and the use of the presented techniques,
while Section 4 describes how we evaluated the proposed methodology and the experi-
mentation of the Q&A application. Finally, Section 5 concludes our study and proposes
directions for future research in this field.
2 Background
2.3 Text-to-SQL
Text-to-SQL is a powerful technique that bridges the gap between natural language queries
and relational database systems by converting user inputs in plain text into executable SQL
commands. This allows users to retrieve precise, structured information from databases
without needing to understand SQL syntax [Liu et al., 2023]. By leveraging the capabilities
of LLMs, Text-to-SQL systems can parse and interpret natural language questions and
map them to the appropriate database schema, significantly improving the accessibility of
data for non-expert users.
A key advantage of Text-to-SQL systems is their ability to handle complex database
queries while shielding users from the intricacies of database schemas and SQL commands.
For instance, as discussed by [Pinheiro et al., 2023], LLMs can be used to construct nat-
ural language database (conversational) interfaces. They do this detecting entities, map-
ping them to corresponding tables and columns, and generating syntactically correct SQL
queries based on the database structure. This approach is particularly useful in domains
where the underlying data is stored in complex databases, such as contract management
systems or healthcare databases, where queries may involve multiple tables and relation-
ships.
According to [Seabra et al., 2024], the main distinction between RAG and text-to-SQL
techniques lies in their approach to retrieving information. RAG focuses on retrieving text
segments from a vectorstore that are semantically similar to the user’s question, and it
uses these segments to generate a coherent and contextually appropriate answer. This
approach is well-suited for questions where the answer can be synthesized from existing
unstructured text. However, it may not always provide the precise information expected
if the answer cannot be directly inferred from the retrieved text segments. On the other
hand, Text-to-SQL translates natural language queries into SQL commands, as demon-
strated in [Pinheiro et al., 2023], which are then executed against a structured database
14
International Journal on Cybernetics & Informatics (IJCI) Vol.13, No.6, December2024
to return exact data matches. This ensures that when the text-to-SQL translation is ac-
curate, the user receives a highly specific, structured answer derived directly from the
relevant database fields.
Therefore, while RAG operates on the principle of textual similarity and utilizes gen-
erative capabilities to synthesize responses from retrieved text, Text-to-SQL provides a
more direct and precise mechanism for data retrieval. By translating natural language
queries into executable SQL commands, Text-to-SQL allows for exact matches based on
the user’s intent, retrieving highly specific information directly from structured databases.
This makes Text-to-SQL particularly effective for data investigations where precise, query-
based access to relational data is crucial, such as financial reports, contract details, or
inventory systems. Unlike RAG, which depends on finding semantically similar text, Text-
to-SQL guarantees an exact match from database fields, ensuring that the user receives
accurate, factual answers without ambiguity. As a result, it is a valuable tool in scenarios
where precision and structure are paramount, complementing the generative and flexible
nature of RAG for a more comprehensive information retrieval system.
accuracy of responses is improved. For example, by using instructions like “Do not use prior
knowledge”, prompt engineering can restrict the LLM’s answer to specific sources, thereby
reducing the risk of factual hallucinations and ensuring that responses are grounded in
the desired information.
Recent studies have begun to explore the synergistic integration of these techniques
with LLMs to create more sophisticated Q&A systems. For example, [Jeong, 2023] rein-
forces the importance of using Prompt Engineering with RAG to improve the retrieval
of relevant documents, which are then used to generate both contextually relevant and
information-rich answers. Similarly, [Gao et al., 2023a] explores the integration of Text-
to-SQL with Prompt Engineering to enhance the model’s ability to interact directly with
relational databases, thereby expanding the scope of queries that can be answered accu-
rately.
2.5 Agents
3 Our Methodology
In designing our multi-source question-answer methodology, we employ a combination
of advanced techniques to access diverse data sources and provide accurate responses tai-
lored to the query and the specific source of information, integrating Retrieval-Augmented
Generation (RAG), Text-to-SQL, Dynamic Prompt Engineering, and Agent-based orches-
tration to effectively manage the complexities of interacting with both structured and
unstructured data sources. Each component plays a critical role in handling various as-
pects of information retrieval, ensuring that the system can dynamically adapt to the
requirements of each query.
RAG enables the retrieval of relevant information from large volumes of unstructured
text, while Text-to-SQL facilitates precise access to structured data within relational
databases. Dynamic Prompt Engineering customizes the query context, ensuring that re-
sponses are tailored to user intent, and Agent-based orchestration coordinates these tech-
niques, directing queries to the appropriate modules and managing workflows seamlessly.
17
International Journal on Cybernetics & Informatics (IJCI) Vol.13, No.6, December2024
In this section, we detail the approaches and challenges associated with implementing each
of these techniques, along with the strategies we used to optimize their integration.
Our methodology was implemented and tested in a real-world project called Con-
trato360 [Seabra et al., 2024], a question-answer system designed specifically for Contract
Management. Contrato360 leverages the combined techniques of Retrieval-Augmented
Generation (RAG), Text-to-SQL, Dynamic Prompt Engineering, and Agent-based orches-
tration to address the unique challenges of navigating and retrieving information from
complex contract documents and structured databases. By integrating these advanced
methods, Contrato360 enables users to efficiently query contract-related data, such as
penalty clauses, deadlines, and contractual obligations, across diverse sources. This prac-
tical application demonstrates the effectiveness of our methodology in a domain where
accuracy, relevance, and contextual understanding are critical.
Chunking strategy One of the first decisions to be made when applying RAG is to
choose the best strategy to segment the document, that is, how to perform the chunking
of the PDF files. A common chunking strategy involves segmenting documents based on
a specific number of tokens and an overlap (overlap). This is useful when dealing with
sequential texts where it is important to maintain the continuity of the context between
the chunks.
There is a common type of document with well-defined sections; contracts are a prime
example. The have a standardized textual structure, organized into contractual sections.
Therefore, sections with the same numbering or in the same vicinity describe the same
contractual aspect, that is, they have similar semantics. For example, in the first section
of contract documents, we always find the object of the contract. In this scenario, we can
assume that the best chunking strategy is to separate the chunks by section of the docu-
ment. In this case, the overlap between the chunks occurs by section, since the questions
will be answered by information contained in the section itself or in previous or subsequent
sections. For the contract page in the example in Figure ??, we would have a chunk for
the section on the object of the contract, another chunk for the section on the term of
the contract, that is, a chunk for each clause of the contract and its surroundings. This
approach ensures that each snippet represents a semantic unit, making retrievals more
accurate and aligned with queries.
Using predefined sections as the boundaries for chunks enhances the relevance of re-
sponses within a single contract. However, this approach presents two main challenges: (1)
18
International Journal on Cybernetics & Informatics (IJCI) Vol.13, No.6, December2024
within a single document, when a term appears repeatedly, it can be difficult to identify
the specific chunk that answers a question; and (2) as the number of documents increases,
accurately selecting the correct document to address becomes more challenging for the
system. In the Contract Management domain, consider a scenario where the user asks,
”Who is the contract manager of contract number 123/2024?”. This query is intended to
retrieve the specific name of the contract manager for the given contract. However, the
term “contract manager” can appear in various clauses of the contract document, often
in sections that do not contain the name of the actual manager but refer to responsibili-
ties or general rules related to contract management. For instance, multiple clauses across
different sections of the contract might mention the term ”contract manager” in contexts
like assigning responsibilities, explaining the duties of a manager, or defining roles in con-
tract supervision. Even though these clauses contain the term ”contract manager,” they
do not answer the user’s question, which is specifically asking for the name of the contract
manager for contract 123/2024.
Due to the similarity between
the query and these irrelevant sec-
tions, the Retrieval-Augmented
Generation (RAG) system may
retrieve a chunk from one of
these irrelevant clauses that does
not actually contain the required
name. For example, instead of re-
trieving the clause that explic-
itly names the contract manager,
the system might retrieve a clause
that discusses the general duties
of a contract manager. This hap-
pens because the chunk embed-
ding for a clause about the role
or responsibilities of the manager
may be semantically similar to the
query, even though it lacks the
Fig. 2. Chunking based on Contract’s clauses
specific information requested. In
this case, the chunk retrieved is re-
lated to the term ”contract manager” but does not include the answer the user expects.
As a result, the system could return an incorrect response, such as a general description
of the role of a contract manager, rather than identifying the actual manager for contract
123/2024. This illustrates the challenge of relying solely on textual similarity in chunk re-
trieval, as it can lead to the retrieval of information that is similar to the query in wording
but not relevant to the specific context of the user’s question. To mitigate this, additional
filtering mechanisms, such as metadata checks or contract-specific identifiers, are required
to ensure that the system retrieves the most contextually appropriate information from
the correct contract section.
To overcome this issue, several strategies can be applied. One approach is to add
metadata to the chunks and, when accessing the vectorstore, use this metadata to filter
the information returned. This method improves the relevance of the retrieved texts by
narrowing the search to only those chunks that match specific metadata criteria. Figure ??
displays the most relevant metadata attributes for the contracts: source, contract, and
clause. Here, source represents the name of the contract’s PDF file, contract refers to
19
International Journal on Cybernetics & Informatics (IJCI) Vol.13, No.6, December2024
the contract number, and clause indicates the section title. For instance, when querying,
”Who is the contract manager of contract 123/2024?” the system first filters for chunks
that belong to contract number 123/2024 and clauses related to the contract manager.
Once these chunks are filtered, a similarity calculation is applied to identify the most
relevant text segments, which are then sent to the LLM to generate the final response.
to the query. With 1536 dimensions, the embeddings can better represent the complex
relationships between terms in the text, especially in documents where meaning often
depends on subtle distinctions in wording. This is particularly useful in distinguishing
between similar but contextually different terms, such as contract manager vs. contract
supervisor, ensuring that the system retrieves the most relevant chunks.
Vectorstore The need to store and query high-dimensional vectors efficiently has led
to the development of specialized vector databases, also known as vectorstores. These
databases allow for the storage and retrieval of vector embeddings, making it possible
to perform similarity searches - a key operation in tasks such as Retrieval-Augmented
Generation (RAG) and semantic search. Unlike traditional databases that are optimized
for structured, tabular data, vector databases are designed to handle embeddings generated
by models like text-davinci-002, which represent semantic relationships in high-dimensional
space.
When choosing the right vector database for a project, several factors come into play,
including scalability, ease of use, latency, and integration with machine learning models.
In our work, we evaluated three popular vector databases: Pinecone, Weaviate, and Chro-
maDB. Pinecone is a cloud-native vector database that excels in providing a fully managed
service for high-performance similarity search. Weaviate is an open-source vector database
that provides a highly flexible, schema-based approach to storing and querying vectors
alongside structured metadata. ChromaDB is an open-source, lightweight vector database
that focuses on simplicity and tight integration with machine learning workflows, making
it ideal for embedding-based retrieval tasks in research and smaller projects. Our choice
was the last one, specially because ChromaDB is easy to set up and integrate into a project
without requiring extensive configuration or overhead. Given that our system is heavily
Python-based, ChromaDB’s Python-first design allowed us to quickly embed it into our
machine learning pipelines. This streamlined our development process, enabling rapid it-
eration and testing, which was especially important in the early stages of system design.
Also, by using ChromaDB, we can directly connect our text-davinci-002 embeddings with
the vectorstore, enabling efficient similarity searches and accurate retrieval of contextually
relevant information.
important as the direction. Euclidean distance is the most common metric for measuring
the straight-line distance between two points (or vectors) in a multi-dimensional space. It
calculates the “as-the-crow-flies” distance between two vectors, treating each dimension
as an axis in a Cartesian plane. Euclidean distance is widely used in geometric tasks or
where the actual distance between points matters. Manhattan distance, also known as
L1 distance or taxicab distance, measures the sum of the absolute differences between
the corresponding coordinates of two vectors. Instead of measuring the direct straight-line
distance (as in Euclidean), Manhattan distance measures how far one would have to travel
along the axes of the space.
In our work, we chose cosine similarity for its ability to prioritize semantic align-
ment between query embeddings and document embeddings. Its strength in handling
high-dimensional data, minimizing the influence of vector magnitude, and focusing on
the directionality of vectors makes it the ideal choice for our Q&A system methodology.
Cosine similarity is widely recognized as one of the best similarity measures for text-based
applications, especially when using vector embeddings generated from NLP models like
text-davinci-002. Since our system heavily relies on textual data, cosine similarity was the
natural choice for ensuring that user queries are matched with the most relevant sections
of the text, even if the exact phrasing differs. Whether we are retrieving specific sections
in documents or providing general answers based on lenghty documents, cosine similarity
ensures that the system is aligned with the semantic intent of the query.
3.3 Agents
Agents are central to the functionality and adaptability of our multi-source question-
answer system, enabling it to handle diverse query types efficiently. By leveraging spe-
cialized agents, the system dynamically routes each query to the most suitable processing
pathway, ensuring that user questions are handled with precision and contextual relevance.
In our architecture, the Router Agent serves as the primary decision-maker, evaluating
each incoming query and directing it to the appropriate agent based on predefined criteria.
The Router Agent uses regular expressions to identify keywords, patterns, or struc-
tures within the query. If the query is specific to a clause within a contract, the Router
Agent recognizes this pattern and assigns the query to the RAG Agent. The RAG Agent
is optimized for handling unstructured text data, retrieving relevant text chunks from
the vectorstore. By focusing on textual similarity, the RAG Agent retrieves semantically
aligned information and generates responses that incorporate precise, contextually relevant
excerpts from the documents, addressing the specifics of the the user’s question.
Conversely, if the Router Agent detects that the question involves broader contract
information, such as dates, financial details, or other exact values, it directs the query to the
SQL Agent. The SQL Agent translates the natural language question into a structured SQL
query, which is then executed against the database to retrieve exact data. This approach
is particularly effective for queries requiring precise, structured responses, ensuring that
the system provides accurate and up-to-date information directly from the database.
This dynamic agent-based architecture enables our system to handle both unstructured
and structured data seamlessly. The Router Agent’s decision-making process allows the
system to optimize query processing based on the context and specific needs of each query.
By directing contract-specific questions to the RAG Agent and structured data queries
to the SQL Agent, the Router Agent ensures that user questions are handled efficiently,
providing relevant answers whether they require interpretive text or exact data values.
23
International Journal on Cybernetics & Informatics (IJCI) Vol.13, No.6, December2024
This modular design not only improves response accuracy but also enhances the system’s
flexibility in adapting to a wide range of contract-related queries.
4 Evaluation
The architecture depicted in the figure represents the implementation of our multi-source
question-answer methodology, combining structured and unstructured data from con-
24
International Journal on Cybernetics & Informatics (IJCI) Vol.13, No.6, December2024
tracts. The system is built using a modular approach, where each component plays a
critical role in the data retrieval and response generation process. At the core of the archi-
tecture is the User Interface, built with Streamlit, as shown in figure 8, which allows users
to input their queries and view responses in a user-friendly interface. Users can submit
both broad questions or specific contract-related queries, which are then processed by the
backend system.
The Backend Agents act as the decision-making layer of the system, handling queries
based on their type and content. These agents include the Router Agent, which determines
whether to route the query to the RAG Agent (for unstructured text retrieval) or the
SQL Agent (for structured data queries using Text-to-SQL). The agents communicate
bidirectionally with the user interface, allowing for interactive feedback during the query
resolution process.
For the unstructured data flow, contract documents in PDF format undergo processing
in the PDF Documents Processing component. This involves extracting text and metadata
from the documents, which is then passed to the Chunking and Metadata Generation
module. This module divides the documents into manageable chunks, enriching them with
metadata for easier retrieval. These chunks are further processed through the Embeddings
Generation component, where each chunk is transformed into a high-dimensional vector
representation using an embedding model. These embeddings are stored in the Vectorstore
(implemented using ChromaDB) for efficient similarity search during retrieval.
On the structured data side, the Contracts Database (implemented using SQLite)
stores relevant contract data such as specific terms, clauses, dates, and financial informa-
tion. When a query requires precise data retrieval, such as asking for contract values or
deadlines, the SQL Agent retrieves the necessary information directly from this database.
By integrating both the vectorstore and structured database, the Backend Agents can
provide comprehensive answers to user queries, dynamically choosing the most appropriate
data source based on the type of question. This hybrid approach ensures that the system
can handle both semantically complex queries and direct database queries, offering flexible
and accurate responses.
The system was evaluated through experiments conducted with two IT contract spe-
cialists from BNDES, who validated its performance using a set of 75 contracts. These
25
International Journal on Cybernetics & Informatics (IJCI) Vol.13, No.6, December2024
contracts, including both PDFs and associated metadata, were processed to assess the
system’s ability to retrieve relevant information from both unstructured documents and
structured data. To evaluate the system’s effectiveness in answering various query types, a
set of benchmark questions was developed, divided into two categories: direct and indirect
questions.
Direct questions refer to those that could be answered using information directly avail-
able in the contract PDFs and their metadata. Examples include questions about contract
subjects, suppliers, managers, and contract terms. The results demonstrated that for these
direct questions, the system consistently provided complete and relevant responses, meet-
ing the users’ expectations for accuracy and comprehensiveness.
Indirect questions, however, required information that would yield better relevance
when retrieved from the database. Examples include questions about the number of ac-
tive contracts, upcoming contract expirations, and specific details regarding exemptions
from tender processes. The results for these indirect questions were generally satisfactory,
although in certain cases, such as questions about contract inflexibility and exemptions,
the answers provided were marked as incomplete. This is likely due to the more com-
plex semantics of the terms involved. For example, the term ”Waiver of Bidding” proved
challenging for the system, as its meaning was not fully captured in the retrieval pro-
cess. Adjustments to the prompts or query structure are expected to improve the system’s
ability to interpret and respond accurately to these nuanced questions.
26
International Journal on Cybernetics & Informatics (IJCI) Vol.13, No.6, December2024
User feedback highlighted that one of the system’s most valuable features is its ability
to seamlessly integrate information from both the structured data store and the unstruc-
tured text in contracts. This feature significantly reduces the time users spend locating
and accessing relevant contract data, as they would typically need to identify the con-
tracts, open the PDFs, and manually search for information. For instance, the system
efficiently retrieves answers regarding contract managers and outlines any penalties re-
lated to contractual non-compliance, eliminating the need for users to sift through lengthy
documents. By directly addressing questions with specific details, the system enhances the
user experience, providing critical information quickly and effectively.
Additionally, users appreciated the system’s capacity to automatically generate visual
summaries through its Plotly agent when a table of values was included in the response.
This feature was positively received, as it not only provides immediate visual insights but
also supports users in preparing professional presentations. By integrating dynamic graph
generation directly into the response process, the system offers users a more comprehen-
sive analytical experience, enabling clearer communication and a deeper understanding of
contract-related data.
References
[Chen et al., 2024] Chen, J., Lin, H., Han, X., and Sun, L. (2024). Benchmarking large language models
in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence,
volume 38, pages 17754–17762.
[Feng et al., 2024] Feng, Z., Feng, X., Zhao, D., Yang, M., and Qin, B. (2024). Retrieval-generation synergy
augmented large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pages 11661–11665. IEEE.
[Gao et al., 2023a] Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., and Zhou, J. (2023a). Text-to-sql
empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308.15363.
[Gao et al., 2023b] Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang,
H. (2023b). Retrieval-augmented generation for large language models: A survey. arXiv preprint
arXiv:2312.10997.
[Giray, 2023] Giray, L. (2023). Prompt engineering with chatgpt: a guide for academic writers. Annals of
biomedical engineering, 51(12):2629–2633.
[Jeong, 2023] Jeong, C. (2023). A study on the implementation of generative ai services using an enterprise
data-based llm application architecture. arXiv preprint arXiv:2309.01105.
[Jin et al., 2024] Jin, H., Huang, L., Cai, H., Yan, J., Li, B., and Chen, H. (2024). From llms to llm-
based agents for software engineering: A survey of current, challenges and future. arXiv preprint
arXiv:2408.02479.
[Lewis et al., 2020] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H.,
Lewis, M., Yih, W.-t., Rocktäschel, T., et al. (2020). Retrieval-augmented generation for knowledge-
intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.
[Liu et al., 2023] Liu, A., Hu, X., Wen, L., and Yu, P. S. (2023). A comprehensive evaluation of chatgpt’s
zero-shot text-to-sql capability. arXiv preprint arXiv:2303.13547.
[Mialon et al., 2023] Mialon, G., Dessı̀, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R.,
Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023). Augmented language models:
a survey. arXiv preprint arXiv:2302.07842.
[OpenAI, 2023a] OpenAI (2023a). Chatgpt fine-tune description. https://help.openai.com/en/
articles/6783457-what-is-chatgpt. Accessed: 2024-03-01.
[OpenAI, 2023b] OpenAI (2023b). Chatgpt prompt engineering. https://platform.openai.com/docs/
guides/prompt-engineering. Accessed: 2024-04-01.
[Pinheiro et al., 2023] Pinheiro, J., Victorio, W., Nascimento, E., Seabra, A., Izquierdo, Y., Garcıa, G.,
Coelho, G., Lemos, M., Leme, L. A. P. P., Furtado, A., et al. (2023). On the construction of database
interfaces based on large language models. In Proceedings of the 19th International Conference on Web
Information Systems and Technologies - Volume 1: WEBIST, pages 373–380. INSTICC, SciTePress.
[Seabra et al., 2024] Seabra, A., Nepomuceno, J., Lago, L., Ruberg, N., and Lifschitz, S. (2024). Con-
trato360: uma aplicação de perguntas e respostas usando modelos de linguagem, documentos e bancos
de dados. In Anais do XXXIX Simpósio Brasileiro de Bancos de Dados.
29
International Journal on Cybernetics & Informatics (IJCI) Vol.13, No.6, December2024
[Singh et al., 2024] Singh, A., Ehtesham, A., Kumar, S., and Khoei, T. T. (2024). Enhancing ai systems
with agentic workflows patterns in large language model. In 2024 IEEE World AI IoT Congress (AIIoT),
pages 527–532. IEEE.
[Wang et al., 2023] Wang, M., Wang, M., Xu, X., Yang, L., Cai, D., and Yin, M. (2023). Unleashing
chatgpt’s power: A case study on optimizing information retrieval in flipped classrooms via prompt
engineering. IEEE Transactions on Learning Technologies.
[White et al., 2023] White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-
Smith, J., and Schmidt, D. C. (2023). A prompt pattern catalog to enhance prompt engineering with
chatgpt. arXiv preprint arXiv:2302.11382.
Authors
João Nepomuceno received his Bachelor’s Degree in Physics from Universidade Fed-
eral Fluminense, Brazil, and he is currently pursuing his Bachelor’s Degree in Computer
Science at Universidade Federal Fluminense, Brazil. His research interests include Data
Engineering, Artificial intelligence and Natural Language Processing.
Lucas Lago is currently pursuing his Bachelor’s Degree in Computer Science at Uni-
versidade do Estado do Rio de Janeiro, Brazil. His research interests include Artificial
Intelligence and Natural Language Processing.
30