Enhancing Text-to-SQL Capabilities of Large Language Models:

A Study on Prompt Design Strategies

Linyong Nan1 Yilun Zhao1 Weijin Zou1 Narutatsu Ri2 Jaesung Tae1
Ellen Zhang1 Arman Cohan1,3 Dragomir Radev1
1
Yale University 2 Columbia University 3 Allen Institute for AI
{linyong.nan, yilun.zhao}@yale.edu

Abstract systems for two main reasons. First, neural mod-

els have inherent limitations, including a limited
In-context learning (ICL) has emerged as a working memory that is costly to access during in-
new approach to various natural language pro- ference and a long-term memory that is unreliable
cessing tasks, utilizing large language models to read from or write to, making it impractical to
(LLMs) to make predictions based on context
have them directly read from large-scale knowl-
that has been supplemented with a few exam-
ples or task-specific instructions. In this paper, edge sources. Second, understanding how a system
we aim to extend this method to question an- decides which information to retrieve and how to
swering tasks that utilize structured knowledge aggregate it is crucial for assessing its reliability
sources, and improve Text-to-SQL systems by and robustness.
exploring various prompt design strategies for Recent investigations have demonstrated the ef-
employing LLMs. We conduct a systematic
fectiveness of the neural symbolic approach in pro-
investigation into different demonstration se-
lection methods and optimal instruction for- ducing transparent reasoning process in formal lan-
mats for prompting LLMs in the Text-to-SQL guage sequence (such as Text-to-SQL) for question
task. Our approach involves leveraging the syn- answering tasks based on databases or knowledge
tactic structure of an example’s SQL query to graphs (Berant et al., 2013; Zhong et al., 2017; Yu
retrieve demonstrations, and we demonstrate et al., 2018; Yin and Neubig, 2018; Yu et al., 2019;
that pursuing both diversity and similarity in Ren et al., 2021; Cheng et al., 2023). A typical sys-
demonstration selection leads to enhanced per-
tem comprises a neural semantic parsing module
formance. Furthermore, we show that LLMs
benefit from database-related knowledge aug- that translates user queries in natural language to
mentations. Our most effective strategy outper- formal language sequences (e.g., logical forms or
forms the state-of-the-art system by 2.5 points executable code) and a symbolic reasoner module,
(Execution Accuracy) and the best fine-tuned such as database management system (DBMS), that
system by 5.1 points on the Spider dataset. executes the code on structured knowledge sources
These results highlight the effectiveness of our to extract the result. The primary objective of this
approach in adapting LLMs to the Text-to-SQL work is to improve the semantic parsing module, as
task, and we present an analysis of the factors
it is essential in extracting answers from relational
contributing to the success of our strategy.
databases using SQL as the formal language.
1 Introduction Current semantic parsing modules can be
broadly categorized based on their learning strate-
Question answering using structured knowledge gies. State-of-the-art systems involve fine-tuning
source is a critical function of information retrieval a pretrained language models on a large corpus
systems that act as an interface between humans of {question, SQL} pairs, enabling the model to
and vast structured data repositories. Extracting generate code (Wang et al., 2020; Yin et al., 2020;
and aggregating information accurately is a funda- Scholak et al., 2021; Xie et al., 2022; Li et al.,
mental requirement of these systems and is thus 2023). Alternatively, the in-context learning (ICL)
a primary goal in their design. In recent years, approach exploits the inherent capabilities of large
the neural symbolic design approach (Berant et al., language models (LLMs) to directly produce SQL
2013; Yao and Van Durme, 2014; Liang et al., 2017; code by providing a well-defined task prompt (Xie
Gardner et al., 2018; Yu et al., 2018; Cheng et al., et al., 2022; Chen et al., 2022; Rajkumar et al.,
2023) has become the preferred choice for such 2022; Ni et al., 2023). Existing research indicates
14935
Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14935–14956
December 6-10, 2023 ©2023 Association for Computational Linguistics
that LLMs using prompt-based semantic parsing strations from a pool of annotated examples for
underperform their fine-tuned counterparts (Liu each test instance is another critical design choice
et al., 2023), while recent studies also suggest that that can impact the ICL performance. We proposed
performance of ICL-trained LLMs is significantly enhancements for each of these components and
affected by the structure of the prompt (Liu et al., evaluated them against existing methods.
2022; Rubin et al., 2022; Lu et al., 2022; Wei et al.,
2022; Fu et al., 2023; Ye et al., 2023). This moti- 2.1 Demonstration Selection
vates us to examine various prompt configurations The goal is to select a subset of annotated examples
for semantic parsing tasks, taking advantage of the from a pool that offers the best context for solving
latest advancements of LLMs pertaining to our do- the test problem. While random selection from
main of interest. the pool is one option, Liu et al. (2022) proposed
Our study focused on exploring various prompt kNN-augmented example selection (KATE), which
design strategies for semantic parsing tasks in the retrieves k nearest neighbors from the pool based
Text-to-SQL domain. We conducted a systematic on the input of the compared instances. To achieve
investigation into different demonstration example this, all the pool instances are first transformed
selection criteria and instruction formats on Text-to- into continuous vectors using a sentence encoder.
SQL datasets. Specifically, we propose to employ During inference, the input of a test instance is
an example’s SQL syntactic structure as the basis projected into a latent space using the same encoder
for retrieving demonstrations, thereby facilitating a and then compared to the pool of vectors using
more accurate representation of the problem struc- a similarity measure, such as negative Euclidean
ture. Our approach revealed that selecting demon- distance or cosine similarity. Finally, the top k most
stration examples with a dual emphasis on diversity similar annotated examples are selected from the
and similarity objectives yields maximized gain in pool.
performance. Our study also showed that LLMs
benefit from database-related knowledge augmen- Structured Prediction as Basis for Retrieval
tation in certain circumstances. Through experi- We propose utilizing the output SQL queries to
ments, we identified the most effective strategy, select the demonstration examples, rather than us-
which resulted in an Execution Accuracy score of ing the input questions. This is because, unlike
84.4 on the Spider dataset (Yu et al., 2018). This many tasks where the output is a classification la-
score is 2.5 points higher than the state-of-the-art bel or extracted entity with little information about
system (Ni et al., 2023) and 5.1 points higher than the problem structure, Text-to-SQL demands struc-
the best fine-tuned system (Scholak et al., 2021) tured prediction which contains more explicit in-
at the time of writing.1 These results demonstrate formation about the problem structure than that
the effectiveness of our in-context learning scheme provided in the input question. Furthermore, un-
in adapting LLMs to our target task. Furthermore, like natural language questions that can only be
we present the empirical findings and analysis on converted into continuous semantic vectors, SQL
the factors that contributed to the success of our queries can be easily transformed into discrete fea-
strategy.2 ture vectors based on their syntax, making their
comparison more efficient and transparent. To im-
2 Methods plement our proposal, we begin by converting the
SQL queries of all pool instances into discrete syn-
To design prompts for in-context learning in zero-
tax vectors. This is done by parsing the queries and
shot or few-shot settings, it is important to find an
identifying their syntactic elements, including key-
optimal way to represent, augment, and arrange all
words, operators, and identifiers. Each SQL query
resources in the input-output mapping. Addition-
is then mapped to a "Bag-of-Syntactic-Elements"
ally, the task instructions should be formulated to
feature vector, each entry of which indicates the
align with these resources. When few-shot learning
presence of a syntactic element in the query, i.e.,
is employed, the selection of a subset of demon-
we assign 1 (instead of the count) if an element
1
Our comparison focuses on fine-tuning studies that em- is present in the SQL query. During inference,
ployed the standard Transformer architecture without any layer we first generate a draft of the SQL query using
modifications or the inclusion of additional modules.
2
We will open source our code for experiments: https: a preliminary predictor. We then apply the same
//anonymous.url process to convert this draft query into a discrete
14936
vector, which is used to represent the test instance knowledge sources such as databases or tables have
for retrieving demonstration examples. been linearized into a “text” sequence. Building on
previous methods (Rajkumar et al., 2022), we adopt
Balancing Diversity and Similarity We propose
a representation of the database using a “code” se-
a new demonstration selection strategy that differs
quence, specifically the CREATE query employed to
from Liu et al. (2022), which retrieves the most
construct the table initially, as illustrated in listing 1
similar examples with continuous-valued measure-
and 2 of the Appendix. This linearization approach
ments for each test instance. In contrast, our strat-
provides data type information for each column
egy seeks to balance similarity and diversity of
and encompasses all foreign key constraint details
the demonstrations. This is achieved by chang-
within the database. Moreover, we modify other
ing the representation of the given example from
resources in the instructions, such as the question
a continuous-valued vector denoting the question
and example entries in the database, to conform
semantics to a discrete-valued vector that captures
to the code sequence style by appending them as
the SQL syntax. To obtain demonstration examples
comments.
that are similar to the given example, we first split
the pool of annotated examples into disjoint par- Schema-related Knowledge Augmentation
titions that represent different categories. Specifi- The ontology of a database delineates the structure
cally, we use the difficulty-level based categoriza- and semantics of the database by offering defini-
tion derived from the Spider dataset (Yu et al., tions for a set of classes (tables), their attributes
2018), because it is developed strictly based on (columns), and the relationships among them. We
syntactic coverage and structure of a SQL query, initially enhance the semantics of each class and
ensuring that queries satisfying the same condi- attribute by elaborating on their meanings within
tions are grouped into the same category. While the context of the entire database. Specifically,
alternative categorization options may exist, we we employ OpenAI’s gpt-3.5-turbo engine3 to
leave this for exploration in future work. Given generate a natural language definition for each
a test instance, we use a preliminary predictor to column in every table, considering all its values
generate a draft SQL query and, based on its cat- and other columns. We then incorporate these
egory, retrieve candidate examples that belong to definitions into the input either by appending
the relevant partition. Next, to select diverse exam- them as a block comment or inserting them
ples from the candidate partitions, we implement within the CREATE query as inline comments.
k-means clustering on the discrete vectors of exam- Furthermore, we suggest augmenting the repre-
ples, selecting k diverse examples that are closest sentation of the database structure by providing
to each centroid of the cluster. The resulting ex- an Entity-Relationship summary that outlines the
amples exhibit similarity to the test example by connections between tables and specifies how
sharing the same category, yet maintain diversity they can be joined. As depicted in Figure 9 of
in problem structures. These demonstrations are the Appendix, an Entity-Relationship diagram of
then used to construct the prompt. The procedure a database is utilized to enumerate all possible
for our demonstration selection strategy is outlined paths between distinct tables. These paths are
in Algorithm 1 of the appendix. subsequently arranged in descending order based
on their respective lengths. The resulting summary
2.2 Schema Representation in Instruction
has shown to be useful in our experiments for
Instructions are crucial to designing prompts, as test instances where multiple tables need to be
they define the task by clarifying how provided re- combined. Listing 5 further demonstrates our
sources can aid the inference process (Dong et al., augmentations and how we arrange them to
2023). Our primary focus lies in determining the construct the prompt.
optimal way to represent a structured knowledge
source within the instruction and identifying sup- 2.3 Integrated Strategy for Text-to-SQL
plementary resources that can enhance the infer- Upon examination, we found that models trained
ence process. with ICL exhibit sensitivity to the number of
Linearization of Structured Knowledge We be- demonstration examples, resulting in noticeable
gin by altering the linearization of structured knowl- variance in performance across models provided
edge. In prior research (Xie et al., 2022), structured 3
Public API available at https://openai.com/api/.

14937
with various numbers of demonstrations. To estab- • Spider-Syn (Gan et al., 2021a) replaced
lish substantial conclusions when comparing dis- schema-related words in the questions of Spi-
tinct prompting approaches, we present the mean der examples with manually selected syn-
and standard deviation for models sharing identi- onyms that reflect real-world question para-
cal configurations except for the varying number phrases to evaluate the robustness of systems.
of demonstrations. In addition, we employ a ma-
jority vote on these models exhibiting diverse per- • Spider-DK (Gan et al., 2021b) defined five
formances. Specifically, we obtain the execution types of domain knowledge and modified
results of different models’ greedy decoding predic- some Spider examples by adding domain
tions, eliminate those with execution errors by de- knowledge to evaluate the cross-domain gen-
terministic database management system (DBMS), eralization capability of a given system.
and choose the prediction that receives the major- • Spider-Realistic (Deng et al., 2021) removed
ity vote. Alternative integration methods, such as explicit mentions of column names from Spi-
the self-consistency sampling (Wang et al., 2023), der examples to reflect more realistic text-
are also available, but we reserve their exploration table alignment settings, and selected eight ex-
for future research. The comprehensive results are isting Text-to-SQL datasets for cross-domain
available in Figures 10, 11, 12 of the Appendix for evaluation.
reader’s perusal.
We propose the following procedure for con- Model We evaluate different ICL strategies with
structing prompts for the Text-to-SQL task. Given Codex (Chen et al., 2021), a GPT-3 variant that was
a set A of annotated examples, we first establish finetuned on code data on the web and has demon-
a categorization that divides the pool into disjoint strated state-of-the-art performance as the time of
partitions Aα , Aβ , . . . ,, with each partition contain- writing (Ni et al., 2023). Specifically, we use the
ing examples whose SQL queries share a relatively code-davinci-002 engine and present the results
similar syntax structure. Next, we apply the k- of systems with prompts ranging from 1 to 10-shot.
Means strategy detailed in Section 2.1 to obtain Additionally, we report the few-shot results utiliz-
diverse demonstration examples Dj for partition ing the ChatGPT (gpt-3.5-turbo) model. How-
Aj . For each example, the demonstration is con- ever, due to its maximum context length limitation
structed by transforming the database into multi- of 4096, we only obtain results for systems pro-
ple CREATE queries and augmenting with schema- vided with prompts ranging from 1 to 5-shot.4
related knowledge. During inference, we employ a Evaluation Metric We use execution accuracy
preliminary model to generate a draft SQL query, as the evaluation metric for all experiments, which
which is used to determine the problem category measures the percentage of system predictions lead-
and thus the corresponding Dj for building the ing to the gold execution result.
prompt. We obtain multiple predictions using vari-
ous numbers of shots in Dj and perform majority Baselines We compare the following prompting
voting to arrive at the final prediction. Details of strategies for generating SQL queries in few-shot
this approach are shown in Algorithm 2 of the ap- and zero-shot settings.
pendix.
Few-shot
3 Experiments
• Random sampling (R): Select demonstration ex-
3.1 Experimental Settings amples randomly from the pool.
• Similarity sampling (S)
Dataset We conduct comprehensive experiments
• Diversity sampling (D): Select diverse examples
on the following four semantic parsing datasets:
from k-Means clusters of the pool.
• Spider (Yu et al., 2018) is a cross-domain se- • Similarity-Diversity sampling (SD): Select ex-
mantic parsing dataset that contains complex amples based on Algorithm 1.
Text-to-SQL problems. The data originates • SD + schema augmentation (SA): Enhance in-
from 200 databases covering 138 different do- structions with schema knowledge (semantic aug-
mains. We use the 7,000 training data as our mentation or structure augmentation).
pool of annotated examples. 4
Public API available at https://openai.com/api/.

14938
 (a) Few-shot results

(b) Zero-shot results

Figure 1: Few-shot and zero-shot results of Codex for all datasets. In the few-shot setting, error bars indicate
means and standard deviations over performances of systems provided with prompts ranging from 4-shot to 10-shot.
To obtain the error bars for the random sampling approach, we conducted 3 independent runs using different
random seeds. Schema augmentation utilized for the reported results in (a) is structure augmentation - add ontology
summary. In the zero-shot setting, the error bars indicate means and standard deviations over 3 independent runs.
Our results suggest that 1) using similarity and diversity objectives in the sampling process, 2) including schema
representation in instructions, and 3) employing model voting with different shot outcomes both contribute to the
improvement of ICL performance.

• SD + SA + Voting: Integrated strategy described efficacy across multiple datasets. The evaluation
in Algorithm 2. of demonstration sampling strategies in a few-
shot setting testing on code-davinci-002 is illus-
Zero-shot trated in Figure 1a, and more few-shot results of
gpt-3.5-turbo are shown in Figure 2. We com-
• Baseline - DB as text-seq: Standard prompt for
pared different demonstration selection strategies,
Text-to-SQL task, where structured knowledge is
including random selection, k-nearest neighbors
linearized as text sequence.
selection (similarity sampling)5 , k-means selec-
• Baseline - DB as code-seq: Improve instructions
tion (diversity sampling), and our proposed ap-
by linearizing structured knowledge source as
proach, which combines both similarity and di-
multiple SQL CREATE queries.
versity. Moreover, we examined the impact of
• Baseline - DB as code-seq + SA: Enhance in-
augmenting schema representation within the task
structions with schema knowledge.
instructions and assessed the performance of our
3.2 Main Results
5
Due to the deprecation of the Codex API in March 2023,
In this section, we present a comprehensive analy- similarity sampling experiments were only conducted on the
sis of various prompting strategies, assessing their Spider dataset.

14939
integrated strategy. Our findings indicate that em- demonstrations in order to effectively understand
ploying similarity and diversity objectives in the and utilize the provided information. In future
sampling process leads to better performance on study, we will explore better structural schema aug-
average across all datasets. Furthermore, incorpo- mentation that aligns to the zero-shot setting.
rating schema representation within the instructions
enhances performance, and the implementation of 4 Analysis
voting of models with different shot results in a
4.1 Prediction-Syntax based Retrieval
marked improvement in overall performance.
The existing method for selecting demonstrations
relies on the semantic representations of the ques-
tion and the database. We propose an alterna-
tive method specifically for code generation tasks,
which focuses on the syntax of the solution code.
We examined syntax coverage and syntax similarity
of the prompts produced with different strategies.
Syntax coverage is computed by counting the occur-
rence of syntactic elements (keywords, operators,
and identifiers), and dividing it by the total number
of all syntactic elements. Syntax similarity, on the
other hand, is measured by the mean Euclidean
distance between the discrete vector representation
of the predicted SQL and vectors that represent
Figure 2: Few-shot results of gpt-3.5-turbo for Spi-
the gold SQLs of the demonstrations selected. As
der. Error bars indicate means and standard deviations indicated in Table 1 of the appendix, both of these
over performances of systems provided with 1-shot to metrics contribute to the quality of the examples
5-shot prompts. Schema augmentation utilized for the selected. Furthermore, a simple summation of the
reported results is semantic augmentation - add column two measurements suggests a correlation with the
summary as block-comment. system’s performance, as illustrated in Figure 6 of
the appendix. We argue the efficacy of our strat-
The efficacy of schema augmentation is fur- egy through the following rationale: (1) in cases
ther supported by experiments in a zero-shot set- where the pool of annotated examples is limited
ting, as illustrated in Figure 1b. We compared in diversity of the problem structures, certain test
systems using different linearization methods for problems may lack similar examples available for
prompts: one that transforms the database into retrieval; and (2) neither the semantic represen-
a text sequence, and another that uses multiple tation of the question/database nor the distance
CREATE queries to represent the database. The metric inherently support encapsulation and com-
latter method shows noticeable improvement in parison of different problem structures, whereas
performance. We also contrasted two separate SQL syntax provides direct measurement of the
techniques for augmenting schema representation: problem structures. Given these constraints, the
one that adds semantic information to each col- optimal strategy is to select similar examples while
umn within each table, and another that incorpo- ensuring the coverage of as many syntax demon-
rates entity-relationship knowledge into the schema. strations as feasible to mitigate potential failures in
The results suggest that structural augmentation similarity-based retrieval.
(adding ontology summary) brings a slight greater
improvement in the few-shot setting for Codex 4.2 Comparative Analysis of Retrieval
(shown in Figure 5), while semantic augmenta- Methods
tion (adding column summary as block comments) We conducted an examination of various similarity-
proves more beneficial in the zero-shot setting for based retrieval methods and presented a compar-
Codex and also the few-shot setting for ChatGPT ative analysis of their performance in Figure 3.
(gpt-3.5-turbo). We hypothesize that this dif- The primary variable in this investigation was the
ference may arise from the less descriptive nature representation extracted for each example, with a
of structural augmentation, which calls for more focus on extracting and comparing the following
14940
Figure 3: Comparison between various similarity based demonstration selection methods. Q indicates the embedding
model employed to extract representation for the question; D stands for database, and S stands for SQL query.

CodeT5-base (Wang et al., 2021) or CodeBERT-

base (Feng et al., 2020) for encoding databases; (3)
syntactic embeddings of predicted SQL, generated
by either binary coding to indicate the presence
of SQL syntactic elements or by quantifying their
occurrences; and finally, (4) embeddings that en-
code questions, databases and predicted SQL using
text-embedding-ada-002.
The following conclusions can be drawn
about the similarity-based retrieval methods
for Text-to-SQL task: (1) questions alone ef-
Figure 4: Comparison between various diversity based fectively represent distinct examples for re-
demonstration selection methods. trieval purposes; (2) RoBERTa-base provides
better embeddings for comparisons relative to
text-embedding-ada-002; (3) it is feasible to
employ models that have not been fine-tuned
on Text-to-SQL examples for similarity-based re-
trieval, while still achieving comparable perfor-
mance to fine-tuned models; (4) the linearization of
databases as SQL queries facilitates the extraction
of enhanced embeddings.
Figure 5: Comparison between various schema augmen- Additionally, we conducted a comparison be-
tations in few-shot and zero-shot settings. tween multiple embeddings utilized for diversity-
based demonstration selection, encompassing em-
beddings that encode the semantics of questions,
embedding types: (1) question embeddings gener- databases and predicted SQL, as well as embed-
ated by Sentence-BERT (Reimers and Gurevych, dings that capture the syntactic features of pre-
2019)6 , RoBERTa-base (Liu et al., 2020), and Ope- dicted SQL. As depicted in Figure 4, the syntactic
nAI’s text-embedding-ada-002; (2) combined embeddings of predicted SQL serve as the most
question and database embeddings obtained by effective basis for contrasting different examples
(i) employing a single model (i.e., T5-base (Raf- for diversity-based retrieval purposes.
fel et al., 2020) finetuned on the Spider train-
ing split and text-embedding-ada-002) with the 4.3 Schema Augmentation
database linearized as a text sequence or CREATE
queries, and (ii) utilizing separate models, specif- Figure 5 presents the outcomes of various schema
ically RoBERTa-base for encoding questions and augmentations applied to the instruction. It is ob-
served that improvement is not apparent in the few-
6
HuggingFace model name: all-MiniLM-L6-V2 shot setting; however, in the zero-shot setting, the
14941
semantic augmentation incorporating descriptions aspect of enhancing model performance. Several
of all table columns proves to be beneficial. studies (Sorensen et al., 2022; Gonen et al., 2022;
Wu et al., 2022; Hu et al., 2022; Lu et al., 2022)
4.4 Effectiveness Analysis have proposed metrics to measure the suitability of
In order to determine the problem types that ben- examples with respect to the target objective and to
efit most or least from our proposed methods, we determine the optimal ordering of them. Liu et al.
also evaluate the performance of different mod- (2022) suggest selecting examples that are seman-
els across various problem categories within the tically similar to the test example by employing
Spider dataset. As indicated in Figure 7 of the a k-NN approach in the embedding space. Rubin
appendix, our similarity-diversity strategy proves et al. (2022) train a prompt retriever based on con-
beneficial for most problem types, with the excep- trastive learning, wherein examples are classified as
tion of the medium split, which includes the most either positive or negative if they are ranked among
diverse problems. This is the case where similarity- the top-k or bottom-k probabilities of a language
based retrieval fails and syntax coverage becomes model generating the target output, conditioned on
more crucial. Furthermore, we observe that aug- the retrieved example and the input. Zhang et al.
menting schema semantics is more effective for (2022) suggests to actively select demonstrations
the easy and medium splits (albeit with high vari- using Q-Learning. Su et al. (2023) introduces the
ance), while augmenting schema structure is more Vote-k approach to selectively annotate diverse and
effective for more complex problems. This ob- representative examples for pool construction, then
vervation leads us to hypothesize that challenging retrieve based on the similarity. In contrast, our
problems necessitate addressing a higher number approach retrieve a diverse set of examples given a
of tables, thus requiring a more comprehensive un- pre-established pool. As the authors demonstrate
derstanding of the entire database structure. Lastly, that having a diverse and representative pool is im-
the integrated approach is effective across all ex- portant for the success of ICL, we posit that a sim-
amples, offering increased benefits especially for ilar characteristic is equally important when com-
those difficult problems. posing the prompt, as this approach increases the
likelihood of including various syntactical usages
4.5 Preliminary Models or similar problem structures within the prompt.
To assess the impact of the choice of preliminary
model used to generate the draft SQL on our ap-
proach, we conducted tests involving our methods 5.2 Prompt Formatting
for preliminary models with varying performance
levels. Figure 8 of the appendix reveals that the
preliminary models have a relatively minor effect Prompt engineering is concerned with investigat-
on the performance of the similarity-diversity or ing the impact of prompt structure on downstream
integrated approaches, exhibiting gradual improve- task performance. For tasks that involve multi-step
ments as better preliminary models are utilized. reasoning and higher complexity, Chain-of-thought
prompting has been developed (Wei et al., 2023;
5 Related Work Kojima et al., 2023). This approach involves lay-
ing out the generation process over multiple steps
Existing literature indicates the ability of large lan- and using the model’s own intermediate process as
guage models to adapt to new tasks at inference input. Wang et al. (2023) proposes to sample mul-
time by learning from a few example demonstra- tiple different chain-of-thoughts then selects the
tions (Brown et al., 2020; Radford et al., 2019). most consistent answer through marginalization of
This new capability has been referred to as in- all possible reasoning paths. Press et al. (2023)
context learning. In this paper, we expand on pre- suggests that prompting LLMs to ask follow-up
vious works that investigate the optimal representa- questions is an effective way to construct the chain-
tions for prompt inputs. of-thoughts process. Zhou et al. (2023) proposes an
automatic approach to identify the optimal prompt
5.1 Prompt Organization by searching over a pool of model generated in-
Prompt organization investigates the task of select- structions, assigning scores to them, and selecting
ing and organizing in-context examples, a critical the prompt with the highest score.
14942
6 Conclusions Gretchen Krueger, Tom Henighan, Rewon Child,
Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens
In this study, we investigated various prompt de- Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma-
sign approaches for semantic parsing tasks in the teusz Litwin, Scott Gray, Benjamin Chess, Jack
Text-to-SQL domain. We proposed an approach Clark, Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei. 2020.
that leverages an example’s SQL syntactic structure Language models are few-shot learners. In Ad-
for demonstration examples selection, emphasising vances in Neural Information Processing Systems,
both diversity and similarity as the sampling ob- volume 33, pages 1877–1901. Curran Associates,
jectives. Additionally, We found that LLMs gain Inc.
benefits from database-related knowledge augmen- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming
tations. Future research can build upon our findings Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka-
to examine the transferability of our approach to plan, Harri Edwards, Yuri Burda, Nicholas Joseph,
other domains. Through ongoing improvement of Greg Brockman, Alex Ray, Raul Puri, Gretchen
Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas-
LLMs’ capabilities in semantic parsing, we aim to try, Pamela Mishkin, Brooke Chan, Scott Gray,
contribute to the development of QA systems that Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz
are more accurate, robust and comprehensible. Kaiser, Mohammad Bavarian, Clemens Winter,
Philippe Tillet, Felipe Petroski Such, Dave Cum-
Limitations mings, Matthias Plappert, Fotios Chantzis, Eliza-
beth Barnes, Ariel Herbert-Voss, William Hebgen
One of the main limitations of this study is the re- Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie
Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain,
producibility problem. The experiments presented
William Saunders, Christopher Hesse, Andrew N.
in this paper relied on the use of OpenAI APIs, Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan
which were available at the time of our research Morikawa, Alec Radford, Matthew Knight, Miles
but have since been or will be deprecated. This Brundage, Mira Murati, Katie Mayer, Peter Welinder,
means that the results of our experiments cannot Bob McGrew, Dario Amodei, Sam McCandlish, Ilya
Sutskever, and Wojciech Zaremba. 2021. Evaluating
be replicated using the same APIs, which hinders large language models trained on code.
the reproducibility of our findings. To address this
limitation, we will focus on providing experiments Wenhu Chen, Xueguang Ma, Xinyi Wang, and
results that are based on open-sourced LLMs (Tou- William W. Cohen. 2022. Program of thoughts
prompting: Disentangling computation from reason-
vron et al., 2023; Taori et al., 2023; Chiang et al., ing for numerical reasoning tasks. arXiv preprint
2023) for greater transparency and reproducibility. arXiv:2211.12588.
Another limitation is that it is not clear how our
approach will benefit LLMs given smaller or more Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu
Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong,
constrained pools of annotated examples. Although Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer,
we postulate that our approach offers the advantage Noah A. Smith, and Tao Yu. 2023. Binding language
of providing a prompt with maximal coverage of models in symbolic languages. In The Eleventh Inter-
similar problem structures when identically struc- national Conference on Learning Representations.
tured problems cannot be found in the pool, we Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng,
could not substantiate this due to our limited bud- Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan
get and access to the OpenAI APIs. Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion
Stoica, and Eric P. Xing. 2023. Vicuna: An open-
source chatbot impressing gpt-4 with 90%* chatgpt
References quality.

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Xiang Deng, Ahmed Hassan Awadallah, Christopher
Liang. 2013. Semantic parsing on Freebase from Meek, Oleksandr Polozov, Huan Sun, and Matthew
question-answer pairs. In Proceedings of the 2013 Richardson. 2021. Structure-grounded pretraining
Conference on Empirical Methods in Natural Lan- for text-to-SQL. In Proceedings of the 2021 Con-
guage Processing, pages 1533–1544, Seattle, Wash- ference of the North American Chapter of the Asso-
ington, USA. Association for Computational Linguis- ciation for Computational Linguistics: Human Lan-
tics. guage Technologies, pages 1337–1350, Online. As-
sociation for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie
Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong
Neelakantan, Pranav Shyam, Girish Sastry, Amanda Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and
Askell, Sandhini Agarwal, Ariel Herbert-Voss, Zhifang Sui. 2023. A survey on in-context learning.

14943
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- (Volume 1: Long Papers), pages 23–33, Vancouver,
aocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Canada. Association for Computational Linguistics.
Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code-
BERT: A pre-trained model for programming and Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu.
natural languages. In Findings of the Association 2023. A comprehensive evaluation of chatgpt’s zero-
for Computational Linguistics: EMNLP 2020, pages shot text-to-sql capability.
1536–1547, Online. Association for Computational
Linguistics. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan,
Lawrence Carin, and Weizhu Chen. 2022. What
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and makes good in-context examples for GPT-3? In
Tushar Khot. 2023. Complexity-based prompting for Proceedings of Deep Learning Inside Out (DeeLIO
multi-step reasoning. In The Eleventh International 2022): The 3rd Workshop on Knowledge Extrac-
Conference on Learning Representations. tion and Integration for Deep Learning Architectures,
pages 100–114, Dublin, Ireland and Online. Associa-
Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew tion for Computational Linguistics.
Purver, John R. Woodward, Jinxia Xie, and Peng-
sheng Huang. 2021a. Towards robustness of text- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-
to-SQL models against synonym substitution. In dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
Proceedings of the 59th Annual Meeting of the Asso- Luke Zettlemoyer, and Veselin Stoyanov. 2020.
ciation for Computational Linguistics and the 11th Ro{bert}a: A robustly optimized {bert} pretraining
International Joint Conference on Natural Language approach.
Processing (Volume 1: Long Papers), pages 2505–
2515, Online. Association for Computational Lin- Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel,
guistics. and Pontus Stenetorp. 2022. Fantastically ordered
prompts and where to find them: Overcoming few-
Yujian Gan, Xinyun Chen, and Matthew Purver. 2021b. shot prompt order sensitivity. In Proceedings of the
Exploring underexplored limitations of cross-domain 60th Annual Meeting of the Association for Compu-
text-to-SQL generalization. In Proceedings of the tational Linguistics (Volume 1: Long Papers), pages
2021 Conference on Empirical Methods in Natural 8086–8098, Dublin, Ireland. Association for Compu-
Language Processing, pages 8926–8931, Online and tational Linguistics.
Punta Cana, Dominican Republic. Association for
Computational Linguistics. Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov,
Wen tau Yih, Sida I. Wang, and Xi Victoria Lin. 2023.
Matt Gardner, Pradeep Dasigi, Srinivasan Iyer, Alane Lever: Learning to verify language-to-code genera-
Suhr, and Luke Zettlemoyer. 2018. Neural semantic tion with execution.
parsing. In Proceedings of the 56th Annual Meeting
of the Association for Computational Linguistics: Tu- Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt,
torial Abstracts, pages 17–18, Melbourne, Australia. Noah A. Smith, and Mike Lewis. 2023. Measuring
Association for Computational Linguistics. and narrowing the compositionality gap in language
Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, models.
and Luke Zettlemoyer. 2022. Demystifying prompts
Alec Radford, Jeffrey Wu, Rewon Child, David Luan,
in language models via perplexity estimation.
Dario Amodei, and Ilya Sutskever. 2019. Language
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Models are Unsupervised Multitask Learners.
Noah A. Smith, and Mari Ostendorf. 2022. In-
context learning for few-shot dialogue state tracking. Colin Raffel, Noam Shazeer, Adam Roberts, Kather-
In Findings of the Association for Computational ine Lee, Sharan Narang, Michael Matena, Yanqi
Linguistics: EMNLP 2022, pages 2627–2643, Abu Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the
Dhabi, United Arab Emirates. Association for Com- limits of transfer learning with a unified text-to-text
putational Linguistics. transformer. Journal of Machine Learning Research,
21(140):1–67.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu-
taka Matsuo, and Yusuke Iwasawa. 2023. Large lan- Nitarshan Rajkumar, Raymond Li, and Dzmitry Bah-
guage models are zero-shot reasoners. danau. 2022. Evaluating the text-to-sql capabilities
of large language models.
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen.
2023. Resdsql: Decoupling schema linking and Nils Reimers and Iryna Gurevych. 2019. Sentence-
skeleton parsing for text-to-sql. In AAAI. BERT: Sentence embeddings using Siamese BERT-
networks. In Proceedings of the 2019 Conference on
Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. For- Empirical Methods in Natural Language Processing
bus, and Ni Lao. 2017. Neural symbolic machines: and the 9th International Joint Conference on Natu-
Learning semantic parsers on Freebase with weak ral Language Processing (EMNLP-IJCNLP), pages
supervision. In Proceedings of the 55th Annual Meet- 3982–3992, Hong Kong, China. Association for Com-
ing of the Association for Computational Linguistics putational Linguistics.

14944
Hongyu Ren, Hanjun Dai, Bo Dai, Xinyun Chen, Michi- and Denny Zhou. 2023. Self-consistency improves
hiro Yasunaga, Haitian Sun, Dale Schuurmans, Jure chain of thought reasoning in language models. In
Leskovec, and Denny Zhou. 2021. Lego: Latent The Eleventh International Conference on Learning
execution-guided reasoning for multi-hop question Representations.
answering on knowledge graphs. In Proceedings of
the 38th International Conference on Machine Learn- Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H.
ing, volume 139 of Proceedings of Machine Learning Hoi. 2021. CodeT5: Identifier-aware unified pre-
Research, pages 8959–8970. PMLR. trained encoder-decoder models for code understand-
ing and generation. In Proceedings of the 2021
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Conference on Empirical Methods in Natural Lan-
2022. Learning to retrieve prompts for in-context guage Processing, pages 8696–8708, Online and
learning. In Proceedings of the 2022 Conference of Punta Cana, Dominican Republic. Association for
the North American Chapter of the Association for Computational Linguistics.
Computational Linguistics: Human Language Tech-
nologies, pages 2655–2671, Seattle, United States. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
Association for Computational Linguistics. Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le,
and Denny Zhou. 2022. Chain of thought prompt-
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- ing elicits reasoning in large language models. In
danau. 2021. PICARD: Parsing incrementally for Advances in Neural Information Processing Systems.
constrained auto-regressive decoding from language
models. In Proceedings of the 2021 Conference on Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
Empirical Methods in Natural Language Processing, Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and
pages 9895–9901, Online and Punta Cana, Domini- Denny Zhou. 2023. Chain-of-thought prompting elic-
can Republic. Association for Computational Lin- its reasoning in large language models.
guistics.
Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Ling-
Taylor Sorensen, Joshua Robinson, Christopher Ryt- peng Kong. 2022. Self-adaptive in-context learning.
ting, Alexander Shaw, Kyle Rogers, Alexia Delorey,
Mahmoud Khalil, Nancy Fulda, and David Wingate. Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong,
2022. An information-theoretic approach to prompt Torsten Scholak, Michihiro Yasunaga, Chien-Sheng
engineering without ground truth labels. In Proceed- Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Vic-
ings of the 60th Annual Meeting of the Association tor Zhong, Bailin Wang, Chengzu Li, Connor Boyle,
for Computational Linguistics (Volume 1: Long Pa- Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming
pers), pages 819–862, Dublin, Ireland. Association Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith,
for Computational Linguistics. Luke Zettlemoyer, and Tao Yu. 2022. UnifiedSKG:
Unifying and multi-tasking structured knowledge
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, grounding with text-to-text language models. In Pro-
Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, ceedings of the 2022 Conference on Empirical Meth-
Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. ods in Natural Language Processing, pages 602–631,
Selective annotation makes language models better Abu Dhabi, United Arab Emirates. Association for
few-shot learners. In The Eleventh International Con- Computational Linguistics.
ference on Learning Representations.
Xuchen Yao and Benjamin Van Durme. 2014. Infor-
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann mation extraction over structured data: Question an-
Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, swering with Freebase. In Proceedings of the 52nd
and Tatsunori B. Hashimoto. 2023. Stanford alpaca: Annual Meeting of the Association for Computational
An instruction-following llama model. https:// Linguistics (Volume 1: Long Papers), pages 956–966,
github.com/tatsu-lab/stanford_alpaca. Baltimore, Maryland. Association for Computational
Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timothée Lacroix, Seonghyeon Ye, Hyeonbin Hwang, Sohee Yang,
Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Hyeongu Yun, Yireun Kim, and Minjoon Seo. 2023.
Azhar, Aurelien Rodriguez, Armand Joulin, Edouard In-context instruction learning.
Grave, and Guillaume Lample. 2023. Llama: Open
and efficient foundation language models. Pengcheng Yin and Graham Neubig. 2018. TRANX:
A transition-based neural abstract syntax parser for
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr semantic parsing and code generation. In Proceed-
Polozov, and Matthew Richardson. 2020. RAT-SQL: ings of the 2018 Conference on Empirical Methods
Relation-aware schema encoding and linking for text- in Natural Language Processing: System Demonstra-
to-SQL parsers. In Proceedings of the 58th Annual tions, pages 7–12, Brussels, Belgium. Association
Meeting of the Association for Computational Lin- for Computational Linguistics.
guistics, pages 7567–7578, Online. Association for
Computational Linguistics. Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Se-
bastian Riedel. 2020. TaBERT: Pretraining for joint
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, understanding of textual and tabular data. In Proceed-
Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, ings of the 58th Annual Meeting of the Association

14945
 for Computational Linguistics, pages 8413–8426, On-
line. Association for Computational Linguistics.
Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue,
Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze
Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga,
Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan
Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vin-
cent Zhang, Caiming Xiong, Richard Socher, Walter
Lasecki, and Dragomir Radev. 2019. CoSQL: A
conversational text-to-SQL challenge towards cross-
domain natural language interfaces to databases. In
Proceedings of the 2019 Conference on Empirical
Methods in Natural Language Processing and the
9th International Joint Conference on Natural Lan-
guage Processing (EMNLP-IJCNLP), pages 1962–
1979, Hong Kong, China. Association for Computa-
tional Linguistics.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga,
Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn-
ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir
Radev. 2018. Spider: A large-scale human-labeled
dataset for complex and cross-domain semantic pars-
ing and text-to-SQL task. In Proceedings of the 2018
Conference on Empirical Methods in Natural Lan-
guage Processing, pages 3911–3921, Brussels, Bel-
gium. Association for Computational Linguistics.
Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. Ac-
tive example selection for in-context learning. In Pro-
ceedings of the 2022 Conference on Empirical Meth-
ods in Natural Language Processing, pages 9134–
9148, Abu Dhabi, United Arab Emirates. Association
for Computational Linguistics.

Victor Zhong, Caiming Xiong, and Richard Socher.

2017. Seq2sql: Generating structured queries
from natural language using reinforcement learning.
CoRR, abs/1709.00103.

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han,

Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy
Ba. 2023. Large language models are human-level
prompt engineers. In The Eleventh International
Conference on Learning Representations.

14946
Appendix

Algorithm 1: Similarity-Diversity Demonstration Selection

Input: Set of annotated examples A, test examples T , # demonstrations k, categorization
{α, β, ...}
Result: Set of prompts P , where Pi is the prompt for test example Ti
/* Split A into disjoint partitions Aα,β,... */
for Ai in annotated set A do
ci = get_category(Ai .SQL);
Aci .append(Ai );
Vi = get_syntax_vectors(Ai .SQL);
end
/* Prepare demonstrations Dj for each partition Aj */
for partition Aj in Aα,β,... do
M = k-Means_clustering(V j , k);
/* V j is set of discrete vectors for examples in Aj , M has k centroids
µ1 , ..., µk */
for µi in M do
Dj .append(get_nearest(A, µi ));
end
end
/* Build test prompts */
for Ti in test set T do
Ti .SQL = initial_predictor(Ti );
ci = get_category(Ti .SQL);
Pi = build_prompt(Dci , Ti );
end
return P

14947
 Algorithm 2: Integrated Strategy
Input: Set of annotated examples A, test examples T , # demonstrations k, categorization
{α, β, ...}, and from Algorithm 1: disjoint partitions {Aα , Aβ , ...} and corresponding
demonstrations {Dα , Dβ , ...}
Result: Set of SQL predictions SP, where SPi is the final prediction for test example Ti
for Ti in test set T do
Ti .SQL = initial_predictor(Ti );
ci = get_category(Ti .SQL);
for n = 4 to k do
Pin = build_prompt(Dci [: n], Ti );
∗
Pin = augment_schema(Pin );
∗
SPni = Model(Pin );
ERni = DBMS(SPni );
end
ER∗i = Remove_Exec_Errors(ERi );
SPi = Majority_Vote(ER∗i );
end
return SP

Coverage Similarity Execution Accuracy

Random 0.38 0.24 76.03
Similarity 0.35 0.30 78.33
Diversity 0.43 0.23 78.64
Similarity-Diversity 0.50 0.26 80.32

Table 1: Average syntax coverage and similarity measures of the prompt for different demonstration selection
strategies and the corresponding execution accuracies.

14948
Figure 6: Correlation between syntax coverage and similarity measures of prompts and execution accuracy.

Figure 7: Effects of various prompting strategies on Text-to-SQL problems of different difficulty levels.

Figure 8: Effects of preliminary model on proposed strategies.

14949
 1 Given the following database schema :
2 gymnast : gymnast_id , floor_exercise_points , pommel_horse_points , rings_points ,
vault_points , parallel_bars_points , horizontal_bar_points , total_points |
people : people_id , name , age , height , hometown
3
4 Answer the following : Return the total points of the gymnast with the lowest
age .
5
6 select t1 . total_points from gymnast as t1 join people as t2 on t1 . gymnast_id =
t2 . people_id order by t2 . age asc limit 1
Listing 1: Baseline prompt with text representation of the database.

1 /* Given the following database schema : */

2 CREATE TABLE IF NOT EXISTS " gymnast " (
3 " Gymnast_ID " int ,
4 " Floor_Exercise_Points " real ,
5 " Pommel_Horse_Points " real ,
6 " Rings_Points " real ,
7 " Vault_Points " real ,
8 " Parallel_Bars_Points " real ,
9 " Horizontal_Bar_Points " real ,
10 " Total_Points " real ,
11 PRIMARY KEY (" Gymnast_ID " ) ,
12 FOREIGN KEY (" Gymnast_ID " ) REFERENCES " people " (" People_ID ")
13 );
14 CREATE TABLE IF NOT EXISTS " people " (
15 " People_ID " int ,
16 " Name " text ,
17 " Age " real ,
18 " Height " real ,
19 " Hometown " text ,
20 PRIMARY KEY (" People_ID " )
21 );
22
23 /* Answer the following : Return the total points of the gymnast with the lowest
age . */
24
25 select t1 . total_points from gymnast as t1 join people as t2 on t1 . gymnast_id =
t2 . people_id order by t2 . age asc limit 1
Listing 2: Baseline prompt with code representation of the database.

14950
 1 /* Given the following database schema : */
2 CREATE TABLE IF NOT EXISTS " department " (
3 " Department_ID " int , -- a unique identifier for a department
4 " Name " text , -- the name of the department
5 " Creation " text , -- the date the department was created
6 " Ranking " int , -- the ranking of the department within the organization
7 " Budget_in_Billions " real , -- the department ' s budget in billions of
dollars
8 " Num_Employees " real , -- the number of employees in the department
9 PRIMARY KEY ( " Department_ID " )
10 );
11 CREATE TABLE IF NOT EXISTS " head " (
12 " head_ID " int , -- a unique identifier for the head of a department
13 " name " text , -- the name of the head of the department
14 " born_state " text , -- the state where the head of the department was born
15 " age " real , -- the age of the head of the department
16 PRIMARY KEY (" head_ID " )
17 );
18 CREATE TABLE IF NOT EXISTS " management " (
19 " department_ID " int , -- the unique identifier for the department being
managed
20 " head_ID " int , -- the unique identifier for the head of the department
21 " temporary_acting " text , -- whether the head of the department is serving
in a temporary or acting capacity
22 PRIMARY KEY (" Department_ID " , " head_ID ")
23 FOREIGN KEY (" Department_ID " ) REFERENCES ` department `( " Department_ID " )
24 FOREIGN KEY (" head_ID " ) REFERENCES `head `( " head_ID " )
25 );
26
27 /* Answer the following : What are the distinct creation years of the
departments managed by a secretary born in state ' Alabama '? */
28
29 select distinct t1 . creation from department as t1 join management as t2 on t1 .
department_id = t2 . department_id join head as t3 on t2 . head_id = t3 . head_id
where t3 . born_state = ' Alabama '
Listing 3: Prompt with semantic augmentation of the schema as inline comment.

14951
 1 /* Given the following database schema : */
2 CREATE TABLE IF NOT EXISTS " department " (
3 " Department_ID " int ,
4 " Name " text ,
5 " Creation " text ,
6 " Ranking " int ,
7 " Budget_in_Billions " real ,
8 " Num_Employees " real ,
9 PRIMARY KEY ( " Department_ID " )
10 );
11 CREATE TABLE IF NOT EXISTS " head " (
12 " head_ID " int ,
13 " name " text ,
14 " born_state " text ,
15 " age " real ,
16 PRIMARY KEY (" head_ID " )
17 );
18 CREATE TABLE IF NOT EXISTS " management " (
19 " department_ID " int ,
20 " head_ID " int ,
21 " temporary_acting " text ,
22 PRIMARY KEY (" Department_ID " ," head_ID ") ,
23 FOREIGN KEY (" Department_ID " ) REFERENCES ` department `( " Department_ID " ) ,
24 FOREIGN KEY (" head_ID " ) REFERENCES `head `( " head_ID " )
25 );
26
27 /* Table column descriptions :
28 { ' department ': {' Department_ID ': 'a unique identifier for a department ' , ' Name
': ' the name of the department ' , ' Creation ': ' the date the department was
created ' , ' Ranking ': ' the ranking of the department within the organization
' , ' Budget_in_Billions ': " the department ' s budget in billions of dollars ",
' Num_Employees ': ' the number of employees in the department '} , ' head ': {'
head_ID ': 'a unique identifier for the head of a department ' , ' name ': ' the
name of the head of the department ' , ' born_state ': ' the state where the
head of the department was born ' , 'age ': ' the age of the head of the
department '} , ' management ': {' department_ID ': ' the unique identifier for
the department being managed ' , ' head_ID ': ' the unique identifier for the
head of the department ' , ' temporary_acting ': ' whether the head of the
department is serving in a temporary or acting capacity '}} */
29 /* Answer the following : What are the distinct creation years of the
departments managed by a secretary born in state ' Alabama '? */
30
31 select distinct t1 . creation from department as t1 join management as t2 on t1 .
department_id = t2 . department_id join head as t3 on t2 . head_id = t3 . head_id
where t3 . born_state = ' Alabama '
Listing 4: Prompt with semantic augmentation of the schema as block comment.

14952
continents.contid -> countries.continent, countries.countryid ->
car_makers.country, car_makers.id -> model_list.maker, model_list.model ->
car_names.model, car_names.makeid -> cars_data.id

employee.emp_num -> department.emp_num, department.dept_code -> course.dept_code,

course.crs_code -> class.crs_code, class.class_code -> enroll.class_code
department.dept_code -> student.dept_code, student.stu_num -> enroll.stu_num
employee.emp_num -> class.prof_num
employee.emp_num -> professor.emp_num
department.dept_code -> professor.dept_code

Figure 9: Examples of schema structure representation construction.

14953
 1 /* Given the following database schema : */
2 CREATE TABLE IF NOT EXISTS " continents " (
3 " ContId " INTEGER PRIMARY KEY ,
4 " Continent " TEXT
5 );
6 CREATE TABLE IF NOT EXISTS " countries " (
7 " CountryId " INTEGER PRIMARY KEY ,
8 " CountryName " TEXT ,
9 " Continent " INTEGER ,
10 FOREIGN KEY ( Continent ) REFERENCES continents ( ContId )
11 );
12 CREATE TABLE IF NOT EXISTS " car_makers " (
13 " Id " INTEGER PRIMARY KEY ,
14 " Maker " TEXT ,
15 " FullName " TEXT ,
16 " Country " TEXT ,
17 FOREIGN KEY ( Country ) REFERENCES countries ( CountryId )
18 );
19 CREATE TABLE IF NOT EXISTS " model_list " (
20 " ModelId " INTEGER PRIMARY KEY ,
21 " Maker " INTEGER ,
22 " Model " TEXT UNIQUE ,
23 FOREIGN KEY ( Maker ) REFERENCES car_makers ( Id )
24
25 );
26 CREATE TABLE IF NOT EXISTS " car_names " (
27 " MakeId " INTEGER PRIMARY KEY ,
28 " Model " TEXT ,
29 " Make " TEXT ,
30 FOREIGN KEY ( Model ) REFERENCES model_list ( Model )
31 );
32 CREATE TABLE IF NOT EXISTS " cars_data " (
33 " Id " INTEGER PRIMARY KEY ,
34 " MPG " TEXT ,
35 " Cylinders " INTEGER ,
36 " Edispl " REAL ,
37 " Horsepower " TEXT ,
38 " Weight " INTEGER ,
39 " Accelerate " REAL ,
40 " Year " INTEGER ,
41 FOREIGN KEY ( Id ) REFERENCES car_names ( MakeId )
42 );
43
44 /*
45 Database ontology :
46 continents . contid -> countries . continent , countries . countryid -> car_makers .
country , car_makers . id -> model_list . maker , model_list . model -> car_names .
model , car_names . makeid -> cars_data . id
47 */
48 /* Answer the following : How many continents are there ? */
49
50 select count (*) from continents ;
Listing 5: Prompt with structure augmentation of the schema.

14954
Figure 10: Few-shot results for comparing different sampling strategies with different number of demonstra-
tion examples selected for the prompt.

Figure 11: Few-shot results for comparing different schema representation augmentation methods with
different number of demonstration examples selected for the prompt.

14955
Figure 12: Few-shot results for comparing different sampling strategies on Text-to-SQL problems of different
difficulty levels, with different number of demonstration examples selected for the prompt.

14956