V-Star: Learning Visibly Pushdown Grammars from Program Inputs (Extended Version)
Abstract.
Accurate description of program inputs remains a critical challenge in the field of programming languages. Active learning, as a well-established field, achieves exact learning for regular languages. We offer an innovative grammar inference tool, V-Star, based on the active learning of visibly pushdown automata. V-Star deduces nesting structures of program input languages from sample inputs, employing a novel inference mechanism based on nested patterns. This mechanism identifies token boundaries and converts languages such as XML documents into VPLs. We then adapted Angluin’s L-Star, an exact learning algorithm, for VPA learning, which improves the precision of our tool. Our evaluation demonstrates that V-Star effectively and efficiently learns a variety of practical grammars, including S-Expressions, JSON, and XML, and outperforms other state-of-the-art tools.
1. Introduction
In recent years, there has been a growing interest in learning grammars from a set of sample strings. This interest stems from a wide range of applications in fuzzing, program validation, and other areas (Arefin et al., 2024; Bastani et al., 2017; Kulkarni et al., 2022; Wu et al., 2019; Bendrissou et al., 2022). Despite significant progress, the challenge of learning the input grammars for black-box programs remains, particularly when considering grammars with inherent complexities. This challenge is part of a broader problem that has been extensively studied for regular languages but is significantly more difficult when dealing with broader classes of grammars.
Recently, GLADE (Bastani et al., 2017) (followed by a replication study (Bendrissou et al., 2022)) and ARVADA (Kulkarni et al., 2022) have been proposed to learn context-free grammars (CFGs) under active learning. Both approaches require positive seed inputs and utilize enumeration and heuristics to reduce the search space. However, methods in CFG learning such as Glade and Arvada do not fully utilize the concept of nesting structures, which could potentially improve a grammar-learning process’s accuracy.
Nesting structures are widely observed in practical languages, where recursions are explicitly delimited in their sentences. For example, an XML document’s open and close tags delimit a component of the document and can be nested within other open and close tags. These nesting structures often carry valuable insights into the grammars’ structure and could potentially be a powerful tool for learning grammars.
To achieve accurate learning, we model the nesting structures together with the target language as Visibly Pushdown Grammars (VPGs) (Alur and Madhusudan, 2009), a subclass of CFGs. VPGs formally specify nesting structures, and despite being slightly weaker than CFGs they can specify many practical format languages such as XML and JSON; they also enjoy all desirable closure properties; e.g., the set of visibly pushdown languages is closed under intersection, concatenation, and complement (Alur and Madhusudan, 2009). We posit that these properties position VPGs as an ideal mechanism for learning practical grammars, which is the focus of this paper.
Learning VPGs is a problem that fits nicely into the well-studied active learning field. In this field, Angluin (Angluin, 1987) first demonstrated that it is possible to efficiently learn regular languages from a minimally adequate teacher (MAT), which answers (1) whether a string is in the language, which is called a membership query, and (2) whether a finite state automaton accepts exactly the language held by the teacher, which is called an equivalence query; if not, the teacher provides a counterexample accepted by either the automaton or the language, but not both.
Despite advancements in VPG learning facilitated by a MAT, current techniques, as highlighted in previous work (Barbot et al., 2021; Michaliszyn and Otop, 2022; Isberner, 2015), are constrained by assumptions that do not match practical settings. Specifically, they assume the availability of known nesting patterns, or more technically, a predefined tagging function—a cornerstone of the VPG formalism detailed in Section 3. This tagging function defines a set of call and return symbols and is assumed to operate on individual characters. In contrast, in many practical settings the tagging must be inferred and it operates on sequences of characters (i.e., tokens) rather than on individual characters; a more detailed comparison with prior work and the limitations of these assumptions are discussed in Section 2.
To address these limitations, we introduce V-Star, a novel grammar-inference framework designed to learn VPGs from a black-box program using a collection of sample seed strings. V-Star’s algorithm is inspired by (Angluin, 1987), which learns a finite-state automaton using a MAT and seed strings. We develop V-Star in several pivotal steps. First, we develop an -like algorithm for learning VPGs when tags are known and are on single characters; this algorithm utilizes -SEVPA (Alur et al., 2005) to define a set of congruence relations, critical to the algorithm. Second, we develop a tag-inference algorithm, which utilizes a novel notion of nesting patterns to infer call and return symbols, assuming they are of single characters. In the third step, we lift the restriction that tags are on single characters and develop an algorithm for inferring token-based tags. Finally, we remove the requirement of equivalence queries by simulating them using membership queries via sampling. These steps all together result in a practical framework for learning VPGs from seed strings. The main contributions of V-Star are summarized as follows:
-
(1)
Innovative Tool and Methodology: V-Star is a novel tool for VPG inference. Its algorithm adapts Angluin’s algorithm and integrates a set of novel techniques such as nesting patterns to infer call and return tokens. To our best knowledge, this is the first VPG-learning algorithm without knowing what call/return tokens are a priori.
-
(2)
Theoretical Reasoning: We provide a theoretical analysis elucidating the conditions under which V-Star achieves accurate learning. We first prove that for any character-based visibly pushdown language with uniquely paired call/return symbols, there exists a finite set of seed strings from which V-Star can learn a tagging function that achieves exact learning. We further show that under some realistic assumptions V-Star can infer tagged tokens for token-based visibly pushdown languages.
-
(3)
Accuracy: Our evaluation of V-Star demonstrates its better accuracy in learning practical grammars, in comparison with state-of-the-art grammar learning tools. The accuracy result highlights the benefit of utilizing the concept of nesting structures and V-Star’s ability to simulate the equivalence queries by sampling test strings from the seed strings.
2. Related Work
In the realm of automata learning, learning finite state automata is a well-studied field. The Algorithm by Angluin (Angluin, 1987), which learns a finite state automaton held by a MAT in polynomial time, is a seminal work in this area. Following , various adaptations focusing on the active learning problem have been proposed, such as (Rivest and Schapire, 1993; Maler and Pnueli, 1991; Irfan et al., 2010; Howar, 2012; Isberner, 2015), to list a few.
Glade (Bastani et al., 2017), targeting learning a CFG from an oracle, employs a two-step algorithmic approach. Initially, it enumerates all substrings of seed strings and attempts to generalize these substrings into regular expressions. Subsequently, nonterminals are created and merged based on learned regular expressions. Arvada (Kulkarni et al., 2022) also aims to learn a CFG by using a technique that exchanges two substrings from seed strings. If two substrings are interchangeable, they are assigned the same nonterminal, and the process gradually constructs parse trees. Arvada employs heuristics to exchange substrings with similar contexts. In its evaluation, the context is considered to be the surrounding substrings of length four. While context strings bear some similarity to call and return symbols in VPGs, call and return symbols are more flexible than context strings in Arvada: they can wrap contexts of an arbitrary length. Moreover, call and return symbols have a stronger implication on the recursive structure of the oracle language, a feature that V-Star capitalizes on for better grammar-inference accuracy, which is evidenced by V-Star’s experimental comparison with Glade and Arvada discussed in Section 6. As an extension of GLADE, REINAM (Wu et al., 2019) refines the grammar learned by GLADE using reinforcement learning. This process allows for the potential replacement of GLADE by other learning tools such as Arvada (Kulkarni et al., 2022) or our tool V-Star.
Learning VPGs, a subset of CFGs, has been the focus of several systems. For example, assuming that the set of call and return symbols is known, VPL* (Barbot et al., 2021) learns VPGs with membership and equivalence queries. The approach taken by VPL* is indirect, first using TL* (Drewes and Högberg, 2007; Drewes et al., 2011) to learn a tree automaton, which is then converted to a VPG. Another work by Michaliszyn and Otop (2022) also assumes that the set of call and return symbols is known and attempts to learn a visibly pushdown automaton (VPA). Moreover, it requires a stronger teacher who, in addition to providing membership and equivalence queries, can also report the stack content during the VPA’s execution. TTT (Isberner, 2015) is another VPA-inference tool under the active-learning setting, based on discrimination trees. V-Star differentiates itself from these systems by learning call and return symbols from the oracle and seed strings.
In practice, a MAT is often instantiated by a black-box program and the oracle language comprises input strings that do not trigger program errors (assuming the program always terminates). Thus, membership queries require only program execution. However, equivalence queries are much harder to answer. Simulating equivalence queries has a long history, often under the name of conformance testing (Aichernig et al., 2024). Chow (Chow, 1978) first proposed the so-called W-method for Mealy machines; we note that the FSA version of the W-method is essentially a brute force approach of enumerating suffix strings that distinguish the representative prefix strings in the Nerode relation. The space of suffix strings is restricted from to , under the assumption that the difference between the size of the oracle FSA and the size of the learned FSA is . The W-method has many variants (Vasilevskii, 1973; Raffelt et al., 2005; Fujiwara et al., 1991). For more information, we refer to Aichernig et al. (2024).
3. Background
3.1. Grammar Inference
In grammar inference, we assume an oracle for the grammar being inferred, denoted as , which maps strings to booleans—true for valid strings and false for invalid ones. The set of valid strings according to the oracle forms the oracle language, denoted as . When the oracle language is defined by a grammar, we refer to this grammar as the oracle grammar, denoted as . Similarly, when the oracle language can be recognized by a deterministic finite automaton (DFA), we refer to this DFA as the oracle automaton, denoted as .
We define the active learning problem as follows:
Inputs: The problem takes two inputs:
-
(1)
A set of terminals, and
-
(2)
A minimally adequate teacher (MAT), which can answer both membership and equivalence queries. For an equivalence query with a hypothesis grammar, the teacher returns true when the hypothesis grammar is equivalent to the oracle grammar, meaning they generate the same language; if they are not equivalent, it provides a counterexample, which is a string accepted by either the hypothesis grammar or the oracle grammar, but not by both.
Output: The goal of the active learning problem is to construct a grammar, denoted as , such that the language it generates, denoted as , is identical to the oracle language .
3.2. Visibly Pushdown Grammars
The expressive power of VPGs (Alur and Madhusudan, 2009) is between regular grammars and context-free grammars, and VPGs are sufficient for describing the syntax of many practical languages, such as JSON, XML, and HTML. Application wise, VPGs have been used in program analysis, XML processing, and other fields (Harris et al., 2012; Heizmann et al., 2010; Chaudhuri and Alur, 2007; Nguyen and Sudholt, 2006; Kumar et al., 2007; Alur, 2007; Thomo and Venkatesh, 2011; Gauwin et al., 2008; Mozafari et al., 2010, 2012). Besides, since they can be efficiently parsed, VPGs are also found valuable to specify practical languages (Jia et al., 2021, 2023).
A language is called a visibly pushdown language (VPL) if it can be generated by a VPG. VPLs enjoy the same appealing theoretical closure properties as regular languages; e.g., the set of VPLs is closed under intersection, concatenation, and complement (Alur and Madhusudan, 2009). Further, since VPLs are a subset of deterministic context-free languages, it is always possible to build a deterministic pushdown automaton from a VPL.
A VPG (Alur and Madhusudan, 2009) is formally defined as a tuple , where is a set of nonterminals, a set of terminals, a set of production rules, and the start nonterminal. The set of terminals is partitioned into three kinds: , , , which contain plain, call, and return symbols, respectively. The stack action associated with an input symbol is fully determined by the kind of the symbol: an action of pushing to the stack is always performed for a call symbol, an action of popping from the stack is always performed for a return symbol, and no stack action is performed for a plain symbol. Notation-wise, a terminal in is tagged with \guilsinglleft on the left, and a terminal in is tagged with \guilsinglright on the right. For example, \guilsinglleft is a call symbol in , and \guilsinglright is a return symbol in .
Well-matched VPGs produce strings where each call symbol is always paired with a return symbol. They are formally defined as follows:
Definition 3.0 (Well-matched VPGs).
A grammar is a well-matched VPG if every production rule in adheres to one of the following forms:
-
(1)
, where stands for the empty string.
-
(2)
, where .
-
(3)
, where and .
Note that in terminal must be a plain symbol, and in a call symbol must be matched with a return symbol; these requirements ensure that any derived string must be well-matched. This is useful in languages like XML, where tags open and close in a nested, well-matched manner. For instance, the grammar rule “” represents an XML element that either contains content within matched open and close tags or is an empty single tag.
In this paper, we consider only well-matched VPGs, and use the term VPGs for well-matched VPGs. We also call rules in the form of matching rules, and rules of the form linear rules.
3.3. Visibly Pushdown Automata
A Visibly Pushdown Automaton (VPA) (Alur and Madhusudan, 2004) on finite strings over symbols , , and is a tuple where is a finite set of states, is the initial state, is a finite stack alphabet that contains a special bottom-of-stack symbol , is the transition function, where , , and , and is a set of final states.
The function means that upon reading \guilsinglleft, state is changed to state and is pushed onto the stack. Similarly, means that upon reading \guilsinglright and the stack top is , is changed to and is removed from the top of the stack (if is , the empty stack remains unaltered). Finally, means state is changed to upon reading symbol .
A stack is a non-empty finite sequence over ending in the bottom-of-stack symbol . The set of all stacks is denoted as . A configuration is a pair of state and stack . We define the single-step transition of configurations as tuple , based on the type of symbol and the transition functions:
-
(1)
If , then and for certain ;
-
(2)
If , then for certain and either , or and ;
-
(3)
If , then and .
We extend the single-step transition for string as . A string is accepted by VPA if . The language of is the set of strings accepted by .
3.4. Angluin’s L-Star Algorithm
We next briefly discuss the algorithm (Angluin, 1987), which learns a finite state automaton from a MAT in polynomial time. We first introduce a notion of equivalence. We define two strings equivalent w.r.t. to language if extending them with any suffix has the same membership result in :
We also introduce a notion of approximate equivalence relative to suffixes in a test-string set :
The algorithm operates in iterations and maintains two sets of strings: and , both of which start with . The set contains a set of test strings. The set contains a set of strings that are separable by , which means any two different strings in are not equivalent: . In addition, is closed in the sense that for any and any symbol there exists such that .
Given separable and closed , we can construct a hypothesis DFA: each string in becomes a state and we add a transition from with input symbol to the unique state such that ; the initial state is the empty string and acceptance states are those strings that are in . With the hypothesis DFA, we can ask the MAT to check if the DFA is equivalent to the oracle language. If it is, a DFA for the oracle language has been learned and the algorithm terminates. If not, the MAT gives a counterexample. With the counterexample, the algorithm can extend and and then use membership queries provided by the MAT to make and separable and closed again. Details can be found in the paper (Angluin, 1987).
4. V-Star for a Character-Based VPL
V-Star learns a Visibly Pushdown Automaton (VPA) using a MAT, which provides both membership and equivalence queries. For ease of exposition, we divide our discussion into two steps: in this section, we consider grammar inference for a character-based VPL, in which the tagging of call/return symbols is on individual characters; in the next section, we consider grammar inference for a token-based VPL, a more realistic setting in which the tagging is performed on tokens (sequences of characters).
For both steps, we develop an algorithm that not only infers the tagging but also constructs a VPA. We further prove that this constructed VPA achieves exact learning, meaning it recognizes the oracle language. We also provide an analysis of the algorithm’s time complexity.
This section proceeds as follows: we start with a precise problem statement in Section 4.1; we then introduce a new algorithm for learning a VPA in Section 4.2 assuming a given tagging function. Then, in Section 4.3, we study how to infer a tagging function that makes the VPA-learning algorithm terminate and achieve exact learning.
4.1. Problem Statement
V-Star seeks to infer a Visibly Pushdown Grammar (VPG) from a black-box oracle that knows a VPL. We next define the precise knowledge of the oracle and what queries it allows.
We assume is the alphabet set from which valid strings can draw characters. A VPL tags each character in as a call symbol \guilsinglleft, a return symbol \guilsinglright, or a plain symbol. This is modeled by a tagging function , which maps a character to either \guilsinglleft, \guilsinglright, or itself. This function extends to strings: , where is the length of and is its -th character. Given a tagging function , we define the terminal set (also denoted as when is clear from the context) as the set of tagged characters: , where , , and include call, return, and plain symbols defined by , respectively.
An oracle knows a language and a tagging function such that the tagged language is a VPL over terminal set . The oracle can answer membership and equivalence queries in active learning. The oracle’s ability to answer these queries is modeled as two functions. The membership query function, , is defined as follows:
That is, it returns true iff the input string belongs to . Note that input strings to membership queries do not carry tags, which reflects the fact that existing oracles are typically recognizers/parsers that take untagged strings. An example oracle used in our experiments is an off-the-shelf JSON parser, which takes untagged JSON strings; the goal of V-Star is to learn the JSON grammar from this oracle. Also note that we sometimes abuse the notation and pretend that can also take tagged strings, in which case it performs membership testing using the string after tagging is removed; i.e., for a tagging function , is defined as .
We next define the equivalence query function, which checks the equivalence between the oracle language and the language defined by a hypothesis VPA proposed by some learning algorithm. One complication is that the tagging function produced by the learner might be different from the oracle tagging function, even if the underlying untagged language is the same as the oracle one. This is due to the inherent flexibility of VPL tagging. As an example, suppose the oracle language is , then its underlying untagged language is the same as the untagged language of , which tags only and , or of , which tags only and .
Since what is relevant is the underlying untagged language, we should allow a learner to learn a different tagging function. Therefore we assume that the learner produces a hypothesis Visibly Pushdown Automaton (VPA) , as well as a hypothesis tagging function . The learner should achieve exact learning, formally stated as , where
Now the equivalence query function is defined as follows: returns none when the oracle language is equivalent to the untagged language recognized by and otherwise returns some such that .
V-Star’s active learning goal is, with an oracle that provides membership and equivalence queries, to learn a tagging function and a VPA so that exact learning is achieved.
The Unique Pairing assumption for oracle languages
To simplify the tagging inference algorithm that will be discussed in Section 4.3, we assume that in the oracle VPL , a call symbol is uniquely paired with a return symbol; i.e., if \guilsinglleft is matched with \guilsinglright in one sentence, then \guilsinglleft can be matched with only \guilsinglright in every sentence of the language. This assumption simplifies our algorithm design, and is satisfied by languages we experimented with (e.g., XML and JSON). We now represent pairs in a tagging function as a tagging , where , . While Algorithm 3 technically can be adjusted to operate without the above assumption, efficiency would be significantly decreased.
4.2. Learning VPA with Known Tagging
This subsection outlines an algorithm for learning a Visibly Pushdown Automaton (VPA) using a MAT, assuming a tagging function as input. is the tagged alphabet according to and is the oracle language tagged with . We assume that must make contain a set of well-matched strings. To avoid clutter, we will write for and for in this subsection. While there were prior VPA-learning algorithms proposed under this setting, some required more information from the oracle, such as the stack content during VPA execution (Michaliszyn and Otop, 2022). Isberner (2015) used advanced discrimination tree structures to minimize the number of membership queries; however, both Isberner (2015) and Howar (2012) discussed that discrimination tree-based algorithms could significantly raise the number of equivalence queries. Since in our implementation we simulate equivalence queries using membership queries (see Section 6), increasing the number of equivalence queries would escalate the simulation effort.
In this section, we introduce a VPA learning algorithm based on -SEVPA ((Alur et al., 2005; Kumar et al., 2007)) and demonstrate its polynomial-time efficiency in Theorem 4.7. Although the concept of polynomial-time VPA learning has been previously explored, as in Isberner (2015)’s TTT-VPA, our approach differs by adopting a table-based methodology, inspired by the clarity and directness of the algorithm (Angluin, 1987). This shift not only simplifies the presentation but also makes it easy to interface with tag-inference algorithms that we will discuss later in this paper.
Our algorithm is outlined in Algorithm 1. At every iteration, it maintains a set of separable and closed equivalence states and test strings in . The current states are used to produce a hypothesis VPA through . It then queries the oracle using an equivalence query. If the query does not produce a counterexample, then the iterative process terminates with the hypothesis VPA as the result; otherwise, the returned counterexample is tagged through the assumed tagging function and employed to refine the current set of equivalence states and test strings, through and . Next we describe , , and , starting with some background information.
4.2.1. Background: -SEVPA and Congruence Relations
Unlike regular languages, a VPL may not have a unique minimum-state deterministic pushdown recognizer. Nonetheless, partitioning the call symbols into distinct groups and mandating the following ensure the existence of a unique minimal VPA: (1) the states are partitioned to a set of modules (each is a set of states), with the -th module as the base module with the initial state and the -th module for the -th group of call symbols with ; (2) the machine stays in the same module when encountering a plain symbol; the machine transitions to a unique entry state in the -th module when encountering a call symbol from the -th group; the machine transitions back to the caller module when encountering a return symbol. Such a VPA is known as a -Single Entry VPA (-SEVPA (Alur et al., 2005)) and is similar to a control-flow graph with the -th module for the main function and the -th module for the -th function in a program.
The minimal -SEVPA can be defined with a set of congruence relations.
Definition 4.0 ().
[Congruence Relations for the Minimal -SEVPA (Alur et al., 2005)] Given a VPL over , let , , represent the -th group of call symbols. Given well-matched strings and , we introduce congruence relations:
(1) | |||||
(2) |
Note that is the Myhill-Nerode right congruence and can be used to construct the minimal DFA for a regular language. For when , the context strings assume specialized forms: the left context string ends with a call symbol from the -th group, and the string is a well-matched string, since both and are well matched. From the above congruence relations, we can construct the minimal -SEVPA: the equivalence classes of become the states of the -th module, with being the unique entry state of the -th module; transition edges can also be added (see Alur et al. (2005)): e.g., transitions to for plain symbol , and to for call symbol .
In our algorithm, we set to be the number of call symbols decided by the input tagging function so that each call symbol is in its own group. We write for the -th call symbol. This partitioning is practical because call symbols often fulfill diverse roles and hence find themselves in separate contexts. Further, Proposition 2 in Alur et al. (2005) tells that enlarging can lead to a more compact VPA.
4.2.2. Access Words and Test Words
At each step, V-Star maintains , which contains
-
(1)
a set of modules to , each containing empty string and a set of well-matched access words in ,
-
(2)
and a set of test words to , with containing strings in for testing and containing strings in the form of for testing , where and are in , is the -th call symbol, and is well matched.
Given test words , two well-matched strings and are equivalent, denoted as , if (1) when , , iff ; and (2) when , , iff . These are essentially the same equivalence relations as those in Definition 4.1, relative to test words in .
We define the following two properties of :
-
(1)
Separability: no two distinct strings in are -equivalent, meaning .
-
(2)
Closedness: for every and (defined below), there is some such that .
Definition 4.0 (Nested Words and ).
Given , along with , we define the nested words for , denoted as , as
where is the -th call symbol. We define .
Our learning algorithm is then based on the following set of propositions.
Definition 4.0 ( function).
For separable and closed , we can construct a hypothesis -SEVPA, denoted as as follows. The set of states of is . We write as . The initial state is . Define the set of acceptance states, , to be , which can be constructed via membership queries. The transition function from the current state , , and the next input symbol is defined as follows:
-
(1)
For plain symbol , the transition is , where .
-
(2)
For call symbol , the transition is , the unique entry state for module .
-
(3)
For return symbol \guilsinglright, the transition is , where .
The target state in each transition exists by closedness and is unique by separability. To run the VPA on a string , start with the initial state and an empty stack and use for transitions. The automaton accepts a string if it terminates in a configuration with a state within and an empty stack.
Proposition 4.0 ().
If is separable and language is a VPL, then the number of states in is bounded above by the number of states in the minimal -SEVPA for VPL .
Proof.
For two strings and , if (Definition 4.1), then . Hence, the number of equivalence classes of is less than that of , which corresponds to the number of states of the -th module in the minimal -SEVPA. Further, since is separable, each element of corresponds to a unique equivalence class of . Therefore, is bounded above by the number of equivalence class of , which is bounded above by the number of states of the -th module in the minimal -SEVPA. Since the number of states in is , it is upper bounded by the number of states in the minimal -SEVPA for . ∎
Proposition 4.0 ().
If is separable but not closed, then using membership queries one can find and such that and the rest remain separable.
Proof.
Since are not closed, there exists for certain and such that is not -equivalent to any state in . Using membership queries (by enumerating all test strings in ) we can find such a and , and then add to , which remains separable by construction. ∎
Algorithm 2 outlines the function, which keeps applying Proposition 4.5 until becomes separable and closed.
Proposition 4.0 ().
Suppose that is separable and closed, and let be the hypothesis automaton (Definition 4.3). Suppose the oracle returns a counterexample for an equivalence query with . Using membership queries, one can find and and such that is separable, or find when such that is separable.
Proof.
Let be the length of . Let be the initial state of , and be the transition function of . For , define to be the state in reached by after reading the prefix of , and define as the corresponding stack. For convenience, we write for the state in module .
We define the context of as follows. When is empty, we define the context to be . Otherwise, let . We define the context as
We denote the context of as . We say that state is correct if .
State is obviously correct since its context is . However, state must be incorrect because of the following. First, state must be in module : if the counterexample is accepted by , then is well-matched under ; otherwise, the counterexample is in , which is a well-matched language. Therefore, we write . Next, since is a counterexample, we have . By the construction of , we have . Therefore, we have , which means state is incorrect.
Now we can then use binary search (using membership queries) to find such that is correct, while is incorrect. We first show that cannot be a call symbol. Otherwise, we must have and . The context of is . We have
but , a contradiction.
Assume is a plain symbol. Since is a state in module , we let and . By definition of the transition function of , is the unique element of that is -equivalent to . On the other hand, the test distinguishes from , since we can get .
Otherwise, is a return symbol \guilsinglright. Let . Recall that at state , reads \guilsinglright and transfers to such that . Notice that . Therefore, is correct. We let and . By definition of the transition function of , is the unique element of that is -equivalent to . On the other hand, the test distinguishes from . We conclude that , and that is separable. ∎
We call the procedure in Proposition 4.6 , which takes a separable and closed and a counterexample and returns a separable .
With these lemmas, we can prove the following theorem; its proof is provided in Appendix A.1.
Theorem 4.7 ().
Given a tagging function such that language is a VPL, the minimal -SEVPA of language can be learned in polynomial numbers of equivalence and membership queries.
Therefore, if language is a VPL, then we can use Algorithm 1 to learn the minimal -SEVPA of language . However, in general, a tagging function does not necessarily introduce a VPL , even if each sentence in is well-matched. For example, consider the language and the tagging that maps and to plain symbols. The resulting language is trivially well matched (as it does not have call/return symbols), but it is not a VPL. In the next section, we discuss the procedure to find a right tagging function that makes a VPL.
4.3. Tagging Inference
Seed strings |
An oracle VPG and a seed string.
We next propose an algorithm that infers a tagging function so that the tagged language is a VPL. Then with Theorem 4.7, we can use the tagging function in Algorithm 1 to learn the VPL efficiently. Our algorithm takes a set of seed strings for inference. In practice, the seed strings can be collected via existing corpora of data (e.g., a corpus of XML strings) or via valid inputs to black-box implementations of oracles (e.g., an XML parser).
We will use the oracle VPG in Figure 1 as a running example, which includes a single seed string. Note that seed strings are untagged: it is the task of our algorithm to infer the tagging. As discussed earlier, the inferred and the oracle tagging functions may differ. For the example VPG in Figure 1, we can remove the tags on either or and the resulting grammar is still a VPG and generates the same untagged language. As we will explain, in such a case the algorithm opts for the outermost tags in its inferred VPG (i.e., \guilsinglleft and \guilsinglright), while treating and as plain symbols.
This section unfolds as follows. We first introduce a VPL pumping lemma, which enables a nesting test to filter out invalid taggings. We then present a tagging inference algorithm based on the nesting test and state theorems discussing properties of the algorithm. We leave most of the proofs of these theorems to Appendix B.
One straightforward observation is that a tagging is invalid if some seed string after tagging through is not well matched. However, this test alone would not eliminate too many possibilities. We next introduce a nesting test to filter out more invalid taggings. For this, we first propose a pumping lemma for VPLs. This lemma diverges from traditional pumping lemmas for regular languages and context-free languages by focusing on the unique requirements of call and return symbols in VPLs.
Lemma 4.0 (Pumping Lemma for VPLs).
For any VPL , there exists a positive number such that, for any string in with length greater than , it is possible to express according to one of the following conditions:
-
(1)
(Regular Pumping) We can partition into for strings and , with being non-empty, such that remains in for all .
-
(2)
(Nesting Pumping) We can partition into for strings and , with and being non-empty, containing a call symbol, and containing a return symbol, such that is valid for all .
For example, consider the VPG in Figure 1. Any string in with length greater than 6 can be decomposed based on the above two ways; e.g., for the tagged seed string , we have in the language, for ; it happens it can also be decomposed through regular pumping: we have in the language, for . We now extend the concept of nesting pumping to untagged strings, calling them nesting patterns.
Definition 4.0 (Nesting Patterns).
For an untagged string in the oracle language , a nesting pattern is a partitioning of , where (1) and are non-empty, (2) for all , (3) but for (both ), . The third condition precludes the possibility that represents a regular pumping, which allows and for all . When , , and are not the focus, we may succinctly write a nesting pattern in a string as a pair .
Definition 4.0 (Compatible Tagging).
We say that tagging is compatible with a nesting pattern , if there exists a pair in , such that (1) includes and includes , and (2) includes an unmatched \guilsinglleft and includes an unmatched \guilsinglright in .
We say that tagging is compatible with a set of seed strings , if (1) strings in are well-matched under , and (2) is compatible with all nesting patterns of . We say that tagging is compatible with language if it is compatible with each string in the language.
Theorem 4.11 ().
Given an oracle language , for any tagging compatible with , language is a VPL.
For the example in Figure 1, the single seed string’s nesting patterns include
One compatible tagging is : firstly, the tagging would make the seed string well matched; secondly, each nesting pattern includes . By Theorem 4.11, when we tag as a call symbol and as a return symbol, the oracle language becomes a VPL. Other compatible taggings include , , , and . In contrast, the tagging is incompatible since this tagging would not make the seed string well matched.
Recall that Theorem 4.7 tells us that, if a tagging makes a VPL, we can efficiently learn the VPL under active learning through Algorithm 1. Now Theorem 4.11 tells us that a compatible tagging makes a VPL. Therefore, what is remaining is to infer a compatible tagging. With such a tagging, we can use it in Algorithm 1 to learn a VPL whose untagged strings are the same as the oracle language.
We next describe Algorithm 3, which performs inference of a compatible tagging using an input set of seed strings. Its runtime complexity is exponential in the worst case; however, as will be discussed in our evaluation section, its runtime performance on practical grammars is efficient. As an overview, starting with a bound , the algorithm (1) employs a bounded checking approach in the function to compute candidate nesting patterns for seed strings , and (2) for , the function tries to find a compatible tagging using a search algorithm (which may backtrack). If the function fails to find a compatible tagging, we increase by and start anew. In more detail, in , for each disjoint substring pair in each seed string, we check if for and check if for and . If so, is a candidate nesting pattern. In the function, we begin with an empty tagging , which tags each character as a plain symbol. We then check if a candidate nesting pattern is already covered by the current tagging; if not, we treat a symbol in as a call symbol and a symbol in as a return symbol and continue the search process.
Returning to our example shown in Figure 1, the seed string includes the nesting pattern . Our algorithm prioritizes the outermost characters for pairing. Consequently, the pair is selected and it covers all nesting patterns of the seed string, resulting in a compatible tagging .
The following theorem states that for some bounded Algorithm 3 terminates and returns a compatible tagging.
Theorem 4.12 (Termination and Correctness of Algorithm 3).
Let be the number of states of the minimal -SEVPA for the oracle VPL. There exists a number , such that returns a tagging that is compatible with a finite set of seed strings .
Note that the theorem is with respect to a finite set of seed strings . It does not say whether the found tagging is compatible with all strings in the oracle language . We address this by demonstrating the existence of a finite set of seed strings , for which a compatible tagging with ensures compatibility with all strings in .
Theorem 4.13 (Finite and Sufficient Seed Strings).
For any given oracle language , there exists a finite set of seed strings, denoted as , such that any tagging that is compatible with is also compatible with .
The proof of Theorem 4.13 is provided in Appendix B.7. We illustrate how is constructed using the VPG shown in Figure 1. In general, for each matching rule where the nonterminal can be recursively rewritten into via a set of derivations, we generate a string reflecting this recursion and incorporate it into . The example VPG includes matching rules and . We start by expanding to reveal , followed by expanding to unveil , resulting in the pattern . Then the expansion of to gives us a seed string . We also generate a seed string witnessing the recursive transition from to and back to , which would lead to strings like .
In conclusion, it is established that a finite set of seed strings exists, enabling our algorithm to identify a tagging that is compatible with the oracle language . This compatibility guarantees that the tagged oracle language is a VPL. With the learned tagging as an input, Algorithm 1 can be employed to achieve exact learning.
5. V-Star for a Token-Basd VPL
In Section 4, we assumed that tagging of the oracle language is on individual characters; i.e., each character is uniquely tagged. This assumption does not always align with real-world scenarios. For instance, in JSON, a curly bracket { may serve as a call symbol, yet it can also be a key, exemplified by {"{":true}; in XML documents, an opening tag such as <p> functions as a call symbol, but it is composed of multiple characters. In this section, we enhance V-Star to address these scenarios.
5.1. Problem Statement
The oracle language is a VPL when sentences in are converted to sequences of tokens determined by an oracle tokenizer. Formally, a tokenizer function, , translates a string from into a sequence of tagged tokens, where represents the set of tagged tokens given by ; we write when is clear from the context. The language is assumed to be a VPL over tokens in . Each category of token is defined as a regular language, often specified by a regular expression. The notation indicates that string belongs to token . We use metasymbols , , or for call, plain, or return tokens, respectively.
OPEN | |||
CLOSE | |||
TEXT |
An example XML grammar and the associated lexical rules.
A toy XML grammar is shown in Figure 2 and we use the seed string as an example. The tokens are OPEN, TEXT, and CLOSE. The oracle tokenizer converts <p><p>p</p></p> into the token sequence [OPEN,OPEN,TEXT,CLOSE,CLOSE], where OPEN is a call symbol, TEXT is a plain symbol, and CLOSE is a return symbol.
The oracle still provides membership and equivalence queries. The membership query function is as before. However, we change the form of equivalence queries. The reason for the change is to convert the oracle language to a character-based VPL so that we can reuse Algorithm 1 for learning a hypothesis VPA.
To model equivalence queries, we first define a converter function. A tokenizer identifies boundaries of call and return tokens for a string. We then use to transform a valid string into a new string by inserting artificial call and return symbols to mark token boundaries. This process is formalized next. Given a tokenizer with , we first build an extended character set : for the -th pair of call and return tokens and , we generate a pair of call and return symbol and \guilsinglright outside of . We define as . Then, the transformation of into language over proceeds as follows. Let , where . We construct based on tokenization: for each , if belongs to , the call symbol is added before in ; if is in , the return symbol \guilsinglright is added after in . For instance, for the XML grammar in Figure 2, with the call-return token pair being (OPEN,CLOSE), our extended character set have two additional characters, say and . The seed string is converted to . Note that the resulting string after conversion is a well-matched string in a character-based VPL that has the call symbol and return symbol . This allows us to reuse the previous algorithm for learning a character-based VPA.
With the converter function defined, we can model an equivalence query. takes a hypothesis VPA and a hypothesis tokenizer and returns none when the oracle language is equivalent to the unconverted language recognized by and otherwise returns some such that , where
A learner achieves exact learning if .
Similar to the method we discussed in Section 4.3, we utilize nesting patterns to identify the boundaries of call and return tokens. Our objective is to discover a compatible tokenizer, which ensures that the language is a VPL. We will demonstrate the existence of a finite set of substrings from which our algorithm can successfully learn a compatible tokenizer. Then the aforementioned converter function transforms the oracle language into a character-based VPL, which according to Theorem 4.7 can be efficiently learned by Algorithm 1.
Assumptions for oracle languages and tokenizers
We previously defined a tokenizer as a function that maps a string to a list of tokens. However, assuming an arbitrary tokenizer is insufficient, as it has been demonstrated that any CFG can be mapped to a VPG through some tagging (Alur and Madhusudan, 2004). Take, for instance, the CFG . A tokenizer might tag differently based on its position within a string, e.g., maps the string to the token list , where \guilsinglleft and \guilsinglright represent call and return tokens, respectively; the resulting language is a VPG. To simplify tag learning to the setting where tagging is context independent, the oracle tokenizer and the language are assumed to satisfy the following properties:
Tokenization Consistency. For a string , if each substring belongs to a token , then . For example, string <p></p> can be split into [<p>,</p>]; this assumption requires it to be tokenized as [OPEN, CLOSE].
Separation. Strings for different tokens do not overlap. For the previous example of , tagging the first as a call token and the last as a return token would violate this property.
Exclusivity. A prefix or suffix of a call or return token cannot serve as an infix of . Exclusivity is not required for a token that contains only a single character.
Unique Pairing. Each call token is uniquely paired with a return token, similar to an assumption we made for the setting of character-based VPLs.
Token Fixed Prefix and Suffix. For each call or return token , if contains more than a single character, we require that there exists a prefix and a suffix , such that all strings of starts with and ends with . Further, there exists a string of , such that the combination of the prefixes and suffixes of constitutes a sufficient set of test strings for exact learning of the token using , from the membership query function , where are any strings such that .
-Repetition. Given a positive numbr , for each valid string where is a nonempty substring, we say that is -repeatable in , if is also a valid string. A language and its tokenizer are said to satisfy -Repetition if, for any valid string and any substring in , if belongs to a call or return token , but is not tokenized as in , then is -repeatable in .
For example, consider the JSON string {"{":true}. Suppose is the second {; since it is inside quotes, it belongs to part of the token for a JSON key (i.e., "{"), even though itself is a call token. For any , is -repeatable since the string remains a valid JSON string. In our implementation, we set to .
While we make the aforementioned assumptions, our approach is still quite expressive, since the above properties are typically satisfied by practical grammars, including those in our evaluation.
5.2. Tagging Inference for Tokens
In this section, we define the compatibility of tokenizers and present a theorem about the relation between a compatible tokenizer and the converted language . Then, we discuss an algorithm that infers a compatible tokenizer from given seed strings.
To define compatible tokenizers, we introduce some additional definitions. Given a tokenizer , recall that for a string in , is built by inserting artificial call and return symbols to . Now, let . We define as the substring in that corresponds to ; additionally, if inserted a call symbol between and , then includes and starts with that call symbol; if inserted a return symbol between and , then includes and ends with that return symbol. For example, for the seed string , , and .
Definition 5.0 (Compatible Tokenizers).
We say that a tokenizer is compatible with a set of nesting patterns , if for each nesting pattern in , , and , there exists a pair of artificial call and return symbols in , such that (1) includes and includes , and (2) is unmatched in and is unmatched in .
We say that a tokenizer is compatible with a set of seed strings , if (1) for each string in , is well-matched, and (2) is compatible with all nesting patterns of .
Now we present Theorem 5.2 as the basis for exact learning.
Theorem 5.2 ().
Assume the oracle language and the oracle tokenizer satisfy the Tokenization Consistency and Separation properties. Given a tokenizer that is compatible with the oracle language , language is a VPL.
Now, we propose Algorithm 4 to infer a hypothesis compatible tokenizer. Instead of finding a full-fledged tokenizer, the algorithm infers a partial tokenizer, which recognizes only call and return tokens in an input string; the syntax of plain tokens is instead learned during the VPA learning process. As a result of this choice, substrings between call/return tokens recognized by a partial tokenizer are implicitly treated as plain tokens. We represent a partial tokenizer as a set , where and are the regular expressions for the -th paired call and return token, respectively. Function and compatibility are similarly defined for a partial tokenizer ; we omit them for brevity.
At a high level, Algorithm 4 identifies call/return tokens by enumerating potential prefixes and suffixes based on the Token Fixed Prefix and Suffix assumption. Further, under the Exclusivity assumption, we can prove that oracle call/return tokens must appear in for certain nesting pattern ; the proof is provided in the appendix as Lemma C.2. Therefore, we restrict our enumeration to substrings within . Our approach begins by searching within and then progressively expands the search space to . Upon identifying a candidate prefix-suffix pair for a token, we learn the token’s lexical rules as a regular expression within the prefix-suffix pair using at line 6; in this learning, we simulate the equivalence queries using test strings obtained by combining the prefixes and suffixes of and , respectively. We then incorporate the tokens into the partial tokenizer and proceed to assess the tokenizer’s compatibility with the nesting patterns of seed strings (line 7).
One compatibility condition is that the seed strings after tokenization should be well-matched; for that, Algorithm 5 is used to tokenize a string based on a given partial tokenizer. The main challenge of tokenization is that we have only a partial tokenizer and we need to rely on -Repetition to deal with the case when a plain token string contains a call/return token as part of its substring. E.g., in , the second { is actually part of the plain token "{" and should not be treated as a call token. To demonstrate Algorithm 5, consider a partial tokenizer and an input JSON string . Start with string index and token list , Algorithm 5 matches the first { as and pushes it to . Since does not result in any match, is updated to , where the second { is matched as ; however it is not added to the token list since it is -repeatable. Finally, the last } is matched as . As a result, Algorithm 5 returns .
We next illustrate the steps of Algorithm 4 using our XML example. We start with an empty tokenizer . We then iteratively select a nesting pattern , tokenize using Algorithm 5, and verify the compatibility of with the tokenization. In our example, we start with the seed string and pick a nesting pattern. Suppose V-Star picks the outmost pattern . The token list of is empty, since there is no rule to find a token yet. Apparently, this tokenizer is not compatible with .
We then extend by a call-return token pair learned from derived from or . By enumerating candidate prefixes and suffixes within , we can build the first call-return token pair. From , we first pick the outmost as ; then in , we pick the outmost as . By learning the tokens’ lexical rules from membership query functions and , we identify two regular expressions <p> and </p> for the call and return tokens, respectively. Note that if the open tag contained XML attributes, the learned lexical rules would encompass regular expressions that specify these attributes. To check if the partial tokenizer , is compatible with , we need to tokenize following Algorithm 5, which returns the token list . It can be shown that this partial tokenizer is compatible with all nesting patterns of string . Therefore, Algorithm 4 ends here and returns this compatible tokenizer.
Lemma 5.0 (Finite and Sufficient Seed Strings).
Assume the oracle language and the oracle tokenizer satisfy the Tokenization Consistency, Separation, Exclusivity, Unique Pairing, Token Fixed Prefix and Suffix, and -Repetition properties. There exists a finite set of seed strings , with which we can find a tokenizer that is compatible with the oracle language using Algorithm 4.
6. Evaluation
In this section, we discuss V-Star’s implementation, evaluation and its comparison with two other state-of-the-art grammar inference tools, Glade (Bastani et al., 2017) and Arvada (Kulkarni et al., 2022), in the context of inferring grammars from program inputs.
Implementation
While black-box programs naturally support membership queries, direct support of equivalence queries is absent. To instantiate the MAT, we approximate equivalence queries through membership queries. In particular, we construct a set of strings by combining prefixes, infixes, and suffixes of the seed strings; for each such string , if is well-matched, we add it to a set of test strings. The set of test strings is then used to check the consistency between the hypothesis VPA and the oracle language. A test string becomes a counterexample if it witnesses inconsistency (i.e., either the hypothesis VPA or the oracle accepts the string, but not both). Similar ideas have appeared in conformance testing (Kumar et al., 2006; Aichernig et al., 2024).
Our previously discussed algorithm produces a visibly pushdown automaton (VPA), instead of a visibly pushdown grammar (VPG). Upon the successful learning of a VPA, we transform it into a VPG using methods outlined by Alur and Madhusudan (2004).
Datasets
For our experiments, we replicated the evaluation methodology of the Arvada study, utilizing their datasets (Kulkarni et al., 2022), including the oracle grammars, datasets for evaluating the recall (discussed later), and seed strings. We selected the grammars of JSON, LISP, XML, While, and MathExpr, due to their distinct characteristics of being VPGs.
Metrics
We evaluate the performance of V-Star using four key metrics: Recall, Precision, F-1 Score, and Number of Membership Queries. We define each metric as follows:
-
(1)
Recall: This metric is the probability that a string of the oracle grammar is also a string of the learned grammar (Bastani et al., 2017). For finite languages, it can be defined as: . Due to the potential infinity of the languages, it may be impractical to compute recall directly. Instead, we approximate it by using a representative dataset from the oracle language and then calculating the proportion of this dataset that is accepted by the learned grammar.
-
(2)
Precision: Contrary to recall, precision is the probability that a string in the learned language is accepted by the oracle (Bastani et al., 2017). For finite languages, it can be defined as: . As with recall, we approximate precision by sampling strings from the learned grammar and calculating the percentage of strings that are accepted by the oracle. We adopt the same sampling method from Arvada (Kulkarni et al., 2022).
-
(3)
F-1 Score: The F-1 score is the harmonic mean of precision and recall, defined as . where is recall and is precision. The F-1 score serves as a measure of the overall accuracy, only reaching high values when both precision and recall are high.
-
(4)
Number of Unique Membership Queries: This counts the number of unique membership queries, i.e., distinct oracle calls, made during the learning process. Since a particular string might be queried multiple times, we cache the result after the first query, and only count unique queries. This metric serves as an efficiency measure.
Glade | ||||||
---|---|---|---|---|---|---|
#Seeds | Recall | Precision | F1 | #Queries | Time | |
json | 71 | 0.42 | 0.98 | 0.59 | ||
lisp | 26 | 0.23 | 1.00 | 0.38 | ||
xml | 62 | 0.26 | 1.00 | 0.42 | ||
while | 10 | 0.01 | 1.00 | 0.02 | ||
mathexpr | 40 | 0.18 | 0.98 | 0.31 |
Arvada | |||||
---|---|---|---|---|---|
Recall | Precision | F1 | #Queries | Time | |
json | 0.97 ± 0.09 | 0.92 ± 0.08 | 0.94 ± 0.05 | ± 394 | ± |
lisp | 0.38 ± 0.26 | 0.95 ± 0.08 | 0.50 ± 0.18 | ± 307 | ± |
xml | 0.99 ± 0.02 | 1.00 ± 0.00 | 1.00 ± 0.01 | ± | ± |
while | 0.91 ± 0.20 | 1.00 ± 0.00 | 0.94 ± 0.14 | ± 563 | ± |
mathexpr | 0.72 ± 0.24 | 0.96 ± 0.03 | 0.80 ± 0.16 | ± 421 | ± |
V-Star | ||||||||
---|---|---|---|---|---|---|---|---|
Recall | Precision | F1 | #Queries | %Q(Token) | %Q(VPA) | #TS | Time | |
json | 1.00 | 1.00 | 1.00 | 8043 | ||||
lisp | 1.00 | 1.00 | 1.00 | 693 | ||||
xml | 1.00 | 1.00 | 1.00 | 682 | ||||
while | 1.00 | 1.00 | 1.00 | 119 | ||||
mathexpr | 1.00 | 1.00 | 1.00 | 2602 |
Results
Table 1 summarizes the performances of Glade, Arvada, and V-Star on oracle VPGs, with the results of Arvada and Glade assessed on the same platform as V-Star, utilizing the Arvada artifact (Kulkarni, 2023). The table shows that V-Star achieves exact learning for all oracles, exhibiting superior accuracy compared to other tools. However, V-Star issues a greater number of queries than Glade and Arvada, resulting in greater inference time. This primarily stems from (1) the substantial number of test strings used in approximating equivalence queries, and (2) the fact that V-Star consumes seed strings without pre-processing. In contrast, Glade and Arvada employ a pre-tokenization strategy, such as grouping digits or letters as a single terminal, which reduces seed string lengths. We take our approach since V-Star can learn tokens. Overall the evaluation shows that V-Star is more accurate but takes more time to infer grammars. In grammar learning, we believe that accuracy is a more important goal as a more accurate grammar benefits downstream applications greatly. Improving efficiency of V-Star (e.g., using heuristics of target grammars) while not decreasing accuracy is left for future work.
V-Star requires a considerable number of membership queries for the MathExpr grammar. This can be attributed in part to the large number of constant function names (26 in all) within the grammar, such as “sin” or “cos”. In its quest for high accuracy, V-Star explores various combinations of these constant names exhaustively. We acknowledge that this approach could be further optimized and propose this as an avenue for future improvement.
In Table 1, we include data on the percentage of membership queries allocated for token inference (“%Q(Token)”) and for learning VPA (“%Q(VPA)”). It can be observed that the majority of queries are utilized for VPA learning. This is mainly because seed strings tend to be short, leading to fewer potential nesting patterns. One exception is XML, where most queries are for token inference. This is because the XML grammar, primarily based on nested tag pairs, allows for easier inference of the overall grammar once the opening and closing tags (call and return tokens) are identified. Furthermore, many queries are required to infer the lexical rules of XML attributes. Additionally, the table provides information on the count of seed strings (“#Seeds”) used in our evaluation. For the grammars assessed, V-Star requires a relatively small number of seed strings to achieve exact learning, attributed to its strategy of employing a wide range of substring combinations to construct test strings for effective simulating equivalence queries; column “#TS” shows the number of test strings constructed.
7. Future Work
We believe the performance of V-Star can be further improved with more advanced methods for generating counterexamples, such as using machine learning tools to infer counterexamples from seed strings and a VPA. A related direction is to investigate the potential adaptation of V-Star with discrimination trees. Other grammar inference tools that are based on discrimination trees such as TTT (Isberner, 2015) enhance inference efficiency by reducing counterexample lengths and minimizing membership queries. It remains to be seen how V-Star can be adapted in this manner and what improvements this can yield.
The present study focuses primarily on inferring well-matched VPGs using V-Star. However, our preliminary experience suggests that V-Star can also be effectively employed to learn general VPGs (Alur et al., 2005) with open call and return symbols. A general VPG can be used to specify streaming data. As such, the learning problem for general VPGs is a promising direction for future work.
Our method makes the assumption of unique call-return token pairing to ease tokenizer inference and reduce computational complexity, as matching one call token with multiple return tokens complicates tokenizer inference. It would be interesting future work to consider the implications of relaxing this assumption to enhance flexibility.
Experimentally, we focus on languages such as XML and JSON to align with benchmarks used by prior tools for a direct comparison. It would be interesting to evaluate V-Star on more complex programming language grammars to check its effectiveness on those grammars.
Improving the readability of VPGs inferred by V-Star is still a challenge. Currently, the grammars generated tend to be larger and less readable than oracle grammars, due to the inherent rigid requirements of VPG rules, the inclusion of lexical rules, and automatically named nonterminals. Although we have made attempts to refactor grammars using regular expressions, these solutions are largely heuristic and may not consistently yield optimal results. Exploring machine learning-based approaches presents a promising avenue to systematically enhance the clarity and conciseness of inferred grammars, making them potentially more accessible and understandable for users.
Finally, VPGs learned by V-Star may provide a valuable starting point for better inference algorithms of CFGs. For instance, similar to the CFGs learned by Glade (Bastani et al., 2017), the VPGs inferred by V-Star can serve as inputs for machine learning tools such as REINAM (Wu et al., 2019), which improves the input grammar with reinforcement learning. Comparing the improvements enabled by these different starting grammars would be an intriguing line of inquiry.
8. Conclusions
This paper introduces V-Star, an algorithm designed to take advantage of nesting structures in languages to achieve exact learning of visibly pushdown grammars. Through a set of novel techniques to infer token boundaries and tag call/return tokens, V-Star demonstrates its capability to learn a diverse array of practical languages. Our preliminary experiments are promising and show V-Star’s advantages of accurate learning.
Appendix A Proofs of Theorems in Section 4.2
Theorem A.1 ().
Given a tagging function such that language is a VPL, the minimal -SEVPA of language can be learned in polynomial numbers of equivalence and membership queries.
Proof.
We run Algorithm 1 with a target language over the alphabet . Define as the state count of the minimal -SEVPA of and as the maximum length of counterexamples returned by equivalence queries. Proposition 4.4 establishes that the number of equivalence queries does not exceed , as each iteration expands the number of states in by a minimum of one. This also shows that the algorithm must terminate.
A counterexample returned by an equivalence query causes at most membership queries as detailed in Proposition 4.6, resulting in no more than membership queries during Step 4 of Algorithm 1. Membership queries in Steps 2 and 5 involve words either of form or , where , , and . With bounded by at completion, total queries amount to at most .
In conclusion, the number of queries remains polynomially bound by , , and , including membership queries and equivalence queries. ∎
Appendix B Proofs of Theorems in Section 4.3
Given tagging , we say that string is -well-matched, if is well-matched.
Definition B.0 ().
[Parse Tree] Given a grammar , a parse tree with respect to grammar is an ordered tree where (1) the leaves of the tree are terminals in or , and (2) each non-leaf node is a nonterminal in , where the children of the node are , , …, such that is a production rule in , or , such that is a production rule. The root of the tree should be , the start nonterminal of grammar . A parse tree of a string in the language of grammar is a parse tree whose leaves, when concatenated from left to right, form .
Lemma B.0 (Pumping Lemma for VPLs).
For any VPL , there exists a positive number such that, for any string in with length greater than , it is possible to express according to one of the following conditions:
-
(1)
(Regular Pumping) We can partition into for strings and , with being non-empty, such that remains in for all .
-
(2)
(Nesting Pumping) We can partition into for strings and , with and being non-empty, containing a call symbol, and containing a return symbol, such that is valid for all .
Proof.
Let VPG be a grammar of . We define as the length of the longest string that contains no recursion in any of its parse trees. Formally, in a parse tree of , if the subtree rooted with a nonterminal node contains another appearance of , we say there is recursion in the parse tree and we call a recursive nonterminal in the parse tree. is then defined as the length of the longest string whose parse trees do not contain recursion. This is well defined because the number of non-recursive parse trees is finite: any path that goes from the root to a leaf of a parse tree and exceeds the length of must have nonterminals and revisit at least one nonterminal twice.
For any string exceeding in length, one of its parse trees must have a recursive nonterminal; say it is . The derivation of the parse tree can be written as: , where . We have two cases:
-
(1)
If is empty, then cannot be empty since is not a valid VPG rule. Thus, remains valid, where is a terminal string derived from . This satisfies regular pumping.
-
(2)
If is not empty, then a matching rule is used somewhere in the derivation sequence that leads to the second appearance of . This is because by the VPG rules, if only rules of the form or were used, then must be the last symbol in the derived string , which contradicts with that is not empty. This leads to:
Here, . We then select as and as for nesting pumping.
∎
Lemma B.0 ().
Consider an oracle VPL , a VPG for , and an oracle tagging . For each string and , where are substrings, and are nonempty, if string is valid but string is invalid for and , then contains an oracle call symbol, and contains an oracle return symbol, and the two symbols are matched with each other in .
Proof.
In this proof, we abuse the notation to also mean .
We first show that as well as contains unmatched symbol. Otherwise, and contain only plain symbols or well-matched call-return pairs. For each and string , the derivation path of can be written as , where are two nonterminals in . This is because, suppose contains only plain symbols, then the derivation path of is of the form
for certain nonterminals . The case is similar when also contains well-matched substrings; we omit the discussion for brevity. Now, for each , we have the following derivations:
Apparantly, there exist and , such that , and a pair appears twice on both sides, i.e.,
Thus, both string and are valid. Given that , this is a contradiction.
Therefore, and must include unmatched symbols. Consider the type of the unmatched symbol in . If includes a return symbol \guilsinglright, where the matched \guilsinglleft is before , then is invalid, because has no additional call symbol to match \guilsinglright. Thus, includes a symbol \guilsinglleft, whose matched symbol \guilsinglright is after . If \guilsinglright is in , we are done. Otherwise, \guilsinglright is either in or in . Consider . Since the string is valid, the unmatched \guilsinglleft in must match a return symbol in .
In conclusion, contains an oracle call symbol, which matches a return symbol in . ∎
Theorem B.4 (Termination and Correctness of Algorithm 3).
Let be the number of states of the minimal -SEVPA for the oracle VPL. There exists a number , with which function returns a tagging that is compatible with a finite set of seed strings .
Proof.
First, we show that at least the oracle tagging can be found, which must be compatible with the pattern. This is because, from Lemma B.3, when , each candidate nesting pattern must either be invalidated, or contain oracle call-return pair unmatched in and , respectively.
Therefore, since any VPG for the oracle VPL can be used for the checking, we pick the specific VPG converted from the minimal -SEVPA by the method discussed in Alur and Madhusudan (2004), Theorem 5.3 (Visibly pushdown grammars), where is bounded by . ∎
For Lemmas B.5 and B.6, we first introduce another congruence relation by Alur et al. (2005). Two well-matched strings, and , are deemed congruent, denoted as , if their contexts coincide. Specifically,
This congruence is an equivalence relation, and is a VPL on if and only if the congruence relation admits a finite number of equivalence classes.
Given a tagging , denote the congruence relation over as .
Given a compatible tagging function and a string , the following Lemma B.5 shows that, if is well-matched, then has a bounded number of unmatched symbols.
Lemma B.0 ().
Given oracle language and oracle tagging , for each compatible tagging , there exists an upper bound positive number, denoted as , such that for each string , if is -well-matched and there exists context strings such that , then contains at most unmatched oracle call and return symbols.
Proof.
In this proof, to simplify the notation, we use “” or “” to also mean strings tagged by the oracle tagging function . For strings tagged by a compatible tagging function , we use “” and “” explicitly.
To simplify the problem, let us assume there is an oracle VPG that includes only one matching rule; we denote the matching rule as . As an overview, we show that for string that contains unmatched oracle call symbols \guilsinglleft, we can construct equivalence classes for the oracle congruence relation. Therefore, is bounded by the number of oracle equivalence classes. The case for multiple matching rules can be similarly proved, and we omit it for brevity.
For -well-matched string with , if contains no unmatched oracle symbols, then we are done. Otherwise, assume contains no return symbols, and unmatched oracle call symbols \guilsinglleft (the other cases are similar and we omit them for brevity). We can rewrite to reflect the derivation of these oracle call and return symbols as follows:
where , , , , , , , and for are strings, and
and string is derived by:
where .
Let , . Notice that are disjoint nesting patterns. Since those are exchangeable, we denote each of them as when their indices do not matter. With this notation, we can simplify the above formulae as
(3) | ||||
(4) | ||||
(5) |
Since is compatible with , for , by definition, each pattern contains a call-return pair of compatible tagging , where and are unmatched in and , respectively. Without loss of generality, let us assume these are the same, denoted as .
Now consider Equation (3)-(5). For , each contains a symbol , whose matched symbol, denoted as , is after ; similarly, each contains a symbol , whose matched is before . Since is -well-matched, for , each symbol can only locate in , and each symbol cannot locate in .
Now, for , we construct -well-matched strings and their contexts , as follows:
Now, we prove that each represents a different equivalence class. First, it is obvious that is -well-matched. Then, we show that is invalid for . Let us expand as
Denote the number of unmatched and unmatched in string as and , respectively. There are three cases.
If , then, when , we have , thus ’s prefix contains pending return symbol, therefore the string is invalid.
If , then, when , we have
(6) |
On the other hand, since is -well-matched, we have
(7) |
Therefore, based on Equation (6)-(7), we have
Similar to the first case, this shows is invalid.
In the last case, . We show that this is impossible. We first assume , then discuss the other case in the end. First, we rewrite each in as
where does not contain unmatched nor . Therefore, for two adjacent , we have
In string , notice that is matched with , therefore, string is -well-matched. Since is a nesting pattern of the oracle tagging function, for any , we can rewrite as
With the corresponding , we have a new nesting pattern
Since tagging is compatible and is -well-matched, contains an unmatched call symbol of tagging , denoted as , whose matched return symbol of tagging , denoted as , is after .
Now we have come back to a similar situation of comparing and . With a similar analysis, we can show that if the number of unmatched , then, we can construct equivalence classes in . Therefore, we again must have .
Then, we can rewrite by expanding as
And, similarly, rewrite two adjacent as
Again, string forms the first part of a nesting pattern, thus must contain another unmatched call symbol in tagging .
However, notice that is -well-matched. Therefore, the new unmatched symbol must appear in , which is strictly shorter than . Subsequentially, we can find substrings , , …, , …with decreasing lengths that must contain unmatched call symbol in . However, must be empty and contain no symbol, which makes incompatible with , a contradiction.
Above is the case where the numbers of unmatched and in is . When the numbers are greater than (recall that the two numbers should be the same), we rewrite as
where is the last unmatched , and is the first unmatched . Expand again, and we can observe that string is still -well-matched. The rest of the reasoning is the same as above.
In conclusion, for each and -well-matched string with , the number of unmatched oracle call symbols equals or less than the number of equivalence classes of . Given that is a VPL under , the numbers of unmatched oracle symbols in any such have an upper bound. ∎
Theorem B.6 ().
Given oracle language and oracle tagging , if language is a VPL, then for any tagging compatible with , language is a VPL.
Proof.
By Lemma B.5, there exists a positive number , such that any -well-matched string contains at most unmatched oracle symbols.
For a given -well-matched string , without loss of generality, let us assume there are only number of unmatched oracle call symbols in . We prove the theorem by showing that is equivalent to a string within a fixed length, denoted as .
First, we can partition into , where each is an unmatched call symbol and each is well-matched under . Let us denote the length of the longest representative under as . Denote as ’s representative in the oracle congruence relation, we have
Now, we construct a shorter representative for , by replacing with . Formally, for all and ,
Consequently, the length of any representative under tagging is limited by , accounting for unmatched characters and substrings each with a length not exceeding .
In conslusion, language has a finite number of equivalence classes, therefore is a VPL. ∎
Theorem B.7 (Finite and Sufficient Seed Strings).
For any given oracle language , there exists a finite set of seed strings, denoted as , such that any tagging that is compatible with is also compatible with .
Proof.
The strategy is to first construct a set of seed strings that provide information of the oracle call and return symbols, then extend the set with more seed strings to exclude the incompatible tagging functions.
Initialize as an empty set. For each oracle call-return symbol pair , pick a seed string that contains a nesting pattern where \guilsinglleft is unmatched in , and \guilsinglright is unmatched in . Incorporate into . Note that if there is no such nesting pattern for , then it is easy to show that are “redundant” in that they can be treated as plain symbols, which does not change the language with tagging removed.
Then, for each tagging that can be found in , but is incompatible with a nesting pattern of a certain string in , include string in . Given that the set of such tagging functions is finite, remains a finite set.
In conclusion, given such , Algorithm 3 can at least find the oracle tagging (with redundant tagging removed), by iteratively selecting the oracle call-return pair for each nesting pattern. ∎
Appendix C Proofs of Theorems in Section 5
This section is organized as follows. Lemma C.1 shows that given a nesting pattern , for sufficiently large , string contains an unmatched oracle call token in , and contains an unmatched oracle return token in . Since such varies among strings, Lemma C.2 bounds with with the help of Exclusivity. Moving on, Lemma C.3 shows that for oracle language and oracle tokenizer , is a VPL over . Lemma C.4 shows that token-based matching rules lead to nesting patterns. Based on Lemma C.4, Lemma C.5 bounds the number of unmatched tokens between and . Following Lemma C.3 and C.5, Theorem C.6 proves that a compatible tokenizer converts the oracle language into a VPL. We conclude this section with Lemma C.7, which shows that there exists a finite set of seed strings that allows V-Star to find a compatible tokenizer.
Lemma C.0 (Matched Tokens in Nesting Patterns, I).
Given oracle language , oracle tokenizer , and a VPG for the oracle language, for each nesting pattern , there exists an oracle matching rule in , denoted as , and exists a positive number , so that tokens and are included in and , respectively.
Proof.
For each , consider the tokenization of , denoted as .
Let be the token sequence in , i.e., each token in overlaps .
We say a token covers a string, if the string is a substring of the string captured by the token. For example, if token in captures string , then covers .
We show that there exists an upper bound, denoted as , such that for each and each token , if covers , then . Otherwise, notice that there are only a finite number of tokens, therefore, there exists a token such that starts at a character in , and can end at a character in either infinite locations in for , or infinite locations in for (could be both). In the first case, one can see that there exist two strings , of , such that
where is a suffix of either or , and are two numbers such that , and is a prefix of . Apparantly, and can be exchanged, which violates that is a nesting pattern. The second case could be similarly invalidated.
Similarly, we can prove another upper bound for . Let be .
With upper bound , we can consider a new nesting pattern , where , , and :
By doing this, we exclude the first and last tokens in that only partially overlap with . From now on, we assume is contained in for each .
For each positive number , define language as , and token-based language as . Apparantly, is not a regular language. Based on this, we can show that there exists token list that contains unmatched call or return tokens. Otherwise, each sequence can only contain plain tokens or well-matched tokens. We only need to show that the depth of the nested well-matched tokens is bounded for all , then the language is a regular language, a contradiction.
To show that the depth is bounded, notice that otherwise, we would have a nesting pattern in for certain . Denote the pattern as . Nesting pattern , by definition, can be replaced by for any in . This replacement extends to for certain . However, since is a nesting pattern, should be invalid, a contradiction.
Therefore, we have shown that for each , there exists and , such that and contains a non-plain token. In other words, when goes to infinity, contains an infinite number of unmatched call or return tokens. However, the number of unmatched return tokens is bounded in for , otherwise, since is fixed, not enough call tokens can match those return tokens. Therefore, the number of unmatched call tokens is unbounded in for . This means for a sufficiently large , a call token in must be matched with a return token in . We thus have proven the lemma. ∎
Lemma C.0 (Matched Tokens in Nesting Patterns, II).
Based on Lemma C.1, assume Exclusivity. For any nesting pattern , there exists a matching rule, denoted as , such that is a substring of , and is a substring of .
Proof.
From Lemma C.1, we know that a token is in for certain . We can therefore let , so that for certain . Consider ; there are two cases.
If is empty, then . Notice that we must have , otherwise either or becomes both a prefix and an infix of , which violates Exclusivity. Therefore, or . Apparantly, is a substring of .
If is nonempty, must be zero, otherwise is both a prefix and an infix, a violation of Exclusivity. In this case, we have , also a substring of .
In conclusion, we know that is a substring of . The reasoning is quite similar for the return token, and we omit it for brevity. ∎
Lemma C.0 ().
For oracle language and oracle tokenizer , is a VPL over .
Proof.
We prove by building a VPA for language .
First, since the language is a VPL over , we have a VPA for it, denoted as .
Then, since each token is a regular language, denote its finite state automaton as .
Now we build a VPA for language by replacing each token in with its FSA .
First, the set of states is the union of states in and for .
Then, the transitions are defined as follows.
-
(1)
We retain transitions in each FSA for plain token .
-
(2)
For transition , we add transition for the start state and transitions for each acceptance state in .
-
(3)
For transition , we add transition , where \guilsinglleft is the call token corresponding to , and we add transitions for each acceptance state in .
-
(4)
For transition , we add transition , where is the call token corresponding to , and we add transitions for each acceptance state in .
Apparantly, a string must be accepted by . On the otherhand, if a string is accepted by , then it leads to a valid token sequence and , therefore, is also a valid string .
In conclusion, is a VPA that accepts , which means language is a VPL. ∎
Lemma C.0 (Call and Return Tokens in Nesting Patterns).
For any oracle VPL , for each string , if is derived by repeatedly applying a matching rule which exposes a recursion, i.e.,
where and , and are strings derived from nonterminals and , respectively, and . Then there exists a nesting pattern for where is a prefix of and is a substring of .
Proof.
Consider the iterative application of the derivation to . This leads to the deduction . To show that the pair represents a nesting pattern, we only need to prove that is invalid when (both ).
Let the oracle tokenizer be . For , we tokenize string as:
Notice that , and are independent strings of token list. Therefore, we have
We next tokenize string , for (both ):
We applied Tokenization Consistency in the last step above. Apparantly, is invalid because of imbalanced call and return tokens. There are two cases left, where either or equals .
When and , . Assume is valid; we can tokenize as
Again, is apparantly invalid. The case of and is similar and we omit it for brevity.
Therefore, we have shown that the pair is a nesting pattern. ∎
Lemma C.0 ().
Given a compatible tokenizer , there exists an upper bound , so that for each string , if is well-matched and there exists context strings such that , then string contains at most unmatched oracle call or return tokens.
Proof.
The proof parallels the proof of Lemma B.5; we show that this would otherwise result in an infinite number of equivalence classes for , contradicting to that is a VPL, proven by Lemma C.3.
If for any number , there exists a -well-matched string , such that can contain at least unmatched call token, then, by Lemma C.4, a set of nesting patterns appear in , and therefore in . Then, since is compatible with , a call-return token pair appear in each . The rest of the analysis is similar to that of Lemma B.5, and we omit it for brevity. ∎
Theorem C.6 ().
Given oracle language and oracle tokenizer , for each tokenizer that is compatible with the oracle language , language is a VPL.
Proof.
By Lemma C.5, for certain , we can represent as , where is well-matched under . Now, we replace each with its representative in the equivalence class of , and get that . Since has a finite number of equivalence classes, let the length of the longest representative be , the length of is bounded by .
Therefore, the congruence relation has a finite number of equivalence classes, which shows is a VPL. ∎
Lemma C.0 (Finite and Sufficient Seed Strings).
There is a finite set of seed strings , with which we can find a tokenizer that is compatible with the oracle language using Algorithm 4.
Proof.
The construction of involves two phases: we first identify strings that reveal the oracle call and return tokens, then augment this set to exclude taggings incompatible with .
Starting with an empty set , for each oracle call-return token pair , we select a seed string that contains a nesting pattern where and are respectively unmatched in and , then we include in . We denote the new set as . Note that, similar to Lemma B.7, if there is no such nesting pattern for , then it is easy to show that are “redundant”, in that they can be treated as plain tokens, which does not change the language with tagging removed.
Subsequently, we extend with strings from that is incompatible with certain tokenizer that can be found by Algorithm 4 based on . We denote the new set as . is still finite, since Algorithm 4 can only find a finite number of tokenizers from .
For each string in , we modify by replacing call and return tokens and with strings and from Token Fixed Prefix and Suffix, respectively. The new set is our targeted .
Given , we now show that the oracle tokenizer is a possible return of Algorithm 4. Intuitively, the oracle token pair can be incrementally added to the hypothesis partial tokenizer , and the tokenization for any valid string maintains well-matched.
By Lemma C.2, each oracle token pair is contained in for certain nesting pattern . Therefore, from the first nesting pattern , Algorithm 4 could find where belongs to an oracle token pair. Then, because of the Token Fixed Prefix and Suffix assumption, the lexical rules for the two paired tokens are learned accurately. Denote this tokenizer that contains only one token-pair as . Consider Algorithm 5. At line 4, we construct a new match . Now, we show that must be the match given by the oracle tokenizer. Firstly, must correspond to an oracle match, otherwise it will be filtered out by -Repetition. Then, by Unique Pairing and Separation, must correspond to the oracle call/return token. Therefore, contains only well-matched oracle tokens, thus is well-matched.
A similar analysis can be done to show that as long as contains only oracle token pairs, is well-matched for any valid string . We thus have proved that Algorithm 4 can at least find the oracle tokenizer given (with redundant oracle tokens removed), by iteratively selecting the oracle token pair for each nesting pattern.
In conclusion, there exists a finite set of seed strings, where Algorithm 4 can find a compatible tokenizer. ∎
References
- (1)
- Aichernig et al. (2024) Bernhard K. Aichernig, Martin Tappler, and Felix Wallner. 2024. Benchmarking Combinations of Learning and Testing Algorithms for Automata Learning. Form. Asp. Comput. 36, 1, Article 3 (mar 2024), 37 pages. https://doi.org/10.1145/3605360
- Alur (2007) Rajeev Alur. 2007. Marrying words and trees. In Proceedings of the Twenty-Sixth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems (¡conf-loc¿, ¡city¿Beijing¡/city¿, ¡country¿China¡/country¿, ¡/conf-loc¿) (PODS ’07). Association for Computing Machinery, New York, NY, USA, 233–242. https://doi.org/10.1145/1265530.1265564
- Alur et al. (2005) Rajeev Alur, Viraj Kumar, P. Madhusudan, and Mahesh Viswanathan. 2005. Congruences for visibly pushdown languages. In Proceedings of the 32nd International Conference on Automata, Languages and Programming (Lisbon, Portugal) (ICALP’05). Springer-Verlag, Berlin, Heidelberg, 1102–1114. https://doi.org/10.1007/11523468_89
- Alur and Madhusudan (2004) Rajeev Alur and P. Madhusudan. 2004. Visibly pushdown languages. In Proceedings of the Thirty-Sixth Annual ACM Symposium on Theory of Computing (Chicago, IL, USA) (STOC ’04). Association for Computing Machinery, New York, NY, USA, 202–211. https://doi.org/10.1145/1007352.1007390
- Alur and Madhusudan (2009) Rajeev Alur and P. Madhusudan. 2009. Adding nesting structure to words. J. ACM 56, 3, Article 16 (may 2009), 43 pages. https://doi.org/10.1145/1516512.1516518
- Angluin (1987) Dana Angluin. 1987. Learning regular sets from queries and counterexamples. Inf. Comput. 75, 2 (nov 1987), 87–106. https://doi.org/10.1016/0890-5401(87)90052-6
- Arefin et al. (2024) M. Arefin, S. Shetiya, Z. Wang, and C. Csallner. 2024. Fast Deterministic Black-box Context-free Grammar Inference. In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, Los Alamitos, CA, USA, 901–901. https://doi.ieeecomputersociety.org/
- Barbot et al. (2021) Benoît Barbot, Benedikt Bollig, Alain Finkel, Serge Haddad, Igor Khmelnitsky, Martin Leucker, Daniel Neider, Rajarshi Roy, and Lina Ye. 2021. Extracting Context-Free Grammars from Recurrent Neural Networks using Tree-Automata Learning and A* Search. In Proceedings of the Fifteenth International Conference on Grammatical Inference (Proceedings of Machine Learning Research, Vol. 153), Jane Chandlee, Rémi Eyraud, Jeff Heinz, Adam Jardine, and Menno van Zaanen (Eds.). PMLR, 113–129. https://proceedings.mlr.press/v153/barbot21a.html
- Bastani et al. (2017) Osbert Bastani, Rahul Sharma, Alex Aiken, and Percy Liang. 2017. Synthesizing program input grammars. SIGPLAN Not. 52, 6 (jun 2017), 95–110. https://doi.org/10.1145/3140587.3062349
- Bendrissou et al. (2022) Bachir Bendrissou, Rahul Gopinath, and Andreas Zeller. 2022. “Synthesizing input grammars”: a replication study. In Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation (San Diego, CA, USA) (PLDI 2022). Association for Computing Machinery, New York, NY, USA, 260–268. https://doi.org/10.1145/3519939.3523716
- Chaudhuri and Alur (2007) Swarat Chaudhuri and Rajeev Alur. 2007. Instrumenting C programs with nested word monitors. In Proceedings of the 14th International SPIN Conference on Model Checking Software (Berlin, Germany). Springer-Verlag, Berlin, Heidelberg, 279–283.
- Chow (1978) T. S. Chow. 1978. Testing Software Design Modeled by Finite-State Machines. IEEE Trans. Softw. Eng. 4, 3 (may 1978), 178–187. https://doi.org/10.1109/TSE.1978.231496
- Drewes and Högberg (2007) Frank Drewes and Johanna Högberg. 2007. Query Learning of Regular Tree Languages: How to Avoid Dead States. Theor. Comp. Sys. 40, 2 (feb 2007), 163–185. https://doi.org/10.1007/s00224-005-1233-3
- Drewes et al. (2011) Frank Drewes, Johanna Högberg, and Andreas Maletti. 2011. MAT learners for tree series: an abstract data type and two realizations. Acta Informatica 48, 3 (May 2011), 165–189. https://doi.org/10.1007/s00236-011-0135-x
- Fujiwara et al. (1991) S. Fujiwara, G. v. Bochmann, F. Khendek, M. Amalou, and A. Ghedamsi. 1991. Test selection based on finite state models. IEEE Transactions on Software Engineering 17, 6 (1991), 591–603. https://doi.org/10.1109/32.87284
- Gauwin et al. (2008) Olivier Gauwin, Joachim Niehren, and Yves Roos. 2008. Streaming tree automata. Inform. Process. Lett. 109, 1 (2008), 13–17. https://doi.org/10.1016/j.ipl.2008.08.002
- Harris et al. (2012) William R. Harris, Somesh Jha, and Thomas Reps. 2012. Secure programming via visibly pushdown safety games. In Proceedings of the 24th International Conference on Computer Aided Verification (Berkeley, CA) (CAV’12). Springer-Verlag, Berlin, Heidelberg, 581–598. https://doi.org/10.1007/978-3-642-31424-7_41
- Heizmann et al. (2010) Matthias Heizmann, Jochen Hoenicke, and Andreas Podelski. 2010. Nested interpolants. SIGPLAN Not. 45, 1 (jan 2010), 471–482. https://doi.org/10.1145/1707801.1706353
- Howar (2012) Falk Howar. 2012. Active learning of interface programs. Ph. D. Dissertation. https://doi.org/10.17877/DE290R-4817
- Irfan et al. (2010) Muhammad Naeem Irfan, Catherine Oriat, and Roland Groz. 2010. Angluin style finite state machine inference with non-optimal counterexamples. In Proceedings of the First International Workshop on Model Inference In Testing (Trento, Italy) (MIIT ’10). Association for Computing Machinery, New York, NY, USA, 11–19. https://doi.org/10.1145/1868044.1868046
- Isberner (2015) Malte Isberner. 2015. Foundations of active automata learning: an algorithmic perspective. https://api.semanticscholar.org/CorpusID:45690562
- Jia et al. (2021) Xiaodong Jia, Ashish Kumar, and Gang Tan. 2021. A derivative-based parser generator for visibly Pushdown grammars. Proc. ACM Program. Lang. 5, OOPSLA, Article 151 (oct 2021), 24 pages. https://doi.org/10.1145/3485528
- Jia et al. (2023) Xiaodong Jia, Ashish Kumar, and Gang Tan. 2023. A Derivative-based Parser Generator for Visibly Pushdown Grammars. ACM Trans. Program. Lang. Syst. 45, 2, Article 9 (may 2023), 68 pages. https://doi.org/10.1145/3591472
- Kulkarni (2023) Neil Kulkarni. 2023. Arvada. https://github.com/neil-kulkarni/arvada.
- Kulkarni et al. (2022) Neil Kulkarni, Caroline Lemieux, and Koushik Sen. 2022. Learning highly recursive input grammars. In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (Melbourne, Australia) (ASE ’21). IEEE Press, 456–467. https://doi.org/10.1109/ASE51524.2021.9678879
- Kumar et al. (2006) Viraj Kumar, P. Madhusudan, and Mahesh Viswanathan. 2006. Minimization, learning, and conformance testing of boolean programs. In Proceedings of the 17th International Conference on Concurrency Theory (Bonn, Germany) (CONCUR’06). Springer-Verlag, Berlin, Heidelberg, 203–217. https://doi.org/10.1007/11817949_14
- Kumar et al. (2007) Viraj Kumar, P. Madhusudan, and Mahesh Viswanathan. 2007. Visibly pushdown automata for streaming XML. In Proceedings of the 16th International Conference on World Wide Web (Banff, Alberta, Canada) (WWW ’07). Association for Computing Machinery, New York, NY, USA, 1053–1062. https://doi.org/10.1145/1242572.1242714
- Maler and Pnueli (1991) Oded Maler and Amir Pnueli. 1991. On the learnability of infinitary regular sets. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory (Santa Cruz, California, USA) (COLT ’91). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 128–138.
- Michaliszyn and Otop (2022) Jakub Michaliszyn and Jan Otop. 2022. Learning Deterministic Visibly Pushdown Automata Under Accessible Stack. In 47th International Symposium on Mathematical Foundations of Computer Science (MFCS 2022) (Leibniz International Proceedings in Informatics (LIPIcs), Vol. 241), Stefan Szeider, Robert Ganian, and Alexandra Silva (Eds.). Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 74:1–74:16. https://doi.org/10.4230/LIPIcs.MFCS.2022.74
- Mozafari et al. (2010) Barzan Mozafari, Kai Zeng, and Carlo Zaniolo. 2010. From regular expressions to nested words: unifying languages and query execution for relational and XML sequences. Proc. VLDB Endow. 3, 1–2 (sep 2010), 150–161. https://doi.org/10.14778/1920841.1920865
- Mozafari et al. (2012) Barzan Mozafari, Kai Zeng, and Carlo Zaniolo. 2012. High-performance complex event processing over XML streams. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (Scottsdale, Arizona, USA) (SIGMOD ’12). Association for Computing Machinery, New York, NY, USA, 253–264. https://doi.org/10.1145/2213836.2213866
- Nguyen and Sudholt (2006) Dong Ha Nguyen and M. Sudholt. 2006. VPA-Based Aspects: Better Support for AOP over Protocols. In Fourth IEEE International Conference on Software Engineering and Formal Methods (SEFM’06). 167–176. https://doi.org/10.1109/SEFM.2006.39
- Raffelt et al. (2005) Harald Raffelt, Bernhard Steffen, and Therese Berg. 2005. LearnLib: a library for automata learning and experimentation. In Proceedings of the 10th International Workshop on Formal Methods for Industrial Critical Systems (Lisbon, Portugal) (FMICS ’05). Association for Computing Machinery, New York, NY, USA, 62–71. https://doi.org/10.1145/1081180.1081189
- Rivest and Schapire (1993) R.L. Rivest and R.E. Schapire. 1993. Inference of Finite Automata Using Homing Sequences. Information and Computation 103, 2 (1993), 299–347. https://doi.org/10.1006/inco.1993.1021
- Thomo and Venkatesh (2011) A. Thomo and S. Venkatesh. 2011. Rewriting of visibly pushdown languages for XML data integration. Theoretical Computer Science 412, 39 (2011), 5285–5297. https://doi.org/10.1016/j.tcs.2011.05.047
- Vasilevskii (1973) M. P. Vasilevskii. 1973. Failure diagnosis of automata. Cybernetics 9, 4 (July 1973), 653–665. https://doi.org/10.1007/BF01068590
- Wu et al. (2019) Zhengkai Wu, Evan Johnson, Wei Yang, Osbert Bastani, Dawn Song, Jian Peng, and Tao Xie. 2019. REINAM: reinforcement learning for input-grammar inference. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Tallinn, Estonia) (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA, 488–498. https://doi.org/10.1145/3338906.3338958