Robust textual inference via graph matching

Aria D. Haghighi Andrew Y. Ng Christopher D. Manning

Dept. of Computer Science Dept. of Computer Science Dept. of Computer Science
Stanford University Stanford University Stanford University
Stanford, CA Stanford, CA Stanford, CA
[email protected] [email protected] [email protected]

Abstract we must be able to recognize legitimate translations

which structurally differ from our baseline.
We present an automated system for de- One sub-task underlying these applications is the
ciding whether a given sentence is entailed ability to recognize semantic entailment; whether
from a body of text. Each sentence is one piece of text follows from another. In con-
represented as a directed graph (extracted trast to recent work (Moldovan et al., 2003), which
from a dependency parser) in which the have successfully utilized logic-based abductive ap-
nodes represent words or phrases, and the proaches to inference, we utilize a graph-based rep-
links represent syntactic and semantic re- resentation of sentences, and use graph matching
lationships. A learned graph matching techniques to measure the semantic overlap of text.
cost is used to measure how much of the Graph matching techniques have proved to be a use-
semantic content of the sentence is con- ful approach to tractable approximate matching in
tained in the text. We present results other domains including computer vision. Addition-
on the Recognizing Textual Entailment ally in the domain of language, graphs provide a
(RTE) dataset (Dagan et al., 2005), com- natural way to express the lexical dependencies be-
pare to other approaches, discuss common tween words and phrases in a sentence. Further-
classes of errors, and discuss directions for more, graph matching also has the advantage of
improvement. providing a framework for structural matching of
phrases that would be difficult to resolve at the level
of individual words.
1 Introduction
A fundamental goal of NLP is the production of ro-
2 Task Definition and Data
bust and accurate systems semantic inference. This
A fundamental stumbling block for several NLP ap-
plications is the lack of a robust and accurate se- We describe our approach in the context of this years
mantic inference. For instance, question answering “Recognizing Textual Entailment” (RTE) challenge
systems must be able to recognize, or infer, an an- (Dagan et al., 2005), but note that our approach eas-
swer which may be expressed differently from the ily extends to other related inference tasks. The sys-
query. Information extraction systems must also be tem presented here was one component of our re-
able to infer propositions and recognize the variabil- search group’s RTE submission (Suppressed, 2005)
ity of equivalent linguistic expressions. Document which was the top-ranking system according to one
Summarization systems must generate succinct sen- of two evaluation metrics.
tences which express the same content as the orig- In this domain, we are given several pairs, each
inal document. In Machine Translation evaluation, consisting of two parts: 1) the text, a small passage
 ever, in the proposed answer, the expected depen-
dency is missing.
Our approach is to view sentences as graphs be-
tween words and phrases, where dependency rela-
tionships are characterized by the path between the
words. This approach has been successfully used
to characterize semantic relationships such as hyper-
nyms (Snow et al., 2004).
Given this representation, we judge entailment by
Figure 1: An example parse tree and the correspond-
measuring not only how many of the hypothesis ver-
ing dependency graph. Each phrase of the parse tree
tices are matched to the text but also how well the
is annotated with its head word, and the parentheti-
relationships between vertices in the hyposthesis are
cal edge labels in the dependency graph correspond
preserved in their counter parts in the text. For the
to semantic roles.
remainder of the section we outline how we produce
graphs from text, and in the next section we intro-
1, and the hypothesis, a single sentence. Our task duce our graph matching model.
is to decide if the hypothesis is entailed by the text.
Here, entails does not mean strict logical implica- 3.1 From Text To Graphs
tion, but roughly means that a competent speaker Starting with raw English text, we use a version of
with minimal world-knowledge would infer the hy- the parser described in (Klein and Manning, 2003),
pothesis from the the text. For a flavor of the nature to obtain a parse tree. Then, using a slightly modi-
(and difficulty) of the task, see Table 2. fied versions of Collins’ head propagation (Collins,
For purposes of comparison, we give results on 1999), we derive a dependency tree representation of
the data provided for the RTE task which consists the sentence. The labels of the edges in the depen-
of 567 development pairs and 800 test pairs. We dency graph are given according to a hand-created
will use the following toy example to illustrate our set of tgrep expressions. These edges represent
representation and matching technique: “surface” syntax relationships such as subj for sub-
Text: In 1994, Amazon.com was founded by Jeff Bezos. ject and amod for adjective modifier, similar to the
Hypothesis: Bezos established a company. relations in Minipar (Lin and Pantel, 2001). The de-
pendency graph is the basis for our graphical repre-
3 Semantic Representation sentation, but it is enhanced in the following ways:
Perhaps the most common representation of text for 1. Collapse Collocations and Named-Entities: We
assessing content is “Bag-Of-Words” or “Bag-of-N- “collapse” dependency nodes which represent
Grams” (Papineni et al., 2001). However, such rep- named entities (e.g., Jeff Bezos in Figure 2) and
resentations lose syntactic information which can also collocations, including verbs and their par-
be essential to determining entailment. Consider a ticles ( e.g. , blow off in He blew off his work).
Question Answer system searching for an answer
to When was Israel established ? A representation 2. Dependency Folding : As in (Lin and Pantel,
which did not utilize syntax would probably enthu- 2001), we found it useful to fold dependencies
siastically return an answer from: The National In- (such as modifying prepositions) so that mod-
stitute for Psychobiology in Israel was established ifiers became labels connecting the modifier’s
in 1979. governor and dependent directly. For instance,
In this example it’s important to try to match rela- in Figure 2, we have changed in from a word
tionships as well as words. In particular, any answer into a relation between its head verb and the
to the question should preserve the dependency be- head of its NP complement.
tween Israel and established in the question. How-
3. Semantic Role Labeling: We also augment the
1
Usually a single sentence, but occasionally longer graph with the output of the Semantic Role La-
 Task Text Hypothesis Entailed
Question Prince Charles was previously married to Prince Charles and Princess Diana got False
Answer Princess Diana, who died in a car crash in married in August 1997.
(QA) Paris in August 1997.
Machine Sultan Al-Shawi, a.k.a the Attorney, said The Attorney, said at the funeral, ”They True
Translation during a funeral held for the victims, were all Iraqis killed during the brutal
(MT) ”They were all children of Iraq killed dur- shelling.”.
ing the savage bombing.”.
Comparable Napster, which started as an unauthorized Napster illegally offers music downloads. False
Documents song-swapping Web site, has transformed
(CD) into a legal service offering music down-
loads for a monthly fee.
Paraphrase Kerry hit Bush hard on his conduct on the Kerry shot Bush. False
Recognition war in Iraq.
(PP)
Information The country’s largest private employer, Wal-Mart sued for sexual discrimination. True
Retrieval Wal-Mart Stores Inc., is being sued by
(IR) a number of its female employees who
claim they were kept out of jobs in man-
agement because they are women.

Table 1: Some Textual Entailment examples. The first last three demonstrate some of the harder instances

beler of (Toutanova et al., 2005). Each pred- 4 Entailment by Graph Matching

icate adds an arc labeled with the appropri-
ate semantic role to the head of the argument We take the view that a hypothesis is entailed from
phrase. This helps to create links between the text when the cost of matching the hypothesis
words which share a deep semantic relation not graph to the text graph is low. For the remainder of
evident through the surface syntax. Addition- this section, we outline a general model for assign-
ally, modifying phrases are labeled with their ing a match cost to graphs.
semantic types (e.g., the Temporal edge in the For hypothesis graph H, and text graph T , a
Figure 2) which should be useful Question An- matching M is a mapping from the vertices of H
swering type tasks. to those of T . For vertex v in H, we will use M (v)
to denote its “match” in T . As is common in ma-
4. Coreference Links: Using a co-rereference chine translation, we allow nodes in H to map to a
resolution tagger, coref links are added fictitious NIL vertex in T if necessary. Suppose the
throughout the graph. These links allowed con- cost of matching M is Cost(M ). If M is the set of
necting the referent entity to the vertices of the such matchings, we define the cost of matching H
referring vertex. Also in the case of multiple to T :
sentence texts, it is our only “link” in the graph
entities in the two sentences. MatchCost(H, T ) = min Cost(M ) (1)
M ∈M

For the remainder of the paper, we will refer to

the text as T and hypothesis as H, and will speak Suppose we have a model, VertexSub(v, M (v)),
of them in graph terminology. In addition we will which gives us a cost in [0, 1], for substituting vertex
use HV and HE to denote the vertices and edges, v in H for M (v). One natural cost model is to use
respectively, of H. the normalized cost for each of the substitutions our
matching makes:
1 X
Cost(M ) = w(v)VertexSub(v, M (v))
Z v∈H
V
(2)
Here, w(v) represents the weight or relative im-
P
portance for vertex v, and Z = v∈HV w(v) is
a normalization constant. In our implementation,
the weight of each vertex was based on the part-of-
speech tag of the word or the type of named entity, if
applicable. However, there are several other possi-
bilities including using tf-idf weights for words and
phrases.
Notice that when Cost(M ) takes the form of
(2), computing MatchCost(H, T ) is equivalent to
finding the minimal cost bipartite graph-matching, Figure 2: Example graph matching (α = 0.55) for
which can be efficiently computed using linear pro- example pair. Dashed lines represent mapping.
gramming.
As (Punyakanok et al., 2004) demonstrated, mod-
els which also match syntactic relationships be- are more important to match than others. Our fi-
tween words can outperform bag-of-words models nal matching cost is given by a convex mixture of
for TREC QA answer extraction. So it should be ad- the vertex and relational match costs: Cost(M ) =
vantageous for our cost-model to incorporate some αVertexCost(M ) + (1 − α)RelationCost(M ).
measure of how relationships in H are preserved in Notice that minimizing Cost(M ) is computa-
T under M . Ideally, a matching, should preserve tionally hard since if our PathSub model as-
all local relationships; i.e, if v → v 0 ∈ H, then signs zero cost only for preserving edges, then
M (v) → M (v 0 ) ∈ T . When this condition holds RelationCost(M ) = 0 if and only if H is isomor-
for all edges in H, H and T are isomorphic. phic to a subgraph of T . As an approximation, we
What we would like is a approximate notion of can efficiently find the matching M ∗ which mini-
isomorphism, where we penalize the distortion of mizes VertexCost(·); we then perform local greedy
each edge relation in H. Consider an edge e = hill-climbing search, beginning from M ∗ , to approx-
(v, v 0 ) ∈ HE , and let M (e) be the path from M (v) imate the minimal matching. In practice, this ap-
to M (v 0 ) in T . proximation appears to perform rather well.
Again, suppose we have a model,
4.1 Relationship to Machine Translation
PathCost(e, M (e)) for assessing the “cost” of
substituting a direct relation e ∈ M for its coun- The model presented in this section bears many
terpart, M (e), under the matching. This leads to a resemblances to a syntactic translation alignment
formulation similar to (2), where we consider the model, where the source text is the target and the
normalized cost of substituting each edge relation hypothesis the source. In particular, we can think of
in H with a path in T : the matching as an alignment, and our equation (1)
is a typical approximation of the translation model
1 X alignment probability.
RelationCost(M ) = w(e)PathSub(e, M (e))
Z e∈H
E 5 Node and Edge Substitution Models
(3)
P
where Z = e∈HE w(e) is a normalization con- In the previous section we described our graph
stant. As in the vertex case, we have weights matching model in terms our VertexSub model,
for each hypothesis edge, w(e), based upon the which gives a cost for substituting one graph vertex
edge’s label; typically subject and object relations for another, and PathSub, which gives a cost for sub-
stituting the path relationship between two paths in • Ancestor Match: M (v) is an ancestor of M (v 0 ).
one graph for that in another. We now outline these We use an exponential increasing cost for longer
models. distance relationships.
Our VertexSub(v, M (v)) model is based upon a • Kinked Match: M (v) and M (v 0 ) share a com-
sliding scale, where progressively higher costs are mon parent or ancestor in T . We use an exponen-
given based upon the following conditions: tially increasing cost based on the maximum of
• Exact Match: v and M (v) are identical the node’s distances to their least common ances-
words/phrases tor in T .
• Stem Match: v and M (v)’s stems match or one These conditions capture many of the common
is a derivational form of the other e.g., matching way in which relationships between entities are dis-
coaches to coach. torted in semantically related sentences.
Give Justification for these conditions
• Synonym Match: v and M (v)’s are synonyms
according to WordNet (Fellbaum, 1998). In par- 5.1 Learning Weights
ticular we use the top 3 senses of both words to
Is it possible to learn weights for the relative impor-
determine synsets.
tance of the conditions in the VertexSub and PathSub
• Hypernym Match: v is a hypernym of M (v) models? Consider the case where match costs are
according to WordNet. Note that this feature is given only by equation (2) and vertices are weighted
assymmetric. We do not match v to a hyponym uniformly (w(v) = 1). Suppose that Φ(v, M (v))
M (v). is a vector of features 3 indicating the cost accord-
• WordNet Similarity: v and M (v) are ing to each of the conditions listed for matching v
similar according to WordNet::Similarity to M (v). Also let w be weights for each element
(Pedersen et al., 2004). In particular, we use the of Φ(v, M (v)). Then we can model the substitution
measure described in (Resnik, 1995). We found cost for a given matching as :
it useful to only use similarities above a fixed
threshold to ensure precision. exp (wT Φ(v, M (v)))
VertexSub(v, M (v)) =
1 + exp (wT Φ(v, M (v)))
• LSA Match: v and M (v) are distributionally
similar according to a freely available Latent Se- Letting s(·) be the 1-sigmoid function used in the
mantic Indexing package, or for verbs similar right hand side of the equation above, our final
according to VerbOcean (Chklovski and Pantel, matching cost as a function of w is given by
2004). 2
• No Match: M (v) is NIL 1 X
c(H, T ; w) = min s(wT Φ(v, M (v)))
M ∈M |HV |
v∈H
Although the above conditions often produce rea- (4)
sonable matchings between text and hypothesis, we Suppose we have a set of text/hypothesis pairs,
found that the recall of these lexical resources to be {(T (1) , H (1) ), . . . , (T (n) , H (n) )}, with labels y (i)
far from adequate. More robust lexical resources which are 1 if H (i) is entailed by T (i) and 0 other-
would almost certainly boost performance a signifi- wise. Then we would like to choose w to minimize
cant amount. costs for entailed examples and maximize it for non-
Our PathSub model is also based upon a sliding entailed pairs:
scale cost based upon the following conditions: X
`(w) = log c(H (i) , T (i) ; w) +
• Exact Match: M (v) → M (v 0 ) is an en edge in
i:y (i) =1
T with the same label. X
log(1 − c(H (i) , T (i) ; w))
• Partial Match: M (v) → M (v 0 ) is an en edge in
i:y (i) =0
T , not necessarily with the same label.
3
In the case of our “match” conditions, these features will
2
Available at http://infomap.stanford.edu be binary
 Unfortunately, `(w) is not a convex function. No- vice versa. In this instance, the is in H, denoted by
tice that the cost of each matching, M , implicitly isH , is matched to isT which has a negation modi-
depends on the current setting of the weights, w. It fier, notT , absent for isH . So the negation check is
can be shown that since each c(H, T ; w) involves failed.
minimizing M ∈ M, which depends on w, it is not Factive Check
convex (??). Therefore, we can’t hope to globally Text: Clonaid claims to have cloned 13 babies worldwide.
optimize our cost functions over w. Hypothesis: Clonaid has cloned 13 babies.
One approach is to use coordinate ascent over M Non-factive verbs (claim, think, charged, etc.) in
and w. Suppose that we begin with arbitrary weights contrast to factive verbs (know, regret, etc.) have
and given these weights choose M (i) to minimize sentential complements which do not represent true
each c(H (i) , T (i) ; w). Then we use a relaxed form of propositions. We detect such cases, by checking that
the cost function where we use the matchings found each verb in H that is matched in T does not have a
in the last step: non-factive verb for a parent.
1 X Superlative Check
ĉ(H (i) , T (i) ; w) = s(wT Φ(v, M (i) (v)))
|HV | v∈H
Text: The Osaka World Trade Center is the tallest building in
Western Japan.
Then we maximize w with respect to `(w) witch
Hypothesis: The Osaka World Trade Center is the tallest build-
each c(·) replaced with the cost-function ĉ(·). This
ing in Japan.
step involves only logistic regression. We repeat this
procedure until our weights converge. In general superlative modifiers, (most, biggest,
Our preliminary experiments revealed that this etc.), invert the typical monotinicity of entailment
procedure did not yield weights which improved and must be handled as special cases. For any noun,
performance very much our hand-set initializations. n, with a superlative modifier (part-of-speech JJS)
We believe this to be the case largely because of the in H, we must ensure that all modifier relations of
presence of several local maxima and ridges in the M (n) are preserved in H. In this example, build-
parameter space. In the future, we hope to find bet- ingH has a superlative modifier tallestH , so we must
ter approximation techniques to this problem. ensure that each modifier relation of JapanT , a noun
dependent of buildingT , has a WestrenT modifier not
6 Checks in H. So its fails the superlative check.
During error analysis on the development set, we
One systematic source of error coming from our ba- spotted the following cases where our VertexSub
sic approach is the implicit assumption of upwards function erroneously labeled vertices as similair, and
monotonicity of entailment; i.e., if T entails H then required special case consideration:
adding more words to T should also give us a sen-
tence which entails H. This assumption, also made • Antonym Check: We consistently found that the
by recent abductive approaches (Moldovan et al., WordNet::Similarity modules gave high-
2003; Harabagiu et al., 2000), does not hold for sev- similarity to antonyms 5 . We explicitly check
eral classes of examples. Below we outline the most whether a matching involved antonyms and reject
common types of cases 4 that we check for after unless one of the vertices had a negation modifier.
graph matching: • Numeric Mismatch: Since numeric expressions
typically have the same part-of-speech tag (CD),
Negation Check they were typically matched when exact matches
Text: Clinton’s book is not a bestseller could not be found. However, mismatching nu-
Hypothesis: Clinton’s book is a bestseller merical tokens usually indicated that H was not
To catch such examples, we check that each hy- entailed, and so pairs with a numerical mismatch
pothesis verb is not matched to a text word which we rejected.
is negated (unless the verb pairs are antonyms) and 5
Which isn’t necessarily incorrect, but simply not suitable
4
All are actual, or slightly altered, RTE examples for textual inference
 Method Accuracy CWS Task General ByTask
Random 50.0% 0.500 Accuracy CWS Accuracy CWS
Bag-Of-Words 49.5% 0.548 CD 72.0% 0.742 76.0% 0.7714
TF-IDF 51.8% 0.560 IE 55.9% 0.583 55.8% 0.595
IR 52.2% 0.5644 51.1% 0.572
GM-General 56.8% 0.614
MT 50.0% 0.497 43.33% 0.489
GM-ByTask 56.8% 0.621 PP 58.0% 0.741 58.0% 0.746
Table 2: Accuracy and confidence weighted score QA 53.8% 0.537 55.4% 0.556
(CWS) for test set using various techniques. RC 52.1% 0.539 52.9% 0.523

Table 4: Accuracy and confidence weighted score

7 Experiments and Results (CWS) split by task on the RTE test set.

For our experiments we used the devolpement and 8 Conclusion

test sets from the Recognizing Textual Entailment
challenge (Dagan et al., 2005). We give results for We have presented a graph matching based approach
our system as well as for the following systems: to determining semantic entailment.

• Bag-Of-Words: We tokenize the text and hypoth-

esis and strip the functional words, and stem the References
resulting words. The cost is given by the fraction Timothy Chklovski and Patrick Pantel. 2004. Verbo-
of hypothesis not matched in the text. cean: Mining the web for fine-grained semantic verb
relations. In EMNLP.
• TF-IDF: Similar to Bag-Of-Words except that
there is a tf-idf weight associated with each hy- Michael Collins. 1999. Head-driven statistical mod-
els for natural language parsing. Ph.D. thesis.
pothesis word so that more “important” words are Supervisor-Mitchell P. Marcus.
higher weight for matching.
Ido Dagan, Oren Glickman, and Bernardo Magnini.
We also present results for two graph matching 2005. The pascal recognizing textual entailment chal-
(GM) systems. The GM-General system fits a sin- lenge. In Proceedings of the PASCAL Challenges
Workshop Recognizing Textual Entailment.
gle global threshold from the devolement set and
the GM-ByTask system fits a different threshold for C. Fellbaum. 1998. WordNet: An Electronic Lexical
each of the “task”. Database. MIT Press.

Our results are summarized in Table 6 6 As the S. M. Harabagiu, Pasca M. A., and S.J. Mariorano. 2000.
result indicates, the task is particularly hard; all RTE Experiments with open-domain textual question an-
swering. In COLING, pages 292–298.
participants scored between 50% and 60% (Dagan
et al., 2005). Both GM systems perform better than Dan Klein and Christopher D. Manning. 2003. Accurate
either Bag-Of-Words or TF-IDF according to both unlexicalized parsing. In ACL, pages 423–430.
raw accuracy and CWS. Dekang Lin and Patrick Pantel. 2001. Discovery of in-
We also present results on a per-task basis in Ta- ference rules from text. In KDD ’01: Proceedings
ble 7. Interestingly, there is a large variation in per- of the seventh ACM SIGKDD international conference
on Knowledge discovery and data mining, pages 323–
formance depending on the task, suggesting the en- 328, New York, NY, USA. ACM Press.
tailment task may be inherently more difficult than
others. Dan I. Moldovan, Christine Clark, Sanda M. Harabagiu,
and Steven J. Maiorano. 2003. Cogex: A logic prover
for question answering. In HLT-NAACL.
6
CWS (confidence weighted score) represents the aver-
age precision among our most confident predictions. If K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2001.
{c
P1n, . . .1, cn } are our confidence outputs then CWS = Bleu: a method for automatic evaluation of machine
i=1 n
(number of correct predications in c1 , . . . , ci ) translation.
 Text Hypothesis True Answer Our answer Conf Comments
A Filipino hostage in Iraq was re- A Filipino hostage was freed True True 0.84 Verb rewrite is handled. Phrasal
leased. in Iraq. ordering does not affect cost.
The government announced last Oil prices drop. False False 0.95 High cost given for substituting
week that it plans to raise oil word for its antonym.
prices.
Shrek 2 rang up $92 million. Shrek 2 earned $92 million. True False 0.59 Collocation “rang up” is not
known to be similar to “earned”.
Sonia Gandhi can be defeated in Sonia Gandhi is defeated by False True 0.77 “can be” does not indicate the
the next elections in India by BJP. BJP. complement event occurs.
Fighters loyal to Moqtada al- Fighters loyal to Moqtada al- False True 0.67 Should recognize non-Location
Sadr shot down a U.S. helicopter Sadr shot down Najaf. cannot be substituted for Loca-
Thursday in the holy city of Najaf. tion.
C and D Technologies announced Datel Acquired C and D False True 0.64 Failed to penalize switch in se-
that it has closed the acquisition technologies. mantic role structure enough
of Datel, Inc.

Table 3: Analysis of results on some RTE examples along with out guesses and confidence probabilities

Ted Pedersen, Siddharth Parwardhan, and Jason Miche-

lizzi. 2004. Wordnet::similarity - measuring the relat-
edness of concepts. In AAAI.
V. Punyakanok, D. Roth, and W. Yih. 2004. Natural
language inference via dependency tree mapping: An
application to question answering. In Computational
Linguistics.
Philip Resnik. 1995. Using information content to evalu-
ate semantic similarity in a taxonomy. In IJCAI, pages
448–453.
Rion Snow, Dan Jurafsky, and Andrew Y. Ng. 2004.
Learning syntactic patterns for automatic hypernym
discovery. In NIPS.
Author Suppressed. 2005. Title suppressed. In Pro-
ceedings of the First PASCAL Challenges Workshop.
Southampton, UK.
Kristina Toutanova, Aria Haghighi, and Cristiopher Man-
ning. 2005. Joint learning improves semantic role la-
beling. In Association of Computational Linguistics
(ACL).