2009 21st IEEE International Conference on Tools with Artificial Intelligence

A Scalable Problem-Solver for Large Knowledge-Bases

Shaw-Yi Chaw, Ken Barker, Bruce Porter, Dan Tecuci Peter. Z. Yeh
University of Texas at Austin Accenture Technology Labs.
Department of Computer Sciences 50 West San Fernando St. Suite 1200
1 University Station C0500 San Jose, CA 95113, USA
Austin, TX 78712, USA [email protected]
{jchaw,kbarker,porter,tecuci}@cs.utexas.edu

Abstract as rules for automatic classification [4] and computation

methods such as, in physics, an equation to compute ve-
We describe a problem solver built to answer questions locity from distance and time. Domain experts build their
like those on Advanced Placement exams using knowledge knowledge-bases by extending the concepts in an upper-
bases authored by domain experts. The problem solver is ontology called the Component Library [3]. The Compo-
designed to work independently of any particular knowl- nent Library consists of a set of generic E VENT and E N -
edge base or domain. Given a question, the problem solver TITY concepts (so called “components”) and a language for
identifies those portions of the knowledge base that are rel- combining them.
evant to the question. We found that simple heuristics for We have found that attempting to answer questions using
judging relevance significantly improved performance, with an entire knowledge base is intractable. For each question,
no drop in coverage. the problem solver must select only the most relevant por-
tions of the knowledge base to reason with. This paper fo-
1 Introduction cuses on building such a problem-solver to work efficiently
with large knowledge bases.
As part of Project Halo [12], our goal is to build a prob-
lem solver capable of answering questions like those on 2 Examples
Advanced Placement (AP) exams in a variety of domains.
These questions are novel, so we believe traditional infor-
The problem solver is given a question and a knowledge
mation retrieval systems [10] would perform poorly at this
base to use to answer it. Consider the biology question
task.
“How many chromosomes are in a cell?”, shown in panel 1
Answering these questions require automated reasoning
of Fig. 1. The question is translated by a natural-language
with domain knowledge represented with formal logic. Un-
system into a set of triples depicted by the graph in panel 2.
like expert systems - which typically operated in task do-
The problem solver searches the knowledge base for rele-
mains small enough to be covered by a few hundred rules -
vant information. The search includes the space of cell con-
knowledge bases in Project Halo are intended to cover entire
cepts in the knowledge base, such as D IPLOID -C ELL and
AP areas (e.g., freshmen college-level physics, chemistry or
P LANT-C ELL, but also far-flung concepts, such as R IBO -
biology).
SOME and P HOTOSYNTHESIS , which might contain rele-
In Project Halo, the knowledge bases are authored by do-
vant information encoded in these contexts. The search also
main experts, not knowledge engineers. They are intended
includes the space of alternative encodings of information.
to be used by many others who are unfamiliar with their
For example, using the ontology of the Component Library,
content and structure. The users pose questions in simpli-
the assertion that the human cell has 46 chromosomes might
fied English [7].
be encoded in the knowledge base as either
The knowledge bases are conventional, consisting of has−part
a set of frames in an inheritance hierarchy [1]. Each H UMAN -C ELL −→ (46 C HROMOSOME )
frame represents a domain concept, such as EUKARYOTIC - or
CELL in biology, METATHESIS - REACTION in chemistry,
has−part has−part
and FALL - FROM - REST in physics. Frames encode declar- H UMAN -C ELL −→ DNA −→ (46 C HROMO -
ative assertions with associated inference methods, such SOME )

1082-3409/09 $26.00 © 2009 IEEE 461

DOI 10.1109/ICTAI.2009.108
 Figure 1. In panel 1, a biology question is posed to the system in simplified English. The system
interprets the question as shown in Panel 2. The query is on C HROMOSOME node, which has a
question-mark. In panel 3, the problem-solver draws in information from the knowledge-base. The
final answer and explanation is shown in Panel 4.

In the end, the problem solver answers the question us- to the mini-kb. Second, we describe the search heuristics
ing information in the H UMAN -C ELL concept (see panel 3), enabling the problem-solver to preferentially select content
and generates the answer shown in panel 4. it judges to be relevant.
The problem solver answers a physics question in a sim- To ground our discussion, Figure 4 shows a portion of
ilar manner, as shown in Fig. 2. In this case, relevant the search graph for solving the physics question in Section
information is found in the M OTION - UNDER - FORCE and 2.
M OTION - WITH - CONSTANT ACCELERATION concepts.
Details on formulating original questions using simpli- 3.1 Problem-Solver without relevance reasoning
fied English can be found in [6, 5]. Additional details on
generating explanations for derived answers are described
3.1.1 States
in [9, 2].
Each state in our state-space graph is a mini-kb. The initial
3 Approach state contains only the triples resulting from the interpre-
tation of the question. Descendant states are elaborations
The problem solver’s challenge is to efficiently find rele- of the initial state containing additional information drawn
vant information in large knowledge bases. Our approach from the knowledge base.
is to answer each question with a mini-knowledge-base In Fig. 4, State 1 is the initial state containing the
(“mini-kb”), containing just enough information to infer an the physics question in Section 2. State 2 results from
answer. Initially, the mini-kb contains only the information elaborating State 1 using the MOTION - WITH - CONSTANT-
(triples) provided by the question. The problem solver in- ACCELERATION concept from the knowledge base being
crementally extends the mini-kb with frames (domain con- queried. This elaboration introduced an equation for cal-
cepts) drawn from the knowledge base that was queried. culating the acceleration of the MOVE event to be -14.45
The frames include both domain assertions and inference meters-per-second-squared.
methods. The problem solver succeeds if it constructs a
mini-kb that is sufficient to answer the question before a 3.1.2 Goal Test and the Goal State
time bound is reached.
We describe the problem solver in two steps. First, The goal test determines whether a state (itself a knowledge
we describe the problem solver’s method of incrementally base) can be used to answer the query.
growing the mini-kb. The method consists of searching a Consider states 2 and 5 in Figure 4. State 2 fails the goal
state space in which states are mini-kbs and operators select test because it does not answer the question; it lacks the
content from the knowledge-base being queried and add it knowledge needed to infer a value for the net-force of the

462
 Figure 2. In panel 1, a physics question is posed to the system in simplified English. The system
interprets the question as shown in Panel 2. The scenario and query of the question is interpreted
as a M OVE event on an O BJECT having mass 80 kg. The initial and final velocity of the M OVE is 17 m/s
and 0 m/s respectively. The distance of the M OVE is 10 m. There is also an E XERT-F ORCE event whose
object is the same object of the M OVE event. The E XERT-F ORCE event causes the M OVE event. The
query is on the net-force of the E XERT-F ORCE and is the node with a question-mark. In panel 3, the
problem-solver draws in information from the knowledge-base. The final answer and explanation is
shown in Panel 4.

EXERT- FORCE event. In contrast, state 5 contains the nec- Figure 3 shows the initial state and the MOTION - WITH -
essary equations to compute the net-force on the E XERT- CONSTANT- ACCELERATION concept. Their overlapping
F ORCE event to be -1156 Newtons. Thus, state 5 satisfies features, identified by the semantic-matcher, are highlighted
the goal test. in bold. Joining state 1 with M OTION - WITH - CONSTANT-
ACCELERATION results in state 2, shown in panel 3. State 2

3.1.3 Operators contains additional problem-solving knowledge, in this case

equations, to compute the acceleration of the M OVE given
Operators extend a state with additional knowledge from values for initial velocity, final velocity, and distance.
the knowledge-base to create a successor state. We next
describe how operators are created and applied during
3.1.4 Control Strategy
problem-solving.
The states in the search graph are explored in a breadth first
Creating Operators. Operators are created for each con- manner until an answer is found or the process aborts be-
cept in the knowledge-base being queried to elaborate the cause of a time bound. In Project Halo, a question can have
mini-kb of a parent state. We use semantic matching to different answers because the knowledge-bases created by
identify overlapping features between the mini-kb in a state domain experts vary in content and organization. Thus
and the concept. The semantic matcher uses taxonomic we engineered the problem-solver to enable users to step
knowledge in the knowledge-base to find the largest con- through different answers. If the user requests a “retry”,
nected subgraph between a state and a concept. the problem-solver continues the search for additional goal
states.
Applying Operators. Applying an operator to a state cre-
ates a successor state containing an elaborated mini-kb. The 3.2 Problem-Solver with Relevance reasoning
successor state includes new knowledge introduced by the
concept from which the operator was created. The new Because states can be elaborated by potentially many
knowledge from the concept is merged with the mini-kb in concepts in the knowledge base, the problem-solver may
the parent state by joining [11] their overlapping features explore a large number of states and require several retries
found by semantic matching during operator creation. before the correct answer is returned. Our goal is to select

463
 Figure 3. The three panels show the initial state (consisting of only the question interpretation),
the concept M OTION - WITH - CONSTANT- ACCELERATION, and the results of extending the initial state with
the concept. State 2 is formed by joining the graph for M OTION - WITH - CONSTANT- ACCELERATION with
state 1 using their overlapping features (highlighted in bold in panels 1 and 2). State 2 contains the
problem-solving knowledge, in this case equations, to compute the acceleration of the M OVE given
values for initial velocity, final velocity, and distance.

only relevant concepts because we are concerned about the

scalability of the problem-solver on large knowledge-bases.
Toward this end, we do relevance reasoning with a set of
heuristics that control operator creation and ordering.

3.2.1 Creating Operators.

When creating operators, the problem-solver rejects con-
cepts when

1. No new knowledge is added Figure 5. Knowledge from FALL - FROM - REST is

added to State 1 of Fig. 4, resulting in an
2. Knowledge added is unrelated
inconsistency in which the M OVE has multiple
3. Inconsistent knowledge is added values, 17 m/s and 0 m/s.

Fig. 5 shows an example where a concept is rejected

because it adds inconsistent knowledge to a state. The
left hand side of Fig. 5 shows state 1, the original set of
triples for the physics question in Section 2. The right hand
side shows the FALL - FROM - REST concept. The seman- distance, align with FALL - FROM - REST having similar fea-
tic matcher finds the overlap between these two graphs, as tures. FALL - FROM - REST is rejected by the problem-solver
highlighted in bold. In this case, the M OVE event in state 1 because it introduces a contradiction in which the M OVE
and its properties, object, initial-velocity, final-velocity and has multiple values, 17 m/s and 0 m/s.

464
 Figure 4. The search graph created by the problem-solver in solving the example question in Section
2. Panels 2-4 show states 1, 2, and 5 in the search graph. State 1, which is the initial state in the search
graph, contains only the triples from the interpretation of the question. State 2 contains the results
of elaborating State 1 with the MOTION - WITH - CONSTANT- ACCELERATION concept in the knowledge base
being queried. This elaboration introduced an equation to infer the acceleration of the MOVE to be
14.45 meters-per-second-squared. State 2 is elaborated using the M OTION - UNDER - FORCE concept to
create state 5. State 5 contains the necessary equations to infer the net-force causing the M OVE to
be -1156 Newtons.

3.2.2 Control Strategy an entire knowledge base. Following that, we built a second
problem-solver by enhancing the naive problem-solver with
We order the application of operators to prioritize concepts
the relevance reasoning described in Section 3.2 to control
in the knowledge-base having
operator creation and ordering. Both problem-solvers use
1. knowledge structures directly affecting query the Knowledge Machine inference engine [8].
We assess the claim in two ways. First, we assess if the
2. a high degree of similarity heuristics used for relevance reasoning reduce correctness
scores. Second, we compare the number of states explored
3. the least number of assumptions added by both problem-solvers during search to assess the effi-
Fig. 6 shows how operators B and C in Fig. 4 are or- ciency of our problem-solver on large knowledge-bases.
dered based on their concept’s similarity to State 1. Figure
4.1 Setup
6(a) and 6(b) shows how state 1 matches MOTION - UNDER -
FORCE and T WO -D IMENSIONAL -M OVE respectively. The
We pose AP-like exam questions in biology and physics
match with MOTION - UNDER - FORCE is preferred because
on a variety of knowledge-bases. The test set consists of 308
a larger portion of the MOTION - UNDER - FORCE concept
questions in biology and 105 questions in physics. Our test
graph matches state 1.
harness determines whether an answer is correct by com-
paring the generated answer with an answer key. This en-
4 Evaluation ables the test harness to automatically retry a question, caus-
ing the problem-solver to return different answers, until the
We claim that simple relevance reasoning can signif- question is correctly answered or a time-bound of 20 min-
icantly improve performance without sacrificing correct- utes is reached.
ness. Three knowledge-bases are used in the evaluation:
We built two versions of the problem-solver to assess physics-domain-kb, biology-domain-kb, and multi-domain-
this claim. First, we built a problem-solver without rele- kb. The two domain-kbs were authored by domain ex-
vance reasoning to establish a gold-standard on the number perts to cover selected chapters of college-level science text-
of questions that can be correctly answered by searching books. The domain-kb for physics contained 38 concepts

465
 edge base. We found that on all knowledge-bases, the ques-
tions answered in the gold-standard were also answered by
the problem-solver with relevance reasoning. Additionally,
the problem-solver with relevance reasoning required fewer
retries to find the correct answer. This indicates that the
heuristics used in our problem-solver do not sacrifice cor-
rectness and, in fact, enable the problem-solver to find the
correct answer with fewer retries. We were pleasantly sur-
prised to find the problem-solver with relevance reasoning
(a) answering additional questions outside the gold-standard.
This is due to the larger number of states explored by the
problem-solver without relevance reasoning and our setup
aborting an attempt after a time-bound. In summary, the
lower correctness scores and the higher number of retries
required by the problem-solver without relevance reason-
ing motivates the need for relevance reasoning to focus the
search on the most relevant portions of the knowledge base.
Assessment #2: On the significantly larger, multi-
domain-kb, the problem-solver with relevance reasoning
outperform the problem-solver without relevance reason-
(b) ing, especially in physics (see Table 1). Without relevance
reasoning, a number of questions fail to answer due to
Figure 6. Different degrees of match between the significantly larger number of states explored by the
state 1 in Fig. 4 and the MOTION - UNDER - FORCE problem-solver and our setup aborting the attempt after a
and TWO - DIMENSIONAL - MOVE concepts in the time-bound. Table 2 lists the number of states explored
knowledge-base. The match with MOTION - by both problem-solvers to find the correct answers on all
UNDER - FORCE has a higher degree of match knowledge bases. We found relevance reasoning to exhibit
because a larger portion of MOTION - UNDER - good scalability on large knowledge-bases by reducing the
FORCE matches state 1. Thus, Operator B is number of states explored by the problem-solver.
preferred over Operator C in Fig. 4.
5 Summary

We described a problem solver built to answer Advanced

and the biology version contained 202 concepts. To have Placement-like questions using knowledge bases authored
a meaningful assessment on the number of states explored by domain experts. The purpose of the problem solver is to
by both problem-solvers on large knowledge-bases, we cre- identify the portion of the knowledge base that is relevant
ated the multi-domain-kb – containing 736 concepts – by for answering each question. We described an approach
concatenating the knowledge-bases of both the domain-kbs based on state-space search, and a set of simple heuristics
for physics and biology, plus the domain-kb for a novel do- for judging relevance. We showed that relevance reasoning
main, chemistry. significantly improves run-time performance without sacri-
ficing coverage.
4.2 Results
References
Questions in both domains were attempted by both ver-
sions of the problem-solver using the respective domain-kb [1] J. S. Aikins. Prototypical knowledge for expert systems. Ar-
and the multi-domain-kb. We measured the number of cor- tificial Intelligence, (20):163–210, 1983.
rectly answered questions, the number of retries required, [2] K. Barker, S. Y. Chaw, J. Fan, B. Porter, D. Tecuci, P. Yeh,
V. K. Chaudhri, D. Israel, S. Mishra, P. Romero, and P. E.
and the number of states explored by the problem-solvers
Clark. A question-answering system for AP chemistry: As-
to find the correct answer.
sessing KR&R technologies. In Principles of Knowledge
Assessment #1: The correctness scores for both Representation and Reasoning: Proceedings of the Ninth In-
problem-solvers are listed in Table 1. The gold-standard is ternational Conference, 2004.
the set of questions answered by the problem-solver with- [3] K. Barker, B. Porter, and P. Clark. A library of generic
out relevance reasoning, i.e., by searching the entire knowl- concepts for composing knowledge bases. In Proceedings

466
 Question set Problem-Solver Knowledge-base used
Domain-kb Multi-domain-kb
% correct Average % correct Average
retries required retries required
With relevance reasoning 68.51 0.23 65.91 0.19
Biology
Without relevance reasoning 68.18 0.27 64.29 0.26
With relevance reasoning 73.33 0.19 70.67 0.16
Physics
Without relevance reasoning 70.67 0.57 56.00 0.54

Table 1. The correctness scores for the problem-solvers with and without relevance reasoning. Both
problem-solvers achieved similar correctness scores on the domain knowledge-bases. This indi-
cates that relevance reasoning did not sacrifice correctness. The problem-solver without relevance
reasoning recorded lower correctness scores when used with the significantly larger multi-domain-
kb when answering physics questions. This is due to the large number of states explored during
blind-search and our evaluation setup aborting an attempt after a time-bound is reached. This result
highlights the need for the problem solver to select only the most relevant portions of the knowledge
base to reason with.

Question set Problem-Solver Number of states explored

Domain-kb Multi-domain-kb
avg median 75th 90th max avg median 75th 90th max
With relevance reasoning 5 1 2 17 80 5 1 2 16 185
Biology
Without relevance reasoning 12 1 10 17 195 36 1 10 92 704
With relevance reasoning 4 2 4 10 58 3 2 4 6 15
Physics
Without relevance reasoning 10 6 13 22 69 97 1 232 235 243

Table 2. The average, median, 75th percentile, 90th percentile, and the maximum number of states
explored by both problem-solvers – with and without relevance reasoning – for both domains.

467
 of First International Conference on Knowledge Capture,
2001.
[4] R. J. Brachman and J. G. Schmolze. An overview of the kl-
one knowledge representation system. Cognitive Science,
9:171–216, 1985.
[5] S.-Y. Chaw, J. Fan, D. Tecuci, and P. Yeh. Capturing a
Taxonomy of Failures During Automatic Interpretation of
Questions Posed in Natural Language. In Proceedings to
The Fourth International Conference on Knowledge Capture
(KCAP, 2007.
[6] P. Clark, S.-Y. Chaw, K. Barker, V. Chaudhri, P. Harrison,
J. Fan, B. John, B. Porter, A. Spaulding, J. Thompson, and
P. Z. Yeh. Capturing and Answering Questions Posed to
a Knowledge-Based System. In Proceedings of the Fourth
International Conference on Knowledge Capture, 2007.
[7] P. Clark and P. Harrison. Controlled language processing for
Halo question-asking, 2003.
[8] P. Clark and B. Porter. KM - The Knowl-
edge Machine: Reference manual. Techni-
cal report, University of Texas at Austin, 1998.
http://www.cs.utexas.edu/users/mfkb/km.html.
[9] N. Friedland, P. Allen, P. Matthews, M. Witbrock,
D. Baxter, J. Curtis, B. Shepard, P. Miraglia, J. Angele,
S. Staab, E. Moench, H. Oppermann, D. Wenke, D. Israel,
V. Chaudhri, B. Porter, K. Barker, J. Fan, S. Chaw, P. Yeh,
D. Tecuci, and P. Clark. Project Halo: Towards a Digital
Aristotle. AI Magazine, 2004.
[10] S. M. Harabagiu, S. J. Maiorano, and M. A. Paşca. Open-
domain textual question answering techniques. Nat. Lang.
Eng., 9(3):231–267, 2003.
[11] J. F. Sowa. Conceptual Structures: Information Processing
in Mind and Machine. Addison-Wesley, 1984.
[12] Vulcan Inc. Project Halo, 2003. http://projecthalo.com.

468