Mining Knowledge from Text Using Information Extraction

Raymond J. Mooney and Razvan Bunescu

Department of Computer Sciences
University of Texas at Austin
1 University Station C0500
Austin, TX 78712-0233
[email protected], [email protected]

ABSTRACT Production of nitric oxide ( NO ) in endothelial

cells is regulated by direct interactions of
An important approach to text mining involves the use of endothelial nitric oxide synthase ( eNOS ) with
natural-language information extraction. Information ex- effector proteins such as Ca2+ -- calmodulin .
traction (IE) distills structured data or knowledge from un- Here we have ... identified a novel 34 kDa protein
structured text by identifying references to named entities , termed NOSIP ( eNOS interaction protein ) , which
as well as stated relationships between such entities. IE avidly binds to the carboxyl terminal region of the
systems can be used to directly extricate abstract knowl- eNOS oxygenase domain .
edge from a text corpus, or to extract concrete data from a
Figure 1: Medline abstract with proteins underlined.
set of documents which can then be further analyzed with
traditional data-mining techniques to discover more general
patterns. We discuss methods and implemented systems for
reasonable accuracy, able to recognize several types of enti-
both of these approaches and summarize results on min-
ties in text and identify some relationships that are asserted
ing real text corpora of biomedical abstracts, job announce-
between them [14; 25; 53].
ments, and product descriptions. We also discuss challenges
Therefore, IE can serve an important technology for text
that arise when employing current information extraction
mining. If the knowledge to be discovered is expressed di-
technology to discover knowledge in text.
rectly in the documents to be mined, then IE alone can serve
as an effective approach to text mining. However, if the doc-
1. INTRODUCTION uments contain concrete data in unstructured form rather
than abstract knowledge, it may be useful to first use IE
Most data-mining research assumes that the information to
to transform the unstructured data in the document corpus
be “mined” is already in the form of a relational database.
into a structured database, and then use traditional data-
Unfortunately, for many applications, available electronic
mining tools to identify abstract patterns in this extracted
information is in the form of unstructured natural-language
data.
documents rather than structured databases. Consequently,
the problem of text mining, i.e. discovering useful knowledge In this article, we review these two approaches to text min-
from unstructured text, is becoming an increasingly impor- ing with information extraction, using one of our own re-
tant aspect of KDD. search projects to illustrate each approach. First, we intro-
duce the basics of information extraction. Next, we discuss
Much of the work in text mining does not exploit any form
using IE to directly extract knowledge from text. Finally,
of natural-language processing (NLP), treating documents
we discuss discovering knowledge by mining data that is first
as an unordered “bag of words” as is typical in information
extracted from unstructured or semi-structured text.
retrieval. The standard a vector space model of text repre-
sents a document as a sparse vector that specifies a weighted
frequency for each of the large number of distinct words or 2. INFORMATION EXTRACTION
tokens that appear in a corpus [2]. Such a simplified repre-
sentation of text has been shown to be quite effective for a 2.1 IE Problems
number of standard tasks such as document retrieval, clas- Information Extraction (IE) concerns locating specific pieces
sification, and clustering [2; 16; 66; 60]. of data in natural-language documents, thereby extracting
However, most of the knowledge that might be mined from structured information from unstructured text. One type
text cannot be discovered using a simple bag-of-words rep- of IE, named entity recognition, involves identifying refer-
resentation. The entities referenced in a document and the ences to particular kinds of objects such as names of people,
properties and relationships asserted about and between these companies, and locations [4]. In this paper, we consider the
entities cannot be determined using a standard vector-space task of identifying names of human proteins in abstracts of
representation. Although full natural-language understand- biomedical journal articles [10]. Figure 1 shows part of a
ing is still far from the capabilities of current technology, sample abstract in which the protein names are underlined.
existing methods in information extraction (IE) are, with In addition to recognizing entities, an important problem is
extracting specific types of relations between entities. For
example, in newspaper text, one can identify that an orga-
nization is located in a particular city or that a person is

SIGKDD Explorations. Volume 7, Issue 1 - Page 3

 Sample Job Posting: number, publisher, and price of book from an Amazon web
Job Title: Senior DBMS Consultant page.
Location: Dallas,TX IE systems can also be used to extract data or knowledge
Responsibilities: from less-structured web sites by using both the HTML text
DBMS Applications consultant works with project teams in their pages as well as the structure of the hyperlinks be-
to define DBMS based solutions that support the enterprise tween their pages. For example, the WebKB project at
deployment of Electronic Commerce, Sales Force Automa- Carnegie Mellon University has explored extracting struc-
tion, and Customer Service applications. tured information from university computer-science depart-
Desired Requirements: ments [22]. The overall WebKB system attempted to iden-
3-5 years exp. developing Oracle or SQL Server apps using tify all faculty, students, courses, and research projects in
Visual Basic, C/C++, Powerbuilder, Progress, or similar. a department as well as relations between these entities
Recent experience related to installing and configuring such as: instructor(prof, course), advisor(student, prof),
Oracle or SQL Server in both dev. and deployment and member(person, project).
environments.
Desired Skills: 2.2 IE Methods
Understanding of UNIX or NT, scripting language. Know There are a variety of approaches to constructing IE sys-
principles of structured software engineering and project tems. One approach is to manually develop information-
management extraction rules by encoding patterns (e.g. regular expres-
sions) that reliably identify the desired entities or relations.
Filled Job Template: For example, the Suiseki system [8] extracts information on
title: Senior DBMS Consultant interacting proteins from biomedical text using manually de-
state: TX veloped patterns.
city: Dallas However, due to the variety of forms and contexts in which
country: US the desired information can appear, manually developing
language: Powerbuilder, Progress, C, C++, Visual Basic patterns is very difficult and tedious and rarely results in
platform: UNIX, NT robust systems. Consequently, supervised machine-learning
application: SQL Server, Oracle methods trained on human annotated corpora has become
area: Electronic Commerce, Customer Service the most successful approach to developing robust IE sys-
required years of experience: 3 tems [14]. A variety of learning methods have been applied
desired years of experience: 5 to IE.
One approach is to automatically learn pattern-based ex-
Figure 2: Sample Job Posting and Filled Template traction rules for identifying each type of entity or relation.
For example, our previously developed system, Rapier [12;
13], learns extraction rules consisting of three parts: 1) a
affiliated with a specific organization [73; 24]. In biomedical pre-filler pattern that matches the text immediately pre-
text, one can identify that a protein interacts with another ceding the phrase to be extracted, 2) a filler pattern that
protein or that a protein is located in a particular part of the matches the phrase to be extracted, and 3) a post-filler pat-
cell [10; 23]. For example, identifying protein interactions tern that matches the text immediately following the filler.
in the abstract excerpt in Figure 1 would require extracting Patterns are expressed in an enhanced regular-expression
the relation: interacts(NOSIP, eNOS). language, similar to that used in Perl [72]; and a bottom-up
IE can also be used to extract fillers for a predetermined set relational rule learner is used to induce rules from a corpus
of slots (roles) in a particular template (frame) relevant to of labeled training examples. In Wrapper Induction [37] and
the domain. In this paper, we consider the task of extract- Boosted Wrapper Induction (BWI) [30], regular-expression-
ing a database from postings to the USENET newsgroup, type patterns are learned for identifying the beginning and
austin.jobs [12]. Figure 2 shows a sample message from ending of extracted phrases. Inductive Logic Programming
the newsgroup and the filled computer-science job template (ILP) [45] has also been used to learn logical rules for iden-
where several slots may have multiple fillers. For exam- tifying phrases to be extracted from a document [29].
ple, slots such as languages, platforms, applications, and An alternative general approach to IE is to treat it as a se-
areas usually have more than one filler, while slots related quence labeling task in which each word (token) in the docu-
to the job’s title or location usually have only one filler. ment is assigned a label (tag) from a fixed set of alternatives.
Similar applications include extracting relevant sets of pre- For example, for each slot, X, to be extracted, we include a
defined slots from university colloquium announcements [29] token label BeginX to mark the beginning of a filler for X
or apartment rental ads [67]. and InsideX to mark other tokens in a filler for X. Finally,
Another application of IE is extracting structured data from we include the label Other for tokens that are not included
unstructured or semi-structured web pages. When applied in the filler of any slot. Given a sequence labeled with these
to semi-structured HTML, typically generated from an un- tags, it is easy to extract the desired fillers.
derlying database by a program on a web server, an IE sys- One approach to the resulting sequence labeling problem is
tem is typically called a wrapper [37], and the process is to use a statistical sequence model such as a Hidden Markov
sometimes referred to as screen scraping. A typical applica- Model (HMM) [57] or a Conditional Random Field (CFR)
tion is extracting data on commercial items from web stores [38]. Several earlier IE systems used generative HMM mod-
for a comparison shopping agent (shopbot) [27] such as MySi- els [4; 31]; however, discriminately-trained CRF models have
mon (www.mysimon.com) or Froogle (froogle.google.com). recently been shown to have an advantage over HMM’s [54;
For example, a wrapper may extract the title, author, ISBN 65]. In both cases, the model parameters are learned from

SIGKDD Explorations. Volume 7, Issue 1 - Page 4

 Pre-filler Pattern: Filler Pattern: Post-filler Pattern: lack of formal structure in the natural-language narrative
1) syntactic: {nn,nnp} 1) word: undisclosed 1) semantic: price
2) list: length 2 syntactic: jj in these documents. Automatically extracting information
from biomedical text holds the promise of easily consoli-
Figure 3: Sample Extraction Rule Learned by Rapier dating large amounts of biological knowledge in computer-
accessible form. IE systems could potentially gather infor-
a supervised training corpus and then an efficient dynamic mation on global gene relationships, gene functions, protein
programming method based on the Viterbi algorithm [71] is interactions, gene-disease relationships, and other important
used to determine the most probable tagging of a complete information on biological processes. Consequently, a grow-
test document. ing number of recent projects have focused on developing IE
Another approach to the sequence labeling problem for IE systems for biomedical literature [33; 23; 7; 55; 32].
is to use a standard feature-based inductive classifier to pre- We recently compared many of the IE methods introduced in
dict the label of each token based on both the token itself section 2.2 on the task of extracting human protein names
and its surrounding context. Typically, the context is rep- [10; 58]. Specifically, we evaluated Rapier, BWI, MBL,
resented by a set of features that include the one or two TBL, SVM, MaxEnt, HMM, and CRF by running 10-fold
tokens on either side of the target token as well as the la- cross-validation on set of approximately 1,000 manually-
bels of the one or two preceding tokens (which will already annotated Medline abstracts that discuss human proteins.
have been classified when labeling a sequence from left to Overall, we found that CRF’s gave us the best result on this
right). Using this general approach, IE systems have been particular problem. However, although CRF’s capture the
developed that use many different trained classifiers such dependence between the labels of adjacent words, it does
as decision trees [3], boosting [15], memory-based learning not adequately capture long-distance dependencies between
(MBL) [43], support-vector machines (SVMs) [40], maxi- potential extractions in different parts of a document. For
mum entropy (MaxEnt) [17], transformation-based learning example, in our protein-tagging task, repeated references to
(TBL)[68] and many others [64]. the same protein are common. If the context surrounding
Many IE systems simply treat text as a sequence of un- one occurrence of a phrase is very indicative of it being a
interpreted tokens; however, many others use a variety of protein, then this should also influence the tagging of an-
other NLP tools or knowledge bases. For example, a number other occurrence of the same phrase in a different context
of systems preprocess the text with a part-of-speech (POS) which is not typical of protein references. Therefore, we
tagger (e.g. [18; 9]) and use words’ POS (e.g. noun, verb, recently developed a new IE method based on Relational
adjective) as an extra feature that can be used in hand- Markov Networks (RMN’s) [69] that captures dependencies
written patterns [8], learned extraction rules [13], or induced between distinct candidate extractions in a document [11].
classifiers [64]. Several IE systems use phrase chunkers (e.g. Experimental evaluation confirmed that this approach in-
[59]) to identify potential phrases to extract [64; 73]. Others creases accuracy of human-protein recognition compared to
use complete syntactic parsers (e.g. [21]), particularly those a traditional CRF.
which try to extract relations between entities by examin- We have also evaluated several approaches to extracting pro-
ing the synactic relationship between the phrases describ- tein interactions from text in which protein names have al-
ing the relevant entities [24; 61]. Some use lexical semantic ready been identified [10]. Blaschke et al. [7; 8] manually de-
databases, such as WordNet [28], which provide word classes veloped patterns for extracting interacting proteins, where
that can be used to define more general extraction patterns a pattern is a sequence of words (or POS tags) and two
[13]. protein-name tokens. Between every two adjacent words
As a sample extraction pattern, Figure 3 shows a rule learned is a number indicating the maximum number of arbitrary
by Rapier [13] for extracting the transaction amount from words that can be skipped at this position. Therefore, we
a newswire concerning a corporate acquisition. This rule developed a new IE learning method, ELCS [10], that au-
extracts the value “undisclosed” from phrases such as “sold tomatically induces such patterns using a bottom-up rule
to the bank for an undisclosed amount” or “paid Honeywell learning algorithm that computes generalizations based on
an undisclosed price”. The pre-filler pattern matches a noun longest common subsequences [35]. Below is a sample pat-
or proper noun (indicated by the POS tags ’nn’ and ’pn’, tern that it learned for extracting protein interactions:
respectively) followed by at most two other unconstrained
- (7) interactions (0) between (5) PROT (9)
words. The filler pattern matches the word “undisclosed”
PROT (17) .
only when its POS tag is “adjective.” The post-filler pat-
tern matches any word in WordNet’s semantic class named where PROT matches a previously tagged protein name. The
“price”. induced patterns were able to identify interactions more pre-
cisely than the human-written ones.
3. EXTRACTING KNOWLEDGE Another approach we have taken to identifying protein inter-
If the information extracted from a corpus of documents actions is based on co-citation [39]. This approach does not
represents abstract knowledge rather than concrete data, try to find specific assertions of interaction in the text, but
IE itself can be considered a form of “discovering” knowl- rather exploits the idea that if many different abstracts ref-
edge from text. For example, an incredible wealth of bio- erence both protein A and protein B, then A and B are likely
logical knowledge is stored in published articles in scientific to interact. Particularly, if two proteins are co-cited signifi-
journals. Summaries of more than 11 million such articles cantly more often than one would expect if they were cited
are available in the Medline database;1 however, retrieving independently at random, then it is likely that they interact.
and processing this knowledge is very difficult due to the In order to account for the case where the co-citing abstracts
1
Available on the web at http://www.ncbi.nlm.nih.gov/entrez/

SIGKDD Explorations. Volume 7, Issue 1 - Page 5

 • Oracle ∈ application and QA Partner ∈ application →
do not actually concern protein interactions but cite multi-
SQL ∈ language
ple proteins for other reasons, we also used a Bayesian “bag
of words” text classifier trained to discriminate between ab- • Java ∈ language and ActiveX ∈ area and Graphics ∈ area
→ Web ∈ area
stracts that discuss protein interactions from those that do
not [39]. In order to find interactions, protein pairs that are • ¬(UNIX ∈ platf orm) and ¬(Windows ∈ platf orm) and
highly co-cited were filtered for those which are specifically Games ∈ area → 3D ∈ area
co-cited in abstracts that the Bayesian text-classifier assigns • AIX ∈ platf orm and ¬(Sybase ∈ application) and DB2
high-probability of discussing protein interactions. ∈ application → Lotus Notes ∈ application
Using these techniques, we recently completed the initial
phase of a large-scale project to mine a comprehensive set Figure 4: Sample rules mined from CS job postings.
of human protein interactions from the biomedical litera-
ture. By mining 753,459 human-related abstracts from Med- • HTML ∈ language and DHTML ∈ language → XML ∈
language
line with a combination of a CRF-based protein tagger, co-
citation analysis, and automatic text classification, we ex- • Dreamweaver 4 ∈ application and Web Design ∈ area →
tracted a set of 6,580 interactions between 3,737 proteins. Photoshop 6 ∈ application
By utilizing information in existing protein databases, this
automatically extracted data was found to have an accu- • ODBC ∈ application → JSP ∈ language
racy comparable to manually developed data sets. Based • Perl ∈ language and HTML ∈ language → Linux ∈
on comparisons to these existing protein databases, the co- platf orm
citation plus text-classification approach was found to be
more effective at identifying interactions than our IE ap- Figure 5: Sample rules mined from CS resumés.
proach based on ELCS. We are currently developing an im-
proved IE system for identifying direct assertions of inter-
actions in biomedical text using an SVM that exploits a ing database to discover interesting relationships. Specifi-
relational string kernel similar to that utilized by Zelenko et cally, DiscoTEX induces rules for predicting each piece of
al. [73]. information in each database field given all other informa-
By consolidating our text-mined knowledge with existing tion in a record. In order to discover prediction rules, we
manually-constructed biological databases, we have assem- treat each slot-value pair in the extracted database as a dis-
bled a large, fairly comprehensive, database of known human tinct binary feature, such as “graphics ∈ area”, and learn
protein interactions containing 31,609 interactions amongst rules for predicting each feature from all other features.
7,748 proteins. More details on our database of protein in- We have applied C4.5rules [56] to discover interesting rules
teractions have been published in the biological literature from the resulting binary data. Discovered knowledge de-
[58] and it is freely available on the web.2 Therefore, us- scribing the relationships between slot values is written in
ing automated text mining has helped build an important the form of production rules. If there is a tendency for
knowledge base of human proteins that has been recognized “Web” to appear in the area slot when “Director” appears
as a contribution worthy of publication in Genome Biology in the applications slot, this is represented by the produc-
and will hopefully become a valuable resource to biologists. tion rule, “Director ∈ application → Web ∈ area”. Sam-
ple rules that C4.5rules mined from a database of 600
4. MINING EXTRACTED DATA jobs that Rapier extracted from the USENET newsgroup
austin.jobs are shown in Figure 4. The last rule illustrates
If extracted information is specific data rather than abstract
the discovery of an interesting concept which could be called
knowledge, an alternative approach to text mining is to first
“the IBM shop;” i.e. companies that require knowledge of
use IE to obtain structured data from unstructured text and
an IBM operating system and DBMS, but not a competing
then use traditional KDD tools to discover knowledge from
DBMS, also require knowledge of Lotus Notes, another IBM
this extracted data. Using this approach, we developed a
product.
text-mining system called DiscoTEX (Discovery from Text
We also applied Ripper [20] and Apriori [1] to discover
EXtraction) [48; 49] which has been applied to mine job
interesting rules from extracted data. Sample rules mined
postings and resumés posted to USENET newsgroups as
from a database of 600 resumés extracted from the USENET
well as Amazon book-description pages spidered from the
newsgroup misc.jobs.resumes by BWI are shown in Fig-
web.
ure 5. The first two rules were discovered by Ripper while
In DiscoTEX, IE plays the important role of preprocess-
the last two were found by Apriori. Since any IE or KDD
ing a corpus of text documents into a structured database
module can be plugged into the DiscoTEX system, we also
suitable for mining. DiscoTEX uses two learning systems
used an information extractor (wrapper) manually devel-
to build extractors, Rapier [12] and BWI [30]. By training
oped for a book recommending system [44] to find interest-
on a corpus of documents annotated with their filled tem-
ing patterns in a corpus of book descriptions. Sample asso-
plates, these systems acquire pattern-matching rules that
ciation rules mined from a collection of 1,500 science fiction
can be used to extract data from novel documents.
book descriptions from the online Amazon.com bookstore are
After constructing an IE system that extracts the desired shown in Figure 6.
set of slots for a given application, a database can be con-
Ghani et al. [34], have developed a system similar to Disco-
structed from a corpus of texts by applying the extractor to
TEX and used it to mine data about companies extracted
each document to create a collection of structured records.
from the web. One of the patterns they discovered was “Ad-
Standard KDD techniques can then be applied to the result-
vertising companies are usually located in New York.”
2
http://bioinformatics.icmb.utexas.edu/idserve Unfortunately, the accuracy of current IE systems is limited,

SIGKDD Explorations. Volume 7, Issue 1 - Page 6

 • Sign of the Unicorn ∈ related books and American Science
Fiction ∈ subject ⇒ Knight of Shadows ∈ related books
5. FUTURE RESEARCH
Information extraction remains a challenging problem with
• Spider Robinson ∈ author ⇒ Jeanne Robinson ∈ author many potential avenues for progress. In section 4, we dis-
cussed mining knowledge from extracted data; this discov-
• Roger Zelazny ∈ author ⇒ 5 ∈ average rating
ered knowledge can itself be used to help improve extraction.
The predictive relationships between different slot fillers dis-
Figure 6: Sample rules mined from book descriptions. covered by KDD can provide additional clues about what
information should be extracted from a document. For ex-
ample, suppose we discover the rule “MySQL∈ language”
and therefore an automatically extracted database will in- → “Database ∈ area”. If the IE system extracted “MySQL
evitably contain a fair number of errors. An important ques- ∈ language” but failed to extract “Database ∈ area”, we
tion is whether the knowledge discovered from a “noisy” ex- may want to assume there was an extraction error and add
tracted database is significantly less reliable than knowledge “Database” to the area slot. We have developed methods for
discovered from a “clean” manually-constructed database. using mined knowledge to improve the recall of extraction
We have conducted experiments on job postings showing but not the precision [48; 52]. McCallum and Jensen [41]
that rules discovered from an automatically extracted data- propose using probabilistic graphical models to unify IE and
base are very close in accuracy to those discovered from a KDD; however, actual results on this approach are a goal of
corresponding manually-constructed database [49]. These on-going research.
results demonstrate that mining extracted data is a reliable
Most IE systems are developed by training on human anno-
approach to discovering accurate knowledge from unstruc-
tated corpora; however, constructing corpora sufficient for
tured text.
training accurate IE systems is a burdensome chore. One
Another potential problem with mining extracted data is approach is to use active learning methods to decrease the
that the heterogeneity of extracted text frequently prevents amount of training data that must be annotated by select-
traditional data-mining algorithms from discovering useful ing only the most informative sentences or passages to give
knowledge. The strings extracted to fill specific data fields to human annotators. We presented an initial approach to
can vary substantially across documents even though they active learning for IE [70]; however, more research is needed
refer to the same real-world entity. For example, the Mi- to explore methods for reducing the demand for supervised
crosoft operating system may be referred to as “Windows,” training data in IE.
“Microsoft Windows,” “MS Windows,” etc.. We developed
Another approach to reducing demanding corpus-building
two approaches to addressing this problem [47].
requirements is to develop unsupervised learning methods
One approach is to first “clean” the data by identifying all for building IE systems. Although some work has been done
of the extracted strings that refer to the same entity and in this area [19; 36], this is another promising area for future
then replacing sets of equivalent strings with canonical entity research. Developing semi-supervised learning methods for
names. Traditional KDD tools can then be used to mine IE is a related research direction in which there has been
the resulting “clean” data. Identifying textually distinct only a limited amount of work [62].
items that refer to the same entity is an instance of the
With respect to handling textual variation when mining ex-
general database “deduping” or record-linkage problem [42;
tracted data, it would be nice to see experimental compar-
5]. We have developed methods for learning string-similarity
isons of the two approaches mentioned in section 4; i.e. au-
measures and classifiers that identify equivalent items by
tomated data cleaning versus mining “soft matching” rules
training on a sample of manually “deduped” data [6].
from “dirty” data. Do both approaches discover equally ac-
Another approach to handling heterogeneity is to mine “soft curate knowledge with similar computational efficiency?
matching” rules directly from the “dirty” data extracted
When mining “soft-matching” rules, our current methods
from text. In this approach, the rule induction process is
use a fixed, predetermined similarity metric for matching
generalized to allow partial matching of data strings in order
rule antecedents to variable text data. However, we have
to discover important regularities in variable textual data.
developed adaptive learned similarity metrics for data clean-
We developed two novel methods for mining soft-matching
ing and “deduping’ [6]. It would be interesting to use such
rules. First, is an algorithm called TextRISE 3 that learns
learned similarity metrics when discovering “soft-matching”
rules whose conditions are partially matched to data using
rules since judging the similarity of textual strings is often
a similarity metric [50]. We also developed SoftApriori,
domain dependent.
a generalization of the standard Apriori algorithm for dis-
covering association rules [1] that allows soft matching us-
ing a specified similarity metric for each field [51]. In both 6. CONCLUSIONS
methods, similarity of textual items is measured using ei-
In this paper we have discussed two approaches to using
ther standard “bag of words” metrics [63] or edit-distance
natural-language information extraction for text mining. First,
measures [35]; other standard similarity metrics can be used
one can extract general knowledge directly from text. As an
for numerical and other data types. Experimental results in
example of this approach, we reviewed our project which
several domains have demonstrated that both TextRISE
extracted a knowledge base of 6,580 human protein inter-
and SoftApriori allow the discovery of interesting “soft-
actions by mining over 750,000 Medline abstracts. Second,
matching” rules from automatically-extracted data. These
one can first extract structured data from text documents
rules accurately capture regularities not discovered by tra-
or web pages and then apply traditional KDD methods to
ditional methods [46].
discover patterns in the extracted data. As an example of
3
A variant of Domingos’ RISE algorithm [26], an elegant ap- this approach, we reviewed our work on the DiscoTEX sys-
proach to unifying rule-based and instance-based methods. tem and its application to Amazon book descriptions and

SIGKDD Explorations. Volume 7, Issue 1 - Page 7

 computer-science job postings and resumés. [10] R. Bunescu, R. Ge, R. J. Kate, E. M. Marcotte, R. J.
Research in information extraction continues to develop more Mooney, A. K. Ramani, and Y. W. Wong. Compara-
effective algorithms for identifying entities and relations in tive experiments on learning information extractors for
text. By exploiting the lastest techniques in human-language proteins and their interactions. Artificial Intelligence in
technology and computational linguistics and combining them Medicine (special issue on Summarization and Informa-
with the latest methods in machine learning and traditional tion Extraction from Medical Documents), 33(2):139–
data mining, one can effectively mine useful and important 155, 2005.
knowledge from the continually growing body of electronic
documents and web pages. [11] R. C. Bunescu and R. J. Mooney. Collective information
extraction with relational Markov networks. In Proceed-
ings of the 42nd Annual Meeting of the Association for
7. ACKNOWLEDGEMENTS Computational Linguistics (ACL-04), pages 439–446,
Our research on bioinformatics extraction was done in col- Barcelona, Spain, July 2004.
laboration with Arun Ramani, Edward Marcotte, Ruifang
Ge, Rohit Kate, and Yuk Wah Wong. The DiscoTEX [12] M. E. Califf and R. J. Mooney. Relational learning of
system was developed primarily by Un Yong Nahm. This pattern-match rules for information extraction. In Pro-
research was partially supported by the National Science ceedings of the Sixteenth National Conference on Arti-
Foundation under grants IIS-0325116 and IIS-0117308. ficial Intelligence (AAAI-99), pages 328–334, Orlando,
FL, July 1999.
8. REFERENCES [13] M. E. Califf and R. J. Mooney. Bottom-up relational
learning of pattern matching rules for information ex-
[1] R. Agrawal and R. Srikant. Fast algorithms for mining traction. Journal of Machine Learning Research, 4:177–
association rules. In Proceedings of the 20th Interna- 210, 2003.
tional Conference on Very Large Databases (VLDB-94),
pages 487–499, Santiago, Chile, Sept. 1994. [14] C. Cardie. Empirical methods in information extrac-
tion. AI Magazine, 18(4):65–79, 1997.
[2] R. Baeza-Yates and B. Ribeiro-Neto. Modern Informa-
tion Retrieval. ACM Press, New York, 1999. [15] X. Carreras, L. Màrquez, and L. Padró. A simple
named entity extractor using AdaBoost. In Proceedings
[3] S. W. Bennett, C. Aone, and C. Lovell. Learning to tag
of the Seventh Conference on Natural Language Learn-
multilingual texts through observation. In Proceedings
ing (CoNLL-2003), Edmonton, Canada, 2003.
of the Second Conference on Empirical Methods in Nat-
ural Language Processing (EMNLP-97), pages 109–116,
[16] S. Chakrabarti. Mining the Web: Discovering Knowl-
Providence, RI, 1997.
edge from Hypertext Data. Morgan Kaufmann, San
Francisco, CA, 2002.
[4] D. M. Bikel, R. Schwartz, and R. M. Weischedel. An al-
gorithm that learns what’s in a name. Machine Learn-
[17] H. L. Chieu and H. T. Ng. Named entity recognition
ing, 34:211–232, 1999.
with a maximum entropy approach. In Proceedings of
[5] M. Bilenko, R. Mooney, W. Cohen, P. Ravikumar, the Seventh Conference on Natural Language Learn-
and S. Fienberg. Adaptive name matching in informa- ing (CoNLL-2003), pages 160–163, Edmonton, Canada,
tion integration. IEEE Intelligent Systems, 18(5):16–23, 2003.
2003.
[18] K. W. Church. A stochastic parts program and noun
[6] M. Bilenko and R. J. Mooney. Adaptive duplicate de- phrase parser for unrestricted text. In Proceedings of the
tection using learnable string similarity measures. In Second Conference on Applied Natural Language Pro-
Proceedings of the Ninth ACM SIGKDD International cessing, pages 136–143, Austin, TX, 1988. Association
Conference on Knowledge Discovery and Data Min- for Computational Linguistics.
ing (KDD-2003), pages 39–48, Washington, DC, Aug.
2003. [19] F. Ciravegna, A. Dingli, D. Guthrie, and Y. Wilks. Min-
ing web sites using unsupervised adaptive information
[7] C. Blaschke and A. Valencia. Can bibliographic point- extraction. In Proceedings of the 10th Conference of the
ers for known biological data be found automatically? European Chapter of the Association for Computational
protein interactions as a case study. Comparative and Linguistics, Budapest, Hungary, Apr. 2003.
Functional Genomics, 2:196–206, 2001.
[20] W. W. Cohen. Fast effective rule induction. In Proceed-
[8] C. Blaschke and A. Valencia. The frame-based mod- ings of the Twelfth International Conference on Ma-
ule of the Suiseki information extraction system. IEEE chine Learning (ICML-95), pages 115–123, San Fran-
Intelligent Systems, 17:14–20, 2002. cisco, CA, 1995.

[9] E. Brill. Transformation-based error-driven learning [21] M. J. Collins. Three generative, lexicalised models for
and natural language processing: A case study statistical parsing. In Proceedings of the 35th Annual
in part-of-speech tagging. Computational Linguistics, Meeting of the Association for Computational Linguis-
21(4):543–565, 1995. tics (ACL-97), pages 16–23, 1997.

SIGKDD Explorations. Volume 7, Issue 1 - Page 8

 [22] M. Craven, D. DiPasquo, D. Freitag, A. McCallum, edge Discovery and Data Mining (KDD-2000) Work-
T. Mitchell, K. Nigam, and S. Slattery. Learning to ex- shop on Text Mining, pages 29–36, Boston, MA, Aug.
tract symbolic knowledge from the World Wide Web. 2000.
In Proceedings of the Fifteenth National Conference on
Artificial Intelligence (AAAI-98), pages 509–516, Madi- [35] D. Gusfield. Algorithms on Strings, Trees and Se-
son, WI, July 1998. quences. Cambridge University Press, New York, 1997.

[23] M. Craven and J. Kumlien. Constructing biological [36] T. Hasegawa, S. Sekine, and R. Grishman. Discovering
knowledge bases by extracting information from text relations among entities from large corpora. In Proceed-
sources. In Proceedings of the 7th International Con- ings of the 42nd Annual Meeting of the Association for
ference on Intelligent Systems for Molecular Biology Computational Linguistics (ACL-04), pages 416–423,
(ISMB-1999), pages 77–86, Heidelberg, Germany, 1999. Barcelona, Spain, July 2004.
[24] A. Culotta and J. Sorensen. Dependency tree kernels for
[37] N. Kushmerick, D. S. Weld, and R. B. Doorenbos.
relation extraction. In Proceedings of the 42nd Annual
Wrapper induction for information extraction. In Pro-
Meeting of the Association for Computational Linguis-
ceedings of the Fifteenth International Joint Conference
tics (ACL-04), Barcelona, Spain, July 2004.
on Artificial Intelligence (IJCAI-97), pages 729–735,
[25] DARPA, editor. Proceedings of the Seventh Message Nagoya, Japan, 1997.
Understanding Evaluation and Conference (MUC-98),
Fairfax, VA, Apr. 1998. Morgan Kaufmann. [38] J. Lafferty, A. McCallum, and F. Pereira. Conditional
random fields: Probabilistic models for segmenting and
[26] P. Domingos. Unifying instance-based and rule-based labeling sequence data. In Proceedings of 18th Interna-
induction. Machine Learning, 24:141–168, 1996. tional Conference on Machine Learning (ICML-2001),
pages 282–289, Williamstown, MA, 2001.
[27] R. B. Doorenbos, O. Etzioni, and D. S. Weld. A scalable
comparison-shopping agent for the World-Wide Web. [39] E. Marcotte, I. Xenarios, and D. Eisenberg. Mining lit-
In Proceedings of the First International Conference on erature for protein-protein interactions. Bioinformatics,
Autonomous Agents (Agents-97), pages 39–48, Marina Apr;17(4):359–363, 2001.
del Rey, CA, Feb. 1997.
[40] J. Mayfield, P. McNamee, and C. Piatko. Named entity
[28] C. D. Fellbaum. WordNet: An Electronic Lexical Data-
recognition using hundreds of thousands of features. In
base. MIT Press, Cambridge, MA, 1998.
Proceedings of the Seventh Conference on Natural Lan-
[29] D. Freitag. Toward general-purpose learning for infor- guage Learning (CoNLL-2003), Edmonton, Canada,
mation extraction. In Proceedings of the 36th Annual 2003.
Meeting of the Association for Computational Linguis-
tics and COLING-98 (ACL/COLING-98), pages 404– [41] A. McCallum and D. Jensen. A note on the unifica-
408, Montreal, Quebec, 1998. tion of information extraction and data mining using
conditional-probability, relational models. In Proceed-
[30] D. Freitag and N. Kushmerick. Boosted wrapper induc- ings of the IJCAI-2003 Workshop on Learning Statis-
tion. In Proceedings of the Seventeenth National Con- tical Models from Relational Data, Acapulco, Mexico,
ference on Artificial Intelligence (AAAI-2000), pages Aug. 2003.
577–583, Austin, TX, July 2000. AAAI Press / The
MIT Press. [42] A. McCallum, S. Tejada, and D. Quass, editors. Pro-
ceedings of the KDD-03 Workshop on Data Cleaning,
[31] D. Freitag and A. McCallum. Information extraction Record Linkage, and Object Consolidation, Washington,
with HMM structures learned by stochastic optimiza- DC, Aug. 2003.
tion. In Proceedings of the Seventeenth National Con-
ference on Artificial Intelligence (AAAI-2000), Austin, [43] F. D. Meulder and W. Daelemans. Memory-based
TX, 2000. AAAI Press / The MIT Press. named entity recognition using unannotated data. In
Proceedings of the Seventh Conference on Natural Lan-
[32] C. Friedman, P. Kra, H. Yu, M. Krauthammer, and
guage Learning (CoNLL-2003), Edmonton, Canada,
A. Rzhetsky. GENIES: A natural-language processing
2003.
system for the extraction of molecular pathways from
journal articles. Bioinformatics, 17:S74–S82, 2001. Sup-
[44] R. J. Mooney and L. Roy. Content-based book recom-
plement 1.
mending using learning for text categorization. In Pro-
[33] K. Fukuda, T. Tsunoda, A. Tamura, and T. Takagi. ceedings of the Fifth ACM Conference on Digital Li-
Information extraction: Identifying protein names from braries, pages 195–204, San Antonio, TX, June 2000.
biological papers. In Proceedings of the 3rd Pacific Sym-
posium on Biocomputing, pages 707–718, 1998. [45] S. H. Muggleton, editor. Inductive Logic Programming.
Academic Press, New York, NY, 1992.
[34] R. Ghani, R. Jones, D. Mladenić, K. Nigam, and
S. Slattery. Data mining on symbolic knowledge ex- [46] U. Y. Nahm. Text Mining with Information Extraction.
tracted from the Web. In D. Mladenić, editor, Proceed- PhD thesis, Department of Computer Sciences, Univer-
ings of the Sixth International Conference on Knowl- sity of Texas, Austin, TX, Aug. 2004.

SIGKDD Explorations. Volume 7, Issue 1 - Page 9

 [47] U. Y. Nahm, M. Bilenko, and R. J. Mooney. Two ap- [59] L. A. Ramshaw and M. P. Marcus. Text chunking us-
proaches to handling noisy variation in text mining. In ing transformation-based learning. In Proceedings of the
Papers from the Nineteenth International Conference Third Workshop on Very Large Corpora, 1995.
on Machine Learning (ICML-2002) Workshop on Text
Learning, pages 18–27, Sydney, Australia, July 2002. [60] E. M. Rasmussen. Clustering algorithms. In W. B.
Frakes and R. Baeza-Yates, editors, Information Re-
[48] U. Y. Nahm and R. J. Mooney. A mutually beneficial trieval. Prentice Hall, Englewood Cliffs, NJ, 1992.
integration of data mining and information extraction. [61] S. Ray and M. Craven. Representing sentence structure
In Proceedings of the Seventeenth National Conference in hidden Markov models for information extraction. In
on Artificial Intelligence (AAAI-2000), pages 627–632, Proceedings of the Seventeenth International Joint Con-
Austin, TX, July 2000. ference on Artificial Intelligence (IJCAI-2001), pages
1273–1279, Seattle, WA, 2001.
[49] U. Y. Nahm and R. J. Mooney. Using information ex-
traction to aid the discovery of prediction rules from [62] E. Riloff. Automatically generating extraction patterns
texts. In Proceedings of the Sixth International Confer- from untagged text. In Proceedings of the Thirteenth
ence on Knowledge Discovery and Data Mining (KDD- National Conference on Artificial Intelligence (AAAI-
2000) Workshop on Text Mining, pages 51–58, Boston, 96), pages 1044–1049, Portland, OR, 1996.
MA, Aug. 2000.
[63] G. Salton. Automatic Text Processing: The Transfor-
[50] U. Y. Nahm and R. J. Mooney. Mining soft-matching mation, Analysis and Retrieval of Information by Com-
rules from textual data. In Proceedings of the Seven- puter. Addison-Wesley, 1989.
teenth International Joint Conference on Artificial In- [64] E. F. T. K. Sang and F. D. Meulder. Introduction to
telligence (IJCAI-2001), pages 979–984, Seattle, WA, the CoNLL-2003 shared task: Language-independent
July 2001. named entity recognition. In Proceedings of the Seventh
Conference on Natural Language Learning (CoNLL-
[51] U. Y. Nahm and R. J. Mooney. Mining soft-matching
2003), Edmonton, Canada, 2003.
association rules. In Proceedings of the Eleventh In-
ternational Conference on Information and Knowledge [65] S. Sarawagi and W. W. Cohen. Semi-markov condi-
Management (CIKM-2002), pages 681–683, McLean, tional random fields for information extraction. In Ad-
VA, Nov. 2002. vances in Neural Information Processing Systems 17,
Vancouver, Canada, 2005.
[52] U. Y. Nahm and R. J. Mooney. Using soft-matching
mined rules to improve information extraction. In Pro- [66] F. Sebastiani. Machine learning in automated text cate-
ceedings of the AAAI-2004 Workshop on Adaptive Text gorization. ACM Computing Surveys, 34(1):1–47, 2002.
Extraction and Mining (ATEM-2004), pages 27–32, San
[67] S. Soderland. Learning information extraction rules
Jose, CA, July 2004.
for semi-structured and free text. Machine Learning,
[53] National Institute of Standards and Technol- 34:233–272, 1999.
ogy. ACE - Automatic Content Extraction. [68] L. Tanabe and W. J. Wilbur. Tagging gene and protein
http://www.nist.gov/speech/tests/ace/. names in biomedical text. Bioinformatics, 18(8):1124–
1132, 2002.
[54] F. Peng and A. McCallum. Accurate information ex-
traction from research papers using conditional random [69] B. Taskar, P. Abbeel, and D. Koller. Discriminative
fields. In Proceedings of Human Language Technology probabilistic models for relational data. In Proceedings
Conference / North American Association for Com- of 18th Conference on Uncertainty in Artificial Intelli-
putational Linguistics Annual Meeting (HLT-NAACL- gence (UAI-2002), pages 485–492, Edmonton, Canada,
2004), Boston, MA, 2004. 2002.

[55] C. Perez-Iratxeta, P. Bork, and M. A. Andrade. As- [70] C. A. Thompson, M. E. Califf, and R. J. Mooney. Ac-
sociation of genes to genetically inherited diseases us- tive learning for natural language parsing and informa-
ing data mining. Nature Genetics, 31(3):316–319, July tion extraction. In Proceedings of the Sixteenth Inter-
2002. national Conference on Machine Learning (ICML-99),
pages 406–414, Bled, Slovenia, June 1999.
[56] J. R. Quinlan. C4.5: Programs for Machine Learning. [71] A. J. Viterbi. Error bounds for convolutional codes
Morgan Kaufmann, San Mateo,CA, 1993. and and asymptotically optimum decoding algorithm.
IEEE Transactions on Information Theory, 13(2):260–
[57] L. R. Rabiner. A tutorial on hidden Markov models and
269, 1967.
selected applications in speech recognition. Proceedings
of the IEEE, 77(2):257–286, 1989. [72] L. Wall, T. Christiansen, and R. L. Schwartz. Program-
ming Perl. O’Reilly and Associates, Sebastopol, CA,
[58] A. K. Ramani, R. C. Bunescu, R. J. Mooney, and 1996.
E. M. Marcotte. Consolidating the set of know human
protein-protein interactions in preparation for large- [73] D. Zelenko, C. Aone, and A. Richardella. Kernel meth-
scale mapping of the human interactome. Genome Bi- ods for relation extraction. Journal of Machine Learn-
ology, 6(5):r40, 2005. ing Research, 3:1083–1106, 2003.

SIGKDD Explorations. Volume 7, Issue 1 - Page 10