Capturing knowledge of User Preferences: Ontologies in

Recommender Systems
Stuart E. Middleton, David C. De Roure and Nigel R. Shadbolt
Department of Electronics and Computer Science
University of Southampton
Southampton, S017 1BJ, UK
Email : {sem99r,dder,nrs}@ecs.soton.ac.uk


 insight has led to the utilization of relevance feedback,

Tools for filtering the World Wide Web exist, but they are where people rate web pages as interesting or not
hampered by the difficulty of capturing user preferences in interesting and the system tries to find pages that match the
such a dynamic environment. We explore the acquisition of interesting examples (positive examples) and do not match
user profiles by unobtrusive monitoring of browsing the not interesting examples (negative examples). With
behaviour and application of supervised machine-learning sufficient positive and negative examples, modern machine
techniques coupled with an ontological representation to learning techniques can classify new pages with impressive
extract user preferences. A multi-class approach to paper accuracy.
classification is used, allowing the paper topic taxonomy to Obtaining sufficient examples is difficult however,
be utilised during profile construction. The Quickstep especially when trying to obtain negative examples. The
recommender system is presented and two empirical studies problem with asking people for examples is that the cost, in
evaluate it in a real work setting, measuring the terms of time and effort, of providing the examples
effectiveness of using a hierarchical topic ontology generally outweighs the reward they will eventually receive.
compared with an extendable flat list. Negative examples are particularly unrewarding, since there
  could be many irrelevant items to any typical query.
Ontology, recommender system, user profiling, machine Unobtrusive monitoring provides positive examples of what
learning the user is looking for, without interfering with the users
 normal activity. Heuristics can also be applied to infer
The mass of content available on the World-Wide Web negative examples, although generally with less confidence.
raises important questions over its effective use. With This idea has led to content-based recommender systems,
largely unstructured pages authored by a massive range of which unobtrusively watch users browse the web, and
people on a diverse range of topics, simple browsing has recommend new pages that correlate with a user profile.
given way to filtering as the practical way to manage web- Another way to recommend pages is based on the ratings of
based information – and for most of us that means search other people who have seen the page before. Collaborative
engines. recommender systems do this by asking people to rate
Search engines are very effective at filtering pages that explicitly pages and then recommend new pages that similar
match explicit queries. Unfortunately, most people find users have rated highly. The problem with collaborative
articulating what they want extremely difficult, especially if filtering is that there is no direct reward for providing
forced to use a limited vocabulary such as keywords. The examples since they only help other people. This leads to
result is large lists of search results that contain a handful of initial difficulties in obtaining a sufficient number of ratings
useful pages, defeating the purpose of filtering in the first for the system to be useful.
place. Hybrid systems, attempting to combine the advantages of
content-based and collaborative recommender systems,
       
have proved popular to-date. The feedback required for
Now people may find articulating what they want hard, but
content-based recommendation is shared, allowing
they are very good at recognizing it when they see it. This
collaborative recommendation as well. A hybrid approach
Permission to make digital or hard copies of all or part of this work for is used by our Quickstep recommender system.
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
This work follows the tradition of over 30 years of
copies bear this notice and the full citation on the first page. To copy knowledge acquisition. Knowledge acquisition above the
otherwise, or republish, to post on servers or to redistribute to lists, normal workflow is intrusive and counterproductive. We
requires prior specific permission and/or a fee. present a system with a low level of intrusiveness, driven by
K-CAP'01, October 22-23, 2001, Victoria, British Columbia, Canada.
Copyright 2001 ACM 1-58113-380-4/01/0010.$5.00

100
people making explicit choices that reflect the real world to   "   # #  # 
capture profiles. Ontologies are used both to structure the web, as in
Yahoo’s search space categorization, and to provide a
     
common basis for understanding between systems, such as
As the trend to publish research papers on-line increases,
in the knowledge query modelling language (KQML). In-
researchers are increasingly using the web as their primary
depth ontological representations are also seen in
source of papers. Typical researchers need to know about
knowledge-based systems, which use relationships between
new papers in their general field of interest, and older
web entities (bookmarks, web pages, page authors etc.) to
papers relating to their current work. In addition,
infer facts about given situations.
researchers time is limited, as browsing competes with
other tasks in the work place. It is this problem our We use an ontology to investigate how domain knowledge
Quickstep recommender system addresses. can help in the acquisition of user preferences.
Since researchers have their usual work to perform, $ $  ! %& '   
unobtrusive monitoring methods are preferred else they will Quickstep unobtrusively monitors user browsing behaviour
be reluctant to use the system. Also, very high via a proxy server, logging each URL browsed during
recommendation accuracy is not critical as long as the normal work activity. A machine-learning algorithm
system is deemed useful to them. classifies browsed URLs overnight, and saves each
Evaluation of real world knowledge acquisition systems, as classified paper in a central paper store. Explicit feedback
Shadbolt [21] discusses, is both tricky and complex. A lot and browsed topics form the basis of the interest profile for
of evaluations are performed with user log data (simulating each user.
real user activity) or with standard benchmark collections. Each day a set of recommendations is computed, based on
Although these evaluations are useful, especially for correlations between user interest profiles and classified
technique comparison, they must be backed up by real paper topics. Any feedback offered on these
world studies so we can see how the benchmark tests recommendations is recorded when the user looks at them.
generalize to the real world setting. Similar problems are Users can provide new examples of topics and correct
seen in the agent domain where, as Nwana [16] argues, it paper classifications where wrong. In this way the training
has yet to be conclusively demonstrated if people really set improves over time.
benefit from such information systems.
This is why we have chosen a real problem upon which to
evaluate our Quickstep recommender system. World Wide
Users Profile
  !  "       
Web
User modelling is typically either knowledge-based or
behaviour-based. Knowledge-based approaches engineer
static models of users and dynamically match users to the
closest model. Behaviour-based approaches use the users
behaviour itself as a model, often using machine-learning Classifier Recommender
techniques to discover useful patterns of behaviour. Kobsa
[10] provides a good survey of user modelling techniques.
The typical user profiling approach for recommender
systems is behaviour-based, using a binary model
representing what users find interesting and uninteresting. Classified papers
Machine-learning techniques are then used to assess
potential items of interest in respect to the binary model.
There are a lot of effective machine learning algorithms Figure 1 The Quickstep system
based on two classes. Sebastiani [20] provides a good ( ($& 
survey of current machine learning techniques and De The current literature lacks many clear results as to the
Roure [5] a review of recommender systems. extent knowledge-based approaches assist real-world
Although more difficult than the binary case, we choose to systems, where noisy data and differing user opinions exist.
use a multi-class behavioural model. This allows the classes For this reason we decided to compare the use of an
to represent paper topics, and hence domain knowledge to ontology against a simple flat list, to provide some
be used when constructing the user profile. We thus bring empirical evidence as to the effectiveness of this approach.
together ideas from knowledge-based and behaviour-based Two experiments are detailed within this paper. The first
modelling to address the problem domain. has 14 subjects, all using the Quickstep system for a period

101
of 1.5 months. The second has 24 subjects, again over a making it hard to benefit from any prior knowledge we may
period of 1.5 months. know about the domain (such as the paper topics). With
Both experiments divide the subjects into two groups. Quickstep, we have chosen a multi-class representation,
with each class representing a research paper topic. This
The first group uses a flat, extensible list of paper topics. allows profiles that consist of a human understandable list
Any new examples, added via explicit feedback, use this of topics. The classifier assigns each paper a class based on
flat list to select from. The users are free to add to the list as which class vector it is most similar to. Recommendations
needed. are selected from papers classified as belonging to a topic
The second group uses a fixed size topic ontology (based of interest.
on the dmoz open directory project hierarchy [6]). Topics The profile itself is computed from the correlation between
are selected from a hierarchical list based on the ontology. browsed papers and paper topics. This correlation leads to a
Interest profiles of this group take into account the super topic interest history, and a simple time-decay function
classes of any browsed topics. allows current topics to be computed.
Performance metrics are measured over the duration of the
   !  !   )&  
trial, and thus the effectiveness of both groups compared.
Research Paper Representation


 Research papers are represented as term vectors, with term
 %& '    frequency / total number of terms used for a terms weight.
Quickstep is a hybrid recommendation system, combining To reduce the dimensionality of the vectors, frequencies
both content-based and collaborative filtering techniques. less than 2 are removed, standard Porter stemming [18]
Since both web pages and user interests are dynamic in applied to remove word suffixes and the SMART [22] stop
nature, catalogues, rule-bases and static user profiles would list used to remove common words such as “the”. These
quickly become out of date. A recommender system measures are commonly used in information systems; van
approach thus appeared well suited to our problem. Rijsbergen [24] and Harman [9] provide a good discussion
Explicit feedback on browsed papers would be too of these issues.
intrusive, so unobtrusive monitoring is used providing Vectors with 10-15,000 terms were used in the trials along
positive examples of pages the user typically browses. with training set sizes of about 200 vectors. Had we needed
Many users will be using the system at once, so it is more dimensionality reduction, the popular term frequency-
sensible to share user interest feedback and maintain a inverse document frequency (TF-IDF) weighting could be
common pool of labelled example papers (provided by the used (term weights below a threshold being removed) or
users as examples of particular paper topics). latent semantic indexing (LSI).
Since there are positive examples of the kind of papers Only Postscript and PDF formats (and compressed formats)
users are interested in, we have a labelled training set. This are supported, to avoid noisy HTML pages. This makes
is ideal for supervised learning techniques, which require classification easier, at the expense of HTML only papers.
each training example to have a label (the labels are then
Research Paper Classification
used as classification classes). The alternative, unsupervised
The classification requirements are for a multi-class
learning, is inherently less accurate since it must compute
learning algorithm learning from a multi-labelled training
likely labels before classification (e.g. clustering
set. To learn from a training set, inductive learning is
techniques). We shall use a term vector representation,
required. There are quite a few inductive learning
common in machine learning, to represent a research paper.
techniques to choose from, including information theoretic
A term vector is a list of word weights, derived from the
ones (e.g. Rocchio classifier), neural networks (e.g.
frequency that the word appears within the paper.
backpropagation), instance-based methods (e.g. nearest
We could have used a binary classification approach, with neighbour), rule learners (e.g. RIPPER), decision trees (e.g.
classes for “interesting” and “not interesting”. This would C4.5) and probabilistic classifiers (e.g. naive Bayes).
have led to profiles consisting of two term vectors, one
Multiple classifier techniques such as boosting [7] exist as
representing the kind of thing the user is interested in
well, and have been shown to enhance the performance of
(computed from the positive examples) and the other what
individual classifiers.
the user is not interested in (computed from the negative
examples). Recommendations would be those page vectors After reviewing and testing many of the above options, we
that are most similar to the interesting class vector. The decided to use a nearest neighbour technique. The nearest
binary case is the simplest class representation, and neighbour approach is well suited to our problem, since the
consequently produces the best classification results when training set must grow over time and consists of multi-class
compared with multi-class methods. examples. Nearest neighbour algorithms also degrade well,
with the next closest match being reported if the correct one
One problem with such a representation is that the explicit
is not found. The IBk algorithm [1] we chose outperformed
knowledge of which topics the user is interested in is lost,

102
naive Bayes and a J48 decision tree in our tests. We also topics value. The next super class receives 25% and so on
use the boosting technique AdaBoostM1 [7], which works until the most general topic in the is-a hierarchy is reached.
well for multi-class problems if the boosted classifier is In this way, general topics are included in the profile rather
strong enough. We found that boosting always improved than just the most specific ones, producing a more rounded
the base classifiers performance in our tests. profile.
Nearest neighbour algorithms represent instances of Recommendation Algorithm
documents as term vectors within a term vector space. Recommendations are formulated from a correlation
Proximity of vectors within this term vector space indicates between the users current topics of interest and papers
similarity. To classify a new paper, the vector distance from classified as belonging to those topics. A paper is only
each example instance is calculated, and the closest recommended if it does not appear in the users browsed
neighbours returned as the most likely classes. Inverse URL log, ensuring that recommendations have not been
distance weighting is used to decrease the likelihood of seen before. For each user, the top three interesting topics
choosing distant neighbours. are selected with 10 recommendations made in total
AdaBoostM1 extends AdaBoost to handle multi-class cases (making a 4/3/3 split of recommendations). Papers are
since AdaBoost itself is a binary classifier. AdaBoostM1 ranked in order of the recommendation confidence before
repeatedly runs a weak learning algorithm (in this case the being presented to the user.
IBk classifier) for a number of iterations over various parts
Recommendation confidence =classification confidence *
of the training set. The classifiers produced (specialized for
topic interest value
particular classes) are combined to form a single composite
classifier at the end. The classification confidence is computed from the
AdaBoostM1 algorithm’s class probability value for that
Profiling Algorithm paper (somewhere between 0 and 1).
The profiling algorithm performs correlation between the
paper topic classifications and user browsing logs. Research Paper Topic Ontology
Whenever a research paper is browsed that has a classified The research paper topic ontology is based on the dmoz [6]
topic, it accumulates an interest score for that topic. Explicit taxonomy of computer science topics. It is an is-a hierarchy
feedback on recommendations also accumulates interest of paper topics, up to 4 levels deep (e.g. an “interface
values for topics. The current interest of a topic is agents” paper is-a “agents” paper). Pre-trial interviews
computed using the inverse time weighting algorithm formed the basis of which additional topics would be
below, applied to the user feedback instances. required. An expert review by two domain experts validated
the ontology for correctness before use in the trials.
n
Topic interest =
∑ Interest value(n) / days old(n) Feedback and the Quickstep Interface
Recommendations are presented to the user via a browser
1..no of instances web page. The web page applet loads the current
recommendation set and records any feedback the user
Interest values Paper browsed = 1 provides. Research papers can be jumped to, opening a new
Recommendation followed = 2 browser window to display the paper URL. If the user
Topic rated interesting = 10 likes/dislikes the paper topic, the interest feedback combo-
Topic rated not interesting = -10 box allows “interested” or “not interested” to replace the
default “no comment”. Finally, the topic of each paper can
The profile for each user consists of a list of topics and the
be changed by clicking on the topic and selecting a new one
current interest values computed for them (see below). The
from a popup menu. The ontology group has a hierarchical
interest value weighting was chosen to provide sufficient
popup menu; the flat list group has a single level popup
weight for an explicit feedback instance to dominate for
menu.
about a week, but after that browsed URL’s would again
become dominant. In this way, the profile will adapt to
changing user interests as the trial progresses.
Profile = (<user>,<topic>,<topic interest value>)*

e.g. ((someone,hypertext,-2.4)
(someone,agents,6.5)
(someone,machine learning,1.33))
If the user is using the ontology based set of topics, all
super classes gain a share when a topic receives some
interest. The immediate super class receives 50% the main

103
 the first trial. The first trials classified papers were also
kept, allowing a bigger initial collection of papers from
which to recommend in the second trial.
Both groups had their own separate training set of
examples, which diverged in content as the trial progressed.
The classifier was run twice for each research paper,
classifying once with the flat list groups training set and
once with the ontology groups training set. The classifier
algorithm was identical for both groups; only the training
set changed.
The system interface used by both groups was identical,
except for the popup menu for choosing paper topics. The
ontology group had a hierarchical menu (using the
ontology); the flat list group had a single layer menu.
Figure 2 Quickstep’s web-based interface The system recorded the times the user declared an interest
New examples can be added via the interface, with users in a topic (by selecting “interesting” or “not interesting”),
providing a paper URL and a topic label. These are added jumps to recommended papers and corrections to the topics
to the groups training set, allowing users to teach the system of recommended papers. These feedback events were date
new topics or improve classification of old ones. stamped and recorded in a log file for later analysis, along
with a log of all recommendations made. Feedback
All feedback is stored in log files, ready for the profile
recording was performed automatically by the system,
builders run. The feedback logs are also used as the primary
whenever the subjects looked at their recommendations.
metric for evaluation. Interest feedback, topic corrections
and jumps to recommended papers are all recorded. (,  
Since feedback only occurs when subjects check their
(*
+

recommendations, the data collected occurs at irregular
   !    dates over the duration of the trial. Cumulative frequency of
Two trials were conducted to assess empirically both the feedback events is computed over the period of the trial,
overall effectiveness of the Quickstep recommender system allowing trends to be seen as they develop during the trial.
and to quantify the effect made by use of the ontology. Since the total number of jumps and topics differ between
The first trial used 14 subjects, consisting of researchers the two groups, the figures presented are normalized by
from the IAM research laboratory. A mixture of 2nd year dividing by the number of topics (or recommendations) up
postgraduates up to professors was taken, all using the to that date. This avoids bias towards the group that
Quickstep system for a duration of 1.5 months. provided feedback most frequently.
The second trial used 24 subjects, 14 from the first trial and Figure 3 shows the topic interest feedback results. Topic
10 more 1st year postgraduates, and lasted for 1.5 months. interest feedback is where the user comments on a
Some minor interface improvements were made to make the recommended topic, declaring it “interesting” or “not
feedback options less confusing. interesting”. If no feedback is offered, the result is “no
The pre-trial interview obtained details from subjects such comment”.
as area of interest and expected frequency of browser use. Topic interest feedback is an indication of the accuracy of
The purpose of the two trials was to compare a group of the current profile. When a recommended topic is correct
users using an ontology labelling strategy with a group of for a period of time, the user will tend to become content
users using a flat list labelling strategy. Subject selection for with it and stop rating it as “interesting”. On the other hand,
the two groups balanced the groups as much as possible, an uninteresting topic is likely to always attract a “not
evening out topics of interest, browser use and research interesting” rating. Good topics are defined as either “no
experience (in that order of importance). Both groups had comment” or “interesting” topics. The cumulative
the same number of subjects in them (7 each for the pilot frequency figures are presented as a ratio of the total
trial, 12 each for the main trial). number of topics recommended. The not interesting ratio
(bad topics) can be computed from these figures by
In the first trial, a bootstrap of 103 example papers covering subtracting the good topic values from 1.
17 topics was used. The bootstrap examples were obtained
from bookmarks requested during the pre-trial interview. The ontology groups have a 7 and 15% higher topic
acceptance. In addition to this trend, the first trial ratios are
In the second trial, a bootstrap of 135 example papers about 10% lower than the second trial ratios.
covering 23 topics was used. The bootstrap training set was
updated to include examples from the final training sets of

104
Figure 4 shows the jump feedback results. Jump feedback is Figure 5 shows the topic correction results. Topic
where the user jumps to a recommended paper by opening corrections are where the user corrects the topic of a
it via the web browser. Jumps are correlated with topic recommended paper by providing a new one. A topic
interest feedback, so a good jump is a jump to a paper on a correction will add to or modify a groups training set so that
good topic. Jump feedback is an indication of the quality of the classification for that group will improve. The number
the recommendations being made as well as the accuracy of of corrections made is an indication of classifier accuracy.
the profile. The cumulative frequency figures are presented The cumulative frequency figures are presented as a ratio of
as a ratio of the total number of recommendations made. the total number of recommended papers seen.
There is a small 1% improvement in good jumps by the Although the flat list group has more corrections, the
ontology group. Both trials show between 8-10% of difference is only by about 1%. A clearer trend is for the
recommendations leading to good jumps. flat list group corrections to peak around 10-20 days into
the trial, and for both groups to improve as time goes on.

Good jumps / recommendations

1 0.16
Good topics / total topics .

0.95 0.14
0.9 0.12
0.85 0.1
0.8 0.08
0.75 0.06
0.7 0.04
0 10 20 30 40 50 0 10 20 30 40 50
Number of days into trial Number of days into trial

Figure 3 Ratio of good topics / total topics Figure 4 Ratio of good jumps / total recommendations
Corrections / recommendations

0.07
0.06
Trial 2, Subjects using ontology
0.05
Trial 2, Subjects using flat list
0.04
Trial 1, Subjects using ontology
0.03
Trial 1, Subjects using flat list
0.02
0.01
0 10 20 30 40 50
Number of days into trial

Figure 5 Ratio of topic corrections / total recommendations

A cross-validation test was run on each group’s final  &  !    (,  
training set, to assess the precision and recall of the From the experimental data of both trials, several
classifier using those training sets. The results are shown in suggestive trends are apparent. The initial ratios of good
table 1. topics were lower than the final ratios, reflecting the time it
Group (trial) Precision Recall Classes takes for enough log information to be accumulated to let
the profile settle down. The ontology users were 7-15%
Trial 1, Ontology 0.484 0.903 27 happier overall with the topics suggested to them.
Trial 1, Flat list 0.52 1.0 25 Our hypothesis for the ontology group’s apparently superior
Trial 2, Ontology 0.457 0.888 32 performance is that the is-a hierarchy produces a rounder,
Trial 2, Flat list 0.456 0.972 32 more complete profile by including general super class
topics when a specific topic is browsed by a user. This in
Table 1 Classifier recall and precision upon trial completion turn helps the profiler to discover a broad range of interests,
rather than just latching onto one correct topic.

105
The first trial showed fewer good topics than the second Multi-class classification is not normally applied to
trial (about a 10% difference seen by both groups). We recommender systems making direct comparison of similar
think this is because of interface improvements made for systems difficult. We would have liked to compare the
the second trial, where the topic feedback interface was usefulness of our recommender to that of other systems, but
made less confusing. Subjects were sometimes rating the lack of published experimental data of this kind means
interesting topics as not interesting if the paper quality was we can only usefully compare classification accuracy.
poor. As there are more poor quality papers than good
+
quality ones, this introduced a bias to not interesting topic
Most recommender systems use a simple binary class
feedback resulting in a lower overall ratio.
approach, using a user profile of what is interesting or not
About 10% of recommendations led to good jumps. Since interesting to the user. The Quickstep recommender system
10 recommendations were given to the users at a time, on uses a multi-class approach, allowing a profile in terms of
average one good jump was made from each set of domain concepts (research paper topics) to be built. The
recommendations received. As with the topic feedback, the multi-class classification is less accurate than other binary
ontology group again was marginally superior but only by a classification systems, but allows class specific feedback
1% margin. We think this smaller difference is due to and the use of domain knowledge (via an is-a hierarchy) to
people having time to follow only 1 or 2 recommendations. enhance the profiling process.
Thus, although the ontology group has more good topics,
Two experiments are performed in a real work setting,
only the top topic of the three recommended will really be
using 14 and 24 subjects over a period of 1.5 months. The
looked at; the result is a smaller difference between the
results suggest how using an ontology in the profiling
good jumps made and the good topics seen.
process results in superior performance over using a flat list
The flat list group has a poor correction / recommendation of topics. The ontology users tended to have more
ratio 10-20 days into the trial. We think this is due to new “rounder” profiles, including more general topics of interest
topics being added to the system. Most new topics were that were not directly suggested. This increased the
added after the users became familiar with the system, and accuracy of the profiles, and hence usefulness of the
know which topics they feel are missing. The recommendations.
familiarization process appeared to take about 10 days. The
The overall performance compares reasonably with other
classification accuracy of these new topics is poor until
recommender systems.
enough examples have been entered, typically after another
10 days.   # '
Collaborative recommender systems utilize user ratings to
The ontology group has about 1% fewer corrections for
recommend items liked by similar people. Examples of
both trials. This is small difference may indicate the utility
collaborative filtering are PHOAKS [23], which
of imposing a uniform conceptual model of paper topics on
recommends web links mentioned in newsgroups and
the subjects (by using the common topic hierarchy).
Group Lens [11], which recommends newsgroup articles.
Classifying papers is a subjective process, and will surely
be helped if people have similar ideas as to where topics fit Content-based recommender systems recommend items
in a groups overall classification scheme. with similar content to things the user has liked before.
Examples of content-based recommendation are Fab [2],
These preliminary results need to be extended so as to
which recommends web pages and ELFI [19], which
enable the application of more rigorous statistical analysis.
recommends funding information from a database.
Nevertheless, we believe the trend in the data to be
encouraging as to the utility of ontologies in recommender Personal web-based agents such as Letizia [13], Syskill &
systems. Webert [17] and Personal Webwatcher [14] track the users
browsing and formulate user profiles. Profiles are
When compared with other published systems, the
constructed from positive and negative examples of interest,
classification accuracy figures are similar, if on the low side
obtained from explicit feedback or heuristics analysing
(primarily because we use multi-class classification).
browsing behaviour. They then suggest which links are
Nearest neighbour systems such as NewsDude [3] and
worth following from the current web page by
Personal Webwatcher [14] report 60-90% classification
recommending page links most similar to the users profile.
accuracy based on binary classification. The higher figures
tend to be seen with benchmark document collections, not News filtering agents such as NewsWeeder [12] and News
real-world data. NewsWeeder [12] reports 40-60% Dude [3] recommend news stories based on content
classification accuracy using real user browsing data from similarity to previously rated examples.
two users over a period of time, so this would be the best Systems such as CiteSeer [4] use content-based similarity
comparison. If the number of classes we classify is taken matching to help search for interesting research papers
into consideration, our system compares well. within a digital library. Ontologies are also used to improve
content-based search, as seen in OntoSeek [8].

106
Mladenic [15] provides a good survey of text-learning and 9. Harman, D. An Experimental Study of Factors Important
agent systems, including content-based and collaborative in Document Ranking. Proceedings of 1986 ACM
approaches. conference on Research and development in information
retrieval, September 1986, Pisa Italy
-&&    !# '
The next step for this work is to run more trials and perform 10. Kobsa, A. User Modeling in Dialog Systems: Potentials
rigorous statistical analysis on the results. As the subjects and Hazards, AI & Society: The Journal of Human and
increase in number, we can become increasingly confident Machine Intelligence, 4:214-231, 1990
of the power of the effects we are seeing. 11. Konstan, J.A. Miller, B.N. Maltz, D. Herlocker, J.L.
Paper quality ratings will be elicited from users, so once an Gordon, L.R. Riedl, J. GroupLens: applying collaborative
interesting topic has been discovered, good quality papers filtering to Usenet news, Communications of the ACM
can be recommended before poorer quality papers. Volume 40, No. 3 (Mar. 1997)
The idea of building a profile that is understandable by the 12. Lang, K. NewsWeeder: Learning to Filter NetNews,
users could be extended to actually visualizing the ICML’95 Conference Proceedings, Tahoe City, CA, July
knowledge contained within it. This will allow the 1995, pp 331-339
recommender to engage the user in a dialogue about what 13. Lieberman, H. Letizia: An Agent That Assists Web
exactly they are interested in. The knowledge elicited from Browsing, Proceedings of the 1995 International Joint
this dialogue should allow further improvements to the Conference on Artificial Intelligence, Montreal, Canada,
recommendations made. Additionally, visualizing the August 1995
profile knowledge will allow users to build a better 14. Mladenic, D. Personal WebWatcher: Implementation and
conceptual model of the system, helping to engender a Design, Technical Report IJS-DP-7472, Department of
feeling of control and eventually trust in the system. Intelligent Systems, J.Stefan Institute, Slovenia, 1996

#+(.(/( 15. Mladenic, D. Text-Learning and Related Intelligent
This work is funded by EPSRC studentship award number Agents: A Survey, IEEE Intelligent Systems, Vol. 14, No.
99308831. 4, July/August 1999
(-((( 16. Nwana, H.S. Software agents: an overview. The
1. Aha, D. Kibler, D. Albert, M. Instance-based learning Knowledge Engineering Review, Vol. 11:3, 1996, 205-
algorithms, Machine Learning, 6:37-66, 1991 244.
2. Balabanovi, M. Shoham, Y. Fab: content-based, 17. Pazzani, M. Muramatsu J. Billsus, D. Syskill & Webert:
collaborative recommendation, Communications of the Identifying interesting web sites, Proceedings of the
ACM Volume 40, No. 3 (Mar. 1997) National Conference on Artificial Intelligence, Portland,
Oregon, 1996
3. Billsus, D. Pazzani, M. A Personal News Agent that
Talks, Learns and Explains, Proceedings of the Third 18. Porter, M. An algorithm for suffix stripping, Program 14
International Conference on Autonomous Agents (Agents (3), July 1980, pp. 130-137
'99), Seattle, Washington, 1999 19. Schwab, I. Pohl, W. Koychev, I. Learning to Recommend
4. Bollacker, K.D. Lawrence, S. Giles, C.L. CiteSeer: An from Positive Evidence, Proceedings of Intelligent User
Autonomous Web Agent for Automatic Retrieval and Interfaces 2000, ACM Press, pp 241-247
Identification of Interesting Publications, Proceedings of 20. Sebastiani, F. Machine Learning in Automated Text
the Second International Conference on Autonomous Categorization. Consiglio Nazionale delle Ricerche, Italy.
Agents, Minneapolis MN, USA, 1998 21. Shadbolt, N. O’Hara, K. Crow, L. The experimental
5. De Roure, D. Hall, W. Reich, S. Hill, G. Pikrakis, A. evaluation of knowledge acquisition techniques and
Stairmand, M. MEMOIR – an open framework for methods: history, problems and new directions,
enhanced navigation of distributed information, International Journal of Human-Computer Studies (1999)
Information Processing and Management, 37, 53-74, 2001 51, pp 729-755
6. dmoz open directory project, Project home page 22. SMART Staff, User's Manual for the SMART
http://dmoz.org/ Information Retrieval System, Technical Report 71-95,
7. Freund, Y. Schapire, R.E. Experiments with a New Revised April 1974, Cornell University (1974)
Boosting Algorithm, Proceedings of the Thirteenth 23. Terveen, L. Hill, W. Amento, B. McDonald, D. Creter, J.
International Conference on Machine Learning, 1996 PHOAKS: a system for sharing recommendations,
8. Guarino, N. Masolo, C. Vetere, G. OntoSeek: Content- Communications of the ACM Volume 40, No. 3 (Mar.
Based Access to the Web, IEEE Intelligent Systems, Vol. 1997)
14, No. 3, May/June 1999 24. van Rijsbergen, C.J. Information retrieval. Department of
Computing Science, University of Glasgow.

107