Personalizing the User Experience on ibm.

com

Clare-Marie Karat, John Karat, Carolyn Brodie, John Vergo, and Sherman Alpert
IBM T.J. Watson Research
19 Skyline Drive
Hawthorne, NY 10532 USA
{ckarat, jkarat, brodiec, jvergo, and salpert}@us.ibm.com

ABSTRACT
Personalizing a user experience means making use of personal data in a business context

to provide value to the customer and the business. Personalization builds on privacy,

security, and trust in the context of the user task. The goals of this project were

1) to understand the value of personalization to customers and IBM and 2) to develop the

strategy for bringing personalization to the ibm.com public web site which ensures that

the top-priority goals of customers and the business are met. The strategy was formulated

by conducting literature reviews and worldwide brainstorming sessions in the Research

Division; executing heuristic usability evaluations of ibm.com and competitor sites; and

employing user-centered design methods to understand customers' views on the value of

personalization of the ibm.com site as well as IBM's business requirements. Customers

participated in three iterations of user studies (group and individual usability sessions)

that investigated potential personalization features and their relative value to site visitors.

Low and mid-level fidelity prototypes were developed to illustrate these candidate

personalization features and evaluate them in the context of user tasks regarding the

purchase and support of desktop and notebook systems, servers, and personal computer

options and accessories. The research illustrates that personalizing interactions for e-

business requires more than implementing a single function; it involves the development

of a collection of functions that together achieve the larger goal. The personalization

1
strategy and the set of 12 identified personalization features with high value to customers

and the business are described. A Personalization Value Model outlining the value of

personalization to customers and the business was created and validated through

contextual analysis and affinity diagrams of data collected from ibm.com customers and

stakeholders in the business.

INTRODUCTION

Human-computer interaction (HCI) will change when the systems with which we interact

make broad use of personal information about users. Information about a user can be

either explicitly gathered or implicitly obtained. We define the use of information about a

user to alter the content and functionality of the user experience Personalizing

Interaction. While there has been a fair amount of research aimed at enabling systems to

tailor interaction based on some understanding of the user, prior work has examined

fairly narrow contexts. Examples of this research on techniques or methods to infer user

goals include click-stream analysis [2], collaborative filtering [3,12,13], and data mining

of web user logs [6,8,10,14]. Newer techniques include using pattern classification and

developing recommender systems [4,11], combining historical profile data and online

visitation patterns [15] and online heuristic decision-making based on flowchart and rule-

based constructs [1]. In general, these methods attempt to predict user interests or goals

and automatically personalize or adapt the presentation of information. Traditionally,

most interactions with computers take place between a system that understands little of

the particular user (i.e., they have no or a very limited user model) and individuals who

have limited understanding of the system or application (i.e., they have a limited

2
conceptual model of the system). Over the last few decades, the general population has

developed more sophisticated conceptual models of the technology they use, while the

technology has made relatively small advances in understanding the human it serves. We

view a future in which human-computer interaction is greatly enhanced through advances

in the ability of technology to employ personal information about users to realize better,

more valuable interactions for users and providers alike. Although computer systems are

often seen as entities in and of themselves, in e-commerce and many other domains they

are really a set of tools which facilitate business transactions . This research begins to

provide a better understanding of the context in which users will provide various kinds of

information to systems so that the systems can provide value to the interaction between

humans who communicate and interact with each other through the technology.

FRAMING PERSONALIZATION

We define personalizing a website to mean using personal information about an

individual to tailor the experience for that individual on the site. We consider personal

information as including a very broad range of elements - from basic identifying

information such as age and income to information we are just beginning to be able to

collect such as intention or emotional state. We will use the terms "personalize" and

"personalization" here primarily because these terms are most commonly used in current

Web applications and research. The terms "adaptive", "context-aware", and "tailored

experience" have also been used to describe the elements we are addressing. Further, we

define a personalization policy as a decision made by an e-commerce company involving

the handling of personal data on the companys website. A personalization feature is a

3
method for collecting and using personal information in order to tailor a website visitors

experience on the website. A personalization policy applies to the whole website, while a

feature provides functionality for a particular task on the site. Examples of

personalization policies include the degree of visibility and control over personal data

that is given to website visitors. Examples of personalization features include

collaborative filtering and adaptive navigation.

We view personalization for e-commerce as involving an exchange between at least two

parties. In general there are two roles in the interaction - that of customer and that of

provider of the product or service. Any interaction in which information about the parties

involved is used to adapt the interaction, can be said to be "personalized."

Second, we believe the essential goal of personalization is to provide increased value to

both parties though the use of personal information [5,9]. Most research to date has

focused on personalization as involving just the user (customer) of a system. The basic

model is that a person divulges information in return for some promised benefit. This

exchange can be viewed as involving a value proposition in which the value to the

customer is a function of the costs of divulging information and the perceived benefits of

doing so. We extend this notion of a value proposition for personalization to include

consideration of the provider's value proposition - that is, the value of any personalization

feature to the organization responsible for developing the system is a function of the cost

of implementation and the benefits obtained from doing it. Thus, for the Customer, the

value of personalization = f ( cost of divulging, perceived benefits) and for the Provider,

value is a function of (cost of gathering information, perceived value). For the Provider's

4
Value Proposition (PVP), costs and benefits can generally be expressed in monetary units.

For the Customer's Value Proposition (CVP), costs and benefits are more complex, and

can involve other factors. Specifically, we suggest that the costs and benefits must be

viewed within a framework of human values that extends beyond simple economic

benefit and includes concepts of security, privacy, trust, and business relationships. For

example, to go one level deeper in our framework, we view Customer Cost for a

personalization feature to be a function of the information requirements of the feature

(e.g., explicit or implicit information), the context of the interaction (e.g., for one-time

visit or long-term relationship), customer trust in the provider (e.g., well known or new

contact), privacy (how much control does the user have over access to and use of their

personal information), and personal predispositions to divulge information (e.g., no fear

or generally wary).

THE PERSONALIZATION VALUE SPACE

When we talk about personalization we assume we are addressing a whole range of

information types and possible values to customers and businesses. For example, various

projects within IBM Research are aimed at "knowing the user" on an individual level and

as a member of some category of users (e.g., expert web user). These efforts include

everything from identifying product preferences (through explicit questions), to inferring

current goal intention (through gaze or click stream), to attempting to identify emotional

state (though facial expression Technologies vary in computational complexity, including

various rule engines or user model based calculations. Our research was not aimed

specifically at identifying a single "best technique". We do not believe this is a

5
reasonable goal because our work suggests that (1) the value of techniques to any

customer will vary with the role of the customer at any given time, (2) the value of a

technique to a business will depend on the kind of business objectives they have, and (3)

there are likely to be interactions between techniques resulting in a package of techniques

that would be optimally effective. Our research will explore this Personalization Value

Space (PVS) through an examination of personalization policies (e.g., permission

marketing, levels of identity), feature categories (e.g., collaborative filtering, click stream

analysis), user characteristics (e.g., predisposition to trust, interaction goal), and business

context (e.g., product offering, business goals). We believe that the effectiveness of

personalization efforts are a function of these four components (i.e., Effectiveness = f

(policy, feature, user context, business context)).

This conception of personalization does not stand alone; we view personalization as

closely tied to Privacy and Security research. While Privacy deals essentially with users

controls over information about themselves, personalization is concerned with the value

that might be realized by a customer and provider from sharing information with one

another. In general, Security research has to do with the confidence that data cannot be

compromised or taken by unauthorized sources. We believe that both Customers and

Providers view security as essential to proceeding with any interaction between them.

Extending this, we view Trust as an important element of the value propositions for both

customers and providers in any interaction. Customer trust of an e-business develops

through their perception that the data they provide is secure, will be used only as they

allow, and provides them value.

6
PROJECT GOALS

The goals of this project were 1) to understand the value of personalization to customers

and IBM and 2) to develop the strategy for bringing personalization to the ibm.com

public web site which ensures that the top-priority goals of customers and the business

are met. The ibm.com site includes 4 million pages of content on 2,200 subsites. In

consultation with business executives, we decided to limit the scope of our research on

personalization of the ibm.com site to the content areas related to servers and personal

computer information, sales, and support. We also integrated our activities with related

human-computer interaction (HCI) efforts on the site. The multidisciplinary team of five

researchers had eight months to complete the work, and we collaborated with several

groups across IBM to accomplish the goals.

APPROACH AND INITIAL RESULTS

The project team followed IBMs User-Centered Design process to complete the major

set of activities and deliverables for the project [16]. We provide an overview of the

major HCI activities and the initial results that shaped the iterative user studies covered

later in the paper (see Figure 1).

[Place Figure 1. ibm.com Personalization Project Major Activities..about here]

Literature Review and Brainstorming Sessions Regarding Personalization in e-

Commerce

We began the project by completing a literature review of the published research in the

area of personalization. We conducted the review to identify possible personalization

7
features and to understand the state of the art. The review covered the personalization

literature, e-commerce research and literature including one-to-one marketing and

permission marketing, adaptive hypermedia literature, and review of confidential internal

personalization, e-commerce, and HCI research and market intelligence reports. We

enriched this summary of information by conducting a brainstorming session with IBM

researchers around the world who are working in areas related to personalization. We

stated our project goals and requested ideas for personalization features to be considered

in the concept phase user studies with customers who would experience interactive

prototypes of personalized user interfaces to the site. At this point we did not prejudge

techniques for lack of feasibility.

Heuristic Usability Evaluations of Competitive Sites

The team completed a set of heuristic evaluations of the ibm.com site and key

competitors to understand current best practices regarding the user experience of

personalization of an e-commerce site, to expand our feature list, assess IBMs

competitiveness, and understand opportunities for leadership. We reviewed the Dell,

Hewlett Packard, Compaq, IBM, Sun and Amazon sites using a set of six task scenarios

covering the purchase and support of computer hardware and accessories. Each of the

team members was randomly assigned to conduct heuristic reviews using a subset of the

six user task scenarios on a subset of the six sites. The results of the competitive heuristic

analysis of the six sites showed that most sites were in their infancy with regard to

personalization with the obvious exception of Amazon. Many sites had extremely

cumbersome and fractured user experiences. The review generated a list of 18 initial

8
design recommendations that were incorporated into both the base strategy as hypotheses

and the master list of possible personalization features.

Business Requirements Identification

The team employed an adaptation of contextual inquiry methods [17] to identify the

business requirements of ibm.com stakeholders regarding personalization of the site and

the underlying value model of personalization. Contextual inquiry is an HCI method that

enables practitioners to identify user issues through observation of users in context, use of

probing questions, and collection and analysis of key data points. Inductive reasoning is

employed to identify issues through the voice of the customer and build hierarchies

from the bottom up based on data instances, to a larger view of patterns, and affinity

diagrams of the associations that highlight common issues and themes of customer issues

and requirements. The ibm.com stakeholders were the primary user group to identify

business requirements. We adapted contextual inquiry methods to the area of business

requirements identification. We combined the use of probing questions with user

observation to gain a deep understanding of stakeholder goals through analysis of key

data points and the construction of affinity diagrams and the model of personalization

from them. We met with 12 representatives of marketing, sales, development, finance,

solutions, support, and hardware and software brands. Teams of two met with the

stakeholder and sometimes an associate. We asked them to tell us about the business

goals they were responsible for related to the site and potential personalization of it. We

probed for specific examples of statements to ground them in real events. We observed

each user work environment for a maximum of 120 minutes.

9
The resulting affinity diagrams document the stakeholders business requirements

regarding customer experience goals for the site, the quality of customer relationships,

business financial goals, and infrastructure goals for the site. The analysis also identified

the target customers of personalization, themes regarding the personalization pilot on the

site, and identification of obstacles and limitations in achieving the identified goals. With

these data we built the business view of the Value Model of Personalization and

combined it with the customer view obtained during the user studies to complete the

personalization value model discussed later in this paper.

The Master List of Possible Personalization Features

The team gathered information from as many sources as possible about potential

personalization features that might be used to provide value to customers and the

business on the ibm.com site. Space prohibits us from providing the complete list here,

however, there were 75 personalization features that we initially catalogued and we view

this list as a snapshot in time. The list will evolve and change across time. We present a

summary of the general categories into which the personalization polices and features

clustered in Table 1 below.

[Place Table 1. Personalization Feature Clusters ..about here.]

Our goal was to study as wide a range of personalization features as possible. However,

we hypothesized that having a central place on the website around which all the

personalization features could be accessed and all personal data found would be seen as

valuable to website visitors. We choose to use the construct of a Personal Book, created

10
by Dr. Karat in previous e-commerce research [18], to test this hypothesis. The Personal

Book, referenced in the first cluster in Table 1, is a personal space on the website which is

created when a visitor chooses to register with the site. It is available from any page

within the site and provides the visitor with both constant access to his or her profile and

quick links to all of the other personalization features, such as a list of purchased products

that allows users to track transactions, find compatible accessories, find replacements for

discontinued items, and see a history of their IT purchases. Figure 2 shows an illustration

of the Personal Book used in the Study 3 prototype. Other personalization features

available through the Personal Book include the ability to filter products based on user

needs, and the ability to indicate items that the user may wish to purchase in the future so

that they are notified of special offerings involving those products.

[Place Figure 2. The Personal Book ..about here]

In addition, there were three personalization policies that we hypothesized were crucial to

the success of personalization on the site and that formed the base strategy. They were: 1)

Giving website visitors control of the data in their profiles; 2) Asking visitors for the

minimal amount of personal information necessary and providing immediate value to the

customer based on use of it (Permission Marketing); 3) Enabling website visitors to adopt

different levels of identity as appropriate to their tasks on the website. Each of these will

be discussed in more detail below.

Ownership of Data

In the past many companies viewed the data they collected about visitors to their website

as something the company owned and could use in any way it liked. This view has been

11
changing for some time. Both social and legal pressures in Europe have forced companies

to view personal data as being owned by the subject of the data [19]. The United States

has been slower to adopt laws, preferring to have business self-regulate, however, similar

pressures are at work [20, 23]. Given this trend, we wanted to explicitly include the

policy that customers own their own data to understand its value to e-commerce

customers. By own their own data we mean, customers can view, edit, and delete

information about themselves, their purchases, and their actions on the website at any

point in time and give permission for the e-commerce company for specific uses of the

data.

Permission Marketing

Permission Marketing [5] is the concept that a customers profile is built slowly over time

as the individual develops trust in the e-commerce company. The customer is only asked

to provide the information needed to enable specific services and receives immediate

value for all the information that he or she provides. Many personalized websites today

require that anyone who wants to use any personalization features on the site must

register by filling out lengthy questionnaires. Hagan has found that people often defeat

the purpose of these forms by entering incorrect information [22]. We wanted to

determine if only asking for the information needed to provide an immediate service to

website users would increase their willingness to share data.

Levels Of Identity on an E-Commerce Website

Schaffer defines the levels of identity concept as the degree of personal information to

which a website has access based on the type of relationship between the e-commerce

company and the customer at any given point in time [21]. According to Schaffers

definition this ranges from no information (visitor is invisible) when a user has cookies

12
turned off, to knowing which of several possible roles an individual is using during any

given session (visitor has differentiated roles) (see Table 2). Roles a user might have

include a home and work role or perhaps multiple work roles.

[Place Table 2.The Levels of Identityabout here]

Summary of Approach

The initial user-centered design activities generated a master list of 75 possible

personalization features and a base strategy for personalization, essentially, a set of

hypotheses that we explored through a series of three iterative user studies. We leveraged

other HCI activities that had been completed or were underway for the Personal

Computer Division and Server Group. The outcome of the three studies with target

customers was a list of the 12 highest-value features and policies for personalization from

a customer point of view, the definition of the personalization strategy for the site, the full

documentation of the three iterations of user studies, and the business case for

personalization of the site based on customer feedback. This complete set of data enabled

the team to develop the Personalization Value Model outlining the value of

personalization to customers and the business.

USER STUDIES

We executed an iterative series of studies, carried out in laboratory and field settings.

The studies were a mixture of group sessions (similar to Design Walkthroughs) and

individual user evaluation sessions. The user studies were carried out by teams of two

with a facilitator who ran the user session and a colleague who recorded the session on

videotape and collected verbal comments from participants. Some user sessions were run

13
in the Usability Lab at the Watson Research Center, others were run in field settings in

New York City, NY; Raleigh, NC; and Austin, TX. The facilities of the usability

laboratory enabled observation of the user session in the Studio from the Control Room

through use of a one-way mirror and video monitor, as well a range of data collection

activities. In the field settings, both the facilitator and the colleague were in a conference

room setting with the participant, with a laptop computer for the participants use and a

video camera to record the session.

Target Users

Participants for the studies were recruited by an external vendor through use of user

profiles. Study subjects were drawn from the population of people who are comfortable

with the World Wide Web (3+ hours per week usage) and who are at least moderately

tech-savvy in their purchasing behavior. They make purchases on the web themselves,

but may enlist assistance from other technical experts in the selection of technology to be

purchased. Participants received $150 for taking part in a two-hour user session. New

groups of target users were recruited for each study.

User Tasks

The research team created a set of user scenarios that covered purchasing I/T equipment,

maintaining and upgrading the equipment, and getting support for products. The user

scenarios featured Pat User (whose gender was randomly assigned in each session) who

needed to complete the set of tasks that arose across a period of 18 months in Pat's

14
organization. Thus we examined both initial visits to a site with personalization features

as well as repeated-use scenarios with the same site.

Procedure

The experimental procedure for both group and individual sessions began with a pre-

session questionnaire to collect demographic and job-related information. In the group

sessions (Study 1 and 2), the experimenter then read three task scenario scripts to

participants accompanied by presentation of a storyboard prototype projected on a large

screen. The first scenario concerned buying a server and a mix of desktop and notebook

systems for a new department of ten people who were beginning a new project. The

second scenario was about upgrading the server to handle the workload of an additional

10 people and buying additional desktop and notebook systems for them. The third

scenario focused on buying accessories - - in this case zip drives - - for Pat's entire

department. Each scenario was presented using a storyboard approach where participants

saw screen shots and heard how Pat User used different personalization features to

complete specific tasks. Each scenario presented between 5 and 13 personalization

features and policies (e.g., presentation of accessories constrained to those compatible

with a selected machine previously purchased; presentation of servers compatible with

previously determined business characteristics when searching for servers; user control of

data). Each scenario presentation was about 20 minutes long and involved presentation

of about 10 screen shots. Following the presentation, the experimenter facilitated a 5-

minute discussion with the participants covering the features presented. Comments were

recorded on flipcharts in the room. Participants then completed a post-scenario

15
questionnaire and gave their individual ratings for each personalization technique covered

on a 7-point scale ranging from "Highly Valuable" to "Not at all Valuable" and design

comments in writing. After the third scenario, participants filled out a post-session

questionnaire which asked them to identify the most and least valuable features (in

relationship to their jobs) from the entire set of three scenarios.

In the individual user sessions (Study 3), six task scenarios about Pat User were provided

in written form to the participant, and he or she completed the tasks using interactive

personalized prototypes. The order of the presentation of tasks was

counterbalanced using a Latin squares design. An example of a task was "purchase

additional memory for the laptop computers you bought last month". In individual

sessions, participants were encouraged to "think aloud" as they completed tasks. Each

scenario included between 3 and 4 features and policies. Participants read the scenario

description and then attempted to complete the task described using a prototype system

implemented in Microsoft Powerpoint and presented on an IBM Thinkpad. After each

scenario, subjects filled out a questionnaire asking about their reactions to the features

presented in the scenario. Following the discussion period with the facilitator,

participants filled out a post-scenario questionnaire about the features in the scenario.

The participants were asked for written ratings and design comments. At the end of both

group and individual sessions, participants completed a post-session questionnaire form

and were debriefed before receiving payment. The post-session questionnaire asked the

participants to rank order least and most favored features across the scenarios, and they

were also asked about expected future interactions with a personalized site.

16
User Study 1

For User Study 1, we reviewed the feature list and selected features for inclusion in the

following way. First we wanted to make sure that at least one feature from each of the 14

major categories listed in Table 1 was included. Then, we selected a number of features

for each scenario as appropriate and illustrated the use of the features in the context of

complete tasks. We established a specific number of scenarios for presentation (three in

this case) so that we would be able to present the scenario, provide some illustrations of

personalization features in use, have a small group discussion, and allow participants to

fill out a questionnaire about the features before the next scenario. Three scenarios and

the post-session questionnaire fit within the two-hour session time frame.

For Study 1 there were a total of 5 two-hour sessions with 3 to 5 participants in each one,

for a total of 20 participants. The study was carried out in a usability lab specifically set

up for a group session during a week in August, 2001. The participants were all

employees of the same company as the researchers, and volunteered in response to a

request to assist in a study which offered a lunch coupon in return for participation.

Participants were recruited based on their answers to screening questions which

indicated that they had input to the decision process for the purchase of a server or

workstation in the last year.

User Study 2

For User Study 2, we reviewed the results of Study 1 and made several small adjustments

in the features included (adding 4 features), questionnaires, and scenario and storyboard

17
presentations. Study 2 was conducted over a four day period in October of 2001. There

were a total of 5 two-hour group sessions with 2 to 6 participants in each one, for a total

of 23 participants. Participants were recruited by an agency from outside of the

company employing the researchers. Participants were paid an incentive to participate in

the study. The agency used screening questionnaires similar to those used in Study 1.

Participants had been involved in the purchase of a server in the last year. About half of

them had also been involved in the purchase decision for desktop and notebook systems.

The participants were a mix of current ibm.com customers and target customers of the

site. Approximately 25% of the participants were recruited from each of the four groups

below:

* IT decision makers from traditional companies that purchase UNIX servers, with a

company size over 1000 employees

* Business Unit Executives (BUE's) from traditional companies that purchase UNIX

servers, with a company size over 1000 employees

* IT decision makers from traditional companies that purchase Intel servers, with a

company size of 50-99 employees

* IT decision makers from NetGen companies that purchase Intel servers, with a

company size of 50-99 employees

User Study 3

Feedback from User Study 1 and 2 was incorporated into the design of User Study 3.

This study involved individual participants recruited from the external target customer

18
population who interacted with a hands-on interactive mid-level fidelity prototype to

carry out six typical tasks. This is the initial use scenario:

You are Pat User and you have just become the manager of an IT department that

develops and hosts web applications for other companies. You know that you will

need to travel in your new job, so you need to purchase a laptop computer. You

want to spend less than $2000. As part of your shopping for this laptop, you look at

ibm.com. You decide to buy an A Series ThinkPad.

And this is one of the repeated use scenarios:

Three months have passed since you made your first purchases on the ibm.com

site. You are still an IT manager for a small company that provides web-hosting

services for other companies. You are now interested in finding a server to provide

web-hosting services for a new client. The client wants their data hosted on a

separate server for security reasons. You have $8,000 to spend on a server. They

need to support 150 clients at a time.

Study 3 was conducted during two weeks in December, 2001. There were a total of 22

two-hour individual user sessions. Participants were recruited by an agency from outside

of the company that employed the researchers. The agency used the same screening

questionnaires as those used in Study 2. Participants had been involved in the purchase

of a server in the last year. About half of them had also been involved in the purchase

decision for desktop and notebook systems. Participants were paid an incentive to

participate in the study.

19
RESULTS

This research was exploratory in nature. The intention was to look at a number of

techniques and to accumulate evidence about the value of the techniques in advance of

actual development. As such we relied mainly on participant ratings in a context that

simulated real usage as much as possible. We moved from an initial study with internal

users who matched the characteristics of the e-commerce target users, to using typical

external customers in Studies 2 and 3. We also moved from group walkthroughs in Study

1 and 2 to individual sessions involving completion of task scenarios by participants. The

group walkthroughs in Study 1 and 2 gave us critical high-level information from users

that allowed us to filter the personalization possibilities and identify the high value

features. Then, the results of Study 3 provided the researchers with more in-depth design

information about the most highly rated personalization features and the impact of these

features on site visitation and purchases. We began with a master list of 75 features 3

policies and identified a final list of 12 features and policies which together provided a

cohesive and valuable personalized user experience on the site that customers stated

saved them time and did some of the steps in their jobs for them while ensuring their

privacy and control over personal information (see Figure 3).

[Place Figure 3. The Development of the High-Value Personalization.. about here.]

We will summarize the iterative user studies of the personalization policies and features

by focusing on the most highly rated features and policies in each study. In general,

participants rated all of the features and policies presented above the neutral point on a 7-

point scale anchored by "Highly Valuable" and "Not Valuable at all". The average ratings

20
for features over the three studies ranged from 4.4 to 6.4, with 7 being the highest and 1

the lowest possible score. We did not employ a statistical cutoff point for determining

when to call a feature highly rated versus not highly rated. We looked for natural and

large breaks in the data and used that as a determination point. We are following up with

further studies that test for statistically significant differences in the design alternatives

for specific personalization features.

For Study 1, we found that participants wanted to provide only the information necessary,

appreciated being able to access histories of past transactions, valued the possibility of

contact with human representatives in task context, and would like more efficient search

capabilities. Participants reported the highest ratings for the personalization features

below (see Table 3). We provide the wording of the text used on the questionnaires that

participants filled out to describe the features and policies. Please note that the user study

sessions provided much richer descriptions of the features and policies through the

demonstrations contained in the scenarios and storyboards.

[Place Table 3. The Highest Rated Personalization Policies..Study 1about here]

Results from the user ratings for Study 2 and Study 3 are presented in Table 4. These

were in general agreement with the findings from Study 1. Participants attached greatest

value to control over their information and access to past interactions with the company.

[Place Table 4. Mean Ratings for Top 17..Study 2 and 3 about here]

21
Figure 4 gives an example of the features available through inventory-based

personalization. Beyond these features, participants also indicated high value for

proactive support and updates, recommendations on expressed or implied needs,

constrained search capabilities, and interacting with technical representatives in task

context.

[Place Figure 4. Use of the Products That I Own..about here]

Participants thought the Help Me Find What I Need feature, which is an instance on

context-sensitive help combined with constrained search capabilities, was very valuable

(see Figure 5). This innovative features value is based on information provided by the

user and the systems awareness of the web page that the customer is currently looking at.

In the example in Figure 5, the system helps users filter the lines of servers down to the

ones they are interested in by combining customer information with the web page they

are looking at. This user interface design feature has been patented by the team this year.

Participants thought the idea of a Personal Book with all personal information that they

could interact with, and shopping carts with wish list function.

[Place Figure 5. The Help Me Find What I Need Feature..about here]

All three personalization policies which were evaluated through the course of the three

studies received strong support from target and current customers of the site. Quantitative

data was collected on the user control of data and permission marketing policies and they

were in the top ratings by users in each study. The levels of identity policy was evaluated

in several scenarios through user actions to opt-in to personalization and through

22
comments during and after scenarios. This policy also received strong customer support.

Users want separate profiles for their different roles that they can access through the

personal book to make their visits to the site as productive as possible.

Participants were unanimous in stating that they would visit the site more often if

personalization features were implemented, and most said that they would make more

purchases. While there were differences in the orderings of the features across the three

studies, we did not find these to be highly significant. For example, Table 4 shows that

the second ranked feature in Study 2 (Transaction Tracking) was ranked tied for ninth in

Study 3. The feature was still within the highly-rated category for both studies

(remember that Table 4 does not contain all personalization features tested - only those

that were highly rated). We believe that variations in the individual and group tasks

contributed to such minor differences.

Business Case Support

All three studies included post-session questions which measured participants attitudes

concerning their likelihood of increased visits and purchases from a site which was

personalized by adding the features they identified as valuable. While we realize that

such measures might not accurately reflect actual future behavior, we believed they

would offer valuable input to our customer organization in making decisions about

development funding. When asked if the subjects thought they would be more likely to

visit the e-commerce site if the features of highest value were implemented, all 23

subjects in Study 2 and all 22 Subjects in Study 3 responded Yes. The exact question

23
was "If the features you indicated as of highest value to you were implemented on the

XXX site, would you be more likely to use the site?". The modal (most frequent)

response to the follow-up question of how often they thought they would visit the site

was "10+" more times during the course of a year for both Study 2 and Study 3. The

average response across the participants was 4.3 for Study 2 and 4.0 for Study 3, where 4

represented 5-10 visits, and 5 represented 10+ visits. When asked if the participants

thought they would be more likely to purchase from the site if the features of highest

value were implemented and appropriate products were available, 22 out of 23

participants in Study 2 and all 22 participants in Study 3 responded Yes. One

participant responded Maybe. The exact question was "If these features were

implemented on the XXX site and the company had a product that met your needs, was

available, and was within your budget, would you be more likely to purchase from the

site?". The participants' modal and mean response in terms of the number of additional

purchases in a year was the range of 3-4 purchases. The modal response on the amount of

the purchase was the $5,001-$10,000 range. The mean across the participants was 5.8 in

Study 2 and 5.7 in Study 3, with "5" representing $2,501- $5,000 and "6" representing

$5,001- $10,000.

We utilized the results from these questions to build a business case for the financial

value of adding personalization features to the web site. Major assumptions for the

business case were that projections were based on: 1) call center and web based revenue

for the Americas, 2) for the personal computer and server areas of the site only, 3) for the

target population of customers covered in the user studies, and 4) for the period of one

24
year. Projections of increased site traffic and increased purchase transactions were added

to existing mathematical and financial models relating site visits to revenue, and a

significant business case for the value of personalization to the e-commerce site resulted.

The costs of implementation of the top-ranked personalization features were included in

the business case as well. This projection was based on user statements about what they

thought they would do rather than on measures of actual performance. However, since the

participants were asked these questions after immersive personalized experiences where

they completed tasks relevant to their critical day-to-day tasks, such projections are

viewed as better founded than most business case estimates in software development.

The Personalization Value Model

Based on the affinity diagrams from the contextual inquiry data analyses of the business

requirements collected from the ibm.com stakeholders in the initial research for this

project and the comments made by external customers and target customers during the

immersive personalization experience in Study 3, we developed a Personalization Value

Model (see Figure 6). The model begins with a customers first time opt-in to

personalization. The system asks the customer for permission to use the minimal amount

of personal information necessary to more efficiently and effectively complete the task

the customer is currently on the site to complete. The customer agrees and provides the

information, and in exchange, the customer receives immediate value in the high quality

completion of his task. The customer experiences improved ease of use in accomplishing

tasks on the site through personalization. Customers value very highly being in control of

their personal data.

25
[Place Figure 6. Personalization Value Model about here]

One customer, echoing many, told us It makes me feel comfortable to be in control of

my information. It makes me feel I can trust a company that is not looking to control or

sell my information. Customers trust IBM because they are in control of their data and

because IBM asks permission to use their data to provide them better service which they

value. Customers told us they value the ability to complete tasks successfully and quickly

on the site with personalization, and that the personalization functionality simplifies their

jobs in small ways. As one customer expressed it: If you can save me time or do some of

the steps in my job for me so that I dont have to do them, thats real value to me and

youve got my business. Customers also thought that the personalization functionality

such as the Products I Own and the Help Me Find What I Need tabs in the Personal

Book allowed them to solve their own problems and made their decision-making simpler.

Customers expressed increased satisfaction with their personalized user experience on the

site and stated that it would have financial and organizational benefits for them as well.

The customer experience with personalization provides a feedback loop to a progressive

opt-in to personalization, as they receive more value as more personal information is

disclosed. The top loop of the model shows the business value of personalization to

ibm.com. With customer permission, personalization enables ibm.com to serve customers

more effectively and efficiently as they inquire about information and make purchases or

come to the site for after-sales support. Personalization allows ibm.com to improve

marketing effectiveness as customers self-select to receive promotional or other

marketing information on specific products. These combined benefits produce financial

and organizational benefits for ibm.com.

26
Personalization Strategy and Recommendations

Based on its analysis of the both the numerical results and the comments received, the

team recommended to ibm.com that the organization implement personalization on the

site with the three personalization policies including user control of data, permission

marketing, and levels of identity at the core of the approach. There were nine specific

personalization features which together with the three policies made a set of twelve

concepts to implement to provide a highly satisfying personalized user experience to

visitors at the site. The two personalization features about wish lists and saved shopping

carts were combined into one function as customers saw it as one. Similarly, four

personalization features which together formed a cohesive inventory-based

personalization function through the Products I Own tab in the Personal Book were

grouped together as one comprehensive feature. The set of twelve is listed in Table 5

below. They are listed in their relative order of importance as suggested by the customer

data.

[Place Table 5. Top Twelve Personalization Policies and Features Recommended..here]

As can be seen in the table, user control of data was the top valued personalization

function. Customers saw this policy as the essential basis for their willingness to opt-in to

a personalized user experience on the site. Customers were excited about the flexibility of

being able to access and use their data through the Personal Book from any page on the

site. Customers thought that automatic support updates on their products, delivered either

to their Personal Book or to their email address or as a phone message as preferred,

would provide great value and save them a lot of time and trouble by having proactive

27
alerts and updates sent to them, completing steps for them that they normally do

themselves at this time. The Products That I Own inventory-based personalization

function was viewed as providing ongoing superb value to customers. This function

would allow them to quickly and correctly find compatible accessories for products they

own, would simply decision making by enabling customers to get recommendations on

purchasing alternatives that would be compatible with previous purchases (e.g., buying

additional laptops or office systems for an organization over time), and being able to

track delivery of orders and review their information technology purchases by group,

department, or organization. Customers saw great value in being able to share this

information within their organizations and thought that this functionality would handle

responsibilities for them that they were allocating other resources, time and energy to at

this time.

The context-aware Help Me Find What I Need function provides customers a way of

quickly and effectively constraining options based on their needs when searching for

information. Permission marketing is valued as a means of earning customers sense of

trust and reliability in working with the site and in developing business relationships they

can count on. The Login Feedback, Universal Profile, and Levels of Identity together

provide valuable information to customers that the site knows who they are, regardless of

where they are on the site, and that the information in their profiles associated with their

chosen levels of identity for their current tasks can be used with their permission to

provide better and more timely service to them. Customers were excited about the idea of

saving shopping carts and being able to indicate they wanted to hear about special

28
promotions on the items in the cart. They wanted to be able to designate the period of

time for the shopping cart to be saved and to be able to share the shopping cart

information with others in their organization. Customers thought that the use of implicit

navigation data to adapt the presentation of information could be valuable, if use of the

data with permission was limited to a session life span. Customers liked this idea in

concept but thought that the technology had not matured sufficiently to quarantee its

usefulness. And last, but not least, customers valued the ability to contact IBM in the

context of their current task. They thought that being able to ask a specific question in a

chat-like session while sharing the page they were viewing with IBM would enable that

to get a quick and accurate answer to questions. For more involved questions, customers

said they would prefer to speak on the phone with an IBM representative whom they

have had previous interactions with and with whom they would be comfortable sharing

their profile data.

The team suggested to ibm.com that the organization extend the research in other key

area of the site such as software, support, and services to confirm where the results

generalize and where changes are needed. Research has been conducted this year in the

support area which completely validate the research reported here. We also recommended

that ibm.com develop guidelines to inform the space of development work underway and

we have participated in the creation of these standards. The ibm.com organization is

developing and implementing the infrastructure to accommodate the personalization

features recommended above. Planning is underway to define how to roll out the

29
implementation of the function across the 4 million pages and 2,200 subsites of the

organization.

DISCUSSION
This research collected a variety of information which supports personalization as an

important interaction feature. In this work we (1) identified potential personalization

features, (2) explored these features with potential users to understand their value, and (3)

prioritized the list of features with respect to cost and benefit information to customers

and providers. The scope of this work included commerce and support use cases for

servers, desktops, notebooks, and related accessories. We believe the results are valid for

these use cases and the targeted customers for these use cases. The target customers were

people who were comfortable with the web and at least moderately technically

sophisticated in their purchasing behavior. For Study 1, we found that participants

wanted to provide only information necessary, appreciated being able to access histories

of past transactions, valued the possibility of personal contact with company

representatives in task context, and would like more efficient search capabilities.

Participants volunteered that controlling their data was of critical importance. There was

general agreement in the results of all three studies. In all studies, participants attached

greatest value to control over their information and access to past interactions with the

company. We also expanded on the notion of guidance based on the local context in

providing a "Help me find what I need" function which was well received. Beyond these

six features, participants also indicated high value for the idea of a Personal Book with all

personal information that they could interact with, and shopping carts with wish-list

functionality. Participants were unanimous in stating that they would visit the site more

30
often if top-value personalization features were implemented, and all said that they would

make more purchases.

We researched the personalization requirements for a portion of the e-commerce site. We

believe the research needs to be extended to determine what features are best for other

parts of the site as the products and services in other parts of the site represent additional

parts of the personalization value model (i.e., other commerce types and other user

characteristics). There was a wealth of design information contained in the comments

generated during the group sessions and written by participants on the questionnaires. It

is important to note that in almost all cases, participants commented that the features

needed to be well designed to be valuable. Comments made by participants in Study 2

led us to combine the Wish List and Shopping Cart functions into a single feature -

resulting in a greatly improved feature in Study 3 (in the minds of the participants). It is

also interesting to note the general consistency in the findings between Study 2 and Study

3, given the differences in the methodology between the studies. In Study 2, participants

were in a group design walkthrough and did not directly interact with the system - they

reacted to a presentation by the experimenter. Though this presentation was designed to

be engaging, it still represented a passive experience. For Study 3, participants were

provided tasks to complete similar those used in the previous studies, but they interacted

with a prototype personalized system to complete them. The similarity in the results is

note worthy. On the one hand, the cross-validation over user studies strengthens our

confidence in the general findings. On another level, it provides some evidence about the

value of the group technique. Such group studies are often easier to conduct at early

31
design stages, but uncertainty about the validity of the results can cause researchers to

limit the use of such techniques. The research team's experience from this work has

encouraged us to use this technique as a part of future research. These similarities

between the group and individual session results provide additional evidence for the

value of the group design walkthrough method that was recently reported in the research

literature [7].

CONCLUSIONS

Our research in developing a personalization strategy for an e-commerce organization led

us to develop a framework on what it means to personalize (or tailor) an interactive

experience. Our focus is on identifying the overall value of personalization with an

emphasis on the e-commerce environment. Our research has indicated that

personalization should not be thought of as a single feature, but rather should be

considered as a space in which different features can have different values depending on

the user and business contexts. Our ongoing research will explore this Personalization

Feature Space (PFS) through a systematic examination of personalization policies (e.g.,

permission marketing, user control of data), feature categories (e.g., collaborative

filtering, click stream analysis), user characteristics (e.g., predisposition to trust,

interaction goal), and business context (e.g., product offering, business goals). We

believe that the effectiveness of personalization efforts are a function of four components

(i.e., Effectiveness = f (policy, feature, user context, business context)). The identification

of the exact functional relationships is a rich area for future research.

32
REFERENCES
1. Anupam, V., Hull, R., and Kumar, B. (2001). Personalizing e-commerce applications
with online heuristic decision making. Proceedings of the tenth International World Wide
Web Conference on World Wide Web. New York: ACM, 296-307.

2. Barrett, R., Maglio, P., & Kellem, D. (1997). How to personalize the Web.
Proceedings of CHI'97 (Atlanta, GA, March, 1997). New York: ACM, 75-82.

3. Burke, R. (1999). Integrating Knowledge-Based and Collaborative-Filtering. In

4. Duda, R. O., Hart P., and Stork, D. G. (2001). Pattern Classification, New York: John
Wiley and Sons, Inc.

5. Godin, S. (1999). Permission marketing: turning strangers into friends, and friends
into customers. New York: Simon and Schuster.

6. Heer, J, and Chi, E. H. (2002). Separating the swarm: Categorization methods for user
sessions on the web. Proceedings of CHI 2002 (Minneapolis, MN, April, 2002). New
York: ACM, 243-250.

7. Karat, CM. , Karat, J., Vergo, J., Pinhanez, C., Reicken, D., and Cofino, T. (2002).
That's entertainment! Designing streaming, multimedia web experiences. International
Journal of Human-Computer Interaction, 14, 3 & 4, 2002, 369-384.

8. Mobasher, B., Cooley, R., and Srinivastava, J. (2000). Automatic personalization based
on web usage mining. Communications of the ACM Special Issue on Personalization.
Volume 43 Issue 8, 142-151.

9. Peppers, D., and Rogers, M. (1997). Enterprise One to One: Tools for Competing in
the Interactive Age, New York: Doubleday.

10. Perkowitz, M. and Etzioni, O. (2000). Adaptive web sites. Communications of the
ACM Special Issue on Personalization. Volume 43 Issue 8, 152-158.

11. Resnick, P. and Varian, H.R. (1997). Recommender Systems. Communications of the
ACM, 40, 3, 56-58.

12. Schafer, B. J., Konstan, J., and Riedl, J. (1999). Recommender systems in e-
commerce. Proceedings of the first ACM Conference on Electronic Commerce. New
York: ACM, 158-166.

13. Schonberg, E., Cofino, T., Hoch, R., Podlaseck, M., & Spraragen, S. (2000).
Measuring success. Communications of the ACM, 43, 8, 53-57.

33
14. Spiliopoulou, M. (2000). Web usage mining for web site evaluation. Communications
of the ACM, 43, 8, 127-135.

15. VanderMeer, D., Dutta, K., Datta, A, Ramamritham, K., and Navanthe, S. B. (2000).
Enabling scalable online personalization on the web. Proceedings of the Second ACM
Conference on Electronic Commerce. New York: ACM, 185-196.

16. Vredenburg, K., Isensee, S., and Righi, C. (2001). User-Centered Design: An
Integrated Approach. New York: Prentice Hall.

17. Beyer, H., and Holtzblatt, K. (1998). Contextual Design. San Francisco: Morgan
Kaufman.

18. Berreby, D. (1999). Getting to Know You. Special Report: The Rise of eBusiness.
IBM Research Magazine, 1, 1-4.

19. Kobsa, A. (2002).Personalized hypermedia and international privacy.

20. Arent Fox. Online Privacy Law. http://www.arentfox.com

21. Schaffer, J. (2001). Personal Communications.

22. Hagen, P.R. (2000). Personalization versus Privacy. Forrester Report, November, 1-
19.

23. Volokh, E. (2000). Personalization and privacy. Communications of the ACM, 43, 8,
84-88.

34
Figure 1. ibm.com Personalization Project Major Activities and Deliverables.

35
Personal book (portal) place where all personal data can be accessed and modified
Universal profile one account for the entire site
Subscription-based services
Service and support
Recommendations based on profile data
Adaptive presentation tailored to user characteristics
Personal preferences in page layout or format (customization)
Adaptive navigation
Live chat-like or phone-based help or sales support (personal shopper)
Feedback that system recognizes a repeat visitor
Transaction history
Loyalty programs, incentives
Future purchase considerations
Your store, built by an expert
Table 1. Personalization Feature Clusters

36
 Figure 2. The Personal Book

Level of Identity Description

37
Invisible An individual who has not only not registered with the site, but has
his cookies turned off so that the website cannot detect whether he
has ever visited before. Lowest level of trust on the site.
Anonymous An individual who has cookies enabled, but has not registered on
the site. Shows slightly more trust of the site.
Identified An individual has registered with the site, providing personal
information in exchange for the use of personalization features.
Shows a high degree of trust on the site.
Associated An individual has both registered with the site and indicated that
she is associated with a particular team or organization. Shows a
very high degree of trust that the site will provide value to him and
his team.
Differentiated An individual who has created multiple profiles on the website for
different purposes (e.g. home and business, different business
roles). Shows a very high degree of trust on the site.
Table 2. The Levels of Identity

38
Figure 3. . The Development of the High-Value Personalization Feature and Policy List

39
You are asked to provide only the information needed to allow you to access a particular
feature.
A personal "myXXX" site is created for you when you provide information about
yourself.
You are asked to provide information for your profile that will be active across the site.
You can choose to be called by, or chat with, a human representative who has access to
your profile.
You can save shopping carts with price quotes, availability dates, and contact information
in them.
You can view your order history.
You can create a wish list that contains items you may be planning to buy.
You can track transaction on the site.
You have the choice of having search constrained based on current activities and profile
data. The pages displayed are adapted based on your profile.
The pages displayed are adapted based on transient implicit information, such as your
connection speed.
Table 3. The Highest Rated Personalization Policies and Features from Study1.

Personalization Feature Study 2 Study 3

40
User Control of Data 6.4* 6.4*
Automatic Support Alerts 5.9 6.2*
Order History Provided 6.1* 6.1*
Help me find what I need 5.6 6.1*
Suggest Alternate Products 5.6 6.0*
List of Products You Own 5.5 5.8
Login Feedback 5.6 5.8
Wish List 5.1 5.8
Personal Site Personal Book 5.4 5.6
Saved Shopping Carts 5.6 5.6
Transaction Tracking 6.2* 5.6
Only info Needed is Asked For 6.0* 5.5
Constrained Search 5.6 5.5
Adapt Presentation, Transient Data 5.5 5.5
Adaptive Navigation 5.6 5.2
Information Valid Across the Site 5.6 5.0
Contact Company in Context 5.5 5.0
Table 4. Mean Ratings for Top 17 Personalization Policies
and Features in Study 2 and 3.
* = Top-rated items in Study 2 and Study 3

41
Figure 5. The Help Me Find What I Need Feature that Constrains Options Based on
User Needs.

42
Figure 4. The Products That I Own Inventory-based Personalization Feature

43
Figure 6. Personalization Value Model.

44
Personalization Recommendations Policy or Feature
User Control of Data: You control all the data in your profile and can
review and edit it at any time Policy
Automatic Support: You can get automatic updates for the products that
you own. Feature
Products That I Own: You can view Products That I Own and get
alternative recommendations for items that are no longer available, find
compatible accessories and upgrades, and track/review current and past
transactions. Feature
Help Me Find What I Need: You can use Help Me Find What I Need to
help you filter through product choices and make purchase decisions. Feature
Permission Marketing: You are asked to provide only the information
needed to allow you to access the feature that helps you complete a task. Policy
Login Feedback: Once you have logged in, it is clear that the system
knows who you are. Feature
Universal Profile: The information you provide is active across the entire
site. Feature
Future Purchase Considerations: You can save shopping carts and indicate
that you want to hear about special promotions on items in that cart. Feature
Personal Book: A personal my IBM site is created for you when you
provide information about yourself. Feature
Levels of Identity: You can adopt the appropriate level of identity for the
particular task on the site. Policy
Adaptive Presentation: The pages displayed are adapted based on your
recent navigation path (implicit data with session life span). Feature
Contact IBM in Context: You can communicate with IBM in the context of
your profile and your current task. Feature
Table 5. Top 12 Personalization Policies and Features Recommended to ibm.com

45
46