>cases

NetConversions Influences
Kelley Blue Book
>Abstract
Kelley Blue Book (KBB) is one of the most visited automotive sites on the Web.
Visitors flock there to estimate the price of a car they might buy or sell. KBB needed
to enhance its site’s performance for advertisers, who had become a major source
of revenue as sales of the printed Kelley Blue Book had declined. NetConversions
is one of the new Web analytic services to evaluate Web site performance. This case
reveals how Web sites are evaluated so that new design elements can be developed
and tested. www.netconversions.com; www.kelleybluebook.com

>The Scenario
When you buy a car, new or used, you want to be sure that you do not overpay.
Kelley Blue Book (KBB), a publisher of detailed cost and other data on cars, is
in the business of assisting buyers determine fair prices. For decades, KBB
annually published a book that flew off bookstore shelves. But like many publish-
ers in the dot-com age, KBB needed to find a new business model. Today, it
makes the same information available via the Internet, deriving its revenue from
advertising exposures (measured by page view impressions) and activity on
partner links on its Web site (measured by visits generated by mouse clicks on the
link). According to Nielsen//NetRatings, the KBB Web site is the number-one
Web site for automotive research. In October 2002, KBB served up its 1 billionth
customer automotive pricing report, the backbone of its Web site.

>TrueUsabilityTM
In late 2001, KBB saw a demonstration of the information-rich actionable data
that NetConversions (NetC) could provide with its TrueUsability™ assessment.
Companies spend hundreds of thousands of dollars developing revenue-produc-
ing Web sites and need to make them as effective as possible. Each time they
change an element on a page, add a page, or delete a page, they take the risk of
endangering their customer satisfaction and loyalty as much as enhancing it. KBB
needs car buyers to return to its site repeatedly; such visits generate a major
source of its revenue.
NetConversions’ Adrian Chiu describes TrueUsability™ as a detailed
research methodology divided into two primary phases (1) site analysis and (2)
Hill Climbing™. Site analysis is further divided into three subphases: (a) visitor
tracking, (b) data analysis, and (c) reporting.
Site Analysis. In the first subphase, NetC tracks site visitor behav-
ior. This process can take between 48 hours and 7 days, depending on the
number of visitors to a site and the level of analysis the client desires on each

Business Research Methods, 11e, Cooper/Schindler

1
 NetConversions Influences Kelley Blue Book

page. Much like retail observation studies, during tracking NetC follows Web site
visitors from the moment they enter the client’s Web site and as they drill down
through the site’s numerous pages. This part of the process provides the data that
are used during the detailed analysis that follows, and later is compared to the Hill
Climbing™ data.
Data Analysis. The data analysis subphase is really a series of
different analytical processes. Most Web site research suppliers provide the big
picture of customer visits with the equivalent of NetC’s path analysis. In this
multistep process, NetC not only reveals which pages are visited but also the
actual paths that users follow when drilling down through the client’s Web site.
“Essentially, we’re looking for ‘low hanging fruit’,” shared Chiu. NetC wants to
know weaknesses along the path, the opportunities for partner exposure (adver-
tisers and other alliance partners from whom the revenue is generated), and where
visitors abandon the site. “Path analysis by itself isn’t enough to provide true
actionable data,” explains Chiu. “If you recommend redesigns based on path
analysis alone, you would be guessing about a great deal of visitor behavior.”
Luckily for KBB, NetConversions has other tools for providing that actionable
data.
Exhibit NetC-1 provides a picture of one such tool, click density analysis.
Every visitor to a given page leaves his or her mark, represented as a red dot for
every mouse-click. “One client seeing a color reproduction of click density
analysis for the first time, thought the printer’s quality control measures weren’t
working,” remembered Chiu. The Web page is seen as a canvas, with the visitor
depositing paint with every mouse click. “From the click density analysis, we can
tell physically were the visitor clicks, showing items that need to be hot-linked,
which content on the page is perceived as most valuable, and which physical
space is allocated the greatest visitor value.” In a retail study the space nearest the
door is leased at a higher cost per square foot than the space farthest removed
from the escalator on the second floor. For a Web site each space on a visitor’s
screen is valuable real estate, but some real estate is far more valuable that others.
Click density revealed that the KBB logo appearing as a graphic on each page
was perceived by many visitors as a logical link to return to the site’s home page.
“It was an obvious recommendation that the logo should be more than a design
graphic; it should be hot-linked,” concluded Chiu. Exhibit NetC-2 shows a page
before its NetC analysis.
Another tool used on the KBB project was scrolling analysis. NetC
divides a Web page into 10 zones. Depending on the detail of a given page, only
some of those zones are initially visible when the page downloads to the visitor’s
monitor. To reach deeper zones, the visitor must scroll down using the screen
scroll bar or the wheel on their mouse. “If their computer offers fast download,
visitors usually prefer a page design that requires limited scrolling,” revealed Chiu,
describing one of many insights NetC has gained from years of meta-analysis of
Web sites. “Scrolling analysis revealed that visitors were hunting for information
via mouse-click that was only available by scrolling. They just weren’t scrolling
down to reach the data.” Thus scrolling analysis revealed where a long page on

Business Research Methods, 11e, Cooper/Schindler

2
 NetConversions Influences Kelley Blue Book

KBB’s site needed to be shorter, offering links to various levels of detail on

subsequent pages. It also revealed where within the KBB site pop-up windows
should be used. The image of a car was one element that the KBB visitor would
scroll to, but advertisers and other partners weren’t as inclined to pay for what
they perceived as less valuable real estate. “Converting the car visual to a pop-up
window increased the revenue opportunities with partners,” shared Chiu. “While
with our archive of page design research data this might seem intuitive, the client
values actual data from their Web site before changing anything.”
Reporting. The last subphase of data analysis is reporting. While
this is sometimes done with personal presentations, to reduce research costs and
travel time for their employees many clients prefer a written report that can be
discussed with the NetC team over the phone. Each report, filled with detailed
images of site visitor data, is accompanied by recommendations. “For some
clients this is enough. They are willing to rely on our extensive research based
knowledge of Web page design and visitor behavior, along with their own knowl-
edge of their product. But for others, those who want to be sure that the indicated
changes will enhance a visitor’s experience, we offer Hill Climbing. KBB chose to
participate at this next level.

>Hill ClimbingTM
Hill Climbing™ is a process for testing individual changes to a Web site. It uses
a script overlay technique that eliminates the need to create a whole page with a
new design to compare any change on that page with the existing design. This is a
major cost savings for the client. For a sample of Web site visitors, when they
download a given page, the script overlay changes what they see. NetC uses an
algorithm to determine which visitors see the change; it generates a random
sample of visitors who participate unknowingly in the test of the change.
Roughly one of every 1,000 visitors to the KBB site was exposed to a
Web site design change during Hill Climbing™. For example, if a visual of the
KBB logo was not a hot-link on the original design, the script overlay could hot-
link the logo for the test. NetC tracks the test group’s behavior and compares it to
the behavior of those seeing the page as it is currently designed. “Depending on
the number of design recommendations that the client wants to test, this process
can take months,” estimates Chiu.

>ResultsTM
KBB chose to use Hill Climbing on many of the recommended changes. As a
result, it added drop-down boxes, shortened several of its pages, and created
pop-up windows for some data. You can see a sample of the orignial page and
the changed design in Exhibit NetC-2. You may see more original pages by
visiting WayBackMachine.org, a digital Web page archive (go to
www.waybackmachine.org then type www.kbb.com in the search window).
Compare these older pages to the KBB current site.

Business Research Methods, 11e, Cooper/Schindler

3
 NetConversions Influences Kelley Blue Book

The changes confirmed by research and implemented on the KBB Web

site increased page views and partner advertisement impressions by 62 percent
for the key New Car Pricing Report page. Other research, shared with advertis-
ers, enhanced the ad click-through rate. Revenue increased more than enough to
offset the cost of the research.

>URLs
www.kbb.com; www.netconversions.com

>Discussion
1 Develop the management-research question hierarchy through investigative
questions for this project.
2 Using the research process model (Exhibit 4-1), describe and evaluate the
research design of this project.
3 Describe and evaluate the sampling design for this project.

>Sources
This case was developed for Business Research Methods 8/e ©2004. Used with permission of
Pamela S. Schindler.
Adrian Chiu, NetConversions, interviewed March 25, 2003, and March 28, 2003.
“Content Site Case Study: Kelley Blue Book,” NetConversions, downloaded March 14, 2003
(http://www.metconversions/case_content.htm).
“Kelley Blue Book Serves 1 Billion Consumer Automotive Pricing Reports: Record Number of
Consumers Flock to Trusted Resource kbb.com for Used and New Car Pricing,” Kelley Blue
Book, downloaded October 28, 2002 (http://www.kbb.com/media/).
“What Is True Usability™?” NetConversions.com, downloaded March 14, 2003
(http://www.netconversions.com/true_usability.htm).

Business Research Methods, 11e, Cooper/Schindler

4
 NetConversions Influences Kelley Blue Book

Exhibit NetC-1 Sample ClickDensity™ Map

Completions Abandonments

Exhibit NetC-2 After and Before Conversion

After Before

Marketing Research, 1/e, Cooper/Schindler 5