Product: OpenText Content Server

Version: 10.0.0
Task/Topic: Performance
Audience: Administrators, Decision Makers
Platform: All
Document ID: 500113
Updated: October 7, 2011

White Paper

Content Server 10.0 Performance

Assessment
Geoff Obbard, Scott Tindal, Senior Performance Analysts, Michelle Quesnel, Performance
Analyst and Alan Robertson, Quality Assurance Analyst, OpenText Performance
Engineering

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

Document Title

Contents
Introduction .................................................................................................................4
Objectives ............................................................................................................... 4
Web Services Test Objectives ................................................................................... 5
Tests and Results ........................................................................................................ 6
CWS Tests .............................................................................................................. 6
1. Ingestion Isolated Operation Tests ..................................................................... 6
2. Ingestion Load Tests ............................................................................................ 7
3. Retrieval Isolated Operation Tests.................................................................... 13
4. Retrieval Load Tests ........................................................................................... 14
Web Services Test Summary ......................................................................... 19
Web User Interface Test Objectives ........................................................................ 20
Web User Interface Isolated Operation Test........................................................... 21
1. Browse Requests ............................................................................................... 22
2. Create Requests ................................................................................................. 23
3. Fetch Requests ................................................................................................... 24
4. Personal Menu Requests ................................................................................... 26
5. Function Menu Requests ................................................................................... 27
6. Search Requests ................................................................................................ 27
Summary .............................................................................................................. 32
Web User Interface Load Tests ................................................................................ 33
Method .................................................................................................................. 33
Load Test Results ................................................................................................. 34
Load Size Summary....................................................................................... 36
Throughput ..................................................................................................... 37
Execute and Response Times ....................................................................... 40
Resource Usage ............................................................................................ 44
Thread Usage .......................................................................................... 44
CPU Usage.............................................................................................. 47
Memory Usage ........................................................................................ 49
Load Test Summary ....................................................................................... 51
Stress Test Results......................................................................................... 52
Stress Test Content Server 9.7.1 ................................................................ 52
Throughput & Response Time................................................................. 52
Resource Usage ...................................................................................... 53
Response Time Correlation ..................................................................... 55
Stress Test Content Server 2010 ................................................................ 56
Throughput and Response Time ............................................................. 56
Resource Usage ...................................................................................... 56
Response Time Correlation Content Server 2010 ............................... 58
Stress Test Summary ..................................................................................... 59
Conclusion................................................................................................................. 61
TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

Document Title

Appendix A: Test Environment Architecture.......................................................... 62

Appendix B: Test Environment Hardware .............................................................. 63
Appendix C: Non-Default Configuration ................................................................. 64
Appendix D: Data-Sets ............................................................................................. 66
Document Sizes Folder: ....................................................................................... 66
Folder Sizes Folders:............................................................................................ 66
News Article Data-Set........................................................................................... 67
Ingested Document Data Set ............................................................................... 68
About OpenText ........................................................................................................ 70

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

Document Title

Executive Summary
This paper describes the testing efforts undertaken by OpenText to determine the key
performance characteristics of Content Server 10.0 relative to those of Content
Server 9.7.1. Tests are broken down into Content Web Services ingestion and
retrieval tests, Web User Interface isolated operation, load, and stress tests.
Results of the Content Web Services ingestion load tests showed that even at eight
threads, Content Server 10.0 achieves 30% higher ingestion throughput than 9.7.1,
and a peak throughput of over 53 documents created per second at 40 threads. This
is more than three times the peak ingestion throughput of Content Server 9.7.1.
Though Content Server 10.0 exhibits almost double the Content Web Services
retrieval throughput of Content Server 9.7.1 while at the same number of threads (5),
the most significant difference occurs at peak throughput, where Content Server 10.0
achieves over 4 times the capacity of Content Server 9.7.1 (over 180 docs/second
in 10.0, compared to less than 40 docs/second in 9.7.1).
Results of the load tests of Content Server 9.7.1 and 10.0 in nearly identical
configurations show that Content Server 10.0 is able to deliver responses to a larger
volume of requests faster than Content Server 9.7.1. Coupled with the increased
scalability of threads, and address space available in a 64-bit process, a single
instance of Content Server 10.0 should be able to handle a significantly higher
volume of requests, and respond to them quicker, as compared to previous releases
of Content Server.
Results of the stress test showed that the software bottle-necks that prevented
Content Server 9.7.1 from using available hardware effectively have been removed
from Content Server 10.0, and that a single instance of Content Server 10.0 is able to
handle two and a half times the load of an instance of Content Server 9.7.1. By
virtually eliminating the need to install multiple instances on a single server (vertical
scaling), Content Server 10.0 significantly reduces the amount of effort needed to
administer larger scale deployments.
For the most part, results of these tests showed either no loss in performance, or
significant improvements to performance with Content Server 10.0. The most notable
areas of improvement in Content Server 10.0 include:
1.
2.
3.
4.

Improved thread scaling

Improved and more stable response times under load
Significantly lower CPU usage on front-end servers
Dramatically higher load carrying capacity of a single instance

Overall, these improvements present compelling reasons for customers to upgrade to

Content Server 10.0.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

Document Title

Introduction
This paper describes the testing efforts undertaken by OpenText to determine the key
performance characteristics of Content Server 10.0 relative to those of Content
Server 9.7.1. For the most part, the same test method used in previous reports of this
type has been used for this release, with the addition of several new tests. The test
suite includes isolated operation tests, load tests, and web services ingestion and
retrieval tests.

Objectives
The objectives of these tests are to:
1. Quantify the difference in performance characteristics between the current
release of Content Server (10.0), and the previous release of Content Server
(9.7.1),
2. Highlight the performance improvements introduced in the current release of
Content Server (version 10.0).
As a result of these tests, we wish to know how the current release of Content Server
(10.0) performs for a variety of functions, compared to currently supported releases
of Content Server (9.7.1):

When executed in isolation.

Under anticipated peak load.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

Document Title

Web Services Test Objectives

The objectives of these tests are to:
1. Compare throughput and performance for ingesting documents through
CWS, for CS 10 and CS 9.7.1.
2. Compare throughput and performance for retrieving documents through
CWS, for CS 10 and CS 9.7.1.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

Document Title

Tests and Results

Meeting the objectives requires two types of testing:
1. Single client thread tests to observe the performance of the software for
individual requests.
2. Load tests to observe the maximum achievable throughput of the software in
the test environment.

CWS Tests
The test is run by an open source test client called soapUI, which makes web service
requests through the Content Web Services (CWS) API. The time recorded by the
client is the time from when the request is submitted to CWS, until the entire
response is retrieved. During the test, summary timing logs are collected from
Content Server, and client timings are collected from the soapUI client.
The initial data set includes English and multilingual files sourced from Reuters news
feed documents and standard load test documents.
During these tests indexing was enabled, and distributed agents for facet calculation
were running. However, the throughput numbers presented below reflect only the
time taken to complete the web service calls to store the documents in CS. Indexing
and facet calculation are an asynchronous task that trails the ingestion, and will be
reported on in a future assessment.

1.

Ingestion Isolated Operation Tests

Ingesting (creating) documents through CWS
DocumentManagement createDocument call.

is

achieved

using

the

An initial test was done using one client thread in the soapUI test client; running 100
iterations of the createDocument scenario, to measure average create document
time under no load. The table below shows that in this no load scenario, the average
time to create a document in Content Server 10.0 is about 20 percent slower than in
CS 9.7.1.
Table 1: CWS Functional Test - Summary CreateDocument
CS 9.7.1

Content Server
10.0

100

100

235.12

279.68

Min (ms)

196

243

Max (ms)

362

351

20.54

24.98

Count
Average (ms)

Std Dev

TOGETHER, WE ARE THE CONTENT EXPERTS

% Difference

-18.9%

CHAMPION TOOLKIT

Document Title

Although this document creation time is slower in Content Server 10.0, there are new
features in Content Server 10.0 such as facets that create additional overhead for
this type of operation. Also, it is important to note that Content Server 10.0 is much
more scalable than 9.7.1 for this type of operation, as illustrated in the load test
section; when configured with 8 threads, Content Server 10.0 has 30 percent higher
throughput and can scale to many additional threads, further improving throughput
and overall performance.
Analysis of the CS summary timing logs from these tests shows a significantly higher
average number of SQL statements and total SQL execution time for Content Server
10.0 than CS 9.7.1. The table below shows average execution time, average SQL
execution time, and number of SQL statements for 9.7.1 and 10.0.
Table 2: CWS Functional Test - Execution Times Create Document
CS 9.7.1

% Difference

Avg Exec Time (sec)

0.210

0.257

-0.047

Avg SQL Exec Time

0.032

0.043

-0.011

129

216

87

Avg SQL Count

2.

Content Server
10.0

Ingestion Load Tests

Additional tests were executed ingesting batches of 100,000 documents into CS, with
different configurations of CS threads, and with soapUI (client) threads set with at
least twice as many threads in order to saturate the server and identify the peak
achievable throughput. CS 9.7.1 tests were done with 5 and then 8 CS threads
(limitations in CS 9.7.1 mean it is not beneficial to run with more than 8 threads),
Content Server 10.0 can scale to use more threads, so tests were done with 5, 8, 16,
24, 32, 40, and 48 threads.
The chart below compares throughput in CS 9.7.1 and 10.0; it illustrates that even at
the level of 8 CS threads, throughput in Content Server 10.0 is considerably higher
(20.8 docs/second compared to 15.88), and as the number of CS threads is
increased in Content Server 10.0, its throughput improves to be more than triple that
of CS 9.7.1.
With CS running with 8 threads, comparing the average
createDocument time from the CS summary timing logs shows 9.7.1 average of
470ms, and Content Server 10.0 23.6 percent faster at 359ms.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

Document Title

Figure 1: CWS Load Test Ingestion - Throughput Content Server 10.0 vs. 9.7.1

CPU usage by the LLServer process is somewhat higher for Content Server 10.0
during the above test with 8 CS threads; this reflects the fact that Content Server
10.0 is processing more documents per second and so better utilizing the available
processing power. It also remains close to 10 percent for an extended period after the
test, which is a result of the new distributed agent processing facet data. The chart
below shows a comparison of CS 9.7.1 and 10.0 CPU usage.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

Document Title

Figure 2: CWS Load Test Ingestion - Content Server Host CPU Usage

Memory usage by LLServer starts at a much higher level, and shows a much larger
increase during the Content Server 10.0 test, increasing by about 400MB, compared
to the minimal 30MB increase during the CS 9.7.1 ingestion. Note that about 1.5
hours after the test completed, Content Server 10.0 memory use decreased by about
100MB. The larger initial memory footprint is partly due to the Content Server 10.0
process being 64-bit (whereas 9.7.1 is 32-bit), and is also due to additional per
thread caching in Content Server 10.0. The chart below compares LLServer private
bytes memory usage for CS 9.7.1 and 10.0 during the 8 thread ingestion load test.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

Document Title

Figure 3: CWS Load Test Ingestion Content Server Memory Usage

SQL Server CPU usage is somewhat higher during the Content Server 10.0 test,
averaging around 18 percent, but with spikes as high as 50 percent. During the CS
971 test, SQL Server CPU usage was more consistently around 10-18 percent with
no spikes higher than 50 percent. Again, this reflects the fact that Content Server
10.0 is processing more documents per second, and that the additional features in
Content Server 10.0 mean that for each document, more SQL statements are
executed.
During an ingestion batch with 8 CS threads, CPU usage on the 10.0 Admin Server is
about twice as high as on the 9.7.1 Admin server. The chart below compares Admin
server CPU usage between the two CS versions. CPU usage continues on the Admin
server well after the ingestion has completed due to indexing of the ingested
documents; as mentioned earlier, further analysis of indexing will be provided in a
future assessment.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

10

Document Title

Figure 4: CWS Load Test Ingestion - Admin Server Host CPU Usage

Memory usage is also higher on the Content Server 10.0 Admin server, with only
about 1GB available memory at the start of the test and remaining at that level
through the test. For CS 9.7.1, the Admin server starts with about 4.1 GB available;
this drops to about 3.9 GB by the end of the test. This is largely due to Content
Server 10.0 processes being 64-bit, and partly to additional features in Content
Server 10.0.
Additional Content Server ingestion batches were executed with CS threads
increased to 16, 24, 32, and 40. CPU usage on the CS server increases with each
test until it is about 75 percent for the 40 thread test. The chart below compares CS
CPU usage during these tests.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

11

Document Title

Figure 5: CWS Load Test Ingestion - Content Server Host CPU Usage for Varying Thread Counts

As the CS thread count is increased, the increase in private bytes memory use by
LLServer also increases. For tests with more than 8 threads, there is also a large
jump in private bytes of 200 MB or more right at the start of the load test; the server is
configured for a minimum of 8 threads, so this jump is likely due to initialization of the
additional threads at the start of the test. As mentioned earlier, the increased memory
usage with additional CS threads is due to additional per thread caching in Content
Server 10.0.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

12

Document Title

Figure 6: CWS Load Test Content Server Memory Usage for Varying Number of Threads

3.

Retrieval Isolated Operation Tests

Retrieving documents through CWS is achieved using the DocumentManagement
getVersion and then getVersionContent call. The soapUI retrieval script loads
log files from the previous ingestion tests, and randomly selects ingested document
IDs to retrieve.
The data set includes English and multilingual Reuters files, and also includes 1
million documents loaded by the ingestion test. The Admin server had only search
engines running, and no index engines running.
Tests were first run using one client thread in the soapUI test client, running 500
iterations, to obtain average retrieval times without any load on the server. The table
below shows that in this no load scenario, Content Server 10.0 is 54 percent faster
than 9.7.1 for the getVersion call, and 20 percent faster for
getVersionContents.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

13

Document Title

Table 3: CWS Functional Test - Client-Side Timings - Retrieval

CS 9.7.1

4.

Content Server
10.0

% Difference

getVersion

116.88

53.7

54%

getVersionContents

162.36

128.79

20%

Combined

279.24

182.49

35%

Retrieval Load Tests

The data set includes English and multilingual Reuters files, and also includes 1
million documents loaded by the ingestion test.
Retrieving documents through CWS is achieved using the DocumentManagement
getVersion and then getVersionContent call. The soapUI retrieval script loads
log files from the previous ingestion tests, and randomly selects ingested document
ids to retrieve. Tests were executed for a duration of 15 minutes, with different
configurations of CS threads, and with soapUI (client) threads set with at least twice
as many threads in order to saturate the server and identify the peak achievable
throughput. CS 9.7.1 tests were done with 5 and then 8 threads (limitations in CS
9.7.1 mean it is not beneficial to run with more than 8 threads). Content Server 10.0
can scale to use more threads, so tests were done with 5, 8, 16, 24, 32, 40, and 48
threads.
The chart below compares throughput in CS 9.7.1 and 10.0. It illustrates that even at
the level of 5 CS threads, throughput in Content Server 10.0 is considerably higher
(40.8 docs/second compared to 22.61), and as the number of CS threads is
increased in Content Server 10.0, its throughput improves to be more than five times
that of CS 9.7.1.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

14

Document Title

Figure 7: CWS Load Test Retrieval Throughput Content Server 10.0 vs. 9.7.1

While Content Server 10.0 achieves much higher retrieval throughput than 9.7.1 with
the same number of threads, it does so with lower CPU usage. The chart below
shows a comparison of CS 9.7.1 and 10.0 CPU usage (overall CPU use on the
server which includes CS and CWS) for the test with 8 CS threads. During this 15
minute test, a total of 64,012 documents were retrieved from Content Server 10.0,
while 29,972 were retrieved from 9.7.1.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

15

Document Title

Figure 8: CWS Load Test Retrieval Content Server Host CPU Usage

For Content Server 10.0, as the number of CS threads are increased, overall CPU
usage increases until it levels off around 70 percent during a test with 40 CS threads.
The chart below illustrates overall CPU usage on the CS server for tests from 5 to 40
threads.

Figure 9: CWS Load Test Retrieval Content Server CPU Usage for Varying Number of Threads

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

16

Document Title

Memory usage by LLServer starts at a much higher level, and shows a larger
increase during the Content Server 10.0 test, increasing by about 40 MB, compared
to the minimal 4 MB increase during the CS 9.7.1 15 minute retrieval test. As
described in the ingestion section, the larger initial Content Server 10.0 memory
footprint is due to it being 64-bit, and to additional per thread caching in Content
Server 10.0. The chart below compares LLServer private bytes memory usage for CS
9.7.1 and 10.0 during the 8 thread ingestion load test.

Figure 10: CWS Load Test Retrieval Content Server Memory Usage

As expected, the increase in Content Server 10.0 private bytes is larger during tests
with more CS threads, where a larger number of documents were retrieved. Also,
similar to the ingestion tests, for tests with more than 8 CS threads (where min
threads was still set to 8), there is a large increase in LLServer private bytes right at
the start of the test that is likely due to initialization of the additional threads. The
table below illustrates the increase in private bytes for retrieval tests with different
numbers of CS threads, and also includes the number of items that were retrieved
during the test.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

17

Document Title

Table 4: CWS Load Test Retrieval Content Server Memory Usage

Private Bytes (MB)
CS Threads

Start

End

Change

# Retrieved

713

738

25

36783

785

825

40

64012

16

955

1031

76

112404

24

1121

1224

103

145063

32

1292

1414

122

166214

40

1455

1589

134

171699

During the retrieval tests, CPU usage on the SQL Server is minimal. The chart below
compares SQL Server CPU usage between Content Server 10.0 and 9.7.1 for the 8
thread retrieval test; both are typically below 5 percent. CPU usage during the 10.0
test is 1-2 percent higher than during the 9.7.1 test.

Figure 11: CWS Load Test Retrieval SQL Server Host CPU Usage

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

18

Document Title

Web Services Test Summary

The objectives of the Web Services Tests were to compare the throughput and
performance of ingesting and retrieving documents through Content Web Services
between Content Server 10.0 and Content Server 9.7.1. These tests were broken
down into Isolated Operation Tests, and Load Tests.
While ingestion of a single document through Content Web Services is slower in
Content Server 10.0, results of the Content Web Services load tests showed that
even at 8 threads, Content Server 10.0 achieves 30 percent higher ingestion
throughput than 9.7.1, and a peak throughput of over 53 documents created per
second at 40 threads. This is more than three times the peak ingestion throughput of
Content Server 9.7.1.
The improvements to Content Web Services retrieval performance in isolation are
even more pronounced under load. While at the same number of threads (5), Content
Server 10.0 exhibits almost double the retrieval throughput of Content Server 9.7.1,
the most significant difference occurs at peak throughput, where Content Server 10.0
achieves over four and a half times the capacity of Content Server 9.7.1 (over 180
docs/second in 10.0, compared to less than 40 docs/second in 9.7.1).

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

19

Document Title

Web User Interface Test Objectives

The objectives of these tests are to compare throughput and performance for
common user requests:
1. In isolation.
2. Under load.
Meeting these objectives requires two types of testing:
1. Isolated Operation performance tests to measure the performance of
requests in isolation.
2. Load tests to measure the performance of the system under load.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

20

Document Title

Web User Interface Isolated Operation Test

The purpose of this test is to isolate the performance of several common requests for
the new release of Content Server compared to the current release. These requests
include:
1.
2.
3.
4.
5.
6.

Browsing
Creating Objects
Fetching Documents
Personal Menu Items
Function Menus
Searching

The test is run by a proprietary test client that makes HTTP requests to the server.
The time recorded by the client is the time from when the request is submitted to the
web server, until the entire response has been received. Timing logs from both
Content Server and the test client are collected after the test. A typical sequence of
steps taken when processing a web request can include:
1. The browser client performs a DNS lookup of the server name and sends the
request to the web server.
2. The web server receives the request, starts the gateway (CGI/ISAPI/Servlet), and
the gateway forwards the request to Content Server.
3. Content Server executes the request, and returns the response to the gateway.
4. The gateway receives the response from the server, and forwards it to the
browser client.
5. The browser parses the response, executes JavaScript, and renders the final
page.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

21

Document Title

This sequence is depicted visually in Figure 12.

Figure 12: Content Server Request Life Time-Sequence Diagram

The Server Time (depicted in blue) is recorded as the Execute Time in summary
timing logs. The Client Time (from the start of the request until the end of the
response) is recorded in the logs of the test client. Non-Server Time (depicted in red)
is the Client Time minus the Server Time.

1.

Browse Requests
Accounting for roughly 60 percent of all requests issued to most Content Server
instances, browse is by far the most common activity. As browse is usually a part of
any task a user wishes to perform, poor browse performance makes many aspects of
TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

22

Document Title

the system appear slow. Conversely, if browse is fast, many tasks a user performs
seem faster.
Figure 13 shows the average Server Time, and Non-Server Time for browse requests
of various sizes on Content Server 9.7.1 and 10.0.

Figure 13: Execute & Response Times for Browse Requests

Figure 13 shows significant improvement to the time spent processing browse

requests inside the server at several container sizes, while the time spent processing
requests outside the server stays relatively constant. This results in an overall
improvement delivering the response to browse requests to the client.

2.

Create Requests
The Add: object page is where users fill out the name, description, and other
properties needed to create objects of a given type.
Figure 14 shows the average Server Time, and Non-Server Time for create requests
of various object types in Content Server 9.7.1 and 10.0.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

23

Document Title

Figure14: Execute & Response Times for Create Requests

These results show that Content Server 10.0 is able to deliver the Create pages for
objects significantly faster than Content Server 9.7.1. They also show that much of
this difference is due to a change in execute time within the server.

3.

Fetch Requests
Figure 15 shows the average Server Time, and Non-Server Time to fetch documents
of various sizes up to 1 GB on Content Server 10.0 and 9.7.1. Note the logarithmic
scale on both the Document Size and Time axis (a log-log scale).

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

24

Document Title

Figure 14: Execute & Response Times for Fetch Requests

These results show a constant base time to execute fetch requests, for documents
up to ~1 MB, and a linear increase in time fetching documents greater than ~1MB.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

25

Document Title

4.

Personal Menu Requests

Figure 16 shows the average Server Time, and Non-Server Time to execute various
requests on the Personal menu.

Figure 15: Execute & Response Times for Personal Menu Requests

These results show a significant reduction in the amount of time needed to execute
these personal menu requests within the server, and a slight increase in the amount
of time needed to process these requests in the CGI. The overall effect is a slight
decrease in the time needed to deliver responses to these requests to the client.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

26

Document Title

5.

Function Menu Requests

Figure 17 shows the average Server Time, and Non-Server Time to execute function
menu requests.

Figure 16: Execute & Response Times for Function Menu Requests

These results show a dramatic decrease (50ms vs. 325ms, or ~6.5x) in the time
needed to execute Function menu requests in Content Server 10.0, as compared to
Content Server 9.7.1. The overall effect is that Content Server 10.0 is able to deliver
responses to Function menu requests to the client significantly faster (150ms vs.
450ms, ~3x) than Content Server 9.7.1.

6.

Search Requests
There are a several parameters that may affect the performance of search requests.
One is the number of terms in the query, and another is the type of operator used.
For this test, we examined six sets of keywords (pulled at random from the keywords
associated with documents) presented in three different ways: a simple token query,
and right or left-truncation versions of the same queries.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

27

Document Title

Following is the list of terms queried:

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.

privata alternativ
*vata alternativ
privata alterna*
aktie andra kvartalet
*ie andra kvartalet
aktie andra kvarta*
fr arbetare inom privat sektor kade
*r arbetare inom privat sektor kade
fr arbetare inom privat sektor k*
att kronan kommer att
*tt kronan kommer att
att kronan kommer at*
vid kp frn
*id kp frn
vid kp fr*
skriver intentia
*iver intentia
skriver inten*

Queries 1, 4, 7, 10, 13, and 16 are the token queries; queries 2, 5, 8, 11, and 17 are
the same sets of tokens, with one of them left-truncated; queries 3, 6, 9, 12, 15, and
18 are the token queries with one of the tokens right-truncated.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

28

Document Title

Figure 18 shows the response times for the same token queries on 9.7.1 and 10.0.

Figure 17: Response Times for Search Queries (All words)

These results show that Content Server 10.0 is able to respond to simple tokenbased search requests significantly faster than 9.7.1, and that the response time
doesnt seem to be affected by the number of terms in the query, as it did in 9.7.1.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

29

Document Title

Figure 19 shows the response times for the same right-truncation queries on 9.7.1
and 10.0.

Figure 18: Response Times for Search Queries (Right Truncation)

These results show that Content Server 10.0 responds to right-truncation search
queries significantly faster than 9.7.1, though both seem to be affected by the number
of terms in the query.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

30

Document Title

Figure 20 shows the response times for the same left-truncation queries on 9.7.1 and
10.0.

Figure 19: Response Times for Search Queries (Left Truncation)

These results show that, for the most part, Content Server 10.0 performs as well as
or faster than 9.7.1, and that the difference increases as the number of terms
increases.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

31

Document Title

Summary
The Web User Interface Isolated Operation Tests tested the performance of several
user requests through the web interface. These tests were executed in isolation
(requests run one-at-a-time, with no other load on the server) in order to determine
the effect of changes to these functions on their performance, independent of any
other factors.
Overall, these tests showed a slight decrease in performance for download requests,
a slight increase in performance for browse and personal menu requests, and
significant improvements to performance for create, Functions menu, and search
requests.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

32

Document Title

Web User Interface Load Tests

The purpose of the load test is to observe how Content Server 10.0 behaves under
an extended period of load, as compared to Content Server 9.7.1. A number of
changes in Content Server 10.0 raise questions about how they will affect
performance.
Some of these changes include:

Migration to 64-bit address space

Significant reduction in contention within the Content Server engine
A new Distributed Agent infrastructure
Facetted browsing
Multilingual browsing
Use of standard log4cxx interface for logging

These changes present many questions related to performance:

How do the improvements and new features in Content Server 10.0 affect
performance?
How does Content Server 10.0 scale under load, compared to 9.7.1?
Each of these questions must be addressed separately.

Method
The software is loaded until it peaks at less than 50 percent of the available hardware
bandwidth (CPU, Disk, Network), for a period of three hours. This simulates an
extended typical peak hour of usage.
As with previous versions, the load test uses WebLoad 1 to simulate several
concurrent clients. However, the way WebLoad is used to generate requests has
changed from previous tests.
In previous tests, relatively few load generators would be used (three load generators
were used for the 9.7.1 test). Each load generator would run a random request, as a
random user, as quickly as possible back-to-back. Each load generation client would
select a mix of requests that is designed to simulate normal usage. However, this test
setup resulted in a maximum of three concurrent requests at any time, and unrealistic
user behavior.

2007 RadView Software Ltd. http://www.radview.com.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

33

Document Title

For the Content Server 10.0 tests, each load generator now behaves like a virtual
user. The load generator loads an agenda, which mimics the way a user might
perform a particular operation.
For example, when running the fetch agenda, the load generator selects a random
document and a random user, then logs on, browses the Enterprise Workspace,
browses the folder tree down to the document, clicks the Functions menu for the
document, fetches the document, and finally logs off.
Between each of these steps is a delay that represents the browser parsing and
rendering the response, and the user deciding what link to click next (the thinktime). After this set of steps, the load generator sleeps for a while (the pace-time),
simulating a user working with the document.
Overall, the result is that the tests more accurately simulate the way users would be
expected to use Content Server, so as to better simulate how Content Server can be
expected to perform under realistic load patterns.

Load Test Results

The purpose of this test is to answer the question: How does the version of the
Content Server affect the performance of end-user requests between Content Server
10.0 and 9.7.1?
To answer this question, we configure a front-end server with instances of Content
Server 9.7.1 and 10.0. Each instance is configured with eight worker threads (a
common value used by customers in Content Server 9.7.1), and all agent threads are
disabled. We also configure back-end instances for search on another machine. The
database for each instance is hosted on the same database server.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

34

Document Title

Figure 21 shows the logical configuration of machines used for this test.

Load Test Logical

Network Diagram

Load Monitor
& Controller

Database
Server

Content Server
Front-End

Search
Server

Figure 20: Load Test Logical Network Diagram

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

35

Document Title

Load Size Summary

Figure 22 shows the load profile used for each load test.

Figure 21: Load Profile, 9.7.1 & 10.0 8 Threads

The test starts by ramping up approximately 12 load generators every minute until
440 load generators have been started. Delaying the load generator start-ups helps
to de-synchronize the virtual users.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

36

Document Title

Throughput
One aspect of performance is throughput. Throughput is concerned with the total
volume of work; it is a measure of how much work can be done in a given amount of
time.
Figure 23: Client-Side Throughput, 9.7.1 & 10.0 - 8 Threads shows the throughput in
pages per second, as measured in the load generation software over the duration of
the test. A page is an explicit request to the Content Server gateway (livelink.exe)
by the load test script, and does not include references to the /support/ directory
mapping.

Figure 22: Client-Side Throughput, 9.7.1 & 10.0 - 8 Threads

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

37

Document Title

Table 5 represents a summary of the data as depicted in Figure 23 for the peak user
load (excluding the periods of ramp-up/ramp-down).
Table 5: Client-Side Throughput, 9.7.1 & 10.0 - 8 Threads
Name

Load Size 9.7.1

Load Size 10.0
Pages per second 9.7.1
Pages per second 10.0

Min

Max

Average

Stdev

4.165

440.00

387.89

113.23

5.414

440.00

388.11

112.66

13.40

21.9

17.713

1.578

11.95

19.1

14.998

1.253

Figure 23: Client-Side Throughput, 9.7.1 & 10.0 - 8 Threads shows that the two tests
(against 9.7.1 and 10.0) use the same configuration of load generators (up to 440
virtual users during the peak), but that there are fewer requests being delivered to the
10.0 instance than to the 9.7.1 instance.
In Content Server 9.7.1, and earlier, logon requests, and requests to browse a folder
with a customview were served by redirecting the user to the desired URL. This
redirect results in two requests being generated for a single user action. In Content
Server 10.0, this behavior has been changed so that the final link is presented to the
user, and the redirects on logon and customview browse are no longer required. The
result of these changes is a significant reduction on the number of requests made to
the server.
Figure 24: Server-Side Throughput, 9.7.1 & 10.0 - 8 Threads shows the throughput,
in transactions per second, as measured from Content Server summary timing logs.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

38

Document Title

Figure 23: Server-Side Throughput, 9.7.1 & 10.0 - 8 Threads

Figure 24: Server-Side Throughput, 9.7.1 & 10.0 - 8 Threads shows that there is little
difference in throughput during the duration of the tests for the 9.7.1 and 10.0
releases.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

39

Document Title

Execute and Response Times

Another important aspect of performance is the responsiveness of a system. Rather
than looking at the total amount of work that can be performed in a given time (as
with throughput), responsiveness is concerned with how quickly individual requests
can be served.
Figure 25: Client-Side Response Time, 9.7.1 & 10.0 - 8 Threads shows the average
response time for all requests, as measured in the load generation software, over the
duration of the tests. The response time includes the time needed to send the
request to the web server, execute the CGI, execute the request inside Content
Server, and return the response to the client; they do not include the time needed to
parse the DOM of the response, execute any client-side JavaScript, or render the
final page that gets displayed to the user (see Figure 13: Content Server Request
Life Time-Sequence Diagram for details).

Figure 24: Client-Side Response Time, 9.7.1 & 10.0 - 8 Threads

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

40

Document Title

Table 6 represents a summary of the data as depicted in Figure 20 for the peak user
load (excluding the periods of ramp-up/ramp-down).

Table 6: Client-Side Response Time, 9.7.1 & 10.0 - 8Threads

Name

Min

Max

4.165

440.00

387.89

113.23

5.414

440.00

388.11

112.66

Pages per second

9.7.1

0.264

0.885

0.375

0.059

Pages per second

10.0

0.304

0.795

0.429

0.073

Load Size 9.7.1

Load Size 10.0

Average

Stdev

These results show that Content Server 10.0 delivers pages to the end-user slightly
slower than 9.7.1 at this level of load, on average. However, it should be noted that
the difference in mean execute time between the two tests (54ms) is smaller than the
variation within the 9.7.1 test alone (where the standard deviation is 59ms). Both
versions still deliver the majority of requests in well under a second, and the stress
test in the next section, Figure 44, shows that Content Server 10.0 is able to maintain
consistently good response times to a much higher level of load than Content Server
9.7.1.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

41

Document Title

Figure 26 shows the server-side execute time of requests over time for both 9.7.1
and 10.0.

Figure 25: Server-Side Execute Time, 9.7.1 & 10.0 - 8 Threads

These results show a significant rise in execute time on 9.7.1 during the ramp-up
period, a steady-state with most requests taking 200 milliseconds to execute during
the body of the test, and a corresponding drop off in execute time at the end of the
test as load decreases.
The 10.0 results show a very slight increase in execute time as load is increased,
achieving steady-state at ~150ms per request throughout the duration of the test,
except for a couple of spikes nearing the mean execute time for 9.7.1 of 200
milliseconds. The steady-state execute time on Content Server 10.0 is approximately
one-third faster than 9.7.1, for the same volume of requests.
Another way of looking at execute times is to show what proportion of requests are
completed in less than a given time. Figure 27 presents the execute time data in this
manner.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

42

Document Title

Figure 26: Server-Side Execute Time CFD, 9.7.1 & 10.0u3 - 8 Threads

These results show that Content Server 10.0 is able to execute far more requests
faster than Content Server 9.7.1. Table 7 shows time under which the given
proportion of requests were executed in the 9.7.1 and 10.0 tests.
Table 7: Execute Time Percentiles - 9.7.1 & 10.0u3

Execute Time
(seconds)

Percentile

90%

95%

99%

9.7.1
10.0

0.6
0.3

0.85
0.45

1.15
1.05

From this table, we can see that Content Server 10.0 executes 95 percent of
requests almost twice as fast as 9.7.1, at the same level of load.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

43

Document Title

Resource Usage
Resource usage is another important aspect of the performance of a system. A
shortage of one or more resources can dramatically affect the performance of a
system. Comparing the resource usage of two versions of a system under load can
help anticipate changes to the resource usage behavior of their system after
upgrade. This can help administrators to plan for extra resources that may be
needed, and possibly reduce the use of resources that arent required.
A typical Content Server deployment makes use of a large variety of resources.
However, bottlenecks typically occur only in a handful of areas. The most common
resources to become bottlenecks on front-end servers are:
1.
2.
3.
4.

Content Server threads

CPU usage on the front-end
CPU usage on the database server
Memory usage on the front-end

Here we compare usage for each of these resources in the 9.7.1 and 10.0 load tests.
Thread Usage
One of the most common areas to become a bottleneck in a front-end deployment is
the number of worker threads. Unfortunately, determining the number of threads
needed for a given increase in load is not linear in Content Server 9.7.1. As more
requests come in, contention between threads within the engine increases, further
increasing the number of threads consumed. This effect can be seen clearly in a plot
of thread usage vs. load.

Figure 27: Thread usage vs. Load - 9.7.1

Figure 28 shows an example of this behavior on Content Server 9.7.1. The test
delivered a varying amount of load to the server, and observed how many threads
TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

44

Document Title

were consumed servicing the load. Doubling the load from ~2Tx/s to ~4Tx/s resulted
in a roughly linear increase in thread usage, from one to two threads. However,
increasing the load from ~4Tx/s to ~6Tx/s resulted in an increase in usage of 1.5
threads; and increasing the load from ~6Tx/s to ~8Tx/s resulted in more than
doubling the thread usage (from 3.5 to 8).
Figure 29 shows similar thread usage vs. throughput data for a late beta build of
Content Server 10.0.

Figure 28: Thread usage vs. Load - 10.0

These results show a much more linear increase in thread usage in Content Server
10.0. Given sufficient hardware to run the threads, this allows Content Server 10.0 to
scale to a much larger number of threads, without significant degradation in
performance. Coupled with a 64-bit address space, this will allow administrators to
run a much larger number of threads in a single instance, reducing the number of
instances needed to service a given level of load.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

45

Document Title

Figure 30 shows the thread usage over the duration of the load tests.

Figure 29: Server-Side Thread Usage, 9.7.1 & 10.0 - 8 Threads

This thread usage data shows that, for the same level and pattern of load, Content
Server 10.0 exhibits significantly lower thread usage than Content Server 9.7.1. With
the linear increase in thread usage vs. load presented earlier, this tells us that
Content Server 10.0 should able to handle 25 percent more load than 9.7.1 at 8
threads, and the load handling capacity of the server should continue to increase well
beyond 8 threads.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

46

Document Title

CPU Usage
Historically, Content Server is a CPU intensive application. So, another important
resource in Content Server deployments is CPU usage, both on the front-end
servers, and on the database server. Figure 31 shows the CPU usage over time for
both 9.7.1 and 10.0 from the eight thread test.

Figure 30: Front-End Server CPU Usage - 9.7.1 & 10.0 - 8 Threads

Table 8 represents a summary of the data as depicted in Figure 31 for the peak user
load (excluding the periods of ramp-up/ramp-down).
Table 8: Front-End Server CPU Usage - 9.7.1 & 10.0 - 8 Threads
Name

Min

Max

Average

Stdev

Load Size 9.7.1

4.165

440.00

387.89

113.23

Load Size 10.0

5.414

440.00

388.11

112.66

Processor:% Processor Time: _Total (9.7.1)

21.659

41.705

30.042

3.352

Processor:% Processor Time: _Total (10.0)

13.11

23.625

17.519

1.761

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

47

Document Title

This chart/table shows that Content Server 10.0 uses significantly less CPU on the
front-end server, while servicing the same volume of requests.
The average front-end CPU usage during these tests shows that Content Server 10.0
consumes roughly 40 percent less CPU time than 9.7.1 consumes, for the same level
of user load.
Another important aspect of resource usage in Content Server is the CPU usage on
the database server. With Content Server instances that service user requests, it is
relatively easy and inexpensive to add more front-end instances to handle additional
user load. However, scaling up a database server to handle the additional load due to
an increase in user requests, either by upgrading the hardware in an existing
instance, replacing it with a larger system, or scaling out by clustering the database
are all expensive propositions.
Figure 32 shows the CPU usage on the database server during the load tests.

Figure 31: DB Server CPU Usage - 9.7.1 & 10.0 - 8 Threads

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

48

Document Title

Table 9 represents a summary of the data as depicted in Figure 32 for the peak user
load (excluding the periods of ramp-up/ramp-down.
Table 9: DB Server CPU Usage - 9.7.1 & 10.0 - 8 Threads
Name

Min

Max

Average

Stdev

Load Size 9.7.1

4.165

440.00

387.89

113.23

Load Size 10.0

5.414

440.00

388.11

112.66

Processor:% Processor Time: _Total 9.7.1 (wlibm3bl04)

5.728

12.97

8.616

1.246

Processor:% Processor Time: _Total 10.0 (wlibm3bl04)

9.485

78.154

41.799

14.284

These results show that Content Server 10.0 requires roughly 40 percent more CPU
time on the database server, as compared to 9.7.1. However, much of this increase
in load on the database server is attributable to background tasks performed by the
distributed agent, rather than front-end load. As will be seen in Figure 42 of the
stress test, this usage does not increase linearly with the amount of load on the
database server.
Memory Usage
A third key aspect of performance is the memory usage of the process. Content
Server 9.7.1 only ships as a 32-bit process, limiting its address space to 2 GB or 4
GB. Memory usage was one of the major factors limiting the number of threads a
given process could run (that, along with a lack of increase in throughput beyond
roughly eight threads).
Content Server 10.0 is a native 64-bit process, so does not have the same address
space limit as 9.7.1 (16TB/18EB depending on the host operating system). Figure 33
shows the memory usage of the llserver process on the front-end server during these
tests.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

49

Document Title

Figure 32: Front-End Server memory Usage - 9.7.1 & 10.0 - 8 Threads

Table 10 summarizes the data as depicted in Figure 33 for the peak user load
(excluding the periods of ramp-up/ramp-down).
Table 10: Front-End Server memory Usage - 9.7.1 & 10.0 - 8 Threads
Name

Min

Max

Average

Stdev

Load Size 9.7.1

4.165

440.00

387.89

113.23

Load Size 10.0

5.414

440.00

388.11

112.66

Process:Private Bytes:llserver (9.7.1)

397,788,770

414,453,760

408,259,474

3,508,463

Process:Private Bytes:llserver (10.0)

752,543,740

793,937,920

777,932,181

11,073,164

These results show that Content Server 10.0 uses roughly twice as much memory as
Content Server 9.7.1, in the same configuration. While this may seem high, recall that
this configuration of Content Server 10.0 can handle significantly more load than
Content Server 9.7.1 in the same configuration (see Figure 30: Server-Side Thread
Usage, 9.7.1 & 10.0 - 8 Threads). Overall, the result is that Content Server 10.0 uses
roughly the same amount of memory needed to handle a given volume of load.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

50

Document Title

Load Test Summary

Results of the baseline tests of Content Server 9.7.1 and 10.0 in nearly identical
configurations show that Content Server 10.0 is able to deliver responses to a larger
volume of requests faster than Content Server 9.7.1. Coupled with the increased
scalability of threads, and address space available in a 64-bit process, a single
instance of Content Server 10.0 should be able to handle a significantly higher
volume of requests, and respond to them quicker, as compared to previous releases
of Content Server.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

51

Document Title

Stress Test Results

The purpose of this test is to determine the peak level of load, where the system is no
longer able to handle more load without response times starting to suffer, and to
isolate the reason for that limit. To accomplish this, the server is put under a gradually
increasing load, until performance starts to suffer.

Stress Test Content Server 9.7.1

Throughput & Response Time
One aspect of performance is throughput. Throughput is concerned with the total
volume of work; it is a measure of how much work can be done in a given amount of
time.
Figure 34 shows the throughput, in pages per second, and response time, in
seconds, as measured in the load generation software, over the duration of the test
for Content Server 9.7.1. A page is an explicit request to the Content Server
gateway (livelink.exe) by the load test script, and does not include references to the
/support/ directory mapping.

Figure 33: Throughput & Response Time 9.7.1

This result shows a fairly constant page time of less than one second for the first two
and a half hours of the test, then a steady climb in response time after that point.
Similarly, the amount of throughput the server can handle increases linearly up to
about 25 pages per second, at which point increasing load doesnt result in any gain
in throughput. So, the peak load capacity of Content Server 9.7.1, in this
configuration, with this mix of requests, is roughly 27 transactions per second.
The next step is to determine whether or not this limitation is caused by a resource
bottleneck on the hardware used to host the test, or by some limitation in the
software.
TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

52

Document Title

Resource Usage
Resource usage is another important aspect of the performance of a system. A
shortage of one or more resources can dramatically affect the performance of a
system.
For this test, we compare performance measurements taken at each of the servers
under test to thresholds set by the load generator software to detect common
resource shortage issues. The thresholds the software monitors are:

High CPU Activity (total CPU usage > 75 percent)

High Privileged Mode CPU Activity (privileged mode CPU usage > 30
percent and total CPU usage > 50 percent)
Low Level of Available Memory (available memory < 100MB)
High Level of Disk Activity (disk idle time <5 percent)

Any of these conditions will generate highlights on the corresponding chart, yellow for
just past the threshold, red for significantly past the threshold.
Figure 35 shows the front-end server resource usage for the Content Server 9.7.1
stress test.

Figure 34: Front End Resource Usage 9.7.1

These results show a peak CPU usage on the front-end server of ~55 percent, peak
privileged mode CPU usage of ~15 percent, minimum disk idle time of ~400 percent,
and minimum available memory of ~15GB. These levels did not trigger any threshold
violations, indicating that the throughput of this test was not constrained by hardware
resources on the front-end server.
Figure 36 shows the back-end server resource usage for the Content Server 9.7.1
stress test.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

53

Document Title

Figure 35: Admin Server Resource Usage 9.7.1

These results show peak CPU usage of ~30 percent, peak privileged mode CPU
usage of ~1 percent, minimum disk idle time of ~300 percent, and minimum available
memory of ~10.5GB. These levels did not trigger any threshold violations, indicating
that the throughput of this test was not constrained by hardware resources on the
back-end server.
Figure 37 shows the database server resource usage for the Content Server 9.7.1
stress test.

Figure 36: Database Server Resource Usage 9.7.1

These results show peak CPU usage of ~7 percent, peak privileged mode CPU
usage of ~2 percent, minimum disk idle time of 63 percent, and minimum available
memory of ~11.3GB. These levels did not trigger any threshold violations, indicating
that the throughput of this test was not constrained by hardware resources on the
database server.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

54

Document Title

Overall, the resource usage levels on all servers involved in the test were well within
the limits defined, indicating that the limitation on throughput was not caused by any
resource constraints in the environment, but most likely by some limitation of the
software.
Response Time Correlation
In order to isolate the most likely cause of the degradation in page time, we present
the page time (in black), the llserver virtual memory size (in dark green) with the top
five measurements that most closely correlate to the page time:
1.
2.
3.
4.
5.

Memory: % Committed Bytes in Use (front-end) light green

Memory: Available MBytes (front-end) red
SQL Server: Transactions, Longest Transaction Running Time yellow
SQL Server: Transactions: Free Space in tempdb orange
SQL Server: Memory Manager: SQL Cache Memory purple

Figure 37: Response Time Correlation 9.7.1

The results in Figure 38 show the top five measurements that most closely correlate
to the page time for the 9.7.1 stress test. Of these, the Percent Committed Bytes in
Use from the front-end server matches the page time almost exactly.
From about 30 minutes into the test, until two and a half hours into the test, the virtual
memory size and % Committed Bytes in Use stay relatively constant. At two and a
half hours into the test, the virtual memory space allocated to the server increases,
but the proportion of memory actually in use stays relatively constant.
By three hours into the test, both the proportion of memory in use, and the time to
deliver pages starts increasing. By four hours into the test, the virtual memory size is
just shy of 2 GB (the upper limit for a 32-bit process on Windows), so the server
cannot increase the amount of virtual memory (which would decrease the % of
Committed Bytes in Use).

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

55

Document Title

Stress Test Content Server 10.0

Throughput and Response Time
Figure 39 shows the throughput in pages per second, and response time in seconds,
as measured in the load generation software, over the duration of the test for Content
Server 10.0.

Figure 38: Throughput & Response Time 10.0u3

These results show that Content Server 10.0 delivers pages consistently in less than
one second for the first six hours of the test. At this point, response time starts to
become more inconsistent as load is increased.
The next step is to examine resource usage on the servers to see if there are any
resource limitations that may be contributing to this behavior.
Resource Usage
As with the 9.7.1 stress test, we compare performance measurements taken at each
of the servers under test to thresholds set by the load generator software to detect
common resource shortage issues. The thresholds the software monitors are:

High CPU Activity (total CPU usage > 75 percent)

High Privileged Mode CPU Activity (privileged mode CPU usage > 30
percent and total CPU usage > 50 percent)
Low Level of Available Memory (available memory < 100 MB)
High Level of Disk Activity (disk idle time <5 percent)

Any of these conditions will generate highlights on the corresponding chart, yellow for
just past the threshold, red for significantly past the threshold.
Figure 40 shows the resource usage on the front-end server during the stress test on
Content Server 10.0, update 3.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

56

Document Title

Figure 39: Front-end Server Resource Usage 10.0u3

These results start to show a warning around six and a half hours into the test, which
persists until the end of the test. The particular threshold that was triggered is the
High Privileged Mode CPU Activity (privileged mode CPU usage > 30 percent and
total CPU usage > 50 percent). Neither the amount of available memory nor the
proportion of time the disk is idle triggered warnings in this test.
This corresponds roughly to the period where page time starts to deteriorate,
however, the CPU usage increases very gradually during the test, while page time
fluctuates significantly during the test. This indicates that privileged mode CPU usage
on the front-end server may be contributing to increased page time, but that it is not
likely the primary cause of the fluctuating page times near the end of the test.
Figure 41 shows the resource usage on the back-end server during the stress test on
Content Server 10.0.

Figure 40: Admin Server Resource Usage 10.0u3

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

57

Document Title

These results show peak CPU usage of ~30 percent, peak privileged mode CPU
usage of ~5 percent, minimum disk idle time of ~350 percent, and minimum available
memory of ~8 GB. These levels did not trigger any threshold violations, indicating
that the throughput of this test was not constrained by hardware resources on the
back-end server.
Figure 42 shows the database server resource usage for the Content Server 10.0
stress test.

Figure 41: Database Server Resource Usage 10.0u3

The total CPU usage on the database server during the Content Server 10.0 stress
test is significantly higher than during the test on 9.7.1. Much of this increased CPU
usage is due to keeping facet counts up to date as content is added to the system.
While there are several CPU usage warnings (>80 percent) triggered throughout the
test, these do not correspond directly to periods of poor response times.
However, the periods where CPU usage exceeds 90 percent correspond very closely
to the spikes in page time observed in Figure 39.
Response Time Correlation Content Server 10.0
Figure 43 shows the page time (in black) and front-end llserver process virtual
memory usage (dark blue) with the top five measurements that correlate most closely
to the page time:
1.
2.
3.
4.
5.

Memory, % Committed Bytes in Use (front-end) green

Memory, % Committed Bytes in Use (sqlserver) red
Memory, Available MBytes (database server) yellow
Memory, Available MBytes (front-end) light blue
Process, % Processor Time (sqlserver) purple

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

58

Document Title

Figure 42: Response Time Correlation 10.0u3

These results show that the amount of available and committed memory on the frontend and database servers were most closely related to the increased response times
in this test, and CPU usage on the database server was also likely to have
contributed to this decrease in response time at the maximum load.

Stress Test Summary

Figure 44 shows the results of the two stress tests combined. Page times for both
tests are presented in green, and throughput is presented in red. Results from the
9.7.1 stress test are presented with solid lines; results from the 10.0 stress test are
presented with dashed lines.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

59

Document Title

Figure 43: Throughput & Response Time 10.0u3 vs. 9.7.1

The results of the 9.7.1 stress test showed subsecond response times until about two
and a half hours into the test. However, after this point, response time increases very
quickly as load is added to the system. The lack of any hardware bottlenecks
indicates that this is a limitation of the software. Many customers address this
bottleneck by installing multiple instance of Content Server 9.7.1 on a single server
(vertical scaling), but this significantly increases the effort needed to administer a
Content Server environment.
The results of the 10.0 stress test showed subsecond response times until roughly
six hours into the test, at two and a half times the user load of 9.7.1. While response
times do increase at times beyond this point, they do not show the same linear
degradation that was present in the 9.7.1 stress test, and there are clear indications
of hardware bottle-necks.
The ability to have a single instance of Content Server 10.0 make use of whatever
hardware resources are available (virtually eliminating the need for vertical scaling),
significantly reduces the amount of effort needed to administer the environment.
Having server resources clearly indicate when there is likely to be degradation in
response time makes it easier for administrators to identify and address potential
performance issues in a Content Server environment before they become an issue
for users.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

60

Document Title

Conclusion
The objectives of the Web Services Tests were to compare the throughput and
performance of ingesting and retrieving documents through Content Web Services
between Content Server 10.0 and Content Server 9.7.1. These tests were broken
down into Isolated Operation Tests, and Load Tests.
Results of the Web Services Isolated Operation Tests showed a significant decrease
in the performance of creating a document through Content Web Services in Content
Server 10.0 as compared to Content Server 9.7.1 (~-19 percent), and a significant
increase in performance when retrieving a document (~35 percent).
While ingestion of a single document through Content Web Services is slower in
Content Server 10.0, results of the Content Web Services load tests showed that
even at eight threads, Content Server 10.0 achieves 30 percent higher ingestion
throughput than 9.7.1, and a peak throughput of over 54 documents created per
second at 40 threads. This is more than three times the peak ingestion throughput of
Content Server 9.7.1.
The improvements to Content Web Services retrieval performance in isolation are
even more pronounced under load. While at the same number of threads (5), Content
Server 10.0 exhibits almost double the retrieval throughput of Content Server 9.7.1,
the most significant difference occurs at peak throughput, where Content Server 10.0
achieves over 4 and a half times the capacity of Content Server 9.7.1 (over 180
docs/second in 10.0, compared to less than 40 docs/second in 9.7.1).
Objectives of the Web U/I tests were to observe how Content Server 10.0 behaves
under an extended period of load, as compared to Content Server 9.7.1, and at what
level of load the two versions start to show a degradation in response time.
Results of the load test showed that Content Server 10.0 is able to deliver responses
to a larger volume of requests faster than Content Server 9.7.1. Results of the stress
test showed that the software bottlenecks that prevented Content Server 9.7.1 from
using available hardware effectively have been removed from Content Server 10.0,
and that a single instance of Content Server 10.0 is able to handle two and a half
times the load of an instance of Content Server 9.7.1. By virtually eliminating the
need to install multiple instances on a single server (vertical scaling), Content Server
10.0 significantly reduces the amount of effort needed to administer larger scale
deployments.
For the most part, results of these tests showed either no loss in performance, or
significant improvements to performance with Content Server 10.0. The most notable
areas of improvement in Content Server 10.0 include:
5.
6.
7.
8.

Improved thread scaling

Improved and more stable response times under load
Significantly lower CPU usage on front-end servers
Dramatically higher load carrying capacity of a single instance

Overall, these improvements present compelling reasons for customers to upgrade to

Content Server 10.0.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

61

Document Title

Appendix A: Test Environment Architecture

Figure 50 shows the architecture used for the Content Web Services tests. The
architecture used for the load tests is shown in Figure 21.
Figure 44: Content Web Services Test Architecture

SQL 2008

Sun 7410 Storage

Hitachi SAN

External File Store

Index Partitions

Oracle 10

WLIBM3BL02

OTSM18-08

Content Server and

Content Web Services

Content Server Admin

(Search)

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

62

Document Title

Appendix B: Test Environment Hardware

The test environment consists of a Content server with EFS stored on a Sun 7410
storage device accessed through the network, a SQL Server 2008 database server
with data stored on SAN attached Hitachi AMS 200 SAN, Admin server with index on
SAN attached Hitachi AMS 200 storage device, and a load generation client.
Note that for the Web UI load tests, the EFS was moved to an AMS 2100 SAN
device, and the SQL Server database files were also moved to an AMS 2100 SAN
device. Specifications for the hardware are listed in the table below:
Table 11: Test Environment Hardware
Content Server

EFS Storage

Admin Server

Index
Storage

SQL Server

Load Client

Make

IBM

Sun

IBM

Hitachi

IBM

IBM

Model

System
7870AC1

7410
Storage

eServer
BladeCentre
HS21
8853C2U

AMS 200
SAN

System X
7870AC1

System X
7870AC1

Processor
s

2 x E5530 Quad
Core 2.4Ghz

2 x Quad
Core AMD
Opteron
2.3GHz

2 x Xeon
E5345 Quad
Core
2.33GHz

2 x E5530
Quad Core
2.4Ghz

2 x E5530
Quad Core
2.4Ghz

Memory

20GB

16GB

10GB

1GB
Cache

20GB

20GB

Network

Broadcom
BCM5709S
NetXtreme
GigE

1Gbps

Broadcom
BCM5708S
NetXtreme II
GigE

1Gbps
SAN

Broadcom
BCM5709S
NetXtreme
II GigE

Broadcom
BCM5709S
NetXtreme II
GigE

5
x
132GB
(RAID 0)

2 x 136 GB
SCSI 4 x
146GB FC
RAID 0

2 x 136 GB
SCSI 2

Disk

2 x 136
SCSI

II
GB

11 Seagate
ST31000
NSSUN1.O
T
932GB
(750GB
assigned)

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

63

Document Title

Appendix C: Non-Default Configuration

Non-Default settings for the CWS Ingestion and Retrieval tests, on CS 9.7.1 and
10.0, are described below.
Table 12: Non-Default Test Settings, all tests
Non-default Settings, all tests
Database

SQL Server database configured with snapshot isolation and read

committed snapshot enabled.

Sessions

CS cookie expiry disabled (Server Configuration > Configure

Security Parameters > in Cookie Authentication Information set to
Never expire).

Table 13: CS10.0 update 3 Web User Interface and Stress Tests
Content Server 10.0 update 3 Web User Interface and Stress Tests

sockserv;javaserver, running on front-end server, all other agents running on back-end server
Index processes were stopped during these tests.
search partition max content disk size changed from default 5 GB to 50 GB (LPAD-23715).

Table 14: CWS Ingestion and Retrieval tests

Content Server 10.0 CWS Tests
Agents

Agents and daagent enabled only on Admin server.

Distributedagent enabled only on front-end server.

Patches

Search/
Indexing

Sept 2010 monthly patch

Patch to resolve slow performance for CWS document upload (LPAD-23528 )

Patch for LPAD-23689

Patch for DA infinite loop issue (LPAD-23386)

After ingestion batch 2, patch to fix long recommender agent query with many new items

After ingestion batch 3, search partition max content disk size changed from default 5 GB to 50
GB (LPAD-23715).
Ingestion batches 1-5 run with 4 index partitions, 6-10 run with 6 partitions.
Ingestion tests run with only index processes running, search processes stopped.
Retrieval tests run with only search processes running, index processes stopped.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

64

Document Title

Table 15: CS 9.7.1 Tests

CS 9.7.1
Agents

Agents enabled only on Admin server.

Patches

June 2010 monthly patch

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

65

Document Title

Appendix D: Data-Sets
This section describes the data sets used during testing.

Document Sizes Folder:

A folder containing various sized files outlined below:
File Name

Actual File Size

1k.txt

1 kb

10k.txt

10 kb

100k.txt

100 kb

1m.txt

1000 kb

10m.txt

10 mb

100m.txt

98 mb

1g.txt

977 mb

Folder Sizes Folders:

Several Folders, containing different number of files, reflected by the folder name and
all sized < 100 KB.
10 Document Folder
100 Document Folder
250 Document Folder
1000 Document Folder
10000 Document Folder
10000 Document Folder Split [Containing 10 folders * 1000 Doc]
100000 Document Folder Split [Containing 9 folders * 11111 Doc]

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

66

Document Title

News Article Data-Set

The News Article data set consists of ~1.28 million news articles, about 800,000 in
English, and about another 480,000 in thirteen other languages. Each article has 1214 attributes applied, so they have a significant amount of meta-data.
Table 16 shows the category configuration, and count of attribute values.
Table 16: News Article Category Definition
Attribute

Data Type

Length

Max. Rows

Count

Category Name 2

1,277,951

Title

Text: Field

254

1,277,951

Headline

Text: Field

254

1,277,951

Byline

Text: Field

254

1,277,951

Dateline

Text: Field

254

1,277,951

Copyright

Text: Field

64

1,277,951

Date Created

Date: Field

N/A

1,277,951

Publisher

Text: Field

64

1,277,951

Date Published

Date: Field

N/A

1,277,951

Source

Text: Field

64

1,277,951

Creator Location

Text: Field

254

798,313

Creator Country

Text: Field

254

798,313

Topic

Text: Field

128

25

4,713,807

Industry

Text: Field

128

50

1,485,225

Region

Text: Field

128

25

3,192,384

Total

23,767,552

The articles are organized by author, with one folder per author. Each author has full
permissions to their folder, while other authors only have read permissions to their
folders. Articles with no author attributed in the source data set are grouped into an
Anonymous folder. There are 798,313 English articles, grouped into 2393 folders,
and 479,638 multilingual articles, grouped into 3 folders.

Each item that has a category assigned stores a copy of the category name as its
first attribute. This attribute is hidden in the interface, but takes up storage in the
database.
TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

67

Document Title

Table 17 shows the distribution of article content sizes.

Table 17: New Article Size Parameters
Parameter

Size (Bytes)

Average

1,456

Std.Dev.

1,351

Min

Max

61,301

Ingested Document Data Set

The documents created during the Web Services ingestion test, and used during the
Web Services retrieval test, and general load tests consist of about one million Word
documents, with randomly generated text. Each document has a category with 14
attributes applied.
Table 18 shows the category configuration, and count of attribute values for the
ingested item category.
Table 18: Ingested Document Category Definition
Attribute

Data Type

Length

Max. Rows

Quantity

Category Name

1,010,189

DocRef

Text:Field

254

1,010,189

Email

Text:Field

254

1,010,189

Approver

Text:Field

254

1,010,189

RequestedBy

Text:Field

254

1,010,189

Spec

Text:Field

50

1,010,189

Code

Text:Field

50

1,010,189

System

Text:Field

100

1,010,189

DocDesc

Text:Field

254

1,010,189

Integer1

Integer: Field

N/A

1,010,189

Integer2

Integer: Field

N/A

1,010,189

ReleaseDate

Date: Field

N/A

1,010,189

CompleteDate

Date: Field

N/A

1,010,189

UniqueAttr

Text:Field

200

1,010,189

FullPath

Text:Field

200

1,010,189

Total

15,152,835
TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

68

Document Title

The documents are organized with 1000 per folder. Table 19 describes the size
distribution for the ingested documents.
Table 19: Ingested Document Size Parameters
Parameter

Size (Bytes)

Average

208,011

Std.Dev.

50,419

Min

6,144

Max

450,560

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

69

Document Title

About OpenText
OpenText is the worlds largest independent provider of Enterprise Content
Management (ECM) software. The Company's solutions manage information for all
types of business, compliance and industry requirements in the world's largest
companies, government agencies and professional service firms. OpenText supports
approximately 46,000 customers and millions of users in 114 countries and 12
languages. For more information about OpenText, visit www.opentext.com.

TOGETHER, WE ARE THE CONTENT EXPERTS

CHAMPION TOOLKIT

70

Visit online.opentext.com for more information about OpenText solutions. OpenText is a publicly traded company on both NASDAQ (OTEX) and the TSX (OTC) Copyright 2011 by OpenText Corporation. Trademarks or registered
trademarks of OpenText Corporation. This list is not exhaustive. All other trademarks or registered trademarks are the property of their respective owners. All rights reserved.