Performance White Paper

OpenText™ Brava!™ Enterprise

Champions Guide for Tuning
Performance
Szymon Reichmann

Product: OpenText™ Brava!™ Enterprise

Version: 16.6.6
Task/Topic Performance
Audience: Administrators, Decision Makers
Platform: All
Document ID: 200006
Updated: May 3, 2022
 Brava! Enterprise Champions Guide for Tuning Performance

Contents
Brava! Enterprise Architecture .................................................................................. 3
Brava! Server (BESS)............................................................................................. 3
Display List Cache (DLC) ....................................................................................... 4
Persisted and temporary entries in Display List Cache ................................... 4
Display List Cache Size ................................................................................... 5
Display List Cache eviction (truncation) strategy. ............................................ 6
Threads .................................................................................................................. 6
Synchronous vs Asynchronous request processing ........................................ 7
Blocking I/O vs asynchronous I/O .................................................................... 7
Blocking HTTP request vs server-side processing .......................................... 7
File monitoring, single page publishing and publishing evasion ............................ 8
Single page publishing ..................................................................................... 8
File monitoring ................................................................................................. 9
Publishing evasion ........................................................................................... 9
Brava! License ...................................................................................................... 10
Job Processor/Publishing Agent (JP/PA) ............................................................. 10
JP threads vs processes ................................................................................ 11
Communication between JP/PA and Brava! Server....................................... 11
CDL characteristics ........................................................................................ 12
XDL format............................................................................................... 12
Print Publishing (Bi2dl)................................................................................... 13
Alternatives for Office documents ........................................................... 14
Search artifacts .............................................................................................. 14
Publishing noise ............................................................................................. 14
Monitoring Tool ......................................................................................................... 16
Live JP/PA jobs tracking ....................................................................................... 16
Queue statistics .................................................................................................... 17
Display List Cache ................................................................................................ 17
Recommendations .................................................................................................... 18
Adjust the default configuration ............................................................................ 18
Monitor and adjust ................................................................................................ 18
Display List Cache size ........................................................................................ 18
Configure your servlet container for best performance ........................................ 19
Keep system I/O clean ......................................................................................... 19
Keep Brava! Server and JP/PA close to each other: ............................................ 20
Use sharding and scale out .................................................................................. 20
Routing Service .............................................................................................. 20
Avoid single page publishing ................................................................................ 21
HTML viewer page precaching ...................................................................... 21
Disable generating heartbeat files ........................................................................ 22

The Information Company™ 2

 Brava! Enterprise Champions Guide for Tuning Performance

Precaching ............................................................................................................ 22
Document upload vs HTML viewer composition ............................................ 22

The Information Company™ 3

 Brava! Enterprise Champions Guide for Tuning Performance

Brava! Enterprise Architecture

Brava! Enterprise consists of three main elements:
• Brava! Server
• Job Processor / Publishing Agent
• Brava! License
Beginning with version 16.6.1, a Routing Service is provided which acts as a load
balancer/session persistence layer for requests coming to multiple Brava! Servers.

This guide and the architecture information uses Brava!

Enterprise version 16.6.6 as a reference point.

Brava! Server (BESS)

Brava! Server is our file server and task queue. From a performance perspective,
Brava! Service does the following:
• Keeps in memory a map of all the jobs created as Display List Cache. It also
manages a file system backup copy of the map. This task is fast, localized, does
not require a lot of CPU power, and memory requirements are relatively low. It
scales relatively well with the number of entries handled by a single Brava!
Server. Each in-memory entry has a filesystem-based folder that holds metadata
(data.txt) and artifacts (published files). Published files (artifacts) are only
accessed when requested by viewers while metadata can be read by various
process. Brava! Server caches entries stored in the metadata files making
access to them relatively fast. The cache for metadata is rather short-lived
(usually measured in seconds).
• Keeps in memory a composition map. A composition is a combination of the
source document, redaction script(s), and burn-in markup(s), and defines what
users can see as a document in the Brava! viewers. This task is fast, localized,
does not require CPU usage, and memory requirements are relatively low. It
scales relatively well with the number of compositions handled by a Brava!
Server. Usually, a composition is held for 8 hours. After that duration,
compositions are removed. The composition map is never persisted on the disk.
• Maintains the ActiveX viewer session manager, which is a map of sessions, in-
memory only and not persisted. Architecturally, it is very similar to the
compositions map. The main difference is that compositions are the same for the
same content definition (the same document will get the same composition id).
Composition id is basically a hash of all the content metadata (viewed document
ids, redaction scripts, burned-in markups).
• Keeps in-memory, non-persisted job queues for Job Processor / Publishing
Agent. There are 5 separate queues, all working as “first in first out” (FIFO)
priority:
o drw - image transformation (thumbnails among others)
o pdf - processing PDF documents to XDL/SVGs

The Information Company™ 3

 Brava! Enterprise Champions Guide for Tuning Performance

o doc - processing doc, generally office-based formats

o single - responsible for publishing on demand single page jobs (XDLs, jpgs,
and SVGs)
o 3D - 3D documents meant for HTML 3D viewer
• Provides artifacts to viewers (file server role). Most of the requests are processed
asynchronously:
o HTML viewers - all endpoints that provide content to HTML viewer are using
asynchronous processing
o AXV - endpoints are not asynchronous, but AXV (ActiveX viewer) in normal
circumstances should not work in a blocking manner
From a performance perspective, Brava! Server requires fast I/O (both network and
filesystem) with strong emphasis on latency and throughput to the configured Display
List Cache location.
The CPU is used mostly to process HTTP requests and cryptographically signing
viewer configuration. Additionally, some CPU processing power is required to handle
JWT (JSON Web Token) and their decryption.

Display List Cache (DLC)

Display List Cache is the part of Brava! Server which is responsible for keeping artifacts
within a server instance.
Display List Cache is a cache, and its persistence is only temporary. Any
document/data stored in it is subject to unannounced deletion. All the file artifacts
stored by the cache must be stored in a directory structure on the disk. This directory
structure is shared between Brava! Server and JP/PA instances. There is a 1:1
relationship between the directory structure and a Brava! Server instance, meaning
that each Brava! Server operates on its own cache independently. The information
pertaining to what is stored in a Brava! Server instance's cache is not shared in any
way among other Brava! Server instances. There are two mechanisms that can bypass
that limitation:
• integration-based sharding - the viewing load is spread by the integration to
multiple Brava! Servers. In that case, the integration must select which Brava!
Server to use to perform a viewing operation, send the viewed document source
there, and configure the viewer to use that Brava! Server instance. In that case,
each Brava! Server instance has a separate address and, from the browser’s
perspective, is completely independent from other instances.
• session persistence - an element in the infrastructure that tracks viewing
sessions, spreads the load among tracked Brava! Server instances, and then
tries to use the same instance whenever the same document is viewed again to
prevent republishing on a different server. The Routing Service acts as such an
element. It tracks uploaded documents, created compositions, and keeps records
of where they were published.

Persisted and temporary entries in Display List Cache

After a new element in the cache is created, regardless of its type, a timeout is created
after which the entry is removed from the system. That timeout can be cancelled and,

The Information Company™ 4

 Brava! Enterprise Champions Guide for Tuning Performance

by doing that, the entry will be persisted. It will not be removed until it is either removed
through an admin action, invalidated through manual folder removal from DLC (not a
supported operation), or deleted through cache truncation. Document entries are
persisted immediately after creation while document sources are never persisted.
Sources will be removed eventually (with one caveat which we will discuss later), and
renditions/jobs after finishing successfully are persisted and will not be automatically
deleted.
The timeout is set to be 3 times the publishing timeout configured in server.properties:
publish.request.timeout=7
Typically, an entry is created when a job/rendition is created, and the timeout is
measured from the creation time. Creating a job does not mean that the task will be
assigned to a JP/PA immediately. It first goes to one of the queues and is then pulled
by a JP/PA instance, and results are published. The timeout spans the whole operation,
which means that it is possible that the timeout’s expiration can be triggered when the
job is still in the queue or when it has already been sent to a JP/PA but has not yet
finished.
Entry types stored in DLC:
• document sources - a temporary entry that holds 1 or more (xrefs or external
references) source documents that usually are uploaded to the system by the
integration.
• documents - metadata entry that keeps the status of the document (if it has
been processed to XDL form already or not) and holds data such as: document
source, XDL representation, and all the related renditions based on this source.
• renditions (publishing jobs) - this entry holds artifacts published by Job
Processor.

Display List Cache Size

Each entry in the Display List Cache (DLC) has a size, which is not necessarily
associated with the total amount of data stored on the filesystem. Depending on the
type of the entry, the size is counted differently:
• documents always have size 0.
• document sources always have size 0.
• rendition/job:
o after an entry is created, and while artifacts are being produced by the JP/PA,
always have size 0.
o successfully finished - after the job is finished, it is marked as persisted and all
the artifacts in the folder, excluding heartbeat file and data.txt, are summed up
and stored as the total size of the entry.
Display List Cache size is a sum of all the entry sizes, meaning it represents only the
size of all the finished/persisted artifacts stored in the finished jobs. Additionally, if an
entry is evicted from the cache and moved to the deletion queue, the size of the entry
is immediately subtracted from the total size of the Display List Cache, even though
the files have not yet been removed from the file system.

The Information Company™ 5

 Brava! Enterprise Champions Guide for Tuning Performance

Because of these specifics, it is important to understand that usually Display List Cache
is taking more space on the disk than the reported total size seen in the Monitoring
Tool.
The maximum Display List Cache that can be set up in the application settings is
enforced based on the total size of the Display List Cache, measured as explained
above.
Once the size of the cache exceeds a certain threshold set up in the configuration,
truncation mechanisms are executed.
The total size of Display List Cache can be monitored using the Monitoring Tool.

Display List Cache eviction (truncation) strategy.

The truncation mechanism works as follows:
Every so often, the deletion thread checks to see that the total size of the DLC does
not exceed a certain threshold. If it does, a portion of entries is removed from the
system. There is a maximum number of entries that can be removed this way in one
sweep. The order of items that are removed is a simple FIFO (first-in first-out) queue.
The first item added to cache is the first that will be removed. The queue is not being
reorganized based on entry usage, meaning that even if the oldest entry is the one that
is the most used, that fact will not prevent the system from deleting it in the first order.

displaylist.cache.maximum.size=10000
displaylist.cache.size.reduction.percent=90
displaylist.cache.max.files.to.delete=20
When an entry is deleted from the in-memory map of entries, it is removed from the
map immediately and moved to a queue of entries that should be cleaned up. The
queue becomes the only element in the system that still knows where the entry has its
files system data. The queue is not persisted and will not survive a restart. The removal
from the map means that the entry is no longer accessible, even if it is still in the
deletion queue.
There is only one thread responsible for cleaning up entries from the deletion queue.
The thread runs in burst mode, deletes some amount of entries from the disk
(displaylist.cache.max.files.to.delete), then goes idle. That ensures that even if the
whole cache is invalidated or removed, a delete operation will not take over the whole
I/O bandwidth of the system. Because of that fact, folder deletion through truncation
should not have a dramatic impact on I/O performance of the system. It is important to
understand that when the system hits the truncation threshold, additional I/O
operations will be happening in the background that will compete with the main
upload/publishing/reading role of Brava! Server and JP/PA in tandem.

Threads
Predominantly, Brava! Server works as event-based system with most endpoints being
processed asynchronously. We recommend at least 200 threads to be assigned to the
Connector that is handling content distribution to the viewers.

The Information Company™ 6

 Brava! Enterprise Champions Guide for Tuning Performance

Additionally, we recommend having a separate Connector/thread pool assigned to

/Pop and /done endpoints that are used by Job Processor. This way, the thread pool is
not shared, and end-users will not be able to (accidentally or not) bring down the
service by requesting too many artifacts at the same time.

Synchronous vs Asynchronous request processing

Brava! Server is a servlet application hosted on a servlet container such as Tomcat
Server. When synchronous processing of HTTP request is in use, a single thread is
assigned to a single request. If you have 200 threads available in a connector thread
pool, you can process only 200 requests at the same time.

Blocking I/O vs asynchronous I/O

Brava! Server is currently not using asynchronous I/O. Even though waiting for the
publication of an artifact can be asynchronous, the actual copying of the file content
from the filesystem to the response happens as a blocking I/O operation. This means
that there should be enough threads assigned for the connector thread pool to be able
to handle the I/O concurrently. Then again, disk I/O is basically synchronous and
blocking anyway. Only one file can be accessed at a time, even if that operation is very
fast. Currently there is no need to keep high amounts of threads in the thread pool.
The default 200 should be enough, but I/O monitoring is still advised.
Additionally, the content is currently sent to clients through user-space memory. This
means that each file must be read by the kernel, made available to our application
running in JVM (user-space), and then copied back to http response. That copying
process has some impact on OS and CPU. It is possible that in the future Brava! Server
will switch to zero-copy file serving when files from the filesystem are copied to http
responses/sockets without copying them to user space.

<Connector
...
maxThreads="200"
...
/>

Blocking HTTP request vs server-side processing

In Brava! Server architecture, a client can request an artifact that is not quite ready.
Brava! Server either detects that and a new job is created, or simply knows that the job
is either currently being published or waiting in the queue to be pulled by JP. The client
does not know the actual state of the server; from its perspective, it is just an HTTP
call. From the client perspective, that operation is a blocking HTTP request. If the
request is made synchronously on the client, the thread/process that calls it will be
blocked. There is a slight difference between the ActiveX viewer and the HTML viewer
in this regard. In general, the AXV tries to make sure that content is already available
by making isready calls and based on the return, requests either single page publish
XDL vs resources coming from the main XDL job. To some extent, AXV decides
whether a single page publish job is created on the server. Because of this, AXV's
requests are usually shorter than HTML viewers. HTML viewer, by contrast, makes

The Information Company™ 7

 Brava! Enterprise Champions Guide for Tuning Performance

identical requests, regardless of the job status on the server side. Brava! Server
decides if the request must wait for some jobs to be finished, some artifacts to be
present on the filesystem, or if a single page publish job should be created.
When asynchronous request processing was not implemented, it meant that such a
request was blocking a single thread on the servlet container (or connector's
threadpool) and idly waiting for publishing to finish. Currently, when a job is not finished,
the threads are not blocked. Instead, the Brava! Server application wait for the proper
event to happen and then the Brava! Server application we inform the servlet container
that the response is ready. Servlet 3.0 API (and its proper implementation by the servlet
container along with spring framework) is responsible for orchestrating different
threads to provide the content in a less blocking way. This allows the Brava! Server to
support more concurrent blocking HTTP requests to the server than threads assigned
to process. Because of this, the asynchronous timeout on the servlet container must
be increased. By default, in Tomcat, that setting is set to just 30 seconds, and that is
not enough time even for the default JP/PA processing timeouts (7 minutes by default).
We recommend increasing the asynchronous timeout to be at least equal to the JP/PA
processing timeout.

<Connector
...
asyncTimeout="420000"
...
/>
The increase of the async timeout can, in theory, have negative effects on the servlet
container because the server must track more request at the same time and is not
capable of timing them out earlier, although the number of threads needed should not
be affected by this setting. Therefore, the increased timeout could potentially increase
the risk of DoS (Denial of Service) but should not have a significant impact on the
overall maximum throughput or general performance of the system for expected load.

File monitoring, single page publishing and publishing

evasion

Single page publishing

Originally, artifacts stored in Display List Cache could not be retrieved from a job that
has not been marked as finished yet. In that situation, a job that could take excessive
time to process (for example, containing hundreds of pages) could not provide results
before all the resulting files were available. CDL (equivalent to a kernel during a
conversion task) produces artifacts page after page meaning that the first page is ready
much faster than the last one.
Therefore, the server had to wait for the whole document to be published before it
could provide the first artifact representing the first page of the document.
To mitigate that issue, Brava! Server provided a concept of single page publishing.
Single page publish is a redundant job that produces an artifact just for a single page.
Single page publishes can help improve latency for artifact accessibility but can have
a negative impact on the overall maximum throughput of the system.

The Information Company™ 8

 Brava! Enterprise Champions Guide for Tuning Performance

From throughput perspective, single page publish jobs should be avoided. Their results
are redundant and not persisted (eventually they will be removed from the Display List
Cache).

File monitoring
Beginning with Brava! Enterprise v.16.5, the server began actively tracking artifacts
created by PA/JP. This allows the server to generate events that a particular artifact is
ready. Additionally, it started checking to see if the requested artifact is already
available - even for jobs that have started but have not finished yet. This way, single
page publishes are avoided for any artifacts that have already been processed.
While this approach was originally optional, it is now the only way to monitor and serve
artifacts to Brava! HTML Viewer.
This feature automatically, without making any changes to Brava! Server configuration
or external API, saves a lot of single page publishes from being created.
This process works based on JVM provided folder monitoring of rendition folders.
JP/PA is not responsible for that monitoring, and the whole process is conducted only
by the server.
With this method, the following new information became available to the system:
• That a job has already started (previously, Brava! Server only knew if it has
been queued, but not if JP has already pulled the job).
• Which artifact is currently being processed and which page of the whole
document will be next to be produced.
Using this data, the server can implement the next step to avoid producing
unnecessary redundant work in the form of single page publishes.
File monitoring is not used by the ActiveX Viewer because this viewer currently decides
if it wants to see a single page publish job or if it can download artifacts from the
finished complete XDL job.

Publishing evasion
Let us consider the following scenario. The client requests the first SVG pages from a
document that has just been uploaded to the server. The SVG job has already been
created and sent to JP/PA. In this case, the page 0 SVG artifact has not been created
yet. It is currently being published by JP.
In this situation, creating a new single page publish job to create that particular SVG
makes no sense. The main job is likely going to finish creating the SVG before the
single page publish job gets to JP/PA. Because of job tracking, the Brava! Server know
that that JP is actively working on the page 0 SVG. The Brava! Server can reasonably
decide not to create the single page publish and let the main job finish working on it.
This way, the single page job is not created, is not taking up JP/PA processing power,
and does not put more pressure on the I/O.
We can expand on that mechanism and assume that creating page 1 (the second
page) through single page publish has no value and decide to skip it as well.
We call this mechanism single page publishing evasion.

The Information Company™ 9

 Brava! Enterprise Champions Guide for Tuning Performance

In the system properties, you can set in advance the number of pages to avoid for
single page publish. You can do this for both SVGs and thumbnails. For thumbnails,
we recommend a much higher value (>20).
Increasing the number of pages skipped should decrease the amount of single page
publish jobs created but doing so could increase the perception of viewer being less
responsive. If the system produces a lot of single page publish jobs and the server
statistics reports that most of the time is spent on creating these jobs, our advice is to
increase the evasion threshold to let main jobs do more work.

#single.page.publishing=off
#single.page.publishing.evasion=1
#single.page.publishing.thumbnails=off
#single.page.publishing.thumbnails.evasion=20

Brava! License
Brava! License provides licensing for Brava! Server only. It is not accessed by JP/PA
or by end-users directly. It provides general licensing for the application. That license
is cached and does not require constant acknowledgements. The second role is client
“slot” tracking, which typically happens one time per document view. Each slot is not
managed individually and the total number of claimed slots is kept and is reset
periodically (refresh window). This approach makes the implementation of the License
Server quite trivial.
Brava! License is typically not CPU intensive, nor does it consume a lot of memory.
Since Brava! License is still using synchronous HTTP request processing; it is
important to understand that there is a potential "deadlock" when the same thread pool
is shared between Brava! License and Brava! Server. To avoid this, our
recommendation is to use a different servlet container for Brava! License or use a
separate Connector other than the Connector used by end-user endpoints.

Job Processor/Publishing Agent (JP/PA)

The main task of JP/PA is to process the provided content. It is the most CPU intensive
part of the system that requires relatively high I/O bandwidth and low I/O latency.
Throughout this guide we use Job Processor and Publishing Agent terms together.
They both have the same function and have similar requirements and architectures.
Job Processor is our legacy product that currently works only on Windows and
Publishing Agent can work on both Microsoft® Windows® (limited deployment
environments as PA is shared with another product line) and Linux platforms.
Queue types:
• single - used to do any single page publishing, including document info jobs
• pdf - used to process PDF sources both into XDL form or SVG
• drw - used to process image files and provide thumbnails (for both pdf and doc)
• doc - used to process "wait for XDL" jobs (jobs that require print publishing)
• 3D - used to process 3d documents using hoops library

The Information Company™ 10

 Brava! Enterprise Champions Guide for Tuning Performance

JP threads vs processes
What is called a thread in JP/PA needs an explanation. A JP/PA thread is responsible
for managing a single worker CDL process (converter). These processes are single-
threaded and cannot use any multithreading capabilities of the system's CPU. A JP
thread works roughly in the following loop:
• request a job from Brava! Server
• if no task is returned, return to job requesting
• initial work related to starting the job (create heartbeat file, I/O checks etc.)
• pass job to converter CDL process
• monitor CDL process
• finish job and send notification back to Brava! Server that the job has finished
• cleanup
• return to job requesting
For example, a configuration with 10 pdf JP "threads" means that there are 10 threads
in JP application that are monitoring at most 10 converter CDL processes. Each of
these processes are autonomic. They do not share any data between each other. While
they work on separate target folders, they can work on the same source document.
They cannot be stopped (even when they time out on the Brava! Server side).

thread.single=2
thread.drw=2
thread.pdf=2
thread.doc=2
thread.3d=1

Communication between JP/PA and Brava! Server

JP must independently configure how many worker threads there are for each Brava!
Server queue type. Brava! Server does not know how many threads/workers there
are. It can only make a determination based on the HTTP request made by JP when a
job is retrieved, and then a notification URL is made to finish the job. Each worker's
thread is using HTTP request to retrieve jobs from Brava! Server. Brava! Server holds
the request, waiting for a job of the requested type. Once a job is available on the
Brava! Server side, it is passed to one of the waiting HTTP connections. If there are no
new jobs during the wait period, an empty response is returned. After returning an
empty response, JP should immediately ask for a new job and continue long polling
jobs from the server.
When a job retrieval request fails (for example when Brava! Server is temporarily
down), it is important that the server does not make another request to pull a job
immediately. That can cause a Denial of Service on the Brava! Server part. The time
that JP/PA should wait after an error is configured through:

job.error.sleep.0=1500
Changing this property to a value near 0 can have negative consequences and adds
virtually no potential gains.

The Information Company™ 11

 Brava! Enterprise Champions Guide for Tuning Performance

It is important to understand that Brava! Server is currently not capable of switching

tasks from one queue to another. Brava! Server does not track if a different queue has
been sitting idle and has, for example, 10 workers waiting idly for a task. If, at the same
time, a different queue is completely overwhelmed with jobs that the server is not going
to use, the other workers can help with the load.
That approach poses certain risks in that, with untasked workers sitting on JP, you can
have a situation where your server is not running at 100% CPU capacity to resolve the
current heavy load.
Because of the CDL characteristics mentioned below, the rule of thumb is that to be
able to utilize 100% of the CPU for JP processing, you should have more running
concurrent CDL processes than CPU cores. Although adding too many can negatively
influence performance and result in higher memory usage, worsened CPU memory
caching, extensive process switching, and increased concurrent I/O.
Additionally, the fact that a worker of one type cannot run a task from a different queue
type makes the process of fine tuning the thread counts quite problematic. Currently,
there is not a ready-to-use common recipe that will work with all or most setups and
for the users’ usage pattern for a portfolio of documents viewed.
The selected viewer type can have an impact as well. AXV viewer composition typically
generates a single complete XDL job and multiple optional single page publishing jobs.
In the case of a PDF document, such a document is sent to the "pdf" queue and
potentially several jobs for "single" queue (including the document info job). With HTML
composition, a document info job is sent to the "single" queue, but the possibility of
generating other single page publish jobs is lower due to file monitoring and single
page publishing evasion. On the other hand, the "pdf" queue will get 1 additional job
(the SVG job) and the "drw" job will handle generating thumbnails. The overall
publishing noise caused by HTML viewer compared to AXV therefore, will usually be
higher.

CDL characteristics
As of 16.6.6, the CDL version that is used is a single-threaded process that relies
heavily on global structures that are not thread safe. The same single thread is
responsible for all the tasks needed to process content: loading loaders into memory,
reading the source file, exporting data to the requested format, and saving the data to
the disk. All these tasks happen sequentially, meaning that CPU-heavy tasks are
intermittent with I/O and therefore a single process of CDL is usually not capable of
maximizing usage of a single CPU core/thread. It also means that the more I/O
bottlenecked your server is operating, the less CPU utilization you will get.
To some extent, you can tell that the CDL process is constantly switching between a
high CPU period (artifact publishing) and a high I/O period (artifact saving to the disk).
The most important conclusion is that if an I/O blocks a CDL process, CPU power
cannot be fully utilized. Additionally, scaling CPU processes and using more of them
concurrently is possible as long as I/O can scale as well.
XDL format
XDL is the internal format of CDL. Through Brava! Enterprise architecture, every
document uploaded to Brava! Server is eventually replaced by its XDL representation.

The Information Company™ 12

 Brava! Enterprise Champions Guide for Tuning Performance

After the XDL representation of the document is ready, it becomes the source for any
other publishes based on that document. The original uploaded source file is not used
anymore, nor are the loader files.
XDL characteristics:
• file based
• multiple files
• page based
• single-threaded
From a performance perspective, generating XDL files is relatively fast; currently faster
than generating SVG files.
Brava! HTML Viewer (BHV) cannot read XDL resources directly. The only resources
that are needed by BHV are search artifacts. All other XDL artifacts are not used.
Typically, when a document is initially uploaded to Brava! Server before the HTML
viewer composition is created, the complete XDL job is not ready before the SVG and
the thumbnail jobs are created and queued. In that case these jobs will be based on
the original source file and not the XDL (and loaders will be used to process the original
document).
When the XDL job is ready, it becomes the representation of the document within the
system. All additional publishing based on the document (such as printing, publish to
PDF or TIFF) will be based on the XDL instead of the original source file.

Print Publishing (Bi2dl)

Print publishing is a process of using an external Windows based application such as
Microsoft® Office’s Word® or Excel®, to "print" to an intermediary format (XPS) and
then read the file by the CDL. The biggest limitation of that process is that only one
document can be printed at a time. Documents (original files extension) that require
such processing should be configured to use the "doc" queue. Typically, there is no
need to have more than 2 workers assigned to the doc queue. That queue can become
a bottleneck for the whole system.
Additionally, print publish does not support single page publishing so the system must
be configured to wait for the main document to complete XDL processing before other
jobs, such as SVG and thumbnails, are created.
To enforce that requirement, the following setting in Brava! Server's server.properties
is used:

extensions.wait.for.XDL.publish=doc,dochtml,docx,htm,html,
mhtml,mht,...

This setting specifies that the listed formats cannot be processed by SVG, thumbnails,
or single page publish jobs directly. The system should first wait for the complete XDL
job to finish, otherwise, responsiveness can be negatively impacted while viewing
these documents.

The Information Company™ 13

 Brava! Enterprise Champions Guide for Tuning Performance

Alternatives for Office documents

Different loaders are available for processing Office documents. They differ not only in
the way that publishing is executed, but also in the fidelity of the end results.
Otf2dl
The Otf2dl library does not have the limitations of being single process only or not
supporting single page publishes, but its fidelity is not as high as other processing
solutions. The Otf2dl technology has a limited release lifecycle, resolving bug fixes
only.
OutsideIn2dl
The OutsideIn2dl library requires additional 3rd party licensing but can process multiple
documents at the same time. It uses an intermediary PDF format before the document
can be fully read by CDL and we recommend using “wait for XDL” in this case.
LibreOffice (Lo2dl)
This library is the latest addition to Brava! Enterprise and it employs LibreOffice to do
the publishing instead of MS Office upon Linux platforms. Currently, the limitations to
the loader are like those of Bi2dl. There is only one concurrent process allowed and
there is also an intermediary format present (PDF).
If the loader you chose supports multiple processes, you should either increase the
count of "doc" threads in the JobProcessor.config or move the supported extension to
a different queue type (such as pdf) in Brava! Server’s server.properties.
Additionally, if the loader you chose does not require publishing to an intermediary
format and can support effective single page publishing, then we advise removing the
extensions supported by it from the extensions.wait.for.XDL.publish list in
server.properties.

Search artifacts
There are 3 search artifacts. The first, Text.idx, is used with ActiveX viewer and can
only by used when you have access to other data from XDL. The other 2 artifacts are
used solely by HTML viewer: textinfo.txtinf and searchtextinfo.txtinf. Both are
uncompressed and can therefore become quite large. These are the last artifacts
created within the complete XDL job.

Publishing noise
Brava! Server has been implemented to publish more artifacts to support the viewing
the file requested by client. When HTML viewer is in use, most of the artifacts produced
by the complete XDL job will not be used as long as the complete XDL job is not being
used as the source for other jobs.
Additionally, if the document contains many pages, a large portion of the published
pages might never be read. If the thumbnail panel is closed, the thumbnails that will be
generated might not be accessible as well.

The Information Company™ 14

 Brava! Enterprise Champions Guide for Tuning Performance

While this approach ensures that artifacts are ready to be sent when needed by an
end-user navigating the content of a file, it also enables JP/PA to process items that
will never be needed. Currently there is no way to decrease this publishing noise.
One option that can help reduce noise is using raster rendering mode in HTML viewer.
In that mode, instead of serving SVG artifacts, raster images (usually jpgs) are
generated. In raster mode, every requested page is provided as a single page publish
and there is not one job that creates all the pages rasterized. The system is not going
to create artifacts for pages that have not been requested.

The Information Company™ 15

 Brava! Enterprise Champions Guide for Tuning Performance

Monitoring Tool
In Brava! Enterprise 16.6.5 we introduced the Monitoring Tool - a new, simple, web-
based UI tool to monitor Brava! Server instance status.
The tool allows you to observe and adjust the whole Brava! Enterprise set up or check
that the environment is not overwhelmed by the number of jobs sent to it.

Live JP/PA jobs tracking

You can observe through the current JP/PA job activity graph alongside a visual
representation of current queue sizes.
Fully saturated JP should generate a solid block of processed jobs:

Figure 1
Live Publishing Chart

In lower sections’ rows, the longer gaps that exist between the rectangles blocks
representing a single job while the queued work size is not 0, may suggest that longer
time was unused by a JP/PA thread between finishing one job and starting another.
In the example above, we can see that there are gaps between single page publish
jobs. The gaps are caused by the fact that the queued work size is basically at 0 size
for quite some time.

The Information Company™ 16

 Brava! Enterprise Champions Guide for Tuning Performance

Queue statistics
The tool provides several basic statistics on the queue types. The stats are not
persisted and reset with every server reset.

Figure 2
Queue Statistics

The total time value can give a highly indicator which queue is receiving the most tasks.

Display List Cache

The DLC section allows you to check the current size of DLC and the available space
(available from the Brava! Server perspective). There is also a pie chart providing
information about the amount of DLC space each job type is using.

The Information Company™ 17

 Brava! Enterprise Champions Guide for Tuning Performance

Recommendations

Adjust the default configuration

The windows installer for Brava! Enterprise sets up the system with configuration ease
in mind. This approach does not always result in the best performance of the system.
Default settings that often should be evaluated for change before the system is used
in production include:
• Display List Cache is on a local system’s disk
• Display List Cache is a network shared resource
• UNC path is used to for Display List Cache path in server.properties
• JP threads numbers are predefined and not based on the actual CPU capabilities
• Display List Cache size is relatively low, and the default value is hardcoded

Monitor and adjust

Brava! Enterprise is an application that works on multiple levels. While it cannot fix or
adjust obvious misconfiguration issue on its own, it will try its best with the settings
provided.
• Monitor CPU usage. A heavily saturated JP should work at ~100% CPU
utilization. Make sure that most of that time is not spent on antivirus, context
switching, and such. It is always worth trying different JP queue settings to see if
that improves performance.
• Monitor memory usage. Increasing the total JP "threads" count can increase
memory usage. In a worst-case scenario, memory swapping can be triggered,
resulting in very poor performance.
• Monitor JVM that runs Brava! Server. Extensive Garbage Collection can have a
negative impact on performance.

Display List Cache size

Make sure that there is enough space for your published documents. Some documents
can be published to very large artifacts. Jobs that produce gigabytes of data, although
rare, do happen. If you produce jobs with gigabytes of data and your DLC size is
relatively low (10 GB), then such a job could negatively impact truncation triggered in
DLC (a huge part of the current cached content would be evicted to fulfill the empty
space requirement).
A rule of thumb is the biggest expected published job should easily fit in the truncation
margin defined in server.properties.

displaylist.cache.maximum.size=10000
displaylist.cache.size.reduction.percent=90
In this example, we have a max size of 10GB with 1GB of margin that is going to trigger
truncation if cache size exceeds 9GB (10GB x 90%). In this case, any job that is around

The Information Company™ 18

 Brava! Enterprise Champions Guide for Tuning Performance

1GB in size can cause significant problems, so increasing the size to 100GB, for
example, should help.

Configure your servlet container for best performance

This guide does not provide any specific steps to ensure the best performance of your
servlet container (such as Tomcat), but these are some areas that can be considered:
• Use a modern JDK. In most cases, more modern JDKs are faster than the old
ones. Try to use the latest version possible.
• Configure your JVM settings:
 use a big enough heap size
 use Garbage Collection that works best with your system and JDK
• Remove any default applications. Any additional application might take resources
from the Brava! Server application and can slow server start up times.
• Remove BravaSDK application. The BravaSDK application is a sample project
and should never be used in production.

Keep system I/O clean

If possible, disable antivirus not only on the directory that holds Display List Cache, but
also on any other folder that keeps files that are managed by Brava! Enterprise
components:
• JP temporary directory. By default, it is the system temporary folder
• JP data folder: c:\ProgramData\OpenText\JobProcessor\
 .request folder specifically
Any process that increases I/O latency or slows down read/write speeds of files stored
on filesystem might negatively affect Brava! Enterprise performance.
Use the fastest disk possible. Disk performance is far more important than its reliability.
A fast SSD should be better in almost all cases than HDD. Using RAID 0 can also help
with getting better performance from multiple disks working concurrently, but that is an
area where proper testing with specific configuration might be required to ensure any
potential benefits.
Use a separate physical disk for Display List Cache. It might be a good idea to use a
separate disk just for Display List Cache. This way, any OS I/O operation should have
less impact on Brava! Server & JP/PA I/O.
Do not share the disk as a network resource. When directories/folders are shared, OS
must make additional checks when any I/O operation is done because it is possible
that other elements in the infrastructure are actively using the resource. In normal
circumstances, that overhead is quite negligible. When there is heavy disk I/O, the
impact might become significant. In that case, different Operating Systems might
behave differently. Note: Currently the BE installer sets up the default Display List
Cache as a shared resource.
Disable 8.3 short file name creation on drives that are managed by Windows. When
many elements exist in a folder system, I/O has an increasingly harder time trying to

The Information Company™ 19

 Brava! Enterprise Champions Guide for Tuning Performance

find a unique 8-character name for it. The impact is minimal, but one more thing that
the OS and filesystem must worry about.

Keep Brava! Server and JP/PA close to each other:

Both Brava! Server and JP are I/O heavy. Brava! Server is read-heavy while Job
Processor is write-heavy.
Brava! Server uploads files to DLC and reads them when they are passed to the clients.
JP/PA reads the uploaded documents and writes published results. Since both sources
and results reside in the same I/O space, both Brava! Server and JP/PA need fast
access to the DLC or else it will slow one or the other.
Because of this, we recommendation keeping Brava! Server and JP on the same
machine and use local paths to specify the shared cache. Do not share the Display L
Cache folder to lower the OS/Filesystem overhead on processing I/O for files accessed
by both Brava! Server and JP/PA.
This is a soft recommendation. If your system can provide a low latency, fast I/O access
for both JP and Brava! Server, then you might try to keep them separately.
One additional advantage of such an environment is that it should be easier to
replicate.

displaylist.cache.root=d:\\
local.displaylist.cache.root=d:\\

Use sharding and scale out

If, in your setup, you see that the best performance gives you a BESS+ JP/PA setup
mentioned above, you might ask how to scale such a system? Adding more Job
Processors can give you better publishing times but can also hurt the I/O for the overall
system. Adding a Job Processor machine takes valuable I/O bandwidth from other JPs
and Brava! Server.
One solution could be a simple multiplication of the BESS+ JP setup and the use of a
load balancing/sharding algorithm on the integration level. Each BESS + JP instance
would be completely independent, which is not an issue for most of the tasks that
Brava! Enterprise is capable of.

Routing Service
Routing Service is a separate independent element in the Brava! Enterprise ecosystem
that provides viewing session persistence for a cluster of Brava! Server instances.
Currently it cannot provide load balancing on its own - that part must be provided by
an additional software or hardware solution. It can, however, track which node the
traffic was routed to and then it can keep that information and use it for subsequent
requests.

The Information Company™ 20

 Brava! Enterprise Champions Guide for Tuning Performance

Avoid single page publishing

Brava! Server's architecture can do additional work even when it is not necessary. To
some extent, the latency is more important than the throughput for Brava! Server.
These additional jobs usually come in the form of single page publish jobs: Jobs that
create a single page artifact that has been requested by end-user but is not yet
available from the main publish job. If you see that your system is producing a lot of
single page publish jobs, it might be worth configuring limitations with the following
steps:
• increase single page publish evasion
• limit the number of preloaded pages in HTML viewer
• increasing the number of threads for main queue types (pdf, drw)
It might be beneficial to check how many single page publishing jobs are being created
using the Monitoring tool.

single.page.publishing=on
single.page.publishing.evasion=4

HTML viewer page precaching

There are two caching mechanisms within HTML viewer that manage displayed pages.
The first defines how many pages around the currently viewed page should be
preloaded from the server, and the second defines how many pages should be kept in
the browser memory. The latter has lower impact on the server performance, but the
former can have a significant impact on the amount of single page publishes generated
on the server side.
As an example of this, let us assume that the setting is 7. In this case, the viewer will
try to load 3 pages before and 3 pages after the current page (3 + 1 + 3 = 7). After
opening any document, the viewer will ask right away for 4 pages. That could make
the scrolling process much smoother but could also put greater pressure on the JP/PA.
We recommend you keep that number relatively low and make sure that the evasion
value from server.properties is higher than the evasion value set for single page
publishing.
Additionally, we have added single page publishes for thumbnails. Thumbnails are
usually generated faster than SVGs, but the viewer typically displays a far greater
number of thumbnails on a view port than on main pages.

single.page.publishing.thumbnails=off
single.page.publishing.thumbnails.evasion=40

By default, single page publishing for thumbnails is enabled. If that is causing far too
many single page publishes on the server side, we recommend disabling that feature
or at least increasing the evasion number to a higher value.
Check the statistics provided by the Monitoring Tool to see how many single page
publishes are being created.

The Information Company™ 21

 Brava! Enterprise Champions Guide for Tuning Performance

Disable generating heartbeat files

Heartbeat files are "log" files generated by both JP/PA and CDL itself. They are
intended for debugging purposes and are not required by any client or Brava! Server.
While they used to be generated for every JP job created by Brava! Server, they now
are disabled by default and can be enabled through server.properties.
We encourage that you do not enable heartbeat files in production environment. There
is a certain performance hit in generating them and, even though that performance hit
is not as big now as it used to be (in the past it could be as high as ~15%), we
recommend disabling this setting to decrease the I/O noise, disk usage, and the CPU
time needed to generate them.

publish.heartbeat=false

Precaching
Precaching is a process of uploading or even creating HTML viewer compositions prior
to the on-demand view triggered by an end-user.
Currently, Brava! Server does not support precaching directly. We do not maintain or
update the LRU (least recently used) list to know which item is the one least needed.
We do not support any external blob storages, so it is quite common to work constantly
in truncation mode when newly uploaded documents and triggered publishing jobs
triggers the eviction of older items.
Precaching can make the system seem more responsive, but that is true only if the
integration can correctly predict which documents will be viewed in the future.
Publishing of the pre-cached documents (which are not currently needed by end-users)
does not affect the current on-demand views of documents.
Still, a simple upload of a document and creation of an HTML composition from it
should effectively prepare all the needed artifacts beforehand.

Document upload vs HTML viewer composition

There is distinct difference between uploading documents to Brava! Server and
creating an HTML viewer composition out of the source document.
A document upload creates a metadata entry for the document in the Display List
Cache and triggers a single publishing job called complete XDL. Complete XDL is a
job that creates an internal representation of the uploaded documents.
Additionally, a document info job might be created. This is a temporary, non-persisted
job with the purpose of getting information about page count and formatting of the
uploaded documents.
An upload does not trigger thumbnail jobs or SVG jobs right away.
Creating an HTML viewer composition by comparison will trigger thumbnail and SVG
jobs. These jobs, if created before the complete XDL job finishes, will be based on the
original source document. If the composition is created later, when the document

The Information Company™ 22

 Brava! Enterprise Champions Guide for Tuning Performance

already has a XDL representation ready, then the source of the document will be the
complete XDL job.
That difference is completely hidden from integrators and end-users. There is a slight
performance hit to processing jobs directly from source documents because in that
case, a loader library is involved. The XDL format, on the other hand, is an internal
format that is recognized by the core CDL library.
In practice, the best option is to upload a document first, wait for complete XDL to
finish, then create the HML viewer composition. This strategy would be based on
current internal implementation details for publishing documents, which is something
that could change at any time in the future.
Be aware of one additional aspect of potential precaching and a general note about
JP/CDL performance versus Brava! Enterprise architecture. From the throughput
perspective, it is better to generate all the artifacts (XDL, SVG, thumbnails) as a single
job than to create multiple jobs to generate them separately. Doing them separately
might make some sense when we can ensure that all the tasks are running
simultaneously, but usually the time spent reading the content using a loader library is
wasted in such a scenario. Brava! Server currently cannot merge these separate jobs
into a single task.

The Information Company™ 23

 Brava! Enterprise Champions Guide for Tuning Performance

About OpenText
OpenText enables the digital world, creating a better way for organizations to work with
information, on premises or in the cloud. For more information about OpenText (NASDAQ:
OTEX, TSX: OTC) visit opentext.com.
Connect with us:

OpenText CEO Mark Barrenechea’s blog

Twitter | LinkedIn

Copyright © 2022 Open Text. All rights reserved. Trademarks owned by Open Text.