User Guide

Monitoring Payara Server

in Production

The Payara® Platform - Production-Ready,

Cloud Native and Aggressively Compatible.
 Monitoring Payara Server in Production

Once an application is running on Payara® Server, it’s important to know when something is not
working as expected, or if you’re experiencing performance issues. To find out what is going on and
what could be improved, Payara Server provides several monitoring options.

Recent versions of Payara Server can even provide support for automatic self- monitoring in order to
detect future problems as soon as possible. When enabled, the automatic checks are lightweight and
run with a negligible impact on performance. Automatic self-monitoring in Payara Server can detect
slow execution of SQL queries and periodically check low level metrics with the Health Check Service.

Introduction
Payara Server offers a comprehensive set of monitoring tools which can broadly be divided into
three aspects:

The Payara Server Monitoring Service

Inherited from GlassFish, the Monitoring Service provides MBeans which are used by many other
monitoring systems. The MBeans can be accessed directly via JMX RMI or through a REST interface.

Payara Server Operations Services

Payara has been gradually adding new services tailored to operations teams to give more insight
into what’s going on “under the covers” of Payara Server. Currently, these include:

•  Request Tracing Service

•  HealthCheck Service
•  Slow SQL Logging
•  JMX Monitoring Logging Service
•  Notification Service

Each of the new operations services can push notifications through various channels provided by
the Notification Service, including log files, emails or SNMP.

MicroProfile Specifications
The Eclipse MicroProfile is a collection of specifications implemented by many application server
vendors. There are currently two MicroProfile specifications which relate to monitoring: Metrics
and Health Check (not to be confused with our own HealthCheck Service which predates the
MicroProfile specification).

1
 Monitoring Payara Server in Production

Payara Server Core Monitoring Service

The Monitoring Service gathers runtime statistics about resources in the server. To do this, Payara
Server needs to intercept invocation of the resources and read their internal information. Naturally,
this would cause a slight degradation in performance, therefore the Monitoring Service is disabled by
default. It can be easily enabled and disabled at any time as needed while Payara Server is running.
You can enable and disable the Monitoring Service using asadmin or the admin web console. Have
a look at our Advanced Administration & Maintenance of Payara Server guide for more information.

Once Monitoring Services are turned on, the following can be monitored:

•  current state of resources on the server

•  historical data and statistics
•  current configuration

This includes resources on all levels, ranging from the core JVM resources like threads, memory
usage, and garbage collection, through resources provided by Payara Server, up to the resources
defined by the application itself. The information is categorised into subsystems, which can be ena-
bled independently. The subsystems include JVM, EJB Container, HTTP Service, and many more.
They need to have the monitoring level set to either LOW or HIGH in order to collect runtime data
which can be set on the Monitoring Service page in the appropriate configuration tree:

2
 Monitoring Payara Server in Production

Finally, there are many ways to access the monitoring data:

•  REST API provided by Payara Server

•  a lightweight web app, which provides an interface to the REST API for humans
•  standard JMX interface, using tools like JConsole or Java Visual VM
•  a user-friendly interface in the web console
•  using the asadmin tool from within a system shell

Monitoring Through the REST API

Each object available for configuration and monitoring is represented as a REST resource. The URLs
to these resources are as follows (for a local connection and the default admin port):

Configuration: http://localhost:4848/management/domain
Monitoring: http://localhost:4848/monitoring/domain

3
 Monitoring Payara Server in Production

Both the REST services can be accessed through HTTP protocol by any REST client (including any
browser). They are capable of providing the information in JSON, XML or HTML format. The default
format is HTML, but it can be specified easily by using the appropriate suffix. Compare the output
of the following:

•  http://localhost:4848/monitoring/domain
(HTML output with links to related resources)
•  http://localhost:4848/monitoring/domain.json
(JSON output)
•  http://localhost:4848/monitoring/domain.xml
(XML output)

The default HTML output is, in fact, a simple web application, which you may use to view the mon-
itoring data in the browser and browse through the whole resource tree by clicking on the links to
the child elements. For example, if we are interested in how many HTTP error responses have been
served, we can use this simple web application to display statistics about the HTTP listener. The
following address will display them in the browser:

http://localhost:4848/monitoring/domain/server/http-service/server/request

The output gives statistics about all response codes and also aggregated statistics for their groups
(e.g. 5xx for all responses with errors).

4
 Monitoring Payara Server in Production

In a similar way, we may display statistics about internal application resources, for example EJBs or
servlets. If we have an application with name “simple” that has an EJB with name “DemoEJB”, we
can even find out how many times a method with the name “executeTimer” executed. The following
REST resource will show it:

http://localhost:4848/monitoring/domain/server/applications/simple/DemoEJB/bean-methods/
executeTimer

This resource gives us statistics about number of method executions, including the number of those
ending successfully and those ending with an error. It also tracks the time of the last execution, and
the total time of all executions.

The potential of the monitoring REST resources is well demonstrated in a short video created by
Adam Bien on his blog.

5
 Monitoring Payara Server in Production

Monitoring Through the JMX RMI Interface

Payara Server also provides monitoring information through a JMX RMI interface. This is a standard
monitoring interface for any Java SE application, and can be accessed by applications available in
the JDK, namely JConsole and Java Visual VM (the latter requires a plugin). These two tools provide
a graphical representation of standard JVM statistics, the rest of the monitoring data is provided as
a tree of JMX Management Beans (MBeans).

Java Visual VM.

In a standard installation, Payara Server exposes the JMX connector locally on port 8686. In order
to allow remote connections, it is necessary to configure a separate remote JMX connector with
additional system properties, as described in Java SE documentation.

The Payara Server JMX management beans are provided under the amx node. If this node is not
available, it has to be initialized first using a special JMX bean provided in amx-support/amx-loader.
Just select amx-loader in the JMX tree and execute the bootAMX operation by pressing the button.

6
 Monitoring Payara Server in Production

The MBeans tree and the button to enable the Payara Server monitoring beans under the amx-sup-
port node when viewing the available operations for that MBean. The example above is Visual VM.

If your application loaded some EJBs then you would find them under appropriate node in the MBeans
tree (e.g. SingletonSessionBean or StatelessSessionBean). There are also other management beans
in the MBeans tree, which provide runtime information about other components and the JVM itself.

Additionally, in order to see nodes that provide live monitoring data, it is necessary to enable moni-
toring MBeans in the admin console. Only after this step will the monitoring data be available through
nodes, whose name usually ends with the suffix “-mon”. For example bean-method-mon provides
the same statistics about method execution as the appropriate REST resource and session-mon
provides information about open sessions as well as past sessions.

In the following screenshot, we can see information about sessions in the admin console.

7
 Monitoring Payara Server in Production

Monitoring in the Administration Console

The administration interface in the browser provides a nice tabular view on the monitored data. The
statistics are accessible through the Monitoring data page.

They are organized into 3 distinct tabs:

•  Applications (info about resources inside the deployed applications)

•  Server (info about the JVM and the Payara Server subsystems)
•  Resources (info about the resources provided by the server)

8
 Monitoring Payara Server in Production

Monitoring from the System Shell Using asadmin

From within a system shell, we may run the asadmin monitor command to display real-time statistics.
The --type parameter specifies which monitoring component should be monitored:

asadmin> monitor --type=jvm

This command will log real-time data in intervals (every 30 seconds by default). We may redirect
the output of the command to a file to archive monitoring information. There are 4 type of statistics
we can monitor (arguments to --type parameter):

•  servlet
•  httplistener
•  webmodule
•  jvm

9
 Monitoring Payara Server in Production

The command asadmin help monitor will provide detailed information about the monitor command.

Using asadmin, we can also access all the monitoring data by executing the following command:

asadmin> get --monitor server.*

This command will return all available server and application monitoring data. This might easily be
several hundred lines of text. Of course, we can use a more precise filter to limit the amount of data:

asadmin> get --monitor server.transaction-service.*

Or we may request data only about a specific module, for example:

asadmin> get --monitor server.transaction-service.committedcount-count

These commands provide exactly the same amount of information that is availa-
ble through the REST interface. The last command corresponds to the following URL:
http://localhost:4848/monitoring/domain/server/transaction-service/committedcount.

Payara Server Operations Services

Payara Server has several services which can be used to monitor the behaviour of Payara Server or
applications deployed to it. The first service to understand is the Notification Service. This service
provides many channels through which to receive monitoring data. There are several services which
can feed data through to the configured notifiers: the JMX Monitoring Logging Service which can
periodically log the values of specific JMX MBeans through the notification service; the Request
Tracing Service which can trace each request as it propagates through the application; and the
HealthCheck Service which can periodically check the status of various server and host metrics.

In addition to these services, there is also a slow SQL logger which operates independently. By
simply specifying a “slow” threshold on a data source, any query made through that data source
which takes longer than the threshold will cause both the problematic query and a stacktrace to
show where the query originated from to be printed to the server.log file.

10
 Monitoring Payara Server in Production

The Notification Service

Payara Server comes with a general Notification Service in order to log events which come from
other services, such as the JMX Monitoring Logging Service, the HealthCheck Service or the Request
Tracing Service.

The Notification Service provides the ability to disseminate notification events through various chan-
nels that are being created by other services. This service is provided with a number of configurable
notifiers, the default being a log notification mechanism alongside these other notifiers:

•  HipChat
•  Slack
•  NewRelic
•  Datadog
•  SNMP
•  XMPP
•  JM
•  Email
•  CDI Event Bus

Because other monitoring services depend on the Notification Service to output their data, both the
Notification Service itself, and at least one notifier, must be enabled for any data to be recorded. If
this does not happen, then the data will be discarded.

It is for this reason that the Log notifier is the default notifier and is enabled out-of-the-box.

In order to configure a service like the HealthCheck Service to send notifications to a specific notifier,
the following configuration steps are needed:

1.  Notification Service needs to be enabled

2.  Notifiers in the Notification Service need to be configured to specify how to process
the notification
3.  After configuring them, notifiers need to be enabled globally in the Notification Service
4.  Notifiers also need to be selected (enabled) for each particular service that should send
them notifications

Notifiers have to be enabled both globally in the Notification Service and also for each service that
sends notifications.

11
 Monitoring Payara Server in Production

Notifiers are configured in the Notification page of the Administration console:

The Notification Service can be enabled in the General tab. Log notifier can be enabled and config-
ured also in the General tab. All other notifiers can be enabled and configured in their respective tabs.

12
 Monitoring Payara Server in Production

Notifiers for each specific service are enabled (selected) in a configuration page for each service.
Here’s an example page for the HealthCheck Service, which selects Log and Email notifiers for it:

You can also configure everything with the asadmin tool. The Notification Service can be configured
with the notification-configure command.

To enable the Notification Service:

notification-configure --enabled=true --dynamic=true

The argument --dynamic is optional and applies the configuration without server restart.

13
 Monitoring Payara Server in Production

Each notifier provides a separate asadmin command to enable and configure it:

•  notification-log-configure for the Log notifier

•  notification-email-configure for the Email notifier
•  notification-slack-configure for the Slack notifier
•  notification-hipchat-configure for the HipChat notifier
•  notification-jms-configure for the JMS notifier
•  notification-newrelic-configure for the New Relic notifier
•  notification-datadog-configure for the Datadog notifier
•  notification-snmp-configure for the SNMP notifier
•  notification-xmpp-configure for the XMPP notifier

For example, to configure the Log and Email notifiers:

notification-log-configure --enabled=true --dynamic=true

notification-email-configure
--jndiName=mail/exampleEmailNotifier
[email protected] --enabled=true
--dynamic=true

Each service that can send notifications also provides one asadmin command for each notifier which
enables the notifier for that service.

For example, the following 2 commands enable the log and email notifiers for the HealthCheck service:

healthcheck-log-notifier-configure --enabled=true –dynamic=true

healthcheck-email-notifier-configure --enabled=true –dynamic=true

The JMX Monitoring Logging Service

Payara Server allows a list of MBean attributes to be supplied and passed to one of more notifiers.
This is particularly useful in situations where RMI connections are not permitted, or there is no
available method of collecting historical JMX data for later analysis.

In order to log information about an MBean you need to know its object name and the name of its
attribute that should be logged. If an MBean provides composite attributes, the JMX Monitoring
Logging Service can either log all composite values or a single specific value. To log a single value,
the attribute name should also contain the name of the specific value separated by a period.

14
 Monitoring Payara Server in Production

As a worked example, we can find and log the total connections to all network listeners (admin-lis-
tener, http-listener-1 and http-listener-2 by default). To do this, we need to find the MBean name
and the attribute we want to monitor. This can be found using VisualVM as shown:

The MBean that we are interested in is highlighted above, which translates to the following MBean
object name:

amx:pp=/mon/server-mon[server],type=connection-queue-mon,
name=network//connection-queue

The attributes available under this MBean are all composite attributes, however. This means that
a number of different values are available for each attribute. We could log all of them for a specific
composite attribute if we reference the attribute by name. Since we are only interested in the raw
count here, we can reference the “count” value with the “.count” suffix as follows:

counttotalconnections.count

15
 Monitoring Payara Server in Production

Using the asadmin tool, we can do the same with the set-monitoring-configuration com-
mand as follows:

set-monitoring-configuration '--addattribute=objectName
=amx:pp=/mon/server-mon[server],type=connection-queue-mon,
name=network//connection-queue attributeName=counttotalconnections.count'

This will result in the following sample output for the counttotalconnections.count attribute:

[2018-04-09T13:55:30.067+0100] [Payara 5.181] [INFO] []

[fish.payara.jmx.monitoring.MonitoringService] [tid: _ThreadID=268
_ThreadName=admin-thread-pool::admin-listener(21)] [timeMillis:
1523278530067] [levelValue: 800] [[ Monitoring Service will startup]]

[2018-04-09T13:55:35.076+0100] [Payara 5.181] [INFO] []

[fish.payara.nucleus.notification.log.LogNotifierService] [tid:
_ThreadID=336 _ThreadName=payara-monitoring-service(1)] [timeMillis:
1523278535076] [levelValue: 800] [[ PAYARA-MONITORING:
- PAYARA-MONITORING: countcounttotalconnections=93 ]]

[2018-04-09T13:55:50.069+0100] [Payara 5.181] [INFO] []

[fish.payara.nucleus.notification.log.LogNotifierService] [tid:
_ThreadID=336 _ThreadName=payara-monitoring-service(1)] [timeMillis:
1523278550069] [levelValue: 800] [[ PAYARA-MONITORING:
- PAYARA-MONITORING: countcounttotalconnections=94 ]]

This output will, by default, be logged to the server.log by the log notifier. The log notifier can be
modified to log to a separate file to keep application logs clean. The information can also be sent to
other notifiers from the list of available notifiers.

16
 Monitoring Payara Server in Production

The Request Tracing Service

The Request Tracing Service provides tracing facilities for multiple protocols and process commu-
nications done by the components of deployed applications.

The service helps users to detect application slowness and performance degradation by logging
requests that exceed a given threshold. The trace data from long-running requests gives insight to
solving bottlenecks and other performance issues.

From the 181 release of Payara Server, Request Tracing has been rewritten for compatibility with
the OpenTracing specification. This will bring Payara Server’s own Request Tracing into line with
the upcoming MicroProfile OpenTracing specification. Planned features for Request Tracing include
ways to interpret and analyse the data with OpenTracing compatible servers.

Understanding Terms
Since OpenTracing is being developed as a collaboration with multiple vendors, the specification
has established a versioned document to track the specification. This specification uses terms in
specific ways which may not be familiar to users of Payara Server. These terms are defined below:

•  Span
A “Span” is equivalent to a single operation, or method and records a start time, end time
and a duration. Each span has a unique “spanId” in its spanContext and a human-readable
operationName which describes the operation that was completed in the span.
•  Span Context
Every span has a “spanContext” property which is used to both identify the span itself as
well as the trace that it belongs to.
•  Trace
A “Trace” can be thought of as a single request. There is no overarching record for the trace,
it is simply a correlation ID which is shared among all the related “spans”. In the spanCon-
text, there is a “traceId” field which is the same among related spans.

17
 Monitoring Payara Server in Production

Quickstart
To begin using Request Tracing, it must first be enabled. This can be done dynamically through the
admin console, so no restart is required:

As well as simply enabling Request Tracing, a valid notifier needs to be added so that the data can
get logged. The log notifier is the most straightforward, because it is enabled in the Notification
Service by default.

Once these steps are complete, it is still likely that no data is logged, which could be for two reasons:

1.  If you have no applications deployed and simply want to trace the activity in the admin con-
sole, you must disable the “Applications Only” filter.
2.  The default threshold value is set to 30 seconds, which means that any request which does
not take that long will not be traced. To trace most things, the threshold can be set to a very
low value, such as 30 milliseconds, for example.

18
 Monitoring Payara Server in Production

Below is a sample output from enabling the Request Tracing Service in the admin console with a
threshold of 30 milliseconds, as shown above. I have extracted the JSON from a single notification
and formatted it for clarity:

{
"traceSpans": [
{
"operationName": "processContainerRequest",
"spanContext": {
"spanId": "5148919a-2413-4cbd-af6c-8cb469f8b0ec",
"traceId": "e7ad7db6-b626-4995-aa10-e2e6a073033e"
},
"startTime": "2018-04-10T11:15:50.018+01:00[Europe/London]",
"endTime": "2018-04-10T11:15:50.164+01:00[Europe/London]",
"traceDuration": "146000000",
"spanTags": [
{ "Server": "server" },
{ "Domain": "domain1" }
]
},
{
"operationName": "authenticateJaspic",
"spanContext": {
"spanId": "374aefba-60ce-4011-98c5-9ade69eeeaeb",
"traceId": "e7ad7db6-b626-4995-aa10-e2e6a073033e"
},
"startTime": "2018-04-10T11:15:50.018+01:00[Europe/London]",
"endTime": "2018-04-10T11:15:50.019+01:00[Europe/London]",

19
 Monitoring Payara Server in Production

"traceDuration": "1000000",
"spanTags": [
{ "Context": "" },
{ "AppContext": "__asadmin " },
{ "AuthResult": "true" },
{ "Principal": "admin" }
]
},
{
"operationName": "processServletRequest",
"spanContext": {
"spanId": "fd66fafe-a741-408c-b99a-cb7a963c07ac",
"traceId": "e7ad7db6-b626-4995-aa10-e2e6a073033e"
},
"startTime": "2018-04-10T11:15:50.019+01:00[Europe/London]",
"endTime": "2018-04-10T11:15:50.164+01:00[Europe/London]",
"traceDuration": "145000000",
"spanTags": [
{ "referer": "[https://localhost:4848/common/index.jsf]" },
{ "QueryString": "configName=server-config&bare=true" },
{ "host": "[localhost:4848]" },
{ "Class": "javax.faces.webapp.FacesServlet" },
{ "Method": "GET" },
{ "URL": "https://localhost:4848/payaraExtras/requestTracing/requestTracing.jsf" },
{ "accept-encoding": "[gzip, deflate, br]" },
{ "accept": "[*/*]" }
]
}
]
}

In it, there are three separate spans which make up the whole trace, indicated by the shared traceId.
It’s important to note that the traceDuration property applies to the span rather than the trace as a
whole. In this case, there was a request that took 146ms, indicated by the first span, and two child
spans which took 1ms and 145ms respectively.

If we were observing this data to find where to focus optimisation efforts, we can see that the pro-
cessContainerRequest operation first triggered a JASPIC authentication, which returned very quickly,
and then a processServletRequest operation which took almost all the time of the request. It would
be in the processServletRequest operation that we would focus optimisation efforts.

20
 Monitoring Payara Server in Production

The HealthCheck Service

Payara Server also has the HealthCheck Service, which periodically checks:

•  Host CPU Usage

•  Host Memory Usage
•  Payara Server’s JVM Garbage Collections
•  Payara Server’s JVM Heap Usage
•  CPU Usage of individual threads
•  Threads which have been stuck for longer than a specified time

If there is a problem with any of these metrics and they exceed a configurable threshold, then a
message is logged to the server’s log file. This helps to rapidly detect problems or work out what
happened after problems have occurred. For most of the checks, threshold configurations can be
specified in 3 different levels: good, warning and critical. When the threshold is not met, this will in
turn log a message with the level of Warning, Error or Critical respectively.

The HealthCheck Service is not enabled by default, but can be easily enabled in the administra-
tion console:

It can also be easily enabled by the asadmin healthcheck-configure command:

asadmin> healthcheck-configure --enabled=true --dynamic=true

21
 Monitoring Payara Server in Production

The argument --dynamic=true is necessary to turn on the service for a running server, otherwise it
would only be applied after a server restart.

Then we need to configure the service for each metric we want to automatically check. This is done
by using the asadmin healthcheck-configure-service command for each metric service.

We can check the actual configuration of the HealthCheck Service and all its metrics using asadmin
get-healthcheck-configuration:

./asadmin get-healthcheck-configuration

This is an example output from the above command, after some Health Check services have
been configured:

Health Check Service Configuration is enabled?: true

Below are the list of configuration details of each checker listed by its name.

Name Enabled Time Unit

GC false 10 SECONDS

Name Enabled Time Unit thresholdPercentage retryCount

HT true 10 SECONDS 95 3

Name Enabled Time Unit Critical Threshold Warning Threshold Good Threshold
CPU false 10 SECONDS 40 20 2
HP false 8 SECONDS - - -
MM false 7 SECONDS - - -

For more information about the HealthCheck Service, see the HealthCheck Service documenta-
tion page.

Slow SQL Logger

Payara Server includes capabilities to detect and log slow SQL queries executed via a connection
pool. The Slow SQL logger monitors all queries executed on the connection pool and if they exceed a
configurable execution time in seconds, a warning message is logged into the server log. The warning
message (see below) logs the SQL query and the stack trace to the code executing the query. This
enables rapid diagnosis, pinpointing the exact lines of code to investigate.

22
 Monitoring Payara Server in Production

To configure slow SQL logging via the administration console, navigate to the connection pool’s
Advanced Properties tab and specify the Slow Query Log Threshold time in seconds. A value of -1
disables logging of slow queries.

Similarly, slow SQL logging can be configured using the asadmin set command, as with other con-
figuration properties.

For more information about the detection of slow SQL queries, see the Slow SQL logger documen-
tation page.

23
 Monitoring Payara Server in Production

Monitoring with Eclipse MicroProfile

As at the 1.3 release of the Eclipse MicroProfile, there are two available specifications which can
be used to monitor both the server and applications. These are designed to solve very different
problems, however.

MicroProfile Health Check

From releases 4.1.2.181 and 5.181, both Payara Server and Payara Micro implement version 1.0 of
the MicroProfile Health Check API. The purpose of the Health Check API is to provide a single end-
point which can be processed automatically by another system (as opposed to by a human) which
could then use the information to make decisions on whether or not to take action in removing and
replacing an unhealthy instance.

MicroProfile Health Check can be used by developers to aggregate a series of important metrics into
a single answer to the question “is my service healthy?”.

As the Health Check specification outlines:

Within CDI contexts, beans that implement HealthCheck and annotated

with @Health are discovered automatically and are invoked by runtime when
the outermost protocol entry point (i.e. http://HOST:PORT/health) receives an
inbound request.

An example of a health check defined by a deployed application:

@Health
@ApplicationScoped
public class CheckDiskspace implements HealthCheck {
@Override
public HealthCheckResponse call() {
return HealthCheckResponse.named("diskspace")
.withData("free", "780mb")
.up()
.build();
}
}

24
 Monitoring Payara Server in Production

The health status can be viewed from a running instance by visiting /health in a web browser. For
an instance with no changes to default settings, this would be http://localhost:8080/health.

The endpoint should respond with one of three HTTP status codes:

•  200
The overall outcome is positive; the state is UP
•  503
The overall outcome is negative; the state is DOWN
•  500
An error occurred while processing the request; the state is UNKNOWN

As can be seen in the example above, the HTTP status can also, optionally, return data which can
help to inform how close the endpoint is to going down, or what may have caused it to go down.

MicroProfile Metrics
From releases 4.1.2.181 and 5.181, both Payara Server and Payara Micro implement version 1.0 of
the MicroProfile Metrics API. The Metrics API was designed to be a standardised alternative to JMX
using a REST endpoint.

Metrics data can be viewed from a running instance by visiting /metrics in a

web browser. For an instance with no changes to default settings, this would be
http://localhost:8080/metrics.

Metrics was designed to be highly compatible with the Prometheus text format, which is a very
commonly implemented format for monitoring systems which pull data from endpoints, rather than
having a remote agent which pushes data to a remote monitoring server.

Using a web browser to visit http://localhost:8080/metrics will result in an output of Prometheus

text format as shown below:

# TYPE vendor:system_cpu_load gauge

# HELP vendor:system_cpu_load Display the “recent cpu usage” for the whole
system. This value is a double in the [0.0,1.0] interval. A value of 0.0 means
that all CPUs were idle during the recent period of time observed, while a
value of 1.0 means that all CPUs were actively running 100% of the time during
the recent period being observed. All values betweens 0.0 and 1.0 are possible
depending of the activities going on in the system. If the system recent cpu
usage is not available, the method returns a negative value.
vendor:system_cpu_load 0.1491192437579042

25
 Monitoring Payara Server in Production

# TYPE base:classloader_total_loaded_class_count counter

# HELP base:classloader_total_loaded_class_count Displays the total number of
classes that have been loaded since the JVM has started execution.
base:classloader_total_loaded_class_count 11250
# TYPE base:cpu_system_load_average gauge
# HELP base:cpu_system_load_average Displays the system load average for the
last minute. The system load average is the sum of the number of runnable
entities queued to the available processors and the number of runnable
entities running on the available processors averaged over a period of time.
The way in which the load average is calculated is operating system specific
but is typically a damped time-dependent average. If the load average is not
available, a negative value is displayed. This attribute is designed to provide
a hint about the system load and may be queried frequently. The load average
may be unavailable on some platform where it is expensive to implement this
method.
base:cpu_system_load_average -1.0
# TYPE base:thread_count counter
# HELP base:thread_count Displays the current number of live threads including
both daemon and non-daemon threads.
base:thread_count 73
# TYPE base:classloader_current_loaded_class_count counter
# HELP base:classloader_current_loaded_class_count Displays the number of
classes that are currently loaded in the JVM.
base:classloader_current_loaded_class_count 10992

...etc

This endpoint is enabled by default, so no other configuration is needed to view this data.

An example configuration file for Prometheus is shown below:

global:
scrape_interval: 5s # Set the scrape interval to every 5 seconds. Default
is every 1 minute.
evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is
every 1 minute.
# A scrape configuration containing exactly one endpoint to scrape:
# Here it’s the default HTTP port for Payara
scrape_configs:
- job_name: 'microprofile-metrics-demo'

26
 Monitoring Payara Server in Production

static_configs:
- targets: ['localhost:8080']

Since MicroProfile Metrics uses a lot of default configuration options, Prometheus doesn’t need any
extra configuration beyond the host and port where metrics are published. Even the path “/metrics”
is the default path that Prometheus will poll.

The data can be scraped from this endpoint by many different monitoring tools as well as Prometheus,
though these tools may need additional configuration.

Metric values are stored in a registry with three scopes:

•  Base (JVM metrics, common to every implementation)

•  Vendor (Vendor specific metrics)
•  Application (User metrics)

There are several types of Metrics. Some of these can be seen in the example output above, and all
can be used to provide data from your own applications in the application scope.

•  Gauge
simple metric type for continuous value, e.g. memory.usedHeap, memory.committedHeap,
memory.maxHeap
•  Counter
simple metric type for discrete value, e.g. thread.count, thread.daemon.count
•  Meter
complex metric type comprised of multiple key/values; values are count, meatRate, one-
MinRate, fiveMinRate, fifteenMinRate
•  Histogram
complex metric type comprised of multiple key/values; values are count, min, max, mean,
stddev, p50, p75, p95, p98, p99, p999
•  Timer
complex metric type comprised of multiple key/values; values are count, meanRate, one-
MinRate, fiveMinRate, fifteenMinRate, min, max, mean, stddev, p50, p75, p95, p98, p99,
p999

27
 Monitoring Payara Server in Production

There is a simple example available in the Payara Examples repository. Below is the TicketResource.
java class, which is a JAX-RS endpoint showing how @Metric can be added to a Counter varia-
ble. Incrementing and decrementing this counter can show the current status of the ticket count.
Annotations like @Metered can be used to provide metrics for method usage:

@Path("tickets")
@ApplicationScoped
public class TicketResource {
private final String SOLD = "sold";
List<String> tickets = Arrays.asList(new String[100]);
@Inject
@Metric
Counter ticketCount;
@PostConstruct
private void init() {
// initialise counter with beginning number
ticketCount.inc(100);
}

/*
We need to set the "name" attribute to make sure the metrics
are differentiated, otherwise there will be a single
Meter for the overloaded methods, rather than one each.
*/
@GET
@Path("buy")
@Metered(name = "Buy any ticket")
public String buyTicket() {
int i = 0;
for (String t : tickets) {
if (t != SOLD) {
tickets.set(i, SOLD);
ticketCount.dec();
return "Success! You have bought ticket number " + i;
} else {
i++;
}
}
return "Error! Sold out";
}
@GET

28
 Monitoring Payara Server in Production

@Path("buy/{id}")
@Metered(name = "Buy specific ticket")
public String buyTicket(@PathParam("id") int id) {
if (tickets.get(id) != SOLD) {
tickets.set(id, SOLD);
ticketCount.dec();
return "Success! You have bought ticket number " + id;
} else {
return "Error! Ticket number " + id + " is not available!";
}
}
... snip ...
}

Since the annotations have been added to overloaded methods, we can get around name clashes
by specifying the “name” property. These are converted to lower case and spaces converted to
underscore to retain compatibility with Prometheus.

Enable Monitoring Tools to Detect Future Problems

Maintaining the health of your technology environment is essential to operating your business effi-
ciently. With the growing list of automatic self-monitoring tools provided by Payara Server, you can
keep a close watch on possible issues that could negatively effect performance and take swift action
to remedy the situation.

  
[email protected] +44 207 754 0481 www.payara.fish

Payara Services Ltd 2016 All Rights Reserved. Registered in England and Wales; Registration Number 09998946
Registered Office: Malvern Hills Science Park, Geraldine Road, Malvern, United Kingdom, WR14 3SZ

29