User Guide

Payara Server in Production

The Payara® Platform - Production-Ready,

Cloud Native and Aggressively Compatible.
 Payara Server in Production

Introduction
Once you have developed applications on Payara Server and moved these applications into a pro-
duction environment, control will pass over to your operations teams. This guide will introduce
some features of Payara Server that you may not know about, which are especially useful for the
operations teams.

The guide covers the following sections:

• Script your Installation

• Production JVM Settings
• Viewing the Server Logs
• Thread Dump
• Operation Monitoring Services
• Request Tracing
• Healthcheck
• Slow SQL Logger
• Securing the server

Production Configuration
Script Your Installation
Payara Server comes with a very powerful administration client called asadmin. When moving to
production, asadmin can be used to create scripts to build out Payara Server domains in a repeat-
able fashion for production, preproduction and test environment. An example asadmin script for
building a domain is available from our examples repository; A simple example to create a domain
containing a separate deployment group in a Hazelcast cluster and an application deployed to that
deployment group:

asadmin create-domain –nopassword –portbase 10000 example-domain

asadmin start-domain example-domain
asadmin -p 10048 create-deployment-group sample-group
asadmin -p 10048 deploy --name=myapp --contextroot=myapp
--target=sample-group /path/to/myapp.war

1
 Payara Server in Production

Think About Your JVM Settings

Before you put Payara Server into production you should think about what JVM settings you need.
First and foremost you should use the latest Java 8 version available to you locally on your operating
system. Then you need to set your JVM settings Ideally you are looking for a balance of Heap Size
versus GC pause times.

Detailed guidance on a specific heap size depends on the needs of the application but some of the
guidelines below should be followed:

• Use the G1 collector for large heaps

• Set your min and max heap sizes to be the same to prevent FullGCs when the heap grows-
Xmx2048M -Xms2048M
• Set -XX:+DisableExplicitGC to disable running garbage collector from an application
• Configure garbage collection logging to enable future diagnostics;
• verbose:gc
• Xloggc:/path_to_log_file/payara-gc.log
• XX:+PrintGCDetails
• XX:+PrintGCDateStamps
• Ensure heap dumps are generated if you run out of memory
• XX:+HeapDumpOnPutOfMemoryError
• XX:+HeapDumpPath=/path_to_dump_file/Payara-dump.hprof

2
 Payara Server in Production

JVM Settings can be set in Payara from the administration console within the JVM Options settings
on the server configuration. Or the configuration can be incorporated into your script for building
domains using create–jvm–options and delete–jvm–options asadmin commands:

> asadmin delete-jvm-options -Xmx512m:-client

> asadmin create-jvm-options -Xmx2048m:-Xms2048m

Consider Using Built-in Production Domain

Payara Server comes with an alternative domain called production (or payaradomain in Payara Server
4), which contains configuration much more suitable for production than the default domain called
domain1. The default domain is not designed to be run in production, but instead to be a usable
example for development and testing purposes and it’s not recommended to use it as a template
for production settings.

3
 Payara Server in Production

In order to use the production domain, just execute the start-domain command with the argu-
ment production:

./asadmin start-domain production

To create a new domain based on the production domain, you can execute the following command:

asadmin> create-domain –template

${PAYARA_HOME}/glassfish/common/templates/gf/
production-domain.jar myNewProductionDomain

To create a new default domain based on the production domain, first, delete the default domain
domain1 and then create domain1 with the production-domain template, such as:

asadmin> delete-domain domain1

asadmin> create-domain –template
${PAYARA_HOME}/glassfish/common/templates/gf/
production-domain.jar domain1

The new domain can be then started without adding the domain name to the start-domain command:

./asadmin start-domain

More information about the production domain in Payara Server Documentation.

4
 Payara Server in Production

Investigation and troubleshooting

Viewing the Server Logs

When you are running Payara in production, especially if you are running a large number of serv-
ers in a domain across multiple hosts, you do not want to login to multiple machines to view and
search the log files. In Payara Server you can see, search and rotate the log files of any instance in
the domain from the administration console. Just display the page about the server and press the
“View Log Files” button. In the new window, you can select an instance in the domain and its log
file you want to examine:

5
 Payara Server in Production

You can also download any log file—select the Domain page, Domain Logs tab, select the instance
to download logs for and press “Collect Logs” in the top right corner to download them:

Get a Thread Dump

If a server appears to be running slowly you can get a Thread Dump for any server in the domain
from the administration console. The thread dump can then be analysed to see why a server is
running slowly.

6
 Payara Server in Production

You can get a thread dump from the Admin Console. Go to the server or instance you want to gener-
ate a thread dump for and press “JVM Report” in the General Information page. Then, in the new
window, select Thread dump from the “View” select box:

You can also generate a thread dump using the generate–jvm–report command:

asadmin generate-jvm-report --type=thread

7
 Payara Server in Production

A thread dump and a lot of other diagnostic information can also be attached to a support ticket if
Payara Server is connected to the Payara Support portal:

Payara Server Operations Services

Payara Server has several services which can be used to monitor the behaviour of Payara Server or
applications deployed to it. The first service to understand is the Notification Service. This service
provides many channels through which to receive monitoring data. There are several services which
can feed data through to the configured notifiers: the JMX Monitoring Logging Service which can
periodically log the values of specific JMX MBeans through the notification service; the Request
Tracing Service which can trace each request as it propagates through the application; and the
HealthCheck Service which can periodically check the status of various server and host metrics.

In addition to these services, there is also a slow SQL logger which operates independently. By
simply specifying a “slow” threshold on a data source, any query made through that data source
which takes longer than the threshold will cause both the problematic query and a stacktrace to
show where the query originated from to be printed to the server.log file.

8
 Payara Server in Production

The Notification Service

Payara Server comes with a general Notification Service in order to log events which come from
other services, such as the JMX Monitoring Logging Service, the HealthCheck Service or the Request
Tracing Service.

The Notification Service provides the ability to disseminate notification events through various chan-
nels that are being created by other services. This service is provided with a number of configurable
notifiers, the default being a log notification mechanism alongside these other notifiers:

• HipChat
• Slack
• NewRelic
• Datadog
• SNMP
• XMPP
• JMS
• Email
• CDI Event Bus

Because other operation monitoring services depend on the Notification Service to output their
data, both the Notification Service itself, and at least one notifier, must be enabled for any data to
be recorded. If this does not happen, then the data will be discarded.

It is for this reason that the Log notifier is the default notifier and is enabled out-of-the-box.

In order to configure a service like the HealthCheck Service to send notifications to a specific notifier,
the following configuration steps are needed:

1. Notification Service needs to be enabled

2. Notifiers in the Notification Service need to be configured to specify how to process
the notification
3. After configuring them, notifiers need to be enabled globally in the Notification Service
4. Notifiers also need to be selected (enabled) for each particular service that should send
them notifications

Notifiers have to be enabled both globally in the Notification Service and also for each service that
sends notifications.

9
 Payara Server in Production

Notifiers are configured on the Notification page of the Administration console:

The Notification Service can be configured with the notification-configure command.

To enable the Notification Service:

notification-configure --enabled=true --dynamic=true

The argument --dynamic is optional and applies the configuration without restart the server.

10