Installation and Configuration Guide

CYMON – CYONE Monitoring

Version 3.8
August 2020
 History of Changes
Date Version Author Changes
07.08.2020 3.8 Vladislav Tatarincev Added new chapter for
hosted environments, with
existing Zabbix or 3rd party
monitoring, some minor
changes
16.03.2020 3.7 Stanislav Svacko New Version of Document
Vladislav Tatarincev

Page 2 / 59
 Table of Contents

1 INTRODUCTION 5

2 SYSTEM REQUIREMENTS 7

3 PREPARATION WORK 8

4 CREATION OF CYMON VIRTUAL SERVER 10

5 ARCHITECTURE OF CYMON MONITORING 11

6 PARTS OF SYSTEM, PORTS AND LIST OF SYSTEM ACCOUNTS 12

7 SERVER TEMPLATES 15

8 REGISTRATION OF SERVER IN CYMON SYSTEM. 17

9 CONFIGURATION OF SETTINGS THAT ARE SPECIFIC TO YOUR DOMINO SERVER 19

10 GRAFANA POST CONFIGURATION 21

11 CREATING YOUR OWN METRICS FOR BUSINESS APP MONITORING 29

12 AUTOMATIC PROBLEM SOLVING 36

13 DEBUG 39

14 UPDATESTATS.NSF – CREATION OF OWN STATISTICS DOMINO 41

15 @FORMULA WIZARD 42

16 FREQUENTLY ASKED QUESTIONS 43

17 APPENDIX А – LIST OF ALL MACROS IN SYSTEM 51

18 APPENDIX B. 3RD PARTY SOFTWARE USED IN CYMON 57

Page 3 / 59
 This document describes installation and configuration of CYONE monitoring solution –
CYMON (CYONE Monitoring).

Ó CYONE, 2020. All Right reserved.

SIA CYONE
www.cyone.eu

Page 4 / 59
1 Introduction
CYMON Monitoring solution (further in document - CYMON) provided as virtual
appliance image for VMWare ESXi version 6.0 or above (VM version 11) as well HYPER-V
image. CYMON works on CentOS 7.7 and uses Zabbix (https://www.zabbix.com/) +
Grafana (https://grafana.com/) + LDSensor (developed by CYONE) + CYONE designed
best practice rules for Domino.

Solution objective is to monitor health of Domino

servers and notify Engineers about issues with visual
Grafana tools. By default, system work with one Domino
domain, but if necessary, system can be adjusted to work
with different domains, as well hosted environment or
non-crosscertified servers under different ID
files(additional information is provided by CYONE
representatives). Monitoring system can notify visually, as
well notifications can be configured to notify by email or SMS or popular messengers (for
example Telegram). Monitoring configuration to support SMS or SMTP is described in
Zabbix documentation:
https://www.zabbix.com/documentation/4.0/manual/config/notifications. Monitoring
keeps an eye on servers, but tuning of servers, administration, configuration is out of
monitoring scope. Our engineers are happy to help with these tasks, but this is not included in
monitoring license. Technical support of CYMON helps with issues related to CYMON only
and does not include consulting about problematic Domino servers.
Threshold values for triggering alerts are based on CYONE engineers experience that has
more than 23 years’ experience in Domino area. You are free to adjust these values based on
your experience and your environment, since every environment may have unique threshold
values.
CYMON presents information about servers in text format and also with the help of
graphical dashboards. For best user experience we recommend using large high resolution
1920 x 1080 (Full High Definition – Full HD) or more.
Page 5 / 59
 Default appliance works in “Free” mode, that reads only following statistics (show stat), and
databases:
1. Mail.*
2. Server.*
3. Mem.*
4. names.nsf
5. admin4.nsf
6. ddm.nsf
7. lndfr.nsf
8. domlog.nsf
9. inetlockout.nsf
Free version of CYMON show only portion of statistics and will notify only about issues in
range of allowed statistics. LNDFR (Lotus Notes Domino Fault Reports) database filename
must have default filename, otherwise reading data from this application will be disabled.
Custom names are supported in Enterprise version of product.
Enterprise or Corporate version of CYMON reads all metrics and has ability to autosolve
issues.

Page 6 / 59
2 System Requirements
Minimal system requirements:
• CPU – CPU 2 х 2.0GHz or higher
• RAM – 2GB (4GB or more recommended)
• HDD – image is supplied with preconfigured disks. 30GB disk is sufficient to monitor
10-15 Domino servers. Each additional Domino server requires to increase partition
/dev/mapper/vg_system-lv_root for +1 GB and /dev/mapper/vg_data-lv_data01 for
+2 GB (check CentOS documentation how to perform this task)

Image of CYMON supplied with preconfigured and up to date operating system when
build is released. For configuration of system according to your security standards you have
full root access, and you are responsible for applying security and other patches to OS.
CYONE does not provide technical assistance for OS nor further customization. We do
recommend ask CYONE before upgrading such components as Zabbix and Grafana, to
double check that update is compatible with all components.
CYMON has auto update module for getting newest templates, triggers, metrics.

Page 7 / 59
3 Preparation work
System requires to create a server.id (Domino Server ID). With the help of this file
CYMON will connect to Domino servers and will collect necessary data using LDSensor –
specially developed by CYONE micro-Notes client. Creation of server ID does not require
additional license, since physically such server will not exist, and server will not provide server
services.
Creation of Server.ID
Server ID is created in Domino Administrator. «Configuration» Tab, Tools -> Register -
> Server

We recommend use CYMON name, while you can choose any. In “Location for storing
server ID” select «In file». File server.id must be create without password, due to some API
specifics, so that LDSensor client can initiate session in multi-thread mode.

When server is registered, status bar will say that registration is successful.

Page 8 / 59
 This server name must have at least Reader ACL rights to the following databases:
o names.nsf
o admin4.nsf
o LNDFR
o domlog.nsf
o ddm.nsf
o userlicenses.nsf.
Server name/Server.ID that is created for CYMON, must be added to Full Remote
Console Administrator field, so CYMON can perform corrective actions (e.g. run a
command).

Page 9 / 59
4 Creation of CYMON virtual server
CYMON server is imported into virtualization based on provided CYMON_latest.ova file.
For testing purposes monitoring can be deployed even on VMWare Player. Time of import
may take 3-7 minutes. System does not require Internet Access, while for OS patches you
might need it later, or you can use local package repository.

Some versions of VMWare may interrupt import, because of OVA/OVF file

(manifest file), in such case unZIP OVA file (it is a regular ZIP archive) and import
all extracted files except manifest file.
By default, CYMON gets IP address by DHCP. If needed you may change with the help
of CentOS 7 documentation https://wiki.centos.org/Documentation.
Before you continue configuration, please confirm that server has:
• IP Address
• This rerves has DNS configured and Domino servers can be reached by 1352 port
and domino server host names are resolved

Page 10 / 59
5 Architecture of CYMON monitoring
CYMON consist of some components that work together. It might help to understand role
of each component:
1. Linux – CentOS virtual machine on which CYMON runs
2. Zabbix – Opensource monitoring solution (open source)
3. Grafana – Opensource solution that visualize data received from Zabbix
4. LD Sensor – micro Notes client, that allow Monitoring to read data from Domino,
or access Notes databases. It works via NRPC protocol with Domino server over
standard 1352 port. With the help of this component data is gathered remotely
without need of local agent. Thus, any Domino OS is supported, as soon it is
supported by vendor.
5. Notes C API – Domino libraries. «LD Sensor» user Notes API to access Domino,
thus we need some non-production libraries from Domino installation.
6. UpdateStats.nsf – optional module for monitoring. With the help of this module
customers can create their own statistics from business apps. It is described in more
details in a separate chapter.
7. Formula Wizard – optional database, that helps to create @Formulas, that later will
be used in Monitoring. This component can be run on Administrators workstation.

UpdateStats.nsf
Formula Wizard Wizzard
Domino Server

Notes C LD Sensor
Domino Server
Zabbix Grafana
Linux

Page 11 / 59
6 Parts of system, ports and list of system accounts
Address and Port Description
http://ip.add.re.ss/ Zabbix server
User «cymon»
Password «cymon»
In older version (2.x) as login was «Admin» and
password «zabbix», starting CyMON 3.0
recommendation is to use cymon/cymon
http://ip.add.re.ss/config/ CYMONCTL (СYMON CONTROL) –
Interface for basic configuration of monitoring and
server configuration
User «cymon» (OS authentication)
Password «cymon»
http://ip.add.re.ss:3000/ Grafana – visual representation of data
User «cymon»
Password «cymon»
ssh ip.add.re.ss:22 ssh access to system
User «root» and/or «cymon»
Password «cymon»

While using system, you may notice that Zabbix has CYMON user (Uppercase) – this is
system user, that Monitoring system use, to integrate different components. Do not change
password for this user, unless you are sure what you do or instructed by CYONE support.
Changing this password will break Zabbix and Grafana integration, as well as CYMONCTL
(config).

Page 12 / 59
 Ports of system
Port Description
:1352 (outgoing) Standard Domino port, system use it to access Domino
servers
:3000 (incoming) By this port Grafana is available, Grafana is main
interface of visual monitoring
:80/443 (incoming) Zabbix interface
:80/443 (outgoing) CYMON uses this port to connect to Traveler and verify
Traveler availability. If Traveler is available on different
port, you have to configure custom port via system
macros, as well protocol (http/https). See description In
Appendix А.
Outgoing 80/443 ports can be used to configure outgoing
notifications to messengers, for example Telegram
25 (outgoing) Used for sending SMTP emails, if notifications are
configured
Ping (outgoing) CYMON must have ability to use ping on network
level
53 (outgoing) System must have access to local DNS server by 53
port

6.1.1 Upload of ID into CYMON

When Server.ID is created, it must be uploaded into CYMON with the help of
CYMONCTL.
You can access CYMONCTL on CYMON server by the following address
http://ip.add.re.ss/config/

To access system, you must use «cymon» user and «cymon» password lower case (users of
CYMONCTL are created on Linux OS level).
After authentication you will see main screen to configure all necessary options.

Page 13 / 59
 In CYMON-ID tab you Upload just registered Server.ID. Under this ID monitoring will
work. Use tab «Upload new serverid file»:

After you upload you will see green notifications on a screen.

In case you issue connection termination after you start upload, investigate
your antivirus settings or WAF (Web Application Firewall) that may control
traffic in network. In some cases, duplicate IP address on Network (this normally
should not happen J ) also result in strange behaviour.

Page 14 / 59
7 Server Templates
Depending on Domino server role, Monitoring will check more or less rules. For example,
names.nsf and admin4.nsf monitoring check only for one server that has Admin_Server role,
since such server in Domino domain is only one. Or if server works in cluster, cluster specific
statistics are checked only if server has corresponding role. Templates (roles) are set during
server registration but can be modified later. Some roles include other roles, some other roles
are independent.

№ Role
1 CYMON_Admin_Server Administrative server of Domino. From all
monitored server only one should have such role.
From this server, monitoring will gather specific
domain statistics, number of users, number of groups,
number of policy documents or pending admin
requests, as well many other domain specific statistics.
(this role does not include CYMON_Domino_Only
role)
2 CYMON_Domino_Only Domino core server, that is not http or traveler
server
3 CYMON_Domino_Traveler Domino server, that also has Traveler and HTTP
tasks running (includes CYMON_Domino_Only and
CYMON_Domino_Web)
4 CYMON_Domino_Web Domino server on which HTTP is running.
(includes role CYMON_Domino_Only)
5 CYMON_Domino_Cluster Additional role for checking Cluster specific
metrics.
(Role is additional and does not include
CYMON_Domino_Only)
6 CYMON_Sametime Optional role to check HCL Sametime

Page 15 / 59
 7 CYMON_LNDFR Server on which “Lotus Notes/Domino Fault
Reports” is located. If you server use LNDFR
database and filename is different from lndfr.nsf in
data root), you must adjust name by changing
corresponding macros {$LNDFR.DB.NAME}. See
Appendix А
8 CYMON_SMSLogin Special template for customers using CYONE
2FA – 2 Factor authentication for HTTP/Verse on
Prem
https://www.cyone.eu/2FA
9 CYMON_Custom This is customer template for customer making
their own changes. Only this template must have
custom settings and will not be overwritten during
newer versions of CYMON.
Please notice that some, templates are mutually excluding:
• CYMON_Domino_only
• CYMON_Domino_web
• CYMON_Domino_Traveler
That means that only one template can be assigned to server.
Template/Role CYMON_Admin_Server may be assigned to only one Domino server in
domain. If you need to monitor several domains contact CYONE for CYMON configuration.

Page 16 / 59
8 Registration of server in CYMON system.
Registration of new monitoring host is performed with the help of CYMONCTL. (
http://YourServer/config/” URL)
In «Target Hosts Tools», use «Prepare server host» form to register new host in CYMON.

There are two ways to register servers in CYMONCTL:

1. Fill form for each server in «Prepare server host» — and for each server HSF-file
(HSF – Host Specification File) will be prepared, then this file will be imported into
monitoring.
2. For Batch import, you can generate HSF files and import them. One file may contain
specification for many files.
First way to do registration of the server is most commonly used, in field of templates you
have to choose corresponding templates to this server.:

Field “Visible name” must contain friendly name of server, for example Traveler or Mail
or Apps05, field “Host DNS name” - Fully qualified Internet host name exactly as it is written
in Domino server document.

Page 17 / 59
 Use Domino FQDN name, otherwise, you may have issues with zero data received from
Domino server. Verify settings with «Submit» button.

If you click «submit» host for verification and process takes 10-20 seconds and
end up with warning, that host cant be resolved you may need to double check
DNS settings. It is likely your DNS servers are not properly setup. Check
/etc/resolv.conf if values of DNS are correct.
After host specification is entered, you can import prepared specification into monitoring
by clicking on “Import into monitoring system”

Page 18 / 59
9 Configuration of settings that are specific to your Domino Server
If on your Domino server run HTTP or Traveler tasks, or if server is responsible for
processing Lotus Notes Domino Fault Reports(analysis of NSD crash files), we have to define
which ports or which filenames monitoring should use to check health of these servers. To do
so, we have to change values of macros defined in system, from default values to your custom
values. For example:
Macro Description
{$HTTPPROTOCOL} Protocol for checking web server HTTP or HTTPS
{$HTTPPORT} Port of web server
{$TRAVELERPROTOCOL} Protocol for checking Traveler HTTP or HTTPS
{$TRAVELERPORT} Port on which Traveler is configured
{$TRAVELERLOGIN} Login, under this account monitoring can login and check that not
only port is opened, but servlet Traveler is available
{$TRAVELERPASSWORD} Password for $TRAVELERLOGIN account
{$LNDFR.DB.NAME} Name and filepath of Fault Reports database

To change default values of macros, go to Zabbix interface http://ip.add.re.ss (or

hostname) and open list of macros for particular host: Configuration Þ Hosts, then select
CYMON group, then select needed server from the registered server list.

Page 19 / 59
 In properties of server select Macros Þ Inherited and host macros. From the list of variables
find needed macro and click «Change», then in field «Effective value» enter new value of
variable, after changes apply changes with «Update».

List of all variable (macros) and default values is described in Appendix А – List of all
Macros in system.

Page 20 / 59
10 Grafana post configuration
Visual dashboards are created with the help of Grafana.

When hoovering over «i» you will see what this graph does as well as some explanation.

For currently opened issues, you can get insight how frequent is this problem, how long it
is opened, is this is a reoccurring event. You can do this by clicking « i » in list of issues.

Page 21 / 59
 You access different dashboards by choosing one from list of available dashboards:

«Management» dashboard is default dashboard when «cymon» user access system.

Informational panel must be configured for each individual server. Out of the box 3 panels
are shipped, while you can clone exiting ones and change according to your needs.
To make another copy of panel, click More Þ Duplicate.

Page 22 / 59
 To configure panel for server or group of servers, click on server title and then «Edit».

In data source tab (see picture below), adjust list of metrics (5 by default but you can add
more complicated rule of server is ok, or in trouble) to select proper server from drop down.
Server must be registered to appear in dropdown list.

Page 23 / 59
 Then on «Visualization» tab, in field «Panel Title» enter server name, then click «Save» or
Left Arrow «» to save changes and quit edit mode. You must change server name in 3 tabs of
dashboard.

After you save changes to dashboard, you will see management dashboard that represents
status of your servers. Start with default, but then you can extend more criteria and make more
sophisticated rules for server availability.

Dashboards «Mail», «Performance», «Network», «Server», «Users» и «CYMON» does not

demand configuration and work out of the box, they might require few hours to gather all
data.

Page 24 / 59
 CYMON is shipped with «Traveler» dashboard, which contains two panels for clustered
Traveler servers. This dashboard must be slightly adjusted to provide more valuable
information, according to your deployment.

To Configure status panel, for each Traveler node, click on dashboard title, and «Edit» and
select propriate node of Traveler. Be sure to change server name in all tabs. Traveler
homepage open dashboard in addition to server name, requires also Traveler login and
password that will allow system to login, check traveler servlet availability and show graphs.
If you run only one server, you can remove second “Traveler” row, if later you add one,
you can duplicate widgets from first row.

Page 25 / 59
 To check if Traveler has sufficient number of threads, you must change «Active Devices vs
HTTP Threads», «Traveler homepage open», «Traveler Device Sync 200». Below you see
how to adjust these graphs:

Page 26 / 59
Page 27 / 59
 Informational dashboard «Databases» require to duplicate some status panels, to monitor
DAOS of Domino server. Each panel consist of two sections. CYMON is shipped with two
panels for two Domino servers. More panels can be added to have needed number of servers.

In settings of each panel, you must select proper Domino server and replace «Title» with
Server name. DAOS synchronization widget must be configured as well to provide
information of DAOS engine state.

Page 28 / 59
11 Creating your own metrics for business app monitoring
In addition to Domino CYMON can monitor also business applications. There are
different ways, to monitor business app. You can track number of documents in a particular
view (request count) and put different triggers on it, processed number of documents should
always grow, or if this is error log it should be static(not grow). Another example is unprocessed
number of documents (starting status), should be less than X, if exceeded then alarm. CYMON
can monitor speed how fast database and view opens for users. Or you can even write your
own @Formula that is executed and get result from your business app.
Idea is to put different sensors or probes and monitoring will check these sensors 24/7.
There is one more way to even create application statistics, it will be covered in next chapters.
Let take few examples, so you get flavor of what can monitoring do with app:
We will create our own metric using two different examples:
1. To gather some statistics, it is required in in «CYMON_custom» template add needed
new metrics. For example, you want to monitor, number of CPU on server, then we
have to read Server.CPU.Count statistic from Domino.
In «CYMON_custom» create new metric with the following values:

On screen above, highlighted with red, we marked fields you need to fill.

Page 29 / 59
 FIELD NAME DESCRIPTION
Name Name of metrics
Key Metrics system name, we suggest using same name as in «Name»
Server {HOST.DNS}
Since value will be read from different servers, we have to type
variable so Domino FQDN is used {HOST.DNS}
Facility First part of statistics, you enter part that is before dot «.»
Stat Remaining part of the string after the dot «.»
Convert Bit mask. Depending on returned value, and some special
characters data may need to be cleaned so monitoring can get expected
value. Some values are returned by Domino as «123 234 344.56»,
while for program this may be not a valid number, as well decimal
delimiter may differ. If you are not sure with one convert flag you need,
start with one (1).

Pay attention, that facility, in our example «Server» is first part of the Domino full
statistics name, but value of «stat» field «CPU.Count» is remaining part of full name.
Value of convert field, is a bit mask:
• 1 – delete non "non-ASCII" symbols
• 2 – delete spaces and quotes (double “ and single ‘)
• 4 – delete "," and "."
• 8 - replace "," with "."
More example:
• delete "non-ASCII" symbols: convert=1
• delete "non-ASCII" symbols, delete quotes, delete commas, and dots and spaces.
convert = 7 (1 + 2 + 4)
• delete "non-ASCII" symbols, quotes, spaces, quotes and replace comas with dots.
convert = 11 (1+2+8)

Page 30 / 59
 Scroll down Zabbix window and specify frequency, how often to collect this metric from
Domino:

CYONE recomends to avoid polling Domino too frequently, 3 minutes or less frequent
time should be ok.
After making changes to template, template must be assigned to server. In Zabbix interface
you have to find needed server:

When needed server is located, open Templates section, and in field «Link new templates»
select «CYMON_custom» and press Enter, then «Add», and finaly click «Update».
Page 31 / 59
 After some time (depends how frequently you selected to read this metric from Domino)
this metric can be depicted in Grafana. For example, graph can be created. Open needed
panel, and click button «Add panel»:

Then «Choose Visualization»:

Then, while Graph is still selected, press Queries on the left:

Page 32 / 59
Fill «Queries» in the following way:

Page 33 / 59
 As you see, Grafana builds graph automatically, based on data received from Domino. We
are almost done, we have to define Graph name in «Panel Title» and may be we will move
graph itself to some area in Dashboard.

Example 2. Number of Documents in a view

One more example of creating metric. To create a metric, that reads number of documens
in particular view. To do this, we open in Zabbix interface CYMON_custom template and
find metric NAB.Servers.Count2 and while in edit mode, we click «Clone»:

Page 34 / 59
 New metric will be created, and you have to make minor changes to achieve your goal,
change fields marked in red as on picture below:

Then click «Add» at the bottom of the screen.

Example 3. @Formula
To create your own metric, that calculates something based on @Formula, repeat same
steps as in example 2, but use another metric as an template for clone
«NAB.Users.IDAttached2».

After making changes, template must be assigned to proper server, if not yet done, as in
previous example.

Page 35 / 59
12 Automatic problem solving
CYMON monitoring system is able to fix issue, if resolution scenario is defined. These
commands/fix scenarios are fired by monitoring trigger. System is shipped with one predefined
trigger, that allow to synchronize DAOS catalog in case DAOS engine is in «Need Resync»
state. This does not require any additional software to be installed on the server. This feature
is available only in Enterprise version of CYMON (Free version of CYMON does not have
such functionality).
Using DAOS Resync trigger as an example, you can define to run any agent, any command
if corresponding trigger has been executed on other metric. For configuring automatic
command, you must use Zabbix interface.
Trigger logics of «tell daosmgr resync» is the following:
When last but one (-2) value and last value of DAOS.Engine.Catalog has value «Needs
Resync» trigger will be fired.
{CYMON_Domino_only:DAOS.Engine.Catalog.str("Needs Resync",#2)}=1 and
{CYMON_Domino_only:DAOS.Engine.Catalog.str("Needs Resync",#1)}=1,

It will open new issue with severity «Problem», and then with Zabbix Þ Actions, configured
the following way:

Page 36 / 59
 This command will be fired by monitoring system after 30 minutes when DAOS engine is
in this state. If needed you may adjust this time.

Page 37 / 59
On third screen you see that «tell daosmgr resync» script will be launched by shell file
/usr/lib/zabbix/externalscripts/start_ldsensor.sh cmd {HOST.DNS} "tell daosmgr resync",

where «tell daosmgr resync» - is same command as we do it in Domino.

Page 38 / 59
13 Debug
You may run into different issues, while using Monitoring. To locate issue, monitoring has
utility for verification of setting and configuration. To run this command, connect via ssh with
«root» or «cymon» user, and then run:
/usr/lib/zabbix/externalscripts/debug.sh

Debug utility will check different configuration settings and will try to help to locate a
problem, or which setting is incorrect. For example:
1. IP address
2. DNS server
3. Result of ping command (IP cant be resolved)
4. Ability of LDSensor to connect to server by IP address
5. Ability of LDSensor connect to Domino by FQDN

When running debug.sh, first you will see some OS settings:

Page 39 / 59
 After this you will see an attempt to connect with LDSensor:

Try to do first level analysis of «debug.sh» output and fix configuration. In case you cant
fix issue, get in touch wit CYONE technical support team that will be happy to assist
([email protected]).

Page 40 / 59
14 UpdateStats.nsf – creation of own statistics Domino
СYMON system has optional download – NSF database «Update Stats». With the
help of this database, you can monitor also business applications, by creating business
statistics in Domino statistics tree. For example, you can track how fast documents or
databases are opened by users or search performed. Another example is number of
documents in particular view (number of requests from customers). СYMON can
execute @Formula queries, as well as amount of documents in a view, but these requests
are not so light as already prepared statistic. In a heavy loaded and large system, it make
sence to prepare statistics so monitoring can just read them. Last version of NSF
database «StatsUpdate.ntf» is available here
https://cyone.box.com/v/StatsUpdateNTF
Database has test agent and script library that allow to create statistics with one line
of code (one function). More detailed information and how to use this database is
described in Help ÞUsing This Application.
15 @Formula Wizard
CYMON Allows not only gather statistics, but also execute @Formula queries.
System supports @Formulas, to select documents without need to create a Domino
views.
For ease of using formulas and debug of formulas, we created @Formila Wizard
database. Create new document, select any view that is close to what you want to get. It
will show a formula for existing view. Then customize this formula in Custom Formula
field. You can test this formula by clicking «Try This formula» button. First 10
documents of a view will be displayed or you can open these documents in UI to verify
if the correspond to formula. Later this formula is copied to monitoring, and formula is
executed on scheduled base.

1. Example of formula, that shows users with attached user IDies in Domino Directory.

CYMON monitoring dashboard «Server» has link to download this database, or it

can be downloaded from https://cyone.box.com/v/FormulaWizard

Page 42 / 59
16 Frequently asked questions
Question 1: many metrics in Zabbix are shown as gray, many triggers has status
«Unknown» Unknown. Any hint for reason of that?
Answer 1: Most possible reason for that, is that Zabbix did not get data in expected
format. For example, got empty string instead of number, or decimal delimiter was
comma while expected «.».
Another possible reason, that Free version is used, that does not allow to read
advanced values. In such case Zabbix will get empty string.
If you still experience No Data issue, please get in touch with CYONE support, we
will get verify your configuration and fix issue.

Question 2: There are triggers about mail, what has caused to them to report a
problem? How can I get more details on this?
CYMON_Domino_only: Dead mail: {ITEM.LASTVALUE} > {$MAIL.DEAD}
CYMON_Domino_only: Hold mail: {ITEM.LASTVALUE} >=
{$MAIL.HOLD}
Answer 2: in this case, check contents of mail*.box databases, you will see
documents that cause triggers to execute.

Вопрос 3: How Server availability index works? I see value always below 30.
CYMON_Domino_only: Server.AvailabilityIndex low: {ITEM.LASTVALUE} <
{$SERVER.AVAILABILITYINDEX}
Ответ 3: Server.Availability is described in HCL documentation:
https://help.hcltechsw.com/domino/10.0.1/admn_theserveravailabilityindex_t.ht
ml
Trigger changes state into «Problem», when value of metrics
Server.AvailabilityIndex becomes less, then value of macro
{$SERVER.AVAILABILITYINDEX}. СYONE’s created rules for CYMON is a list
of IBM and HCL best practice that is checked against your server 24/7.

Page 43 / 59
 Question 4: how to configure to monitor another disk partition?
Answer 4: to configure new partition of disk click on a metric and edit it. Lets take
example how to do it on Linux.

Lets assume, you have different partition, than /local - Disk./local.*. To adjust it and
keep logic of Disk./local.Used.Pct, you must make changes to all three (3) metrics:
Disk./local.Free, Disk./local.Size и Disk./local.Used.Pct. Click on name of metrics, and
then click Clone at the bottom of a screen, this will create new metric for particular
server (based on metric from which it was cloned):

Page 44 / 59
 You have to change here:
1.1. field Name, for example, to Disk./dominonotesdata.Free
1.2. field Key to Disk.dominonotesdata.Free
1.3. Value for «stat» to /dominonotesdata.Free
2. Do same for Disk./local.Size.
3. Then clone and edit Disk./local.Used.Pct so it looks like on screen below

Page 45 / 59
 Question 5: why triggers has priority High and what does it mean?
CYMON_Domino_only: Database.NAMELookupCacheHitRate too high:
{ITEM.LASTVALUE} >
{$DATABASE.NAMELOOKUPCACHEHITRATE.CRITICAL.PERCENT}
CYMON_Domino_only: Database.RM.SinceStartup.Critical.Log.Times:
{ITEM.LASTVALUE} >
{$DATABASE.RM.SINCESTARTUP.CRITICAL.LOG.TIMES.COUNT}
Answer: Based on experience of CYONE people, we classify this trigger with
«High» priority. If for your environment, you believe another priority is needed, you
can adjust it since all metrics are free to adjust.
See HCL documentation on Database.NAMELookup:
Monitoring the database cache
https://help.hcltechsw.com/domino/10.0.1/tune_monitoringthedatabasecache_r.html

Page 46 / 59
 Metrics Database.RM.SinceStartup.Critical.Log.Times says, how many times
transactional log was full, from Domino start. Engineers of CYONE believe that
transactional log should never be full, since it may crash server, impact ability to perform
normally. Thus value of macro
{$DATABASE.RM.SINCESTARTUP.CRITICAL.LOG.TIMES.COUNT} is zero
0 by default.
Database.RM reference Guide
http://publib.boulder.ibm.com/tividd/td/TDSdomino/GC32-0675-
00/en_US/HTML/refmst25.htm (this link may become dead since IBM is removing
documentation. If this happens, search for Database.RM. Reference Guide)

Question 6: how following metric is calculated and for which purposes?

NAB.Users.Expiring in {$NAB.USERS.EXPIRING.DAYS} days:
{ITEM.LASTVALUE} > 0
Answer 6:
This metrics helps to prolong user certificates and avoid situation when certificate is
expired for some users and users are unable to access servers. CYMON has template
CYMON_sys_Macros in which all macros are defined globally for all servers, and there
you can change notification period for certificate expiration:

Page 47 / 59
 Macro that we need to adjust is:

If you change Macro value in template CYMON_sys_Macros, changes will affect all
monitored Domino hosts, unless macro for specific server is defined.
To change {$NAB.USERS.EXPIRING.DAYS} only for particular host
«Macros» Þ «Inherited and host macros» click «Change»:

More information on how macros work find in Zabbix documentation:

https://www.zabbix.com/documentation/current/manual/config/macros/usermacros

Thus, trigger works the following way: if last received value of NAB.Users.Expiring
metrics more than value defined in macro {$NAB.USERS.EXPIRING.DAYS}, then
trigger will turn into «Problem» state.

Page 48 / 59
 Take a look at @Formula with the help of which value of NAB.Users.Expiring is
calculated:
Type = 'Person' & ((@Date(@Certificate([Expiration];Certificate)) - @Date(@Today)) /
86400) > 0 & ((@Date(@Certificate([Expiration];Certificate)) - @Date(@Today)) / 86400) <
{$NAB.USERS.EXPIRING.DAYS}

Question 7: if you have to change to modify templates for CYMON, what should
I know? Will my changes be lost during next update of CYMON?
Answer 7: CYMON is shipped with the following templates:
CYMON_Admin_Server
CYMON_custom
CYMON_Domino_cluster
CYMON_Domino_only
CYMON_Domino_web
CYMON_Domino_Traveler
CYMON_Domino_web
CYMON_LNDFR
CYMON_SMSLogin
CYMON_sys_Macros
CYMON_sys_NRPC
CYMON_sys_SSL_Check
CYMON_sys_Traveler_Web_Check
CYMON_sys_Web_Check
The only template this will be not updated is – CYMON_custom, that contains
samples for customer metrics. Best practice is to have your custom settings in
CYMON_Custom template. While you can do modifications also in CYONE provided
templates, and assigning metric a unique non conflicting name, like
ZYX_Customer.Mail.Waiting, will still be ok and will be not overwritten during
upgrade. To avoid confusion, we strongly suggest to use CYMON_Custom template

Page 49 / 59
 Question 8: How to change timezone in CYMON OS?
Answer 8: list of available timezones you can get by typing Cent OS 7 command:
timedatectl list-timezones.

Then needed timezone is set by following command

timedatectl set-timezone your_time_zone

for Example,
timedatectl set-timezone Europe/Moscow

Or
timedatectl set-timezone Europe/London

Question 9: how to reset a Grafana user password?

Answer 9: connect via ssh, make a backup before you do any changes of
/var/lib/grafana/grafana.db
Then run a command
sqlite3 /var/lib/grafana/grafana.db "update user set password =
'b03333f9cfad69a14598676b2f432df645b993ce54c40422be65e10a94ed09af4286d8924215eea84b5df3447
c794015105f', salt = 'i9nPFG4PT8' where login = 'cymon';"

this will reset password of «cymon» user to «cymon».

Page 50 / 59
17 Appendix А – List of all Macros in system
List of all macros in system:
Macro name Default Description
{$AGENT.DAILY.ACCESSDENIALS.HIGH} 30 Trigger Too much Domino Daily Agents run denials: {ITEM.LASTVALUE} >
{$AGENT.DAILY.ACCESSDENIALS.HIGH}
{$AGENT.DAILY.ACCESSDENIALS.INFO} 0 Trigger Too much Domino Daily Agents run denials: {ITEM.LASTVALUE} >
{$AGENT.DAILY.ACCESSDENIALS.INFO}
{$AGENT.DAILY.ACCESSDENIALS.WARN} 10 Trigger Too much Domino Daily Agents run denials: {ITEM.LASTVALUE} >
{$AGENT.DAILY.ACCESSDENIALS.WARN}
{$AGENT.DAILY.UNSUCCESSFULRUNS.INFO} 0 Trigger Too much Domino Daily Agents Unsuccessful Runs: {ITEM.LASTVALUE}
> {$AGENT.DAILY.UNSUCCESSFULRUNS.INFO}
{$AGENT.DAILY.UNSUCCESSFULRUNS.WARN} 30 Trigger Too much Domino Daily Agents Unsuccessful Runs: {ITEM.LASTVALUE}
> {$AGENT.DAILY.UNSUCCESSFULRUNS.WARN}
{$BUFFERPOOL.PERCENT} 90 Trigger Database.Database.BufferPool.PercentReadsInBuffer
{$DATABASE.DBCACHE.CURRENTENTRIES.PCT.I 90 Trigger Database.DbCache.CurrentEntries: {ITEM.LASTVALUE} is more than
NFO} {$DATABASE.DBCACHE.CURRENTENTRIES.PCT.INFO}% of
Database.DbCache.MaxEntries: {ITEM.VALUE2}
{$DATABASE.DBCACHE.CURRENTENTRIES.PCT. 95 Trigger Database.DbCache.CurrentEntries: {ITEM.LASTVALUE} is more than
WARN} {$DATABASE.DBCACHE.CURRENTENTRIES.PCT.WARN}% of
Database.DbCache.MaxEntries: {ITEM.VALUE2}
{$DATABASE.DBCACHE.EFFECTIVENESS.CRITIC 10 Trigger Database.Dbcache.Effectiveness is too low: {ITEM.LASTVALUE} <
AL.PERCENT} {$DATABASE.DBCACHE.EFFECTIVENESS.CRITICAL.PERCENT}%
{$DATABASE.DBCACHE.EFFECTIVENESS.WARNI 20 Trigger Database.Dbcache.Effectiveness is too low: {ITEM.LASTVALUE} <
NG.PERCENT} {$DATABASE.DBCACHE.EFFECTIVENESS.WARNING.PERCENT}%
{$DATABASE.DBCACHE.OVERCROWDINGREJEC 0 Trigger Database.DbCache.OvercrowdingRejections: {ITEM.LASTVALUE} >
TIONS.COUNT} {$DATABASE.DBCACHE.OVERCROWDINGREJECTIONS.COUNT}
{$DATABASE.NAMELOOKUPCACHEHITRATE.CRI 90 Trigger Database.NAMELookupCacheHitRate too high: {ITEM.LASTVALUE} >
TICAL.PERCENT} {$DATABASE.NAMELOOKUPCACHEHITRATE.CRITICAL.PERCENT}
{$DATABASE.NAMELOOKUPCACHEHITRATE.WA 80 Trigger Database.NAMELookupCacheHitRate too high: {ITEM.LASTVALUE} >
RNING.PERCENT} {$DATABASE.NAMELOOKUPCACHEHITRATE.WARNING.PERCENT}
{$DATABASE.RM.SINCESTARTUP.CRITICAL.LOG. 0 Trigger Database.RM.SinceStartup.Critical.Log.Times: {ITEM.LASTVALUE} >
TIMES.COUNT} {$DATABASE.RM.SINCESTARTUP.CRITICAL.LOG.TIMES.COUNT}
{$DBMT.COMPACT.UNSUCCESSFUL} 0 Trigger DBMT.Compact.Unsuccessful: {ITEM.LASTVALUE} >
{$DBMT.COMPACT.UNSUCCESSFUL}
{$DISK.C.USED.PCT} 90 Trigger HDD C:\ almost full > {$DISK.C.USED.PCT}%
{$DISK.D.USED.PCT} 90 Trigger HDD D:\ almost full > {$DISK.C.USED.PCT}%
{$DISK.LOCAL.USED.PCT} 90 Trigger HDD /local almost full > {$DISK.LOCAL.USED.PCT}%
{$DISK.ROOT.USED.PCT} 90 Trigger HDD / almost full > {$DISK.ROOT.USED.PCT}%
{$DOMLOG.INVALIDLINKREQUESTS} 20 Trigger HTTP task: invalid requests high
{$DOMLOG.REQUESTS.COUNT.PCT.THR} 1000 Trigger Domlog.Requests.Count avg(15m) grow faster for
{$DOMLOG.REQUESTS.COUNT.PCT.THR}% than avg(6h)
{$FT.TIME.VS.ELAPSED.TIME.PERCENT} 25 Trigger FT time vs Elapsed time: {ITEM.LASTVALUE} >
{$FT.TIME.VS.ELAPSED.TIME.PERCENT}%
{$HTTP.INETLOCKOUT.COUNT} 0 Trigger Http.Inetlockout: {ITEM.LASTVALUE} >
{$HTTP.INETLOCKOUT.COUNT}
{$HTTPPORT} 443 Port for checking availability of HTTP server Domino
{$HTTPPROTOCOL} https Protocol (http or https for checking) availability of HTTP
{$ICMP_LOSS_WARN} 20 Trigger High ICMP ping loss >{$ICMP_LOSS_WARN} <100 (large % of lost
packets)
{$ICMP_RESPONSE_TIME_WARN} 0.15 Trigger High ICMP ping response time > {$ICMP_RESPONSE_TIME_WARN}
(in seconds)

Page 52 / 59
{$LDAP.FAILEDLDAPCONNECTIONS} 0 Trigger LDAP.Failed LDAP Connections growth by {ITEM.LASTVALUE}
{$LNDFR.DB.NAME} lndfr.nsf Path to Notes/Domino Fault Reports database
{$LOGICALDISK.1.AVGQUELEN.AVG.PERCENT} 1 Trigger Platform.LogicalDisk.1.AvgQueLen.Avg: {ITEM.LASTVALUE} >
{$LOGICALDISK.1.AVGQUELEN.AVG.PERCENT}
{$LOGICALDISK.2.AVGQUELEN.AVG.PERCENT} 1 Trigger Platform.LogicalDisk.2.AvgQueLen.Avg: {ITEM.LASTVALUE} >
{$LOGICALDISK.2.AVGQUELEN.AVG.PERCENT}
{$LOGICALDISK.3.AVGQUELEN.AVG.PERCENT} 1 Trigger Platform.LogicalDisk.3.AvgQueLen.Avg: {ITEM.LASTVALUE} >
{$LOGICALDISK.3.AVGQUELEN.AVG.PERCENT}
{$LOGICALDISK.4.AVGQUELEN.AVG.PERCENT} 1 Trigger Platform.LogicalDisk.4.AvgQueLen.Avg: {ITEM.LASTVALUE} >
{$LOGICALDISK.4.AVGQUELEN.AVG.PERCENT}
{$LOGICALDISK.5.AVGQUELEN.AVG.PERCENT} 1 Trigger Platform.LogicalDisk.5.AvgQueLen.Avg: {ITEM.LASTVALUE} >
{$LOGICALDISK.5.AVGQUELEN.AVG.PERCENT}
{$LOGICALDISK.6.AVGQUELEN.AVG.PERCENT} 1 Trigger Platform.LogicalDisk.6.AvgQueLen.Avg: {ITEM.LASTVALUE} >
{$LOGICALDISK.6.AVGQUELEN.AVG.PERCENT}
{$MAIL.DEAD} 0 Trigger Dead mail: {ITEM.LASTVALUE} > {$MAIL.DEAD}
{$MAIL.HOLD} 10 Trigger Hold mail: {ITEM.LASTVALUE} >= {$MAIL.HOLD}
{$MAIL.MAILBOX.ACCESSCONFLICTS.PCT} 2 Trigger Mail.Mailbox.AccessConflicts: {ITEM.VALUE1} is more than
{$MAIL.MAILBOX.ACCESSCONFLICTS.PCT}% of Mail.Mailbox.Accesses:
{ITEM.VALUE2}
{$MAIL.MAILBOX.ACCESSWARNINGS.PCT} 10 Trigger Mail.Mailbox.AccessWarnings: {ITEM.VALUE1} is more than
{$MAIL.MAILBOX.ACCESSWARNINGS.PCT}% of Mail.Mailbox.Accesses:
{ITEM.VALUE2}
{$MAIL.TOTALPENDING.HIGH} 30 Trigger Pending Mail Messages: {ITEM.LASTVALUE} >=
{$MAIL.TOTALPENDING.HIGH}
{$MAIL.TOTALPENDING.WARNING} 10 Trigger Pending Mail Messages: {ITEM.LASTVALUE} >=
{$MAIL.TOTALPENDING.WARNING}
{$MAIL.WAITING} 10 Trigger Mail.Waiting: {ITEM.LASTVALUE} > {$MAIL.WAITING}

Page 53 / 59
{$MAIL.WAITINGFORDIR} 0 Trigger Mail.WaitingForDIR: {ITEM.LASTVALUE} >
{$MAIL.WAITINGFORDIR}
{$MAIL.WAITINGFORDNS} 5 Trigger Mail waiting for DNS: {ITEM.LASTVALUE} >
{$MAIL.WAITINGFORDNS}
{$MEM.AVAILABILITY.CHECK.COUNT} #9 Trigger Not enough memory available Количество проверок
{$MEM.AVAILABILITY.CHECK.COUNT} не содержащих результат
Plentiful>{$MEM.AVAILABILITY.THR.COUNT}
{$MEM.AVAILABILITY.THR.COUNT} 3 See previous
{$NAB.USERS.EXPIRING.DAYS} 90 Trigger NAB.Users.Expiring in {$NAB.USERS.EXPIRING.DAYS} days:
{ITEM.LASTVALUE} > 0
{$NODATA.SECS} 180 Used in some triggers, «for some seconds we didnt received any data»
{$NOT.ALIVE.TIMEOUT} 180 Obsolete macro
{$PLATFORM.LOGICALDISK.QUEUE.NAME} AvgQue If server runs on Linux: AvgQueLen, if on Windows: AvgQueueLen
Len On different platforms, disk load is reported by Domino to different statistics
{$PLATFORM.PAGINGFILE.TOTAL.PCTUTIL.AVG 10 Trigger Platform.PagingFile.Total.PctUtil.Avg {ITEM.LASTVALUE}% >
} {$PLATFORM.PAGINGFILE.TOTAL.PCTUTIL.AVG}%
{$PLATFORM.SYSTEM.PCTCOMBINEDCPUUTIL. 3 Trigger Platform.System.PctCombinedCpuUtil {ITEM.LASTVALUE} >
COUNT} {$PLATFORM.SYSTEM.PCTCOMBINEDCPUUTIL.THR}
{$PLATFORM.SYSTEM.PCTCOMBINEDCPUUTIL. 95 See previous
THR}
{$REPLICA.CLUSTER.SECONDSONQUEUE} 15 Trigger Replica.Cluster.SecondsOnQueue: {ITEM.LASTVALUE} >
{$REPLICA.CLUSTER.SECONDSONQUEUE}
{$REPLICA.CLUSTER.WORKQUEUEDEPTH} 15 Trigger Replica.Cluster.WorkQueueDepth: {ITEM.LASTVAUE} >
{$REPLICA.CLUSTER.WORKQUEUEDEPTH}
{$SERVER.AVAILABILITYINDEX} 30 Trigger Server.AvailabilityIndex low: {ITEM.LASTVALUE} <
{$SERVER.AVAILABILITYINDEX}

Page 54 / 59
{$SERVER.CONCURRENTTASKS.WAITING.COUN 10 Trigger Server.ConcurrentTasks.Waiting: {ITEM.LASTVALUE} >
T} {$SERVER.CONCURRENTTASKS.WAITING}.
{$SERVER.CONCURRENTTASKS.WAITING.THR.C 3 See previous
OUNT}
{$SERVER.SESSIONS.DROPPED.THR.PCT} 5 Trigger Server.Sessions.Dropped: {ITEM.LASTVALUE} is more than
{$SERVER.SESSIONS.DROPPED.THR.PCT}% of Server.Users.Active15Min:
{ITEM.VALUE2}
{$SNMP_COMMUNITY} public
{$SSL_PORT} 443 Не используется
{$TRAVELER.DEVICESYNC.COUNT.AVERAGE} 900 Trigger Traveler.DeviceSync.Count.200 not growing. Number of HTTP 200 requests
has not grown during this time (in seconds)
{$TRAVELER.DEVICESYNC.COUNT.HIGH} 7200 Trigger Traveler.DeviceSync.Count.200 not growing. Количество 200 запросов не
росло в течение значения
{$TRAVELER.ERRORS.SECS} 30 Trigger Traveler.Errors is growing. Количество ошибок между измерениями
больше, чем значение
{$TRAVELER.MEMORY.JAVA.USEDPCT} 80 Trigger Traveler.Memory.Java.UsedPct: {ITEM.LASTVALUE} >
{$TRAVELER.MEMORY.JAVA.USEDPCT}%
{$TRAVELERLOGIN} fakeuser Логин для Traveler сервера
{$TRAVELERPASSWORD} fakepass Пароль для Traveler сервера
word
{$TRAVELERPORT} 443 Порт, на котором работает Traveler
{$TRAVELERPROTOCOL} https Протокол, с которым работает Traveler
{$TRIGGER.PROBLEM.GENERATION.TIME.AVG} 1h Используется в некоторых Triggerах в смысле как «Среднее значение за время
{$TRIGGER.PROBLEM.GENERATION.TIME.AVG}»
{$UPDATE.DEFERREDLIST} 20 Trigger FT Indexer too busy. Update.DeferredList: {ITEM.LASTVALUE} >
{$UPDATE.DEFERREDLIST}

Page 55 / 59
{$UPDATE.PENDINGLIST} 20 Trigger FT Indexer too busy. Update.PendingList: {ITEM.LASTVALUE} >
{$UPDATE.PENDINGLIST}
{$VERSION.TIMETOCHECK} 14h Trigger Domino version was changed to: {ITEM.LASTVALUE}. Как часто
проверять версию Domino сервера
{$VERSIONS.TOCHECK} 12 Trigger Domino version was changed to: {ITEM.LASTVALUE}. Сколько раз
провести проверку.

Page 56 / 59
18 Appendix B. 3rd party software used in CYMON
Cent OS
Cent OS is licensed under GNU General Public License, version 2.
http://www.gnu.org/licenses/old-licenses/gpl-2.0.html

ZABBIX
Zabbix monitoring solution GNU (GPL) version 2. Official homepage GPL
http://www.fsf.org/licenses/. Home page of Zabbix product www.zabbix.com

Grafana
Grafana is licensed under the Apache License 2.0.
Apache License Version 2.0, January 2004 http://www.apache.org/licenses/

HCL C API Toolkit for Notes/Domino

Copyright (c) HCL Corporation 2020, All rights reserved.

Node.js
Copyright Node.js contributors. All rights reserved.

End of document
Notes

Page 58 / 59
 COPYRIGHT LICENSE:
This information contains sample application programs in source language, which
illustrate programming techniques on various operating platforms. You may copy,
modify, and distribute these sample programs in any form without payment to CYONE,
for the purposes of developing, using, marketing or distributing application programs
conforming to the application programming interface for the operating platform for
which the sample programs are written. These examples have not been thoroughly
tested under all conditions. CYONE, therefore, cannot guarantee or imply reliability,
serviceability, or function of these programs. You may copy, modify, and distribute these
sample programs in any form without payment to CYONE for the purposes of
developing, using, marketing, or distributing application programs conforming to
CYONE’s application programming interfaces.
Each copy or any portion of these sample programs or any derivative work, must
include a copyright notice as follows:
© CYONE 2020. All rights reserved.

Page 59 / 59