IPC Agility 3.

2
Platform checks

Part Number: B02201019, Release: 00

IPC Confidential
September 8, 2023
© Copyright 2021–2023 IPC Systems, Inc. All Rights Reserved.
 3

Table of Contents
Revision List for IPC Agility v3.2.................................................................................................5
Preface ............................................................................................................................................6

Chapter 1: Introduction ............................................................................................................... 9

Chapter 2: ESXi and virtual machines checks .........................................................................12

2.1 ESXi License .................................................................................................................................................... 12
2.2 VLAN ID configuration ....................................................................................................................................13
2.3 VM hardware profile .........................................................................................................................................14
2.4 VM IP configuration.......................................................................................................................................... 15
2.5 VM names configuration .................................................................................................................................. 18
2.6 VM snapshot presence....................................................................................................................................... 19
2.7 VM Autostart settings........................................................................................................................................ 21

Chapter 3: Agility Tools.............................................................................................................. 24

3.1 Check VM network configuration with Agility Tools.......................................................................................25
3.2 Check Kubernetes installation with Agility Tools............................................................................................. 29
3.3 Check Applications installation with Agility Tools...........................................................................................30
3.4 Check Kurbernetes certificates with Agility Tools............................................................................................ 31
3.5 Check Agility platform connectivity with Agility Tools................................................................................... 33

Chapter 4: Kubernetes checks....................................................................................................38

4.1 Network certificates .......................................................................................................................................... 38

Chapter 5: Applications checking.............................................................................................. 40

5.1 Check Nagios Core installation......................................................................................................................... 40
5.2 Check the Network Diagnostic Tool (NDT) installation .................................................................................. 40
5.3 Check the NGSC installation............................................................................................................................. 42
5.3.1 Check Redis ....................................................................................................................................... 42
5.3.2 Check RabbitMQ configuration on the platform ...............................................................................43
5.3.3 Check USC (NGSC) configuration on the platform ..........................................................................43
5.4 Check the Prometheus installation.....................................................................................................................45
5.5 Check Recording Check.................................................................................................................................... 46
5.6 Check the SCD installation................................................................................................................................48
5.7 Check the SLIC configuration .......................................................................................................................... 49

IPC Confidential IPC Agility 3.2

Platform checks
4

5.7.1 SLIC logs on Splunk dashboard ........................................................................................................ 52

5.7.2 Unigy log forward to SLIC instance ..................................................................................................54

Appendix A: Establish an SSH connection to IPC Agility nodes ........................................... 56

A.1 Establish an SSH connection from customer network ......................................................................................56
A.2 Establish an SSH session using the shelldiag account.......................................................................................57
A.3 Establish an SSH connection with PRIISMS.................................................................................................... 60

Appendix B: Import the CA certificate .................................................................................... 63

Index..............................................................................................................................................65

September 8, 2023 IPC Confidential

 5

Revision List for IPC Agility v3.2

General updates
• Network certificates This topic describes how to c...

IPC Confidential IPC Agility 3.2

Platform checks
6 Preface

Preface
About this guide
This guide is intended for the manufacturing team and the team in charge of the IPC Agility
maintenance.
This document provides a list of tests to perform on the IPC Agility platform after its staging and
installation processes have ended. These tests allow you to check that the IPC Agility platform has been
properly configured before it goes into production. This document can also be used during the platform
life cycle.

Note
With IPC Agility V2.1, this documentation was named 6 - IPC Agility Platform Checks. As checks can be
made at any time during the product life cycle, we have removed the document numbering.

Copyright notices
• IPC, the IPC logo, Alliance MX, IQ/MAX, IQ/MAX TOUCH, MAXaccess, Nexus Suite, Unigy,
Blue Wave, and the Unigy and Blue Wave logos are trademarks of IPC Systems, Inc.
• Microsoft, Windows, Excel, Outlook, Lync, Microsoft OCS, Microsoft Office Communications
Server, Active Directory, and Internet Explorer are trademarks of Microsoft Corporation.
• Oracle, Java, and MySQL are trademarks of Oracle.
• Red Hat, Enterprise Linux, Ansible, and Ansible Tower are registered trademarks of Red Hat, Inc.
• Dell and PowerEdge are trademarks of Dell, Inc.
• Intel and Xeon are trademarks of Intel Corporation in the U.S.
• NICE and the NICE logo are trademarks of NICE Systems Ltd. and/or its subsidiaries.
• Verint is a registered trademark of Verint Systems Inc.
• Radisys is a trademark of Radisys Corporation.
• NetGuardian 832A is a trademark of DPS Telecom.
• ConferenceManager is a trademark of Sonexis Technology Inc.
• Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates.
• Splunk is a registered trademark of Splunk, Inc.
• All other brand and product names used in this document are trademarks or registered trademarks of
their respective owners.

IPC Agility Copyright notices

• Kubernetes is a registered trademark of Linux Foundation's.
• fluentd is a registered trademark of Linux Foundation's.
• Git is a registered trademark of Software Freedom Conservancy, Inc.

September 8, 2023 IPC Confidential

 Preface 7

Document Conventions
This topic describes the typographic conventions used in this manual:
• To indicate a user interface item to select or click:
Click Help. The Help dialog box opens.
Style Note: This is a san-serif bold font.
• To indicate a sequence of UI clicks:
Click File ➤ New ➤ Command.
Style Note: This is a san-serif bold font.
• To indicate window, screen, or panel names:
The Help dialog box opens.
Style Note: This is an italicized font.
• To indicate text to be typed or entered for user input or command names:
Type install at the prompt and press Enter.
Style Note: This is a Fixed-Width Bold font.
Type ls -al at the prompt and press Enter.
Style Note: This is a Fixed-Width Bold font.
• To indicate variables to be typed or entered:
Type yourPassword and press Enter.
Style Note: This is a Fixed-Width Bold Italic font.
• To indicate screen text such as prompts:
At the Enter your password prompt, type your password.
Style Note: This is a Fixed-Width font.
• To indicate file and directory names:
The error.log file is stored in /var/log.
Style Note: This is a Fixed-Width font.
• For references to other documents:
Refer to the Security Administrator Guide.
Style Note: This is an italicized font.
Space Designator
When the user needs to type a space in a command, a single space is used in the command text.
Key Combinations
Many instructions include key combinations where it is necessary to press two keys simultaneously. For
example, when CTRL+C is specified, it means press and hold down the Control key and press C at the
same time.

IPC Confidential IPC Agility 3.2

Platform checks
8 Preface

Note styles

Note
This is a Note and is used to alert you to important information.

Tip
This is a Tip and is used to provide helpful suggestions or hints.

Caution
This is a Caution and is used to alert you to any procedures in which extreme caution must be used.

Warning
This is a Warning and is used to alert you to dangerous situations or procedures that must be completed
in a specific manner to prevent a dangerous or damaging situation.

September 8, 2023 IPC Confidential

 9

1 Introduction
This topic provides a reminder of the deployment models for the IPC Agility platform available from the
release version 2.1. This procedure guide will refer to these models of deployment (standalone and
multi-server models).
IPC identifies three deployments for IPC Agility:
• Deployment on an IPC appliance (standalone or multi-server model)
• Deployment on the client's infrastructure
• Deployment on the cloud provider's infrastructure
From IPC Agility V2.1, two models of Kubernetes cluster can be deployed to have a different level of
fault tolerance and high availability:
• A single master with three worker nodes deployed on a standalone server.
• Three masters with three worker nodes deployed on three servers.
This procedure guide refers to both deployments (standalone and multi-server models).
For more details on Kubernetes cluster, refer to the IPC Agility General Overview document.

Standalone and multi-server deployments on an IPC appliance

The IPC Agility appliance(s) is(are) prepared on the IPC premises, through a staging process, and
shipped to the client's location. For the installation overview during the staging process, refer to the IPC
Agility General Overview document.
The standalone server model hosts the following virtual machines:
• RHEL Master 1
• RHEL Worker 1
• RHEL Worker 2
• RHEL Worker 3

IPC Confidential IPC Agility 3.2

Platform checks
10

Figure 1: Single master on a standalone server model (IPC premises)

The multi-server model (with multi-master) hosts the following virtual machines:
• The first IPC appliance hosts:
• RHEL Master 1
• RHEL Worker 1
• The second IPC appliance hosts:
• RHEL Master 2
• RHEL Worker 2
• The third IPC appliance hosts:
• RHEL Master 3
• RHEL Worker 3

September 8, 2023 IPC Confidential

 11

Figure 2: Multi-master on a multi-server model (IPC premises)

IPC Confidential IPC Agility 3.2

Platform checks
12 2.1: ESXi License

2 ESXi and virtual machines checks

This topic gives the checklist to perform on ESXi and VM nodes. A prerequisite for this topic is
knowledge of the VM hardware profile and network IP addresses, from the earlier VM creation.
This chapter covers a check of the following perimeters:
• ESXi license
• VLAN ID configuration
• Virtual machine hardware profile and network
• Network interface configuration
• NTP server configuration
• Snapshot presence
• Autostart settings

2.1 ESXi License

This topic describes how to check the ESXi License.
Prerequisite: Connect to the ESXi Web UI and click Manage ➤ Licensing.
Figure 3: Check the ESXi License

September 8, 2023 IPC Confidential

 2.2: VLAN ID configuration 13

Table 1: ESXi license checklist

Check Expected result Current

Status
Check the ESXi 1 The License must be valid and never expire.
License
Check the ESXi 2 The License must be valid and never expire.
License
For the multi-server
deployment.
Check the ESXi 3 The License must be valid and never expire.
License
For the multi-server
deployment.

2.2 VLAN ID configuration

This topic describes how to check the VLAN ID on each virtual machine.
Prerequisite: Connect to the ESXi and select Networking from the left menu. On the Port groups tab,
check each configured VLAN ID for the following port group:
• Customer Network
• Local Network
• Management Local Network
Each VLAN ID must correspond with the VLAN ID previously configured on the customer network.
Figure 4: Check the VLAN ID

IPC Confidential IPC Agility 3.2

Platform checks
14 2.3: VM hardware profile

Table 2: VLAN checklist

Check Expected result Current

Status
Check the VLAN ID The VLAN ID corresponds to the VLAN ID configured by the
for the Customer customer on its network.
Network port group.

Check the VLAN ID The VLAN ID corresponds to the VLAN ID configured by the
for the Local Network customer on its network.
port group.
Check the VLAN ID The VLAN ID corresponds to the VLAN ID configured by the
for the Management customer on its network.
Local Network port
group.

2.3 VM hardware profile

This topic describes the check to perform on the hardware settings for each virtual machine (i.e. CPU,
RAM, hard drives sizing and networking settings).
To perform the following checklist, you must connect to the ESXi and the hardware configuration for
each VM.
Figure 5: Checking VM hardware profile

Table 3: VM hardware and network settings checklist

Check Expected result Current

status
Master1 CPU, RAM, hard drive(s) and network settings are
Check the master1 VM properly configured
hardware and network profile
Master2 - multi-server only CPU, RAM, hard drive(s) and network settings are
Check the master2 VM properly configured
hardware and network profile.

September 8, 2023 IPC Confidential

 2.4: VM IP configuration 15

Table 3: VM hardware and network settings checklist (continued)

Check Expected result Current

status
Master3 - multi-server only CPU, RAM, hard drive(s) and network settings are
Check the master3 VM properly configured
hardware and network profile.
Worker1 CPU, RAM, hard drive(s) and network settings are
Check the worker1 VM properly configured
hardware and network profile
Worker2 CPU, RAM, hard drive(s) and network settings are
Check the worker2 VM properly configured
hardware and network profile
Worker3 CPU, RAM, hard drive(s) and network settings are
Check the worker3 VM properly configured
hardware and network profile

2.4 VM IP configuration
This topic describes how to check the network configuration for each VM. This includes checking both
the local (ens192) and the customer (ens224) network interfaces and their associated IP and default
gateways.
Prerequisite: Connect to the ESXi, select a virtual machine, and open its console using Open console in
new window control. Repeat this operation for each expected node.

Note
The Agility Tools check_os_cluster.sh script also gives the same results, but requires the
complete installation of the IPC Agility platform. For more information on the check_os_cluster.sh
script, refer to the section Check VM network configuration with Agility Tools on page 25.

Figure 6: Open a console in a new window for each VM

IPC Confidential IPC Agility 3.2

Platform checks
16 2.4: VM IP configuration

In the console, log in to the Red Hat VM and check the network interface configuration (i.e. IP address,
mask and default gateway) using both the ip address and ip route commands.
Figure 7: IP addresses and IP routes check on VM console

Check the network configuration for each VM as follows:

Table 4: IP address and IP route checklist

Check Expected result Current

status
Master1 • Ens192 (local network): 100.64.152.2/24.
Execute the ip address
command on the master1 VM • Master VIP: 100.64.155.11/32 (only for multi-server)
console. Check the IP • Ens224 (client network): The IP address and mask
addresses and the masks defined in the customer IP plan.
defined for the ens192 and
the ens224 interface.
Master1 The default gateway is set on the ens224 interface.
Execute the ip route
command on the master1 VM
console to check the default
gateway.

September 8, 2023 IPC Confidential

 2.4: VM IP configuration 17

Table 4: IP address and IP route checklist (continued)

Check Expected result Current

status
Master2 - multi-server only • Ens192 (local network): 100.64.152.3/24.
Execute the ip address
command on the master2 VM • Ens224 (client network): The IP address and mask
defined in the customer IP plan.
console. Check the IP
addresses and the masks
defined for the ens192 and
the ens224 interface.
Master2 - multi-server only The default gateway is set on the ens224 interface.
Execute the ip route
command on the master2 VM
console to check the default
gateway.
Master3 - multi-server only • Ens192 (local network): 100.64.152.4/24.
Execute the ip address
command on the master3 VM • Ens224 (client network): The IP address and mask
defined in the customer IP plan.
console. Check the IP
addresses and the masks
defined for the ens192 and
the ens224 interface.
Master3 - multi-server only The default gateway is set on the ens224 interface.
Execute the ip route
command on the master3 VM
console to check the default
gateway.
Worker1 • Ens192 (local network): 100.64.152.5/24.
Execute the ip address
command on the worker1 VM • Ens224 (client network): The IP address and mask
defined in the customer IP plan.
console. Check the IP
addresses and the masks
defined for the ens192 and
the ens224 interface.
Worker1 The default gateway is set on the ens224 interface.
Execute the ip route
command on the worker1 VM
console to check the default
gateway.
Worker2 • Ens192 (local network): 100.64.152.6/24.
Execute the ip address
command on the worker2 VM • Ens224 (client network): The IP address and mask
defined in the customer IP plan.
console. Check the IP
addresses and the masks
defined for the ens192 and
the ens224 interface.
Worker2 The default gateway is set on the ens224 interface.
Execute the ip route
command on the worker2 VM
console to check the default
gateway.

IPC Confidential IPC Agility 3.2

Platform checks
18 2.5: VM names configuration

Table 4: IP address and IP route checklist (continued)

Check Expected result Current

status
Worker3 • Ens192 (local network): 100.64.152.7/24.
Execute the ip address
command on the worker3 VM • Ens224 (client network): The IP address and mask
defined in the customer IP plan.
console. Check the IP
addresses and the masks
defined for the ens192 and
the ens224 interface.
Worker3 The default gateway is set on the ens224 interface.
Execute the ip route
command on the worker3 VM
console to check the default
gateway.

2.5 VM names configuration

This topic describes how to check the names configuration for each VM.
Prerequisite: Connect to the ESXi, select a virtual machine, and open its console using Open console in
new window control. Repeat this operation for each expected node.

Note
The Agility Tools check_os_cluster.sh script also gives the same results, but requires the
complete installation of the IPC Agility platform. For more information on the check_os_cluster.sh
script, refer to the section Check VM network configuration with Agility Tools on page 25.

Figure 8: Open a console in a new window for each VM

In the console, log in to the Red Hat VM and check the VM hostname by using the hostname
command.

September 8, 2023 IPC Confidential

 2.6: VM snapshot presence 19

Figure 9: VM Hostnames check on a VM console

Check the hostname for each VM as follows:

Table 5: Hostnames checklist

Check Expected result Current

status
Master1 The hostname is properly set according to the site prep.
Execute the hostname
command on the master1 VM
console.
Master2 - multi-server only The hostname is properly set according to the site prep.
Execute the hostname
command on the master2 VM
console.
Master3 - multi-server only The hostname is properly set according to the site prep.
Execute the hostname
command on the master3 VM
console.
Worker1 The hostname is properly set according to the site prep.
Execute the hostname
command on the worker1 VM
console.
Worker2 The hostname is properly set according to the site prep.
Execute the hostname
command on the worker2 VM
console.
Worker3 The hostname is properly set according to the site prep.
Execute the hostname
command on the worker3 VM
console.

2.6 VM snapshot presence

This topic describes how to check for the presence of a snapshot for each VM. The snapshot must
include the Red Hat installation and user configuration.
Prerequisite: Connect to the ESXi and check for the presence of the snapshot in the Manage snapshots
menu for each VM.

IPC Confidential IPC Agility 3.2

Platform checks
20 2.6: VM snapshot presence

Figure 10: Manage snapshots menu

After the Red Hat installation and user configuration by an IPC engineer, it is assumed that the VM
snapshot has been taken. The Manage snapshots menu displays the expected snapshot named before
k8s that includes the Red Hat installation.
Figure 11: VM snapshot presence

Check the snapshot presence of each VM, as follows:

Table 6: VM snapshot presence checklist

Check Expected result Current

status
Master1 The snapshot is present (including the Red Hat
Check the master1 VM snapshot installation).
presence.
Master2 - multi-server only The snapshot is present (including the Red Hat
Check the master2 VM snapshot installation).
presence.
Master3 - multi-server only The snapshot is present (including the Red Hat
Check the master3 VM snapshot installation).
presence.

September 8, 2023 IPC Confidential

 2.7: VM Autostart settings 21

Table 6: VM snapshot presence checklist (continued)

Check Expected result Current

status
Worker1 The snapshot is present (including the Red Hat
Check the worker1 VM snapshot installation).
presence.
Worker2 The snapshot is present (including the Red Hat
Check the worker2 VM snapshot installation).
presence.
Worker3 The snapshot is present (including the Red Hat
Check the worker3 VM snapshot installation).
presence.

2.7 VM Autostart settings

This topic describes how to check the Autostart settings for each VM.
Prerequisite: Connect to the ESXi, and verify the following Autostart settings:
• The Autostart feature is enabled on the ESXi (enable the feature on the Manage menu accessible
from the System tab).
• The Autostart order number and the Start delay must be configured as follow:
• Standalone deployment:
• (1) The master VM - Start delay = 120 seconds
• (2) The worker 1 VM - Start delay = 0 second
• (3) The worker 2 VM - Start delay = 0 second
• (4) The worker 3 VM - Start delay = 0 seconds
Figure 12: Standalone - VM Autostart enabling and order setting

• Multi-server deployment:
• ESXi 1/2/3:

IPC Confidential IPC Agility 3.2

Platform checks
22 2.7: VM Autostart settings

• (1) The master 1 VM - Start delay = 120 seconds

• (2) The worker 1 VM - Start delay = 0 seconds
Figure 13: Multi-server - ESXi1 VM Autostart enabling and order setting

Table 7: Standalone deployment autostart checklist

Check Expected result Current

status
Check the following on the master1 VM: Both the Autostart order number
and Start delay parameter are
• The Autostart order number is 1. properly configured.
• The Start delay is set to 120 seconds.
Check the following on the worker1 VM: Both the Autostart order number
and Start delay parameter are
• The Autostart order number is 2. properly configured.
• The Start delay is set to 0 second.
Check the following on the worker2 VM: Both the Autostart order number
and Start delay parameter are
• The Autostart order number is 3. properly configured.
• The Start delay is set to 0 second.
Check the following on the worker3 VM: Both the Autostart order number
and Start delay parameter are
• The Autostart order number is 4. properly configured.
• The Start delay is set to 0 seconds.

September 8, 2023 IPC Confidential

 2.7: VM Autostart settings 23

Table 8: Multi-server deployment autostart checklist - ESXi 1

Check Expected result Current

status
Check the following on the master1 VM: Both the Autostart order number
and Start delay parameter are
• The Autostart order number is 1. properly configured.
• The Start delay is set to 120 seconds.
Check the following on the worker1 VM: Both the Autostart order number
and Start delay parameter are
• The Autostart order number is 2. properly configured.
• The Start delay is set to 0 seconds.

Table 9: Multi-server deployment autostart checklist - ESXi 2

Check Expected result Current

status
Check the following on the master2 VM: Both the Autostart order number
and Start delay parameter are
• The Autostart order number is 1. properly configured.
• The Start delay is set to 120 seconds.
Check the following on the worker2 VM: Both the Autostart order number
and Start delay parameter are
• The Autostart order number is 2. properly configured.
• The Start delay is set to 0 second.

Table 10: Multi-server deployment autostart checklist - ESXi 3

Check Expected result Current

status
Check the following on the master3 VM: Both the Autostart order number
and Start delay parameter are
• The Autostart order number is 1. properly configured.
• The Start delay is set to 120 seconds.
Check the following on the worker3 VM: Both the Autostart order number
and Start delay parameter are
• The Autostart order number is 2. properly configured.
• The Start delay is set to 0 second.

IPC Confidential IPC Agility 3.2

Platform checks
24

3 Agility Tools
Agility Tools is an application that installs scripts used to check, patch and repair the platform.
Scripts are launched, using an SSH connection through PRIISMS or an SSH connection through the
Unigy customer network.

Table 11: Main scripts

Name Description Run on

check_all_platform Checks Kubernetes installation Master(s)
.sh
check_all_applicat Checks all the applications hosted by the IPC Agility platform Master(s)
ions.sh
check_os_cluster.s Checks the RHEL OS and network configuration for all nodes Master(s)
h
check_cert_cluster Checks the expiration date of Kubernetes certificates. Master(s)
.sh
Note
Run this script periodically to check certificate expiration date.

check_all_connecti Checks the network connectivity of all Agility nodes. Master(s)

vity.sh
clean_unused_image Removes all the unused Docker images on worker nodes in Master(s)
s.sh case of application upgrade.
This script is only available from Agility Tools V1.1.0-12

Note
As Docker has been removed from Agility V2.3, the
clean_unused_images.sh is useless with Agility V2.3,
but this script is still delivered with Agility Tools V1.3 that can
be used with previous versions of Agility.

The check_all_platform.sh, check_all_applications.sh,

check_os_cluster.sh, check_cert_cluster.sh and the
check_all_connectivity.sh scripts are automatically launched every hour by a cron job and
the results are logged in Splunk.
You can see all these logs in the RAG logs ➤ All or RAG logs ➤ Operating Systems Splunk menus,
and by using the *-CHECK input in SPL filter. You must also change the Timeframe to at least the
Last 60 minutes to be sure to get some logs.

September 8, 2023 IPC Confidential

 3.1: Check VM network configuration with Agility Tools 25

Figure 14: Filter Agility Tools logs in Splunk

Figure 15: Agility Tools logs in Splunk

You can get more information about Splunk in the IPC Agility Monitoring Using Splunk Guide.

3.1 Check VM network configuration with Agility Tools

This section describes how to check the network configuration of the Agility master(s) and workers
nodes, using the Agility Tools' check_os_cluster.sh script.
1. Connect to the master endpoint to log on to the expected master node (refer to Establish an SSH
connection to IPC Agility nodes on page 56. Use the ipctech account and the su command to log
in with root user privileges.
login as: ipctech
[email protected]'s password:
Last login: Mon May 23 13:55:44 2022 from 100.64.152.252
[ipctech@k8s-master-cao ~]$ su
Password:
[root@k8s-master-cao ipctech]#

2. Run the check_os_cluster.sh script.

[root@cao12-k8s-master1-xl bin]# check_os_cluster.sh

IPC Confidential IPC Agility 3.2

Platform checks
26 3.1: Check VM network configuration with Agility Tools

The script gives the network configuration and the Linux version for masters and workers.
Figure 16: check_os_cluster.sh script result for master1

Figure 17: check_os_cluster.sh script result for worker1

Table 12: check_os_cluster.sh - network configuration checklist

Check Expected result Current

status
Master1 The hostname is properly set according to the site prep
Check the node name document.
Master1 • Ens192 (local network): 100.64.152.2/24.
Check the IP addresses and
the masks defined for the • Master VIP: 100.64.155.11/32 (only for multi-
ens192 and the ens224 server)
interface. • Ens224 (client network): The IP address and mask
defined in the customer IP plan.

Master1 The default gateway is set on the ens224 interface.

Check the default gateway.
Master1 DNS server IP if configured, not found if any.
Check the DNS server
configuration
Master1 Time Zone where the IPC Agility platform is installed
Check the Time Zone
Master1 • NTP synchronized: yes
Check NTP configuration
• List of NTP servers IP addresses

September 8, 2023 IPC Confidential

 3.1: Check VM network configuration with Agility Tools 27

Table 12: check_os_cluster.sh - network configuration checklist (continued)

Check Expected result Current

status
Master2 The hostname is properly set according to the site prep
Check the node name document.
Master2 - multi-master • Ens192 (local network): 100.64.152.3/24.
only
Check the IP addresses and • Ens224 (client network): The IP address and mask
the masks defined for the defined in the customer IP plan.
ens192 and the ens224
interface.
Master2 - multi-master The default gateway is set on the ens224 interface.
only
Check the default gateway.
Master2 - multi-master DNS server IP if configured, not found if any.
only
Check the DNS server
configuration
Master2 - multi-master Time Zone where the IPC Agility platform is installed
only
Check the Time Zone
Master2 - multi-master • NTP synchronized: yes
only
Check NTP configuration • List of NTP servers IP addresses
Master3 The hostname is properly set according to the site prep
Check the node name document.
Master3 - multi-master • Ens192 (local network): 100.64.152.4/24.
only
Check the IP addresses and • Ens224 (client network): The IP address and mask
the masks defined for the defined in the customer IP plan.
ens192 and the ens224
interface.
Master3 - multi-master The default gateway is set on the ens224 interface.
only
Check the default gateway.
Master3 - multi-master DNS server IP if configured, not found if any.
only
Check the DNS server
configuration
Master3 - multi-master Time Zone where the IPC Agility platform is installed
only
Check the Time Zone
Master3 - multi-master • NTP synchronized: yes
only
Check NTP configuration • List of NTP servers IP addresses
Worker1 The hostname is properly set according to the site prep
Check the node name document.

IPC Confidential IPC Agility 3.2

Platform checks
28 3.1: Check VM network configuration with Agility Tools

Table 12: check_os_cluster.sh - network configuration checklist (continued)

Check Expected result Current

status
Worker1 • Ens192 (local network): 100.64.152.5/24.
Check the IP addresses and
the masks defined for the • Ens224 (client network): The IP address and mask
ens192 and the ens224 defined in the customer IP plan.
interface.
Worker1 The default gateway is set on the ens224 interface.
Check the default gateway.
Worker1 DNS server IP if configured, not found if any.
Check the DNS server
configuration
Worker1 Time Zone where the IPC Agility platform is installed
Check the Time Zone
Worker1 • NTP synchronized: yes
Check NTP configuration
• List of NTP servers IP addresses
Worker2 The hostname is properly set according to the site prep
Check the node name document.
Worker2 • Ens192 (local network): 100.64.152.6/24.
Check the IP addresses and
the masks defined for the • Ens224 (client network): The IP address and mask
ens192 and the ens224 defined in the customer IP plan.
interface.
Worker2 The default gateway is set on the ens224 interface.
Check the default gateway.
Worker2 DNS server IP if configured, not found if any.
Check the DNS server
configuration
Worker2 Time Zone where the IPC Agility platform is installed
Check the Time Zone
Worker2 • NTP synchronized: yes
Check NTP configuration
• List of NTP servers IP addresses
Worker3 The hostname is properly set according to the site prep
Check the node name document.
Worker3 • Ens192 (local network): 100.64.152.7/24.
Check the IP addresses and
the masks defined for the • Ens224 (client network): The IP address and mask
ens192 and the ens224 defined in the customer IP plan.
interface.
Worker3 The default gateway is set on the ens224 interface.
Check the default gateway.
Worker3 DNS server IP if configured, not found if any.
Check the DNS server
configuration

September 8, 2023 IPC Confidential

 3.2: Check Kubernetes installation with Agility Tools 29

Table 12: check_os_cluster.sh - network configuration checklist (continued)

Check Expected result Current

status
Worker3 Time Zone where the IPC Agility platform is installed
Check the Time Zone
Worker3 • NTP synchronized: yes
Check NTP configuration
• List of NTP servers IP addresses

3.2 Check Kubernetes installation with Agility Tools

This section describes how to check the Kubernetes installation on the IPC Agility platform, using the
Agility Tools' check_all_platform.sh script.
1. Connect to the master endpoint to log on to the expected master node (refer to Establish an SSH
connection to IPC Agility nodes on page 56. Use the ipctech account and the su command to log
in with root user privileges.
login as: ipctech
[email protected]'s password:
Last login: Mon May 23 13:55:44 2022 from 100.64.152.252
[ipctech@k8s-master-cao ~]$ su
Password:
[root@k8s-master-cao ipctech]#

2. Run the check_all_platform.sh script.

[root@cao12-k8s-master1-xl bin]# check_all_platform.sh

If a pod is not running, you will get a failure (in red characters), example for the helm-operator-
ipc-helm-operator pod.
KUBERNETES-CHECK sub_check=FLUX POD CHECK : FAILURE
DETAILS / REASONS
NOT Running pods: 1
NAME READY
STATUS RESTARTS AGE IP NODE
helm-operator-ipc-helm-operator-75cbdcc694-25lws 0/1
CrashLoopBackOff 277 16h 100.64.146.25 k8s-worker2-cao

At the end of the script execution, you should get the following result (in green characters):
ALL PLATFORM CHECK: SUCCESS
Reason: all checks are successful

3. Find the Virtual IPs range line and control that the customer-vSwitch IP range
addresses correspond to those indicated in the Site Prep document.
Virtual IPs range:
config: |
address-pools:
- name: local-vSwitch
protocol: layer2
addresses:
- 100.64.152.100-100.64.152.109
- name: customer-vSwitch

IPC Confidential IPC Agility 3.2

Platform checks
30 3.3: Check Applications installation with Agility Tools

protocol: layer2
addresses:
- 192.168.81.200-192.168.81.207

4. Find the KUBERNETES-CHECK sub_check=INGRESS-NGINX IP SERVICE CHECK line

and control that the nginx-ingress-customer-ingress-nginx-controller IP
address corresponds to the one indicated in the Site Prep document.
KUBERNETES-CHECK sub_check=INGRESS-NGINX IP SERVICE CHECK : SUCCESS
DETAILS / REASONS
nginx-ingress-customer-ingress-nginx-controller IP: 192.168.81.200
nginx-ingress-local-ingress-nginx-controller IP: 100.64.152.100

3.3 Check Applications installation with Agility Tools

This section describes how to check the applications deployed on the IPC Agility platform, using the
Agility Tools' check_all_applications.sh script.
1. Connect to the master endpoint to log on to the expected master node (refer to Establish an SSH
connection to IPC Agility nodes on page 56. Use the ipctech account and the su command to log
in with root user privileges.
login as: ipctech
[email protected]'s password:
Last login: Mon May 23 13:55:44 2022 from 100.64.152.252
[ipctech@k8s-master-cao ~]$ su
Password:
[root@k8s-master-cao ipctech]#

2. Run the check_all_applications.sh script.

[root@cao12-k8s-master1-xl bin]# check_all_applications.sh

If an application is not running properly, you will get an error (in red characters).
If an application is not enabled, you il get a warning (in yellow characters).
At the end of the script execution, you should get the following result (in green character):
ALL APPLICATIONS CHECK: SUCCESS
Reason: all checks are successful

You can also check the results for each application, example for USC (NGSC):

September 8, 2023 IPC Confidential

 3.4: Check Kurbernetes certificates with Agility Tools 31

Figure 18: check_app_009_ngsc.sh script - USC (NGSC)

For the following applications, you must check that the MetalLB VIPs on customer network
correspond to those indicated in the Site Prep document:

Table 13: check_all_application.sh - MetalLB VIP on customer network

Check Expected result Current

status
NDT ndt-svc-echoserver-tcp IP: IP addresses defined
in the customer IP plan defined in the site prep.
ndt-svc-echoserver-udp IP: IP addresses defined
in the customer IP plan defined in the site prep.
USC (NGSC) ngsc-svc-sipservice IP: IP address defined in the
customer IP plan defined in the site prep.
SLIC slic-filebeat-svc IP: IP address defined in the
customer IP plan defined in the site prep.
slic-syslog-svc IP: IP address defined in the
customer IP plan defined in the site prep.

3.4 Check Kurbernetes certificates with Agility Tools

This section describes how to check the Kubernetes certificates expiration, by using the Agility Tools
check_certs_cluster.sh script.
The Kurbernetes certificates should be automatically renewed before they expire. Nevertheless, if the
certificates are not renewed 30 days before the expiration date, contact your local support. To check the
certificate expiration date with Agility Tools, perform the following steps.
1. Connect to the master endpoint to log on to the expected master node (refer to Establish an SSH
connection to IPC Agility nodes on page 56. Use the ipctech account and the su command to log
in with root user privileges.
login as: ipctech
[email protected]'s password:

IPC Confidential IPC Agility 3.2

Platform checks
32 3.4: Check Kurbernetes certificates with Agility Tools

Last login: Mon May 23 13:55:44 2022 from 100.64.152.252

[ipctech@k8s-master-cao ~]$ su
Password:
[root@k8s-master-cao ipctech]#

2. Run the check_cert_cluster.sh scrip and enter the ipctech password

[root@cao12-k8s-master1-xl bin]# check_certs_cluster.sh

****************************************************
CERTIFICATES CLUSTER CHECK
Enter ipctech user's password:
Check certificates expiration on k8s-master1-cao node
[sudo] password for ipctech:

The script gives the expiration date of all Kubernetes components for all nodes.
Master kubernetes certificates:
• /etc/kubernetes/admin.conf
• /etc/kubernetes/controller-manager.conf
• /etc/kubernetes/scheduler.conf
• /etc/kubernetes/pki/ca.crt
• /etc/kubernetes/pki/apiserver.crt
• /etc/kubernetes/pki/apiserver-kubelet-client.crt
• /etc/kubernetes/pki/front-proxy-ca.crt
• /etc/kubernetes/pki/front-proxy-client.crt
• /var/lib/kubelet/pki/kubelet.crt
• /var/lib/kubelet/pki/kubelet-client-current.pem
• /etc/ssl/etcd/ssl/ca.pem
Worker kubernetes certificates:
• /etc/kubernetes/pki/ca.crt
• /var/lib/kubelet/pki/kubelet.crt
• /var/lib/kubelet/pki/kubelet-client-current.pem
• /etc/ssl/etcd/ssl/ca.pem
If the expiration duration is inferior to 30 days, you will get the following warning
Figure 19: Example of certificate expiration warning

3. If a certificate has not been automatically renewed, contact your local support.

September 8, 2023 IPC Confidential

 3.5: Check Agility platform connectivity with Agility Tools 33

3.5 Check Agility platform connectivity with Agility Tools

This section describes how to check the Agility platform network connectivity, by using the Agility
Tools check_all_connectivity.sh script.
1. Connect to the master endpoint to log on to the expected master node (refer to Establish an SSH
connection to IPC Agility nodes on page 56. Use the ipctech account and the su command to log
in with root user privileges.
login as: ipctech
[email protected]'s password:
Last login: Mon May 23 13:55:44 2022 from 100.64.152.252
[ipctech@k8s-master-cao ~]$ su
Password:
[root@k8s-master-cao ipctech]#

2. Run the check_all_connectivity.sh script.

[root@cao12-k8s-master1-xl bin]# check_all_connectivity.sh

This script performs the following connectivity checks for all Agility master(s) and workers nodes:
1. Ping of the node gateway
2. Ping of the Artifactory server
3. HTTPS curl test of the Artifactory server
4. Ping of the IPC GIT server
5. HTTPS curl test of IPC GIT server
6. Ping of the IPC Splunk deployment server

IPC Confidential IPC Agility 3.2

Platform checks
34 3.5: Check Agility platform connectivity with Agility Tools

Figure 20: check_all_connectivity.sh example

If a check fails, you will get an error (in red characters).

September 8, 2023 IPC Confidential

 3.5: Check Agility platform connectivity with Agility Tools 35

Table 14: check_all_connectivity.sh

Check Expected result Current

status
Master1 - gateway ping 5/5 pings received, RTT < 500 ms
Master1 - Artifactory server 5/5 pings received, RTT < 500 ms (not performed if
ping proxy configured)
Master1 - Artifactory server 200 OK
URL
Master1 - GIT server ping 5/5 pings received, RTT < 500 ms (not performed if
proxy configured)
Master1 - GIT server URL 200 OK
Master1 - Splunk server 5/5 pings received, RTT < 500 ms (not performed if
Ping proxy configured)
Master2 (only for multi- 5/5 pings received, RTT < 500 ms
server) - gateway ping
Master2 (only for multi- 5/5 pings received, RTT < 500 ms (not performed if
server)- Artifactory server proxy configured)
ping
Master2 (only for multi- 200 OK
server)- Artifactory server
URL
Master2 (only for multi- 5/5 pings received, RTT < 500 ms (not performed if
server)- GIT server ping proxy configured)
Master2 (only for multi- 200 OK
server)- GIT server URL
Master2 (only for multi- 5/5 pings received, RTT < 500 ms (not performed if
server)- Splunk server Ping proxy configured)
Master3 (only for multi- 5/5 pings received, RTT < 500 ms
server) - gateway ping
Master3 (only for multi- 5/5 pings received, RTT < 500 ms (not performed if
server)- Artifactory server proxy configured)
ping
Master3 (only for multi- 200 OK
server)- Artifactory server
URL
Master3 (only for multi- 5/5 pings received, RTT < 500 ms (not performed if
server)- GIT server ping proxy configured)
Master3 (only for multi- 200 OK
server)- GIT server URL
Master3 (only for multi- 5/5 pings received, RTT < 500 ms (not performed if
server)- Splunk server Ping proxy configured)
Worker1 - gateway ping 5/5 pings received, RTT < 500 ms

IPC Confidential IPC Agility 3.2

Platform checks
36 3.5: Check Agility platform connectivity with Agility Tools

Table 14: check_all_connectivity.sh (continued)

Check Expected result Current

status
Worker1 - Artifactory server 5/5 pings received, RTT < 500 ms (not performed if
ping proxy configured)
Worker1 - Artifactory server 200 OK
URL
Worker1 - GIT server ping 5/5 pings received, RTT < 500 ms (not performed if
proxy configured)
Worker1 - GIT server URL 200 OK
Worker1 - Splunk server 5/5 pings received, RTT < 500 ms (not performed if
Ping proxy configured)
Worker2 - gateway ping 5/5 pings received, RTT < 500 ms
Worker2 - Artifactory server 5/5 pings received, RTT < 500 ms (not performed if
ping proxy configured)
Worker2 - Artifactory server 200 OK
URL
Worker2 - GIT server ping 5/5 pings received, RTT < 500 ms
Worker2 - GIT server URL 200 OK
Worker2 - Splunk server 5/5 pings received, RTT < 500 ms (not performed if
Ping proxy configured)
Worker3 - gateway ping 5/5 pings received, RTT < 500 ms
Worker3 - Artifactory server 5/5 pings received, RTT < 500 ms (not performed if
ping proxy configured)
Worker3 - Artifactory server 200 OK
URL
Worker3 - GIT server ping 5/5 pings received, RTT < 500 ms (not performed if
proxy configured)
Worker3 - GIT server URL 200 OK
Worker3 - Splunk server 5/5 pings received, RTT < 500 ms (not performed if
Ping proxy configured)

You can also use the --help argument to show the available syntaxes:
[root@cao9-k8s-master-cao ipctech]# check_all_connectivity.sh --help
Usage:
---------------------------------------------------------------------------
-----
--default: CONNECTIVITY AVAILABILITY CHECK
example: check_all_connectivity.sh or check_all_connectivity.sh --default
---------------------------------------------------------------------------
-----
--stability: CONNECTIVITY STABILITY CHECK
example: check_all_connectivity.sh --stability
---------------------------------------------------------------------------
-----

September 8, 2023 IPC Confidential

 3.5: Check Agility platform connectivity with Agility Tools 37

--bandwidth: CONNECTIVITY BANDWIDTH CHECK

example: check_all_connectivity.sh --bandwidth
---------------------------------------------------------------------------
-----
--help: this message

The --stability argument performs all the ping tests (not the curl), but runs 100 pings instead of
five for the default check. This check can take about ten minutes.
The --bandwidth argument downloads the kubelet file (100MB) from the Artifactory server. The
check is considered failed if the download time is superior to 30 seconds, but the download anyhow goes
on, until the file is totally downloaded.
Figure 21: Bandwidth test failure example

Caution
the bandwidth check can degrade the IPC Agility platform functioning in term of platform and application
upgrade.

It is also possible to check manually the connectivity of a master node, with the
check_master_connectivity.sh script.

IPC Confidential IPC Agility 3.2

Platform checks
38 4.1: Network certificates

4 Kubernetes checks
This chapter describes the Kubernetes checks that cannot be done with the Agility Tools
check_all_platform.sh script.
For check_all_platform.sh use, refer to the section Check Kubernetes installation with Agility
Tools on page 29.

4.1 Network certificates

This topic describes how to check the customer network CA certificates. These must be issued by IPC or
the customer CA.
Import the correct certificates onto your laptop. For details, refer to Import the CA certificate on page
63
To check the certificate on the customer network:
1. Open the Web browser on a device (e.g., VM) in the customer network.
2. Log on to an IPC Agility Web UI application (e.g., Unigy Soft Client).
3. Click the padlock icon next to the address, then click Connection is secure.
4. Check the certificate issuer. It must be either IPC or the customer CA. In the following example, the
Recording Check application has a certificate issued by IPC Lab CA.

Note
The browser used in this example is Microsoft Edge. This step may vary for different browsers.

September 8, 2023 IPC Confidential

 4.1: Network certificates 39

Figure 22: Client certificate on the customer network

IPC Confidential IPC Agility 3.2

Platform checks
40 5.1: Check Nagios Core installation

5 Applications checking

5.1 Check Nagios Core installation

This topic describes how you can confirm the Nagios Core installation.
To check Nagios Core, use the check_app_008_nagios.sh script from the Agility Tools
application.
1. Connect to the master node (refer to the section Establish an SSH connection to IPC Agility nodes on
page 56).
2. Run the check_app_008_nagios.sh script.
All the tests should be successful.
Figure 23: check_app_008_nagios.sh script result

5.2 Check the Network Diagnostic Tool (NDT) installation

This topic describes how to check the Network Diagnostic Tool (NDT) installation.
To check NDT, use the check_app_006_ndt.sh script from the Agility Tools application.
1. Connect to the master node (refer to the section Establish an SSH connection to IPC Agility nodes on
page 56).
2. Run the check_app_006_ndt.sh script.
All the tests should be successful.

September 8, 2023 IPC Confidential

 5.2: Check the Network Diagnostic Tool (NDT) installation 41

Figure 24: NDT URL in check_app_006_ndt.sh script result

3. Connect to the NDT Web UI: https://[ingress-nginx customer EXTERNAL-IP]/

ndt, https://172.29.186.39/ndt, for the example above..

IPC Confidential IPC Agility 3.2

Platform checks
42 5.3: Check the NGSC installation

Figure 25: Network Diagnostic Tool Web UI

5.3 Check the NGSC installation

5.3.1 Check Redis
This topic describes how to check the Redis deployment on the IPC Agility platform.
To check Redis, use the check_app_002_redis.sh script from the Agility Tools application.
1. Connect to the master node (refer to the section Establish an SSH connection to IPC Agility nodes on
page 56).
2. Run the check_app_002_redis.sh script.
All the tests should be successful.

September 8, 2023 IPC Confidential

 5.3: Check the NGSC installation 43

Figure 26: USC (NGSC) URL in check_app_002_redis.sh script result

5.3.2 Check RabbitMQ configuration on the platform

This topic describes how to check the RabbitMQ deployment on the IPC Agility platform.
To check RabbitMQ, use the check_app_001_rabbit.sh script from the Agility Tools
application.
1. Connect to the master node (refer to the section Establish an SSH connection to IPC Agility nodes on
page 56).
2. Run the check_app_001_rabbit.sh script.
All the tests should be successful.
Figure 27: USC (NGSC) URL in check_app_001_rabbit.sh script result

5.3.3 Check USC (NGSC) configuration on the platform

This topic describes how to check the USC (NGSC) deployment on the platform.

IPC Confidential IPC Agility 3.2

Platform checks
44 5.3: Check the NGSC installation

To check USC (NGSC), use the check_app_009_ngsc.sh script from the Agility Tools
application.
1. Connect to the master node (refer to the section Establish an SSH connection to IPC Agility nodes on
page 56).
2. Run the check_app_009_ngsc.sh script.
All the tests should be successful.
Figure 28: USC (NGSC) URL in check_app_009_ngsc.sh script result

3. You can also use your Web browser to check that you access to the Unigy Soft Client Sign-in page.

Note
The screenshot below displays the login page of the USC (NGSC) Web server. If SSO is enabled, the
login page returned by the customer IDP should be different from the login page display above.

September 8, 2023 IPC Confidential

 5.4: Check the Prometheus installation 45

Figure 29: Unigy Soft Client login

4. You can also see the USC (NGSC) logs, input the following commands:
• For the USC (NGSC) authentication service:
[root@k8s-master-cao ipctech]# kubectl logs -n ngsc [ngsc-
authentservice-podname]

• For the USC (NGSC) webserver:

[root@k8s-master-cao ipctech]# kubectl logs -n ngsc [ngsc-webserver-
podname]

• For the USC (NGSC) websocket server:

[root@k8s-master-cao ipctech]# kubectl logs -n ngsc [ngsc-websocket-
server-podname]

• For the USC (NGSC) Sipservice:

[root@k8s-master-cao ipctech]# kubectl logs -n ngsc [ngsc-sipservice-
server-podname]

5.4 Check the Prometheus installation

This topic describes how to check the Prometheus installation.
To check Prometheus, use the check_app_005_prometheus.sh script from the Agility Tools
application.
1. Connect to the master node (refer to the section Establish an SSH connection to IPC Agility nodes on
page 56).
2. Launch the check_app_005_prometheus.sh Agility Tools script.

IPC Confidential IPC Agility 3.2

Platform checks
46 5.5: Check Recording Check

Figure 30: Prometheus application test

All the tests should be successful.

5.5 Check Recording Check

This topic describes how to check the Recording Check installation on the IPC Agility platform.
To check SLIC, use the check_app_010_slic.sh script from the Agility Tools application.
1. Connect to the master node (refer to the section Establish an SSH connection to IPC Agility nodes on
page 56).
2. Launch the check_app_004_recordingcheck.sh Agility Tools script.

September 8, 2023 IPC Confidential

 5.5: Check Recording Check 47

Figure 31: Recording Check application test

All the tests should be successful.

3. Check the Recording Check Web UI access: https://<Customer Ingress IP>/
recordingcheck.
4. Log in to the Recording Check Web application. You can ask your local support for log in credentials.
Communication is established and the Recording Check Web application login window is displayed.
Figure 32: Recording Check - launching web application

For Recording Check usage, refer to the Recording Check Administration Guide.

IPC Confidential IPC Agility 3.2

Platform checks
48 5.6: Check the SCD installation

5.6 Check the SCD installation

This topic describes how to check the SCD installation.
To check SCD, use the check_app_007_scd.sh script from the Agility Tools application.
1. Connect to the master node (refer to the section Establish an SSH connection to IPC Agility nodes on
page 56).
2. Run the check_app_007_sdc.sh script.
All the tests should be successful.
Figure 33: check_app_007_scd.sh script result

3. Check the SCD Web page: https://<Customer Ingress IP>/svc/oneclick.

September 8, 2023 IPC Confidential

 5.7: Check the SLIC configuration 49

Figure 34: Login form

4. Log in to the SCD Web UI. You can ask your local support for log in credentials.
5. In SCD Web UI, the Click Enterprise Servers Plan and check that the CCM server is displayed in
the Server Profile menu.

5.7 Check the SLIC configuration

This topic describes how to check the SLIC configuration.
To check SLIC, use the check_app_010_slic.sh script from the Agility Tools application.
1. Connect to the master node (refer to the section Establish an SSH connection to IPC Agility nodes on
page 56).
2. Launch the check_app_010_slic.sh Agility Tools script.

IPC Confidential IPC Agility 3.2

Platform checks
50 5.7: Check the SLIC configuration

Figure 35: SLIC application test (1)

September 8, 2023 IPC Confidential

 5.7: Check the SLIC configuration 51

Figure 36: SLIC application test (2)

All the tests should be successful.

3. You can also check the sending of logs to the IPC Splunk server.
a) Connect to https://splunk/.
b) Click Common Services Platform ➤ Rag logs ➤ Slic and select your platform in the Platform
scrolling list. Refer to the splunk_hostname value defined in Gitea, and confirm the presence
of logs.
For more information about the Splunk monitoring, refer to the IPC Agility Monitoring Guide using
Splunk.

IPC Confidential IPC Agility 3.2

Platform checks
52 5.7: Check the SLIC configuration

Figure 37: Confirm the presence of logs

5.7.1 SLIC logs on Splunk dashboard

This topic describes how to check if the SLIC application is running properly using the Splunk
dashboard and log transmission.
Prerequisite: Log on to the Splunk server and select the Common Services Platform button to
access the IPC Agility main dashboard (named All). For more details, refer to IPC Agility Monitoring
using Splunk guide.
Using the following filtering options, check to confirm that the IPC Agility platform generates log
events from its nodes (masters and workers):
• Platform: Corresponds to the given name of an IPC Agility platform.
• Slic ID: Corresponds with a specific SLIC instance belonging to a given IPC Agility platform.
• Node ID: Corresponds with a specific node (i.e. master or worker) belonging to a given IPC Agility
platform.

September 8, 2023 IPC Confidential

 5.7: Check the SLIC configuration 53

On the Platform filtering option, select your corresponding IPC Agility platform name and check that
the logs are being properly generated.
Figure 38: Platform filtering option on Splunk

Optionally, you can use both Slic ID and Node ID filtering options to check that the logs are being
properly transmitted from a specific node.
Figure 39: Slic ID and Node ID filtering options on Splunk

Check the SLIC application, as follows:

IPC Confidential IPC Agility 3.2

Platform checks
54 5.7: Check the SLIC configuration

Table 15: SLIC application checklist

Check Expected result Current

status
On the Platform filtering option (through Logs are generated properly (by sending info,
Splunk dashboard), select the chosen warning, errors or other/unknown events).
IPC Agility platform. Check that the log The number of logs is not null.
events are being generated properly.
SLIC instance generates logs in Splunk
dashboard.
Optional check: Logs are generated properly (by sending info,
Use both the Slic ID and Node ID filtering warning, errors or other/unknown events).
options that belongs to the tested IPC The number of logs is not null.
Agility platform. Check that the log
events are being generated properly.

5.7.2 Unigy log forward to SLIC instance

This topic describes the checking of SLIC instance receiving properly the Unigy logs.
Prior to perform the following checks, you have to configure a Unigy system to send syslog and filebeat
logs to the SLIC instance. For enabling syslog and filebeat on a Unigy system, refer to Enable Log
Sources on Unigy Node chapter on the SLIC Installation and Administration Guide'.
Execute some actions on the Unigy system (i.e. UMS login/logout, turret restart) and log on to the
Splunk server. Select Search & Reporting menu.
Figure 40: Select Search & Reporting menu

In the search field, type the command index=* host=<platformname> source="/opt/

slicin/<SLICinstanceID>/whitelist/<SLICindex>/<nodetype>/*" where variables
must be replaced as following:
• platformname is your platform name
• SLICinstanceID is the SLIC instance ID used with your platform and set in
slic_details.conf file
• SLICindex is the SLIC index set in slic_details.conf file
• nodetype is the SLIC node set in slic_nodes.conf file
Command example: host=cao6qa index=* source="/opt/slicin/
LAB_PARIS_SLIC001/whitelist/ap_amer_u360/ACCM/*"

September 8, 2023 IPC Confidential

 5.7: Check the SLIC configuration 55

Figure 41: Search command example

Table 16: Unigy log forward to SLIC instance checklist

Checking Expected result Current

status
Generate Unigy logs and check the log Log events are displayed on Search menu
presence on Splunk events with appropriate
search command (i.e. index=*
host=<platformname>
source="/opt/slicin/
<SLICinstanceID>/whitelist/
<SLICindex>/<nodetype>/*")

IPC Confidential IPC Agility 3.2

Platform checks
56 A.1: Establish an SSH connection from customer network

A Establish an SSH connection to IPC

Agility nodes
This section gives the two ways to connect to an IPC Agility node.
Up to IPC Agility V3.0, there are two ways to launch an SSH connection to an IPC Agility node:
• If ION has been installed and configured (refer to the 3 - IPC Agility ION Installation and
Configuration Guide), you can use a remote PRIISMS (ION) connection (refer to the section
Establish an SSH connection with PRIISMS on page 60).
• If an ION connection is not available and if the list of CCMs/ACCMs has been configured on the IPC
Agility platform, you can establish an SSH connection by using a CCM shelldiag account (refer to the
section Establish an SSH connection from customer network on page 56).
From IPC Agility V3.1, only the connection through the Unigy CCMs/ACCMs available.

A.1 Establish an SSH connection from customer network

This section describes how to use a CCM shelldiag to establish an SSH connection with a IPC Agility
master or worker VM.
1. In your SSH client launch the following address to connect to a CCM:
shelldiag@CCM IP address

Figure 42: Example of shelldiag connection with PuTTy

2. Follow the section Establish an SSH session using the shelldiag account on page 57 to enable the
SSH connection.

September 8, 2023 IPC Confidential

 A.2: Establish an SSH session using the shelldiag account 57

3. Once connected to the CCM using shelldiag, connect to the IPC Agility node with the following
command.
ssh ipctech@nodeIPaddress-customer

Where nodeIPaddress-customer is the node IP address on the customer network.

4. Launch your SSH client and connect to the VM with the virtual address displayed in the popup
window. Use the ipctech user account and the password set during the VM creation (refer to 2 - IPC
Agility VM Creation documentation).

Note
Connect as root using the su command:

[ipctech@k8s-master-cao ~]$ su
Password:
[root@k8s-master-cao ipctech]#

A.2 Establish an SSH session using the shelldiag account

Follow this procedure to establish an SSH session on a Unigy appliance in a customer’s Unigy
enterprise. This involves initiating an SSH session, obtaining a token, entering the token in the Password
Manager to generate a password, then returning to the SSH session and using the password to establish
the session.

Note
An SSH session can be initiated from a client, such as PuTTY, or directly (by connecting a laptop to the
console port on the appliance) or remotely (by using the Integrated Dell Remote Access Controller
(iDRAC)) from the CCM console. The following procedure describes the behavior of the interface when
using an SSH client. There are slight differences in the interface and interaction with the interface when
accessing from the CCM console.

The shelldiag account provides SSH access to a CCM, ACCM, or MM.

Note
Sessions time out due to inactivity as follows:
• SSH sessions time out in five minutes
• Password Manager sessions time out in ten minutes

1. Open an SSH session to a CCM, ACCM, or MM and log in using the shelldiag account.

Note
A CCM console session can also be used to obtain the token.

IPC Confidential IPC Agility 3.2

Platform checks
58 A.2: Establish an SSH session using the shelldiag account

Figure 43: SSH shelldiag log in

When you press the Enter key, the start time of the validity period, the warning notice, and the
password token are displayed as illustrated in the following figure.
Figure 44: SSH shelldiag session - with token

2. Left click with the mouse and drag to copy the token from the SSH session. Do not include leading or
trailing spaces. As soon as you release the mouse left-click button, the token is automatically copied
to the clipboard.

Important
Do not press CTRL+C to copy the token. This sends a CTRL+C character to your session and causes
process interruption and causes the SSH session to close.

3. Access the Password Manager web application using either https://nw.dynamic.ipc.com or

https://ny.dynamic.ipc.com.
The Password Manager Login screen is displayed.
4. Enter your IPC Username and Password and click Login.
The Token input box is displayed.
5. Place your cursor in the Token input box, right click then Paste.
You can also manually type in the value.

September 8, 2023 IPC Confidential

 A.2: Establish an SSH session using the shelldiag account 59

Figure 45: Password Manager Generate Output

6. Click Generate Output.

The password is displayed in the Output field as illustrated in the following figure.
Figure 46: Password Manager Output

7. Copy the password from the Output field.

8. Return to the SSH session. Mouse right click the SSH cursor then press the Enter key to paste the
password. Optionally paste/enter the output by pressing the CTRL+Insert keys on your keyboard
or manually typing in the password.

Important
If you paste the output, the SSH client does not display the password, so be careful to only paste it
once.

The session begins.

For example, in the following figure, the shelldiag prompt indicates the user is logged in.

IPC Confidential IPC Agility 3.2

Platform checks
60 A.3: Establish an SSH connection with PRIISMS

Figure 47: SSH shelldiag successful log in

A.3 Establish an SSH connection with PRIISMS

This section describes how to use a PRIISMS session to open a SSH connection with the IPC Agility
master or worker VMs.

1. Launch the PRIISMS Web interface (https://ionv2a.ipc.com/) and log on with your IPC active
directory account.
2. Click Config ➤ Site.
The list of sites is displayed.
3. Click the Edit Site button dedicated to your site.
Figure 48: Log in to an IPC Agility VM using SSH #1

The Site Endpoints list is displayed.

4. Search for the endpoint corresponding to your VM (a master or worker node) and click GEN.

September 8, 2023 IPC Confidential

 A.3: Establish an SSH connection with PRIISMS 61

Figure 49: Log in to an IPC Agility VM using SSH #2

A popup window indicates that a connection is opened with your VM.

Figure 50: Log in to an IPC Agility VM using SSH #3

5. Launch your SSH client and connect to the VM with the virtual address displayed in the popup
window. Use the ipctech user account and the password you set during the VM creation (refer to 2 -
IPC Agility VM Creation documentation).

Note
When configuring IPC Agility applications with the COSA-APP, you must connect as root using the
su command before applying any Kubernetes commands:
ipctech account and the

login as: ipctech

[email protected]'s password:
Last login: Wed Jul 22 15:13:01 2020 from 100.64.152.1
[ipctech@k8s-master-cao ~]$ su
Password:
[root@k8s-master-cao ipctech]#

IPC Confidential IPC Agility 3.2

Platform checks
62 A.3: Establish an SSH connection with PRIISMS

Important
At the end, do not forget to close the PRIISMS session by clicking Close Connection.

September 8, 2023 IPC Confidential

 63

B Import the CA certificate

This section explains how to import a CA certificate in your laptop.
From your computer, execute the following steps to import the CA certificate:
1. Copy the certificate provided by IPC or the customer on your laptop.
2. Click the Start icon, enter the certlm.msc command and press the Enter key.
You can also use the Windows + R keys and then enter the certlm.msc command.
The certificate manager is displayed.
Figure 51: Certificate manager

3. Right-click the Trusted Root Certification Authorities and select All Tasks ➤
Import.
4. In the Certificate Importation Wizard window, click Next.
5. Click Browse and select the CA certificate provided by IPC or the customer and click Open.
Figure 52: Certificate Import Wizard

IPC Confidential IPC Agility 3.2

Platform checks
64

6. Click Next.
7. Click Finish.

September 8, 2023 IPC Confidential

 65

Index
A N
agility tools Nagios
script verifying installation 40
cron 24 NDT
manual check 24 checking installation 40
Agility Tools Network certificates 38
certificate ngsc deployment
expiration 31 checking 43
check_all_applications.sh 30
check_all_connectivity.sh 33 P
check_all_platform.sh 29
check_cert_cluster.sh 31 Prometheus
check_os_cluster 25 checking installation 45
kubernetes 31
master 29, 31, 33 R
network
master rabbitMQ deployment
worker 25 checking 43
RHEL 25, 29, 30 recording check
worker 29, 31, 33 checking installation 46
redis
C checking installation 42

check S
cron 24
manual check 24 SCD
connection checking installation 48
ssh 56 shelldiag
using 57
E SLIC
checking configuration 49
esxi license SLIC configuration
expiration date 12 checking 49

I U
install Unigy log forward to SLIC 54
certificate 63
IP address V
Customer network 15
Local network 15 VLAN ID configuration 13
IPC Agility deployment VM Autostart settings 21
Multi-server model 9 VM hardware profile 14
Standalone server model 9 VM names 18
VM network interface 15
L VM snapshot presence 19

Log presence in Splunk 52

IPC Confidential IPC Agility 3.2

Platform checks
Part Number: B02201019, Release: 00
IPC Systems, Inc.
777 Commerce Drive
Fairfield, CT 06825-5500 USA