Ericsson Internal

SYSTEM ADMINISTRATION GUIDE 1 (30)

Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

VNF-Lifecycle Manager System Administration

Guide

Copyright

© Ericsson AB 2016. All rights reserved. No part of this document may be

reproduced in any form without the written permission of the copyright owner

Disclaimer

The contents of this document are subject to revision without notice due to
continued progress in methodology, design and manufacturing. Ericsson shall
have no liability for any error or damage of any kind resulting from the use
of this document.

Contents

1 About This Document . .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. . 3
1.1 Purpose ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. . 3
1.2 Target Group .. ... ... .. ... .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. . 3
1.3 Prerequisites . .. ... ... .. ... .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. . 3
1.4 Typographic Conventions . ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. . 3

2 VNF-Lifecycle Manager System .. .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. . 4

3 Configuration Information . ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. . 5
3.1 OSS-RC Specific Configuration . ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. . 5
3.2 Cloud Resources Orchestration .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. . 7

4 License Installation . ... .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 13

5 Administering Workflows .. ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 13
5.1 Install Workflows . ... .. ... .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 13

6 System Backup and Restore . .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 14
6.1 Backup . ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 14
6.2 Changing Automatic Backup Schedule Configuration . .. ... ... .. ... .. ... ... .. 16
6.3 Restore . ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 18

7 Standard Maintenance Procedures ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 19
7.1 File System Maintenance . ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 19
7.2 Log Files and Dump Locations on Virtual Machine .. ... .. ... ... .. ... .. ... ... .. 21
7.3 Database Administration Tasks .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 22
7.4 Certificate Maintenance . .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 24

8 Logging and Tracing ... .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 24

A4 XSEIF R6
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 2 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

8.1 View Jboss Server Logs . .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 24
8.2 Add a log category in the JBOSS instance . .. ... ... .. ... .. ... ... .. ... .. ... ... .. 24
8.3 Change the log level of an existing log category in JBOSS Instance .. ... .. 25

9 Corrective Maintenance . ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 27
9.1 Unable to Connect to cloud manager . ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 27
9.2 ECM Certificate Error ... .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 27

10 Troubleshooting ... .. ... .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 28
10.1 VNF Lifecycle Manager UI Not Working . ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 29

11 Terminology . ... ... .. ... .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 29

12 References .. .. ... ... .. ... .. ... ... .. ... .. ... ... .. ... ... .. ... .. ... ... .. ... .. ... ... .. 30
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 3 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

1 About This Document

1.1 Purpose

This document contains information for System Administration tasks for VNF
Lifecycle Manager (VNF-LCM) application in the OSS-RC system

1.2 Target Group

The intended target groups for this document are the following:

• System Administrator

1.3 Prerequisites

Readers of this document must have root authority to perform the tasks listed. It
is also assumed that the readers of this document are familiar with the following:

• Have a working knowledge of Operation and Maintenance

• Is familiar with:

0 OSS-RC

0 Ericsson Cloud Execution Environment

0 Ericsson Cloud Manager

0 Red Hat Linux (RHEL)

0 JBoss Application Server

0 PostgreSQL

0 VNF Lifecycle Manager

1.4 Typographic Conventions

The typographic conventions for all Customer Product Information (CPI) in

OSS-RC are found in OSS Library Typographic Conventions. See Reference[4]
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 4 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

2 VNF-Lifecycle Manager System

Figure 1 VNF Lifecycle Manager Architecture Overview

VNF Lifecycle Manager consists of 2 main components that together provides

lifecycle management for VNFs:

• VNF Lifecycle Management Workflows

VNF Lifecycle Management use cases are realized as workflow with an

ordered sequence of steps. The workflows are deployed over automation
framework, on JBOSS Application server.

• VNF Lifecycle Automation Framework

VNF Lifecycle Automation Framework provides workflow execution

environment and a web based application for managing the VNF lifecycle
management workflows. The framework is made up of following logical
blocks:

0 Apache Httpd - The static UI components of VNF Lifecycle Manager

web UI are installed on the web server, Apache httpd. It handles
HTTPS request from the OSS-RC OEX Users.
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 5 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

0 Workflow Service - The VNF Lifecycle Management workflows are

executed by the Workflow Engine provided by Workflow Service. The
execution of the workflows are processed and visualized in VNF LCM
UI. Workflow Service is deployed in JBOSS Application Server.

0 ServiceFramework - This will provide the common service interfaces

related to configuration, logging, messaging service.It is also deployed
on JBOSS Application Server.

0 VNF LCM Services - Other supporting services like Auto Start Service
are also deployed in JBOSS server. Auto Start Service listens to
OSS-RC Alarms and starts a workflow automatically on receipt of an
alarm matching a trigger criteria.

0 Postgres DB - The Postgres DB provides the support for creating

database schema and tables required for the VNF. It persists the
VNFLCM workflows data.

VNF Lifecycle Manager deployment on cloud consists of two Virtual Machines

that use RedHat Linux as the guest operating system:

• VNFLAF-Services

0 This VM consists of the JBOSS application server where VNF

Lifecycle Manager Services and VNF lifecycle management workflows
are deployed

0 The static UI components are deployed in Apache httpd server.

• VNFLAF-DB

0 This VM is for VNF Lifecycle Manager’s database (PostgresSQL).

3 Configuration Information

3.1 OSS-RC Specific Configuration

Following are the details related to OSS-RC specific configuration applied in
the VNF Lifecycle Manager.

3.1.1 Configuration Parameters.

In VNF Lifecycle Manager System, there are certain configuration parameters,
the administrator can update them using a system utility.

VNF Lifecycle Manager need to connect with the OSS-RC to perform some
operations while executing the workflow.

Following are the configuration Parameters required for communication with

OSS-RC and their default Values
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 6 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

Table 1 Configuration parameters specific to OSS-RC

Name Description Default Value Type

ossType parameter OSSRC String
describes the
type of OSS
ossRcMasterHos parameter String
tName describes the
host name
required to
connect to
OSS-RC
ossRcFtpPortNu parameter 22 Integer
mber describes the
ftp port number
required to
connect to
OSS-RC
ossRcFtpUserNa Authorized user nmsadm String
me of OSSRC Server
for VNF handling
ossRcFtpPasswo parameter - String
rd describes the
password for
above ftp user
required to
connect to
OSS-RC
ossRcSshPortNu parameter 22 Integer
mber describes the
ssh port number
required to
connect to
OSS-RC
ossRcSshUserN Authorized user nmsadm String
ame of OSSRC Server
for VNF handling
ossRcSshPassw parameter - String
ord describes the
password for
above ssh
user required
to connect to
OSS-RC
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 7 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

ossRcVnfArneGe parameter /var/opt/ericsson/ String

neratedFilesPath describes the vnflaf/workflows/
Path for Arne tmp
generated files in
OSS-RC
ossRcVnfArneDir parameter /tmp/vnflaf String
OnOSSMaster describes the
Arne Directory
Path present in
OSS-RC master
ossRcNameServi parameter 49254 String
cePort describes the
OSS-RC service
port
poolingIntervalF parameter 120 Long
M describes the
interval at which
FM Adapter will
keep checking
subscription
status with
OSS-RC
Notification
Service

The command to update the above parameter values is as follows :

/ericsson/pib-scripts/etc/config.py <read/create/update>
--app_server_address <hostname>:8080 --name=<paramName>
--value=<paramValue> --type=<paramType> --scope=GLOBAL

For example: To update the parameter name as ecmTenantId, run the

command to update with required value to be set

/ericsson/pib-scripts/etc/config.py read --app_server_ad

dress localhost:8080 --name=ecmTenantId --value=tenant1
--type=String --scope=GLOBAL

3.2 Cloud Resources Orchestration

3.2.1 Cloud VIM Configuration Properties

Following are the configuration parameters and their default values.
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 8 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

Table 2 Configuration parameters specific to cloud VIM like ECM, CEE,

OpenStack

Name Description Default Value Type

ecmTenantId This specifies the VNF-LAF String
ECM tenant ID
ecmDefaultVdcO The create vdc SAMPLE_VDC_ String
ffer offer name to be OFFER
passed in user
task
ecmBaseURL This specifies https://ieatecm03 String
the base URL .athtem.eei.erics
to connect the son.se/ecm_serv
Ericsson Cloud ice
Manager
ecmUserName The User Name xid String
in ECM to
be used by
VNF-LCM to
create the VNFs.
ecmUserPasswo The Password in - String
rd ECM to be used
by VNF-LCM to
create the VNFs.
cloudType Cloud Manager ECM String
type. ECM/CEE/
Openstack - This
input will be used
by VNF-LCM
to prepare the
environment
to execute
workflows.
Example - ECM
waitPeriod This parameter P0DT0H0M30S String
describes time
period used
to wait for
responses from
ECM
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 9 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

deleteBsvs This parameter false Boolean

describes a
boolean value
which determines
whether to delete
Bsvs or not
cloudIpAddress IP address of <IP address of String
Cloud Manager cloud manager>
ECM/CEE/Open
stack Keystone.
Example -
141.137.212.29

Use the command explained in sec. 3.1.1 to update the cloud Configuration
Parameters.

3.2.2 Cloud VIM Certificate Installation

All web communication with ECM or CEE happens through secure HTTPS
protocol. To enable this, SSL certificates of cloud mangers need to be installed
on VNFLAF service VM.

To install the certificate if it's not installed or expired; follow the steps mentioned

1. Using openssl utility download the cloud manager certificate from the
server

• The command to download the certificate from CEE is

openssl s_client -showcerts -connect <CEE-HOST

or CEE IP ADDRESS>:5000 -CAfile root-ca.cert
< /dev/null | openssl x509 -outform DER >
/tmp/sslcert.cer

Note:

• i) Due to below bug, root-ca certificate of CEE server

is required to be obtained from CEE team and passed
as an argument.

bug HV22698: Atlas: root certificate missing with

-showcerts option on the openssl client connection

• ii) Port number is 5000 for CEE

• The command to download the certificate from ECM is

openssl s_client -showcerts -connect <ECM-HOST or

ECM IP ADDRESS>:443 < /dev/null | openssl x509
-outform DER > /tmp/ecm_cert.cer
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 10 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

The command stores certificate under /tmp/ folder with file extension .cer,
this same file is used while installing the certificat in next steps

2. Change directory to jre/lib/security for example: cd

/usr/java/jdk*/jre/lib/security

3. Use keytool util to import the certificate using the command $JAVAPA
TH/keytool -import -alias ca -file /tmp/ecm_cert.cer
-keystore /usr/java/jdk*/jre/lib/security/cacerts
-storepass changeit Here $JAVAPATH is the java bin directory path

For example:

/usr/java/jdk*/bin/keytool -import -alias ca -file

/tmp/ecm_cert.cer -keystore //usr/java/jdk*/jre/lib/se
curity/cacerts -storepass changeit

Owner: [email protected],
CN=ecmapp.domain.com, OU=ECM, O=Ericsson, L=Piscataway,
ST=NJ, C=US

Issuer: [email protected],
CN=ecmapp.domain.com, OU=ECM, O=Ericsson, L=Piscataway,
ST=NJ, C=US

Serial number: c74cdb4c97942206

Valid from: Mon Jul 06 04:53:11 EDT 2015 until: Tue

Jul 05 04:53:11 EDT 2016

Certificate fingerprints:

MD5: 45:F4:EB:15:13:D0:AF:26:01:0B:4A:10:BE:BF:2A:12

SHA1: DE:A4:77:15:8A:43:4D:62:29:FA:9D:83:CC:D0:E6:BE:
D9:E5:FC:AE

SHA256: 11:A7:AD:B9:C1:20:BE:00:F4:25:3C:4D:18:5F:59:2
2:E3:52:CE:13:D4:77:20:7F:94:30:86:20:96:24:AF:E4

Signature algorithm name: SHA1withRSA

Version: 1

Trust this certificate? [no]: yes

Certificate was added to keystore

 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 11 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

3.2.3 Verify Certificate Installation

To check if cloud manager certificate is already been installed in the system,
use the mentioned steps

1. Check if certificate is installed correctly

• For ECM : /usr/java/jdk*/bin/keytool -list -v

-keystore /usr/java/jdk*/jre/lib/security/cacerts |
grep -i ecm

• For CEE :

/usr/java/jdk*/bin/keytool -list -v -keystore

/usr/java/jdk*/jre/lib/security/cacerts | grep -i
cee

2. b. Enter keystore password:Press Enter

3. Output:

***************** WARNING WARNING WARNING ***********

******

* The integrity of the information stored in your

keystore *

* has NOT been verified In order to verify its

integrity, *

* you must provide your keystore password. *

***************** WARNING WARNING WARNING ***********

******

Note: If certificate is not installed the output of the command results with no
details of certificate, on such case install the certificates by run steps
mentioned in sec. 3.2.2.

If certificate is installed correctly, then the command output provides

the details of certificate installed.

For example:

/usr/java/jdk*/bin/keytool -list -v -keystore

/usr/java/jdk*/jre/lib/security/cacerts | grep -i
ecm

Enter keystore password:Press Enter

 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 12 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

Owner: [email protected]
on.se, CN=ecmapp.domain.com, OU=ECM, O=Ericsson,
L=Piscataway, ST=NJ, C=US

Issuer: [email protected]
son.se, CN=ecmapp.domain.com, OU=ECM, O=Ericsson,
L=Piscataway, ST=NJ, C=US

3.2.4 Check Certificate Expiry

ECM Certificate installed has validity time period, it is required to check the
expiry and install the certificate again.

Run the command /usr/java/jdk*/bin/keytool -list -v

-keystore //usr/java/jdk*/jre/lib/security/cacerts -alias
ca and in the output of this command look for "Valid From" line and check the
certificate expiry date.

Enter keystore password:Press Enter

For Example, the output of the above command is as follows

Alias name: ca
Creation date: Apr 12, 2016
Entry type: trustedCertEntry
Owner: [email protected],
CN=ecmapp.domain.com, OU=ECM, O=Ericsson, L=Piscataway,
ST=NJ, C=US
Issuer: [email protected],
CN=ecmapp.domain.com, OU=ECM, O=Ericsson, L=Piscataway,
ST=NJ, C=US
Serial number: c74cdb4c97942206
Valid from: Mon Jul 06 04:53:11 EDT 2015 until: Tue Jul
05 04:53:11 EDT 2016
Certificate fingerprints:
MD5: 45:F4:EB:15:13:D0:AF:26:01:0B:4A:10:BE:BF:2A:12
SHA1: DE:A4:77:15:8A:43:4D:62:29:FA:9D:83:CC:D0:E6:BE:D9
:E5:FC:AE
SHA256: 11:A7:AD:B9:C1:20:BE:00:F4:25:3C:4D:18:5F:59:22:E
3:52:CE:13:D4:77:20:7F:94:30:86:20:96:24:AF:E4
Signature algorithm name: SHA1withRSA
Version: 1

Note: User must provide the correct alias name of the certificate, in the
example use, the value used is "ca" which is given at the time of
installation of certificate.

Remove Certificate Alias

In a scenario, where the certificate is expired and requires a new certificate to

be installed, User must delete the previously installed certificate. Remove the
alias entry from the keystore before the installation.
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 13 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

To remove the alias use the command /usr/java/jdk*/bin/keytool

-delete -keystore /usr/java/jdk*/jre/lib/security/cacerts
-alias ca

Then follow the steps mentioned in the sec. 3.2.2 to install the new certificate.

4 License Installation
Sentinel license be installed for the new VNF-Lifecycle Manager feature on
OSS-RC server.
Steps to check license

• Log on to the OSS-RC Server and open a command prompt

• Run the command to verify if license is installed or not

/opt/Sentinel/bin/lsmon | grep -i CXC4011979

If the license is installed then command will result in the output as follows

|- Feature name : "CXC4011979"

• If the output is empty, then license installation is required to use

this feature, See Reference [3] for more information on license key
administration for OSS features

5 Administering Workflows

5.1 Install Workflows

The Workflow bundle can be installed by executing the following steps :

• Stop Jboss Server.

• For installation, run the following command on the RPM which has the
workflows

rpm -ivh <rpm_name>

For Example ,

rpm -ivh /tmp/ERICvEPG-workflows_CXP12345678-1.0.1.rpm

• Copy the workflow bundle RPM to the/vnflcm-ext/backups/workfl

ows for backup

For Example
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 14 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

cp /tmp/ERICvEPG-workflows_CXP12345678-1.0.1.rpm
/vnflcm-ext/backups/workflows/

• To check , what all workflow RPMs are already installed and their versions
, run the following command

rpm -qa | grep <keyword>

For Example ,

rpm -qa | grep vEPG-workflows

ERICvEPG-workflows_CXP12345678-1.0.1-1.noarch

6 System Backup and Restore

This section describes the backup and restore of the VNF-Lifecycle Manager .

6.1 Backup
The standard cron job will be used to backup the VNF Lifecycle Manager data.
Each of vnflaf-service and vnflaf-db VMs have their own cron job.

The system back up is taken by the cron job at weekly intervals.The

current default cron entry consists of executing the back-up script at every
Sunday, mid-night. The backup of the system is placed in a directory name
/vnflcm-ext/backups/vnflcm-<date-time>/

For Example,<<vnflcm-2016-08-18-09-46>> is directory name.

Inside this directory , it will have 3 sub directories.

• db :This will have 3 compressed files, containing backups of 3 DBs.

• config : This will contain compressed file which has configuration files.

• jboss_logs: Contains the compressed Jboss logs.

The backup directory , for example ,<<vnflcm-2016-08-18-09-46>> , once

created , is retained for 5 weeks. To change this duration, follow the below
steps.

• Login to vnflaf-service VM as cloud-user and then switch to root user

su - root

• Change directory to /opt/ericsson/ERICvnflafsgservice_CXP90

32544/scripts/admin

cd /opt/ericsson/ERICvnflafsgservice_CXP9032544/scrip
ts/admin
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 15 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

• Open the vnflcm.configuration file and replace the value of

WEEKS_TO_KEEP property to a desired value.

# How many weeks to keep weekly backups

WEEKS_TO_KEEP=5

Caution!
The backups are stored on the same disk as where the current (or live) data of
the system is stored, hence it is highly recommended to transfer the backup
files periodically to an external storage. This will prevent loss of backup data on
disk failure.

Log Files

For each session, a log file is created. This log file is stored in a default location
/vnflcm-ext/current/logs/. The log holds all important actions taken
during the backup and restore process. It can be consulted after each run, to
be sure that the backup is finished as required.

In case of failure while taking the backup , user can check log file to see if there
are any "ERROR" message logged

For example: If Database back up failed due to some error, log file contains
message"[!!ERROR!!] Failed to produce database backup for
<<dbname>>

Take Manual Back Up

Apart from the scheduled automatic backups, it is possible to take a backup of

the system when required by executing the backup script

To take the manual backup , the backup of vnflaf-services VM should be taken

first. Follow the below steps to take the manual backup

Do!
Flow for Manual Backup

-->Take backup of vnflaf-services VM

-->Take backup of vnflaf-db VM.
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 16 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

• Login to vnflaf-service VM as cloud-user and then switch to root user

su - root

• Change directory to /opt/ericsson/ERICvnflafsgservice_CXP90

32544/scripts/admin

cd /opt/ericsson/ERICvnflafsgservice_CXP9032544/scrip
ts/admin

• Run the backup script

./vnflafservices_backup.sh

When the script finishes, it should display a log like below

2016_09_09:12:16:04 : BACKUP VNFLAFSERVICES : Services

Backup Completed...directory : /vnflcm-ext/backups/vnf
lcm-2016-09-09-12-16/

• Note the directory in which the backup has been taken, for example ,
/vnflcm-ext/backups/vnflcm-2016-09-09-12-16/ shown above.

• Now, login to vnflaf-db VM as cloud-user and then switch to root user

su - root

• Change directory to /opt/ericsson/ERICvnflafpersistencedb_CX

P9032663/sh

cd /opt/ericsson/ERICvnflafpersistencedb_CXP9032663/sh

• Run the backup script vnflafdb_backup.shand provide the earlier

noted backup directory path as an argument

./vnflafdb_backup.sh /vnflcm-ext/backups/vnflcm-2016-0
9-09-12-16/

Warning!
If additional workflows are installed on the server while backup activity is in
process, it may result in inconsistencies.

6.2 Changing Automatic Backup Schedule Configuration

The system admin can change the frequency or the cron jobs' configuration

To change the configuration follow the steps as follows

 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 17 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

1. Login to vnflaf-services VM as cloud-user and then switch to root user

su - root

2. Check the cron job entry by executing crontab -l

For example : crontab -l

[root@vnflaf-services ~]# crontab -l

0 0 * * 0 /opt/ericsson/ERICvnflafsgservice_CXP9
032544/scripts/admin/vnflafservices_backup.sh 2>>
/vnflcm-ext/backups/logs/vnflaf_backup.err

3. Enter into the edit mode by executing crontab -e

crontab -e and edit the field 0 0 * * 0 based on the frequency of back

up to be taken.

Refer man page of crontab for valid options available man crontab

4. Now, login to vnflaf-db VM as cloud-user and then switch to root user

su - root

5. Check the cron job entry by executing crontab -l

For example : crontab -l

[root@vnflaf-services ~]# crontab -l

0 0 * * 0 /opt/ericsson/ERICvnflafsgservice_CXP9
032544/scripts/admin/vnflafservices_backup.sh 2>>
/vnflcm-ext/backups/logs/vnflaf_backup.err

6. Enter into the edit mode by executing crontab -e

crontab -e and edit the field 0 0 * * 0 , make the same changes as done
for vnflaf-services VM in step 3 above

Warning!
The cron jobs for vnflaf-service VM and vnflaf-db VM should be configured to
have the exactly same frequency. If a change in the frequency configuration is
made in the cnflaf-service VM, same change should be made in vnflaf-db VM.
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 18 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

6.3 Restore

This section describes the steps to be followed to restore the VNF LAF system
that was previously backed up.

Attention!
Before restore is performed , please be aware that the current state of system
like active workflow instance state, system configuration modified post previous
backup etc. will be lost on applying the restore.
It is recommended to ensure that there are no active workflow instances
executing and take a manual back up of the current system so that, it is
possible to roll back to current state if required.
See Manual Backup for steps to take backup

Do!
Flow for Restore

-->Perform restore of vnflaf-db VM

-->Perform restore of vnflaf-services VM

Following are the steps to perform system restore

1. Login to vnflaf-service VM as cloud-user and then switch to root user

su - root

2. Stop the Jboss server by executing the below command

service jboss-as-standalone stop

3. Login to vnflaf-db as cloud-user and then switch to root user su - root

4. Change directory to /opt/ericsson/ERICvnflafpersistencedb_CX

P9032663/sh

cd /opt/ericsson/ERICvnflafpersistencedb_CXP9032663/sh

5. Invoke the restore script vnflafdb_restore.sh and pass the complete

back-up directory path <<back-up directory

For example ./vnflafdb_restore.sh /vnflcm-ext/backups/vnf

lcm-2016-09-09-12-16/
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 19 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

Executing the above command it will restore the database backup taken
on date 2016-09-19 , taken at 12:16 hours.

6. Now, come back to vnflaf-services VM, user should be root user

7. Change directory to /opt/ericsson/ERICvnflafsgservice_CXP90

32544/scripts/admin

cd /opt/ericsson/ERICvnflafsgservice_CXP9032544/scrip
ts/admin

8. Invoke the restore script vnflafservices_restore.sh and pass the

complete back-up directory path <<back-up directory

For example ./vnflafservices_restore.sh /vnflcm-ext/backu

ps/vnflcm-2016-09-09-12-16/

Executing the above command, it will restore the services backup taken on
date 2016-09-09 , taken at 12:16 hours.

9. To start the Jboss server run the command service jboss-as-standalone

start

Attention!
Always provide same backup directory path to restore both vnflaf-db and
vnflaf-services otherwise there may be inconsistencies.

After successful execution of restore script, the System must be restored back
up and all the required services should be up and running.

7 Standard Maintenance Procedures

The following tasks are advised to be considered as part of general
administration tasks to ensure that the system is operating as efficient as
possible.

7.1 File System Maintenance

Context Description

This section describes the regular system administration tasks with regard to
file system maintenance and how often they are to be performed. It is important
that any file system is not allowed to reach 100% capacity as it can cause
system instability/outage.
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 20 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

Prerequisites

• The root password has already been changed in the installation process
and is known to System Administrator

• A command console has been opened.

Expected Result

The system administrator has checked and verified that the file system usage
is less than 80%.
Steps

1. Login to vnflaf-services VM as cloud-user and then switch to root user

su - root.

From the console , execute df -lh command to check free disk space

[root@vnflaf-services ~]# df -lh

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/VolGroup00-LogVol00

8.1G 2.1G 5.6G 27% /

tmpfs 1.9G 0 1.9G 0% /dev/shm

/dev/vda1 190M 61M 119M 34% /boot

8.1G 2.1G 5.6G 27% /

[root@vnflaf-services ~]#

2. Login to vnflaf-db VM as cloud-user and then switch to root user su - root

From the console, execute df -lh command to check free disk space

[root@vnflaf-db ~]# df -lh

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/VolGroup00-LogVol00

8.1G 1.8G 5.9G 23% /

tmpfs 1.9G 0 1.9G 0% /dev/shm

/dev/vda1 190M 61M 119M 34% /boot

/dev/vdb 976M 76M 849M 9% /opt/rh/postgresql92/r

 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 21 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

[root@vnflaf-db ~]#

3. If any file system has over 80% usage, then the system administrator must
clarify that the disk space usage is consistent with expectations otherwise
action needs to be taken to prevent the file system to continually grow

4. This task should be performed daily.

7.2 Log Files and Dump Locations on Virtual Machine

Context Description

It is important for system administrator to be able to reference all the necessary

log files and application dump if they exist.

Prerequisites

• The root password should have already been changed in the installation
process and must be known to System Administrator

• A command console has been opened.

Expected Result

The system administrator is able to view all generated logs and application
dump files. If the result is not expected , contact local Ericsson support.

Details

JBOSS Logs

All JBOSS logs are stored locally in /ericsson/3pp/jboss/standalo

ne/log

3PP and System Logs

As standard, most 3PP and system logs are available locally in /var/log

command-executor-service- vnflaf provides the service to execute the long

running script asynchronously.

The logs relevant to this task are placed under /var/log/cmd-exec-service

in file cmd-exec-service.log

Along with this file two more logs file will be created with .out and .error
extensions. The .out extension file contains output information logged by the
script exection and .error contains any errors related to this task.

Dumps

All application memory and core dump files are located in /ericsson/oss/
dumps/
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 22 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

7.3 Database Administration Tasks

Context Description

This section describes the regular system administration tasks with regard to
database maintenance and how often they are to be performed. It is important
that database is monitored on a regular basis to have efficient database
performance.

Prerequisites

• The password to vnflaf-db VM and user accounts are known to the System
Administrator.

• Postgres DB is installed and running.

Expected Result

The required database administration tasks are done

PostgresSQL Directory Structure.

PostgresSQL server is installed at path/opt/rh/postgresql92/ on the

vnflaf-db VM. This directory stores all of the PostgresSQL server configuration
files.

PostgresSQL server is initialized at path/opt/rh/postgresql92/root/va

r/lib/pgsql/data on the vnflaf-db where Postgres is running.

PostgresSQL database tasks

Table 3 Database tasks

 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 23 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

TASK COMMENT Time Interval, Task

has to be performed
(Manually/Automatica
lly
Check the Database To ensure efficient It is recommended that
Disk space database performance, this task is performed
the database disk space manually on a weekly
needs to be checked on basis.
weekly basis. Run the
commanddf -kh from
vnflaf-db VM.

The vnflaf-db data is

present at /opt/rh/po
stgresql92/root/v
ar/lib/pgsql/data.
Query the Database • Log on vnflaf-db as It is recommended that
Size cloud-user and this task is performed
then switch to root manually on a weekly
user su - root basis.

• Run su -
postgres

• Change to the
PostgresSQL bin
directorycd /opt/r
h/postgresql92/
root/usr/bin

• Run the command

to get the list
of database
available./psql
-l

• Run the query on

each of the database
to get the database
size./psql
SELECT pg_size_
pretty(pg_datab
ase_size(
'<db_name>'))
For Example: SELE
CT pg_size_pret
ty(pg_database_
size('wfsdb'))
pg_size_pretty
------------
9165 kb
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 24 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

7.4 Certificate Maintenance

This section describes the tasks related to the VNF Lifecycle Manager Licenses
and Certificates validity and expiry check

Table 4 Monitor Certificates

TASK COMMENT Time Interval,Task has

to be performed (Man
ually/Automatically)
Monitor the Certificates Certificates used by It is recommended that
Validity Time the VNF Lifecycle this task is performed
Manager have validity Monthly by the Operator
time and must be
renewed before they
expire. For additional
information refer sec
3.2.4

8 Logging and Tracing

This chapter describes the logging and tracing.

8.1 View Jboss Server Logs

The latest jboss server logs of applications deployed are logged on to the log
file by name server.log. This log file can be read or viewed from path
/ericsson/3pp/jboss/standalone/log/.

8.2 Add a log category in the JBOSS instance

In this example, we will create the new log category on com.ericsson.oss.

services.vnflaf and set the logging level to DEBUG.
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 25 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

Table 5 Steps to add a log category

[root@vnflaf-services ~]# /ericsson/3pp/jboss/bin/jboss-

cli.sh -c

[standalone@localhost:9999 /] /subsystem=logging/logger=
com.ericsson.oss.service.vnflaf:add(level=DEBUG)

{"outcome" => "success"}

[standalone@localhost:9999 /] /subsystem=logging/logger=
com.ericsson.oss.service.vnflaf:read-attribute(name=lev
el)
{
"outcome" => "success",
"result" => "DEBUG"
}

8.3 Change the log level of an existing log category in JBOSS

Instance
The jboss server logs by default is configured with INFO level; this log level can
be modified at each application level.

In this example , we will change the log level of com.ericsson.oss log

category from INFO to DEBUG.

Table 6 Steps to change log level

[root@vnflaf-services ~]# /ericsson/3pp/jboss/bin/jboss-

cli.sh -c

[standalone@localhost:9999 /] /subsystem=logging/logger=
com.ericsson.oss:read-attribute(name=level)
{
"outcome" => "success",
"result" => "INFO"
}

[standalone@localhost:9999 /] /subsystem=logging/logger=
com.ericsson.oss:write-attribute(name=level,value=DEBUG)

{"outcome" => "success"}

[standalone@localhost:9999 /] /subsystem=logging/logger=
com.ericsson.oss:read-attribute(name=level)
{
"outcome" => "success",
"result" => "DEBUG"
}
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 26 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

Note: It should be noted that recommended log level is INFO, when changing
to DEBUG for troubleshooting scenarios it is advised to revert to INFO
once troubleshooting is complete.

Supported Logger Categories

Currently , below listed logger categories are supported for which the log level
can be modified

• com.arjuna

• org.apache.tomcat.util.modeler

• org.jboss.as.config

• sun.rmi

• jacorb

• jacorb.config

• com.ericsson

• com.ericsson.oss

• com.ericsson.oss.services.vnflaf

• com.ericsson.nms

• org.camunda

• org.jgroups.protocols.UDP

• org.jgroups

• org.openliberty.openaz

• com.ericsson.oss.itpf.NETWORK_STATUS_LOGGER

• com.ericsson.oss.itpf.COMMAND_LOGGER

• com.ericsson.oss.itpf.ERROR_LOGGER

• com.ericsson.oss.itpf.EVENT_LOGGER

• com.ericsson.oss.itpf.SECURITY_EVENT_LOGGER
Supported Log Levels

List of log levels that are currently supported for log categories:

• all

• trace
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 27 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

• debug

• info

• warn

• error

• fatal

9 Corrective Maintenance

9.1 Unable to Connect to cloud manager

If Workflow are unable to connect to cloud manager (e.g. ECM) and are failing,
the /ericsson/3pp/jboss/standalone/log/server.log file will have
the ERROR logged , see for the ERROR string as follows

2016-04-21 07:58:19,038 ERROR [com.ericsson.oss.serv

ices.vnflaf.sdk.tasks.reusable.ecm.utils.RestClient]
(http-/0.0.0.0:8080-8) IOException:: No route to host

or it will log the UnknownHostException in ERROR logs

As a corrective action, check ECM host entry is added to hosts file. The hosts
file is present in the location:-

/etc/hosts

run the command as less /etc/hosts | grep "ecmapp" and check

if following output is shown.

• For ECM ,less /etc/hosts | grep "ecmapp" and check if following

output is shown.

1x.xxx.xx.xxx [IP Address of ECM Hostname]ecmapp.doma

in.com

• For ATLAS,,less /etc/hosts | grep "ieatcee" and check if

following output is shown

1x.xxx.xx.xxx [IP Address of Atlas CEE Hostname]athtem.

eei.ericsson.se

9.2 ECM Certificate Error

During the execution of Workflows which invokes ECM Rest call, there is a
possibility of failure in case valid ECM Certificate is not installed in jboss.
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 28 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

The invalid certificate error can be found in the server.logs of JBoss server .
The sample ERROR log to be checked in logs is as follows

2016-02-22 11:22:18,389 ERROR [com.ericsson.oss.services.v

nflaf.sdk.tasks.reusable.ecm.utils.RestClient]
(Thread-1 (HornetQ-client-global-threads-1248218433))
IOException:: sun.security.validator.ValidatorException:
PKIX path building failed: sun.security.provider.certpath
.SunCertPathBuilderException:
unable to find valid certification path to requested
target

To resolve the ECM invalid or no certificate found error, please follow the steps
mentioned in section 3.2.2 Install ECM Certificate

10 Troubleshooting

This chapter describes how to extract data that needs be sent to the local
Ericsson help desk. In the events of faults or problems being encountered with
the feature that cannot be solved as per the previous chapter.

To, extract the information follow the steps

1. Log on to the Server where problem has been reported

2. Reproduce the fault after enabling the respective application logging level
to debug as described in this document.

3. Forward the/ericsson/3pp/jboss/standalone/log/server.log
file and the content of problem description using the normal fault reporting
procedure.

4. For workflow failure , collect the web console log, by selecting the
Tools->Web Developer->Web Console option in firefox browser as shown
as follows.
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 29 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

5. Copy the console log shown by first selecting the Console tab, right clicking
and selecting option copy.

10.1 VNF Lifecycle Manager UI Not Working

In case, the VNF Lifecycle Manager UI is not opening or displaying page
cannot be found error.

Perform the steps:

1. Login to vnflaf-services VM as cloud-user and then switch to root user su -

root and check if Apache httpd server is operational . Run the command
/etc/init.d/httpd status

2. Log on to the OSS-RC server and open a command window.

3. Verify if host name of vnflaf services is configured and entry exist in

host file

run the command as less /etc/hosts | grep "vnflafhost" and

check if following output is shown

1xx.xxx.xx.xx [IP address] vnflafhost

11 Terminology
Table 7 Commonly Used Terminology list

VNF Virtual Network Function

LCM Life Cycle Manager
WFS Workflow Service
ECS Ericsson Cloud System
ECM Ericsson Cloud Manager
 Ericsson Internal
SYSTEM ADMINISTRATION GUIDE 30 (30)
Prepared (Subject resp) No.

XUSMAZH Usman Azhar 1543-APR 901 0578 Uen

Approved (Document resp) Checked Date Rev Reference

Raghavendra Rao Tadepalli ERAGTAD 2016-08-09 C

ECEE Ericsson Cloud Execution

Environment
ENM Ericsson Network Manager
CEE Cloud Execution Environment
NFVO Network Function Virtualization
Orchestrator
BPMN Business Process Model and
Notation
OSS-RC Operations Support System, Radio
& Core

12 References

• [1] Operations Support System (OSS) Glossary 0033-AOM 901 017/2

• [2] Installation and Upgrade Guide of VNF Lifecycle Manager 15372

-CXP9032641

• [3] License Key Administration in OSS-RC 3/1543-AOM 901 122

• [4] OSS Library Typographic Conventions 1/154 43 AOM 901 017/4

• [5] VNF Lifecycle Manager Functional Description