Guideline for new and pre-existing alarms during MMBB Activities

Purpose
The purpose of this document is to have a compile of the most common alarms raised during the MMBB
activity; in this document you will find the basic TS to clear alarms.

It is very important to take in count pre-existing alarms and double check that the alarms present during
the MMBB activity are not due to a pre-existing issue, if that would be the case it is needed to leave the
node as it was before the activity.

Every time you have a new alarm please look into this document so you can perform proper TS before
raise any Ticket to TCM or DNOC team.

Alarm: No Connection                       AntennaUnitGroup=

as you can see here on the original LTE Node prior the migration we had just 6 uniqueids :
Once the migration was performed since 4G to 5G bb on 5G node now we have 8 uniqueids, that means
that we have 2 duplicated RETs that needs to be deleted in order to fix the issue ( before to perform this
step you need to check first with your Market Lead or TCM)
1St step clean all the references:

Delete the duplicated uniqueID:

Deleting Duplicated Radio Reference port and antenna motor Jam Refference:
Deleting RET definition:
Deleting Duplicated antenna:

At the all the RETs defined on the 5G node matchs with the Rets defined on the LTE 4G kget:
 *************************** RETS ***********************************

Posibble alarms:
o 2020-09-22 06:44:38 m Inconsistent Configuration         
AntennaUnitGroup=10,AntennaNearUnit=4,RetSubUnit=1 (Tilt value out of range)
o 2020-09-21 07:49:22 m SW Error                            AntennaUnitGroup=6-
04,AntennaNearUnit=1
o No Connection

If after the migration, we see some RETs disabled please check:

- Check if the the MOs were locked since prechecks. Use the next commands to check on the pre-
kget.
o st near
o st ret
- Check the references between the radio and AntennaNearUnit. Use the next commands:
o hget ant rfport
o invrxb
 o Check the uniqueId, the value must match between the current configuration and prechecks, in
case we have a mismatch, change the value according to the prechecks and do a restart to the
radio. To check that value please use the next command on 5G node and pre-kget 4G.
o hget antenna unique|onunit

For RETs disabled on LTE node, usually the RETs were referenced to the L600/L700 radios and it’s
needed to migrate from the 5G to 4G node. It’s needed to open a TCM ticket. To check the references
use the next commands on pre-kget and 4G node:
o hget near rfport
o lst nn
Alarm: No Connection AntennaUnitGroup=6-01,AntennaNearUnit=1 (Timeout: Failed to get
anuConnectIndication)

NNM01606A> alt
200923-00:54:58 11.201.36.202 20.0f MSRBS_NODE_MODEL_20.Q2_453.28212.66_b919 stopfile=/tmp/21886
Collecting Alarms...
====================================================================================================================
Date & Time (UTC) S Specific Problem MO (Cause/AdditionalInfo)
====================================================================================================================
2020-09-23 06:55:59 m No Connection AntennaUnitGroup=6-01,AntennaNearUnit=1 (Timeout: Failed to get anuConnectIndication)
2020-09-23 06:55:59 m No Connection AntennaUnitGroup=6-02,AntennaNearUnit=1 (Timeout: Failed to get anuConnectIndication)
2020-09-23 06:55:59 m No Connection AntennaUnitGroup=6-03,AntennaNearUnit=1 (Timeout: Failed to get anuConnectIndication)
>>> Total: 3 Alarms (0 Critical, 0 Major)

One of the TS to clear the above alarms is verify the uniqueid and compare with the Kget

NNM01606A> get . uniqueid

200923-01:26:24 11.201.36.202 20.0f MSRBS_NODE_MODEL_20.Q2_453.28212.66_b919 stopfile=/tmp/16461

=================================================================================================================
MO Attribute Value
=================================================================================================================
AntennaUnitGroup=6-01,AntennaNearUnit=1 onUnitUniqueId
AntennaUnitGroup=6-01,AntennaNearUnit=1 uniqueId R1
AntennaUnitGroup=6-02,AntennaNearUnit=1 onUnitUniqueId
AntennaUnitGroup=6-02,AntennaNearUnit=1 uniqueId R1
AntennaUnitGroup=6-03,AntennaNearUnit=1 onUnitUniqueId
AntennaUnitGroup=6-03,AntennaNearUnit=1 uniqueId R1
=================================================================================================================
Total: 3 MOs

These are the prints for the Kget:

OFFLINE_KGET_LNM01606A_PRE_CHECK_200922_2130> get . uniqueid

200923-01:26:32 OFFLINE_KGET_lnm01606a_PRE_CHECK_200922_2130 20.0f MSRBS_NODE_MODEL_20.Q2_453.28212.66_b919

stopfile=/tmp/11050
=================================================================================================================
MO Attribute Value
=================================================================================================================
AntennaUnitGroup=1,AntennaNearUnit=2 onUnitUniqueId AN0000a000C5061774a
AntennaUnitGroup=1,AntennaNearUnit=2 uniqueId
AntennaUnitGroup=13,AntennaNearUnit=1 onUnitUniqueId AN00017CN1026514711
AntennaUnitGroup=13,AntennaNearUnit=1 uniqueId
AntennaUnitGroup=14,AntennaNearUnit=1 onUnitUniqueId AN00017CN1026514641
AntennaUnitGroup=14,AntennaNearUnit=1 uniqueId
AntennaUnitGroup=15,AntennaNearUnit=1 onUnitUniqueId AN00017CN1026514931
AntennaUnitGroup=15,AntennaNearUnit=1 uniqueId
AntennaUnitGroup=2,AntennaNearUnit=2 onUnitUniqueId AN0000a004K3631551a
AntennaUnitGroup=2,AntennaNearUnit=2 uniqueId
AntennaUnitGroup=3,AntennaNearUnit=2 onUnitUniqueId AN0000a000C5091502a
AntennaUnitGroup=3,AntennaNearUnit=2 uniqueId
=================================================================================================================
Total: 6 MOs

As you can see on our current configuration que have the next:
AntennaUnitGroup=6-01,AntennaNearUnit=1 uniqueId R1

But the information on our kget looks different:

AntennaUnitGroup=13,AntennaNearUnit=1 uniqueId

The unique id on precheck is empty, so it is needed to delete the current unique id and then restart the
node.

NNM01606A> set AntennaUnitGroup=6-01,AntennaNearUnit=1 uniqueId

200923-01:26:49 11.201.36.202 20.0f MSRBS_NODE_MODEL_20.Q2_453.28212.66_b919 stopfile=/tmp/16461

Set uniqueId on following 1 MOs ?
===================================================================================
574 Equipment=1,AntennaUnitGroup=6-01,AntennaNearUnit=1
===================================================================================
Set uniqueId on 1 MOs. Are you Sure [y/n] ? y

============================================================================================================
Id MO uniqueId Result
============================================================================================================

$netconf_pid = 18943

574 AntennaUnitGroup=6-01,AntennaNearUnit=1 >>> Set.

============================================================================================================
Total: 1 MOs attempted, 1 MOs set

After the node restart the BB will keep the same unique id’s as prechecks and the alrms will be cleared:

NNM01606A> get . uniqueid

200923-01:30:52 11.201.36.202 20.0f MSRBS_NODE_MODEL_20.Q2_453.28212.66_b919 stopfile=/tmp/16461

=================================================================================================================
MO Attribute Value
=================================================================================================================
AntennaUnitGroup=6-01,AntennaNearUnit=1 onUnitUniqueId AN00017CN1026514711
AntennaUnitGroup=6-01,AntennaNearUnit=1 uniqueId
AntennaUnitGroup=6-02,AntennaNearUnit=1 onUnitUniqueId AN00017CN1026514641
AntennaUnitGroup=6-02,AntennaNearUnit=1 uniqueId
AntennaUnitGroup=6-03,AntennaNearUnit=1 onUnitUniqueId AN00017CN1026514931
AntennaUnitGroup=6-03,AntennaNearUnit=1 uniqueId
=================================================================================================================
Total: 3 MOs
 ********************************PTP ALARMS********************************

Alarm: Sync PTP Time Reachability Fault   

Synchronization=1,RadioEquipmentClock=1,RadioEquipmentClockReference=4 (No PTP messages)

As you reported all the nodes are showing the PTP alarm, on the legacy and MMBB node it just appeared because
the PTP wasn’t configured before the MMBB, on this case is required support on site to verify if the GPS connected
to the IXRe router is correctly set, as well to confirm with the TMO Switch that the router is correctly configured.

The nodes are currently working with the GPS sync for that reason the cells are enabled and processing traffic.

Please update us when the support on site is scheduled to check the PTP issue to can follow up with the
troubleshooting.

LSL01833A>

200922-02:56:06 11.234.47.52 20.0f MSRBS_NODE_MODEL_20.Q2_453.28212.66_b919

stopfile=/tmp/20736

Collecting Alarms...
........
==============================================================================================
======================
Date & Time (UTC)   S Specific Problem                    MO (Cause/AdditionalInfo)
==============================================================================================
======================
2020-09-22 08:50:46 M Sync PTP Time Reachability Fault   
Synchronization=1,RadioEquipmentClock=1,RadioEquipmentClockReference=4 (No PTP messages)
2020-09-22 09:50:39 m External Link Failure               ENodeBFunction=1 (X2 link problem to
one or several neighbouring eNodeBs. AI: PLMN ID-eNB ID 1 : 310120-714366,PLMN ID-eNB ID 2 :
310120-713285,PLMN ID-eNB ID 3 : 310120-712992,PLMN ID-eNB ID 4 : 310120-712965,PLMN ID-eNB ID
5 : 310120-712835,PLMN ID-eNB ID 6 : 310120-712834,PLMN ID-eNB ID 7 : 310120-712833,PLMN ID-
eNB ID 8 : 310120-712832,PLMN ID-eNB ID 9 : 310120-712830,PLMN ID-eNB ID : list truncated)
>>> Total: 2 Alarms (0 Critical, 1 Major)

LSL01833A2> 
200922-02:39:15 11.234.47.53 20.0f MSRBS_NODE_MODEL_20.Q2_453.28212.66_b919
stopfile=/tmp/27583
Connected to 11.234.47.53
(SubNetwork=ONRM_ROOT_MO,SubNetwork=Draper,MeContext=LSL01833A2,ManagedElement=LSL01833A2)
Collecting Alarms...
==============================================================================================
======================
Date & Time (UTC) S Specific Problem MO (Cause/AdditionalInfo)
==============================================================================================
======================
2020-09-21 20:57:23 M Sync PTP Time Reachability Fault
Synchronization=1,RadioEquipmentClock=1,RadioEquipmentClockReference=4 (No PTP messages)
2020-09-22 07:08:48 m External Link Failure ENodeBFunction=1 (X2 link problem to one or
several neighbouring eNodeBs. AI: PLMN ID-eNB ID 1 : 310120-712729,PLMN ID-eNB ID 2 : 310120-
712737,PLMN ID-eNB ID 3 : 310120-712763,PLMN ID-eNB ID 4 : 310120-712770,PLMN ID-eNB ID 5 :
310120-712786,PLMN ID-eNB ID 6 : 310120-712794,PLMN ID-eNB ID 7 : 310120-712812,PLMN ID-eNB ID
8 : 310120-712826,PLMN ID-eNB ID 9 : 310120-712829,PLMN ID-eNB ID : list truncated)
2020-09-22 08:28:50 m External Link to GNodeB Failure ENodeBFunction=1 (X2 link problem to one
or several neighbouring gNodeBs. AI: PLMN ID-gNB ID 1 : 310260-1835118,PLMN ID-gNB ID 2 :
310260-1837525,PLMN ID-gNB ID 3 : 310260-1841599)
>>> Total: 3 Alarms (0 Critical, 1 Major)

LSL01833A3> 
200922-02:39:15 11.234.47.54 20.0f MSRBS_NODE_MODEL_20.Q2_453.28212.68_480e
stopfile=/tmp/20735

Connected to 11.234.47.54
(SubNetwork=ONRM_ROOT_MO,SubNetwork=Draper,MeContext=LSL01833A3,ManagedElement=LSL01833A3)
Collecting Alarms...
==============================================================================================
======================
Date & Time (UTC) S Specific Problem MO (Cause/AdditionalInfo)
==============================================================================================
======================
2020-09-21 20:57:22 M Sync PTP Time Reachability Fault
Synchronization=1,RadioEquipmentClock=1,RadioEquipmentClockReference=4 (No PTP messages)
>>> Total: 1 Alarms (0 Critical, 1 Major)

LSL01833A4> 
200922-02:39:15 11.234.47.50 20.0f MSRBS_NODE_MODEL_20.Q2_453.28212.68_480e
stopfile=/tmp/20734

Connected to 11.234.47.50
(SubNetwork=ONRM_ROOT_MO,SubNetwork=Draper,MeContext=LSL01833A4,ManagedElement=LSL01833A4)
Collecting Alarms...
==============================================================================================
======================
Date & Time (UTC) S Specific Problem MO (Cause/AdditionalInfo)
==============================================================================================
======================
2020-09-21 20:57:23 M Sync PTP Time Reachability Fault
Synchronization=1,RadioEquipmentClock=1,RadioEquipmentClockReference=4 (No PTP messages)
>>> Total: 1 Alarms (0 Critical, 1 Major)

LSL01833A5> 
200922-02:39:15 11.234.47.51 20.0f MSRBS_NODE_MODEL_20.Q2_453.28212.68_480e
stopfile=/tmp/30916

Connected to 11.234.47.51
(SubNetwork=ONRM_ROOT_MO,SubNetwork=Draper,MeContext=LSL01833A5,ManagedElement=LSL01833A5)
Collecting Alarms...
==============================================================================================
======================
Date & Time (UTC) S Specific Problem MO (Cause/AdditionalInfo)
==============================================================================================
======================
2020-09-21 20:57:22 M Sync PTP Time Reachability Fault
Synchronization=1,RadioEquipmentClock=1,RadioEquipmentClockReference=4 (No PTP messages)
>>> Total: 1 Alarms (0 Critical, 1 Major)

LSL01833A6> 
200922-02:39:15 11.234.47.34 20.0f MSRBS_NODE_MODEL_20.Q2_453.28212.68_480e stopfile=/tmp/8163

Connected to 11.234.47.34
(SubNetwork=ONRM_ROOT_MO,SubNetwork=Draper,MeContext=LSL01833A6,ManagedElement=LSL01833A6)
Collecting Alarms...
==============================================================================================
======================
Date & Time (UTC) S Specific Problem MO (Cause/AdditionalInfo)
==============================================================================================
======================
2020-09-21 20:57:23 M Sync PTP Time Reachability Fault
Synchronization=1,RadioEquipmentClock=1,RadioEquipmentClockReference=4 (No PTP messages)
>>> Total: 1 Alarms (0 Critical, 1 Major)

NSL01833A> 
200922-02:39:15 11.234.47.42 20.0f MSRBS_NODE_MODEL_20.Q2_453.28212.66_b919
stopfile=/tmp/30926

Connected to 11.234.47.42
(SubNetwork=ONRM_ROOT_MO,SubNetwork=Draper,MeContext=NSL01833A,ManagedElement=NSL01833A)
Collecting Alarms...
==============================================================================================
======================
Date & Time (UTC) S Specific Problem MO (Cause/AdditionalInfo)
==============================================================================================
======================
2020-09-22 08:51:33 M Sync PTP Time Reachability Fault
Synchronization=1,RadioEquipmentClock=1,RadioEquipmentClockReference=4 (No PTP messages)
>>> Total: 1 Alarms (0 Critical, 1 Major)

NSL01833A2>
200922-02:39:15 11.234.47.43 20.0f MSRBS_NODE_MODEL_20.Q2_453.28212.68_480e stopfile=/tmp/7341

Connected to 11.234.47.43
(SubNetwork=ONRM_ROOT_MO,SubNetwork=Draper,MeContext=NSL01833A2,ManagedElement=NSL01833A2)
Collecting Alarms...
==============================================================================================
======================
Date & Time (UTC) S Specific Problem MO (Cause/AdditionalInfo)
==============================================================================================
======================
2020-09-21 20:57:22 M Sync PTP Time Reachability Fault
Synchronization=1,RadioEquipmentClock=1,RadioEquipmentClockReference=4 (No PTP messages)
>>> Total: 1 Alarms (0 Critical, 1 Major)

********************************PTP ALARMS********************************

- PTP SLAVE: it’s just using between the BB’s (LTE Legacy and MMBB) and use the ERAN (IDL_A)
connection to sync from the LTE to MMBB node.
- PTP Grandmaster: it’s using in sites with more than 2 BB’s (AAS/Anchor), the PTP will sync to
each BB from the IXRe router that is connected through the TN_A port.
If we have PTP_SLAVE alarms, please check the GPS and the ERAN connections, also please check the
configuration in picture below:

If the GPS is disabled it’s an onsite issue and a DNOC ticket it’s needed.
o hget RadioEquipmentClockReference= state|priority|type

Alarm:

Certificate Management a Valid Certificate is Not Available

SecM=1,CertM=1,NodeCredential=oamNodeCredential
To clear the above alarm it is needed to verify first if the node is currently enrolled with the next
command: hget nodecred progress@result$|state

As you can see the node credential that is raising the alarm is not enrolled yet, so we need to proceed
and perform the enrollment with the next commands:

acc NodeCredential=oamNodeCredential startOnlineEnrollment

null

After the node is 100 % enrolled, we give an lt all and check again the Enrollment state and the alarms.
Now the node is alarm free and Enrolled.
2nd TS for enrollment:

*********************************************************************************
Enrollment issues
*********************************************************************************
1. In ENM GUI, goto PKI Entity Management
2. Click on the filter button on the right.
3. Put in node ID + -oam. Example N4WA0534B-oam
4. If none appear, you have to add the Entity.
5. Using a name change script package for the ENM you are working on, open and
modify script step02.xml.
6. drag this file into the ENM CLI and run the command to add the entity
pkiadm entitymgmt --createbulk --xmlfile file:"step02.xml"
7. Try the online enrollment from the node again.
8. Once enrolled, re-sync the node in the ENM and it should sync up.

********************************ERAN********************************

Possible issues:
o 2020-09-21 07:55:26 m Ethernet Link Failure EthernetPort=ERAN
(Connected to physical port TnPort=IDL_A1)
o 2020-09-21 09:26:06 m Inter Node Carrier Aggregation Service Unavailable
EUtraNetwork=1,ExternalENodeBFunction=310260-81119 (Elastic RAN Carrier Aggregation
is not available towards remote eNodeB.)
o 2020-07-29 07:40:37 m Inconsistent Configuration
EUtraNetwork=1,ExternalENodeBFunction=310260-127959 (Elastic RAN cannot be
activated towards remote eNodeB. VLAN IDs do not match. Expected Vlan Id: 4050,
Please check the eRAN Vlan Id at the remote eNodeB)
o BBLinks disabled

Please check:
 o FEATURES (CARRIER AGG/ERAN)
o VLAN CONFIG
o SCELLCANDIDATES
o ERAN PORT
o Router

Troubleshoot:

o We shouldn’t have ERAN on TDD nodes.

set EutrancellFDD=.*,EutranFreqRelation=.*,EutrancellRelation=<externalENodeBFunctionId>
scellcandidate 2

If we see the EtehernetPort=ERAN disabled, it’s not needed to open a ticket with DNOC due to the issue
is onsite. (Below an example)

For alarms regarded Inter Node Carrier Aggregation Service Unavailable if after the checks above the
configuration is the correct, it’s probably that the port on the router is no enabled. This issues it’s
needed to escalate to Switch/Market to verify/enabled the port.

o VLAN MISMATCH

Please be sure that both nodes (LTE/MMBB) have the same VLAN configured to ERAN. Please use the
next command
o get vlan id

 MME’S AMF’S DISABLED.

Please check that the MMEs IPs between LTE Legacy and MMBB match. It’s normal that the MMBB node
has lees IPs than the LTE node due to some MMEs are dedicated to the NBIOT Cells. Use the next
command:
o hget mme ipadd

If the IPs match, please do a ping from the MMBB to the router. Use the next commands:
a. get Transport=1,Router=vr_TRAFFIC,RouteTableIPv4Static=1,Dst=1,NextHop=1
address > $TRAFFICIP
b. mcc Transport=1,Router=vr_TRAFFIC ping $TRAFFICIP
If the ping is successfully, it’s a routing issue and we need support from Switch team.
If the ping is unsuccessfully, the switch team needs to check the IP due to a possible mismatch with the
inputs.

Also, it’s probably that there is a mismatch in the inputs. (SAS Router on site and inputs for IXRe
router). Please use the next command.
o get ipv4 address

SAS Router:

IXRe router:

AMF’s Disabled:

Possible alarm:

o PLMN service unavailable.

Check:

o Check if it in prechecks is enabled.

o st amf
o Lock / Unlock.
 ********************************NGS Issues********************************

Possible alarms:
Node Group Sync Loss of All SoCC
Synchronization=1,RadioEquipmentClock=1,NodeGroupSyncMember=1 (Absence of NGSC_BCM)

Node Group Sync Loss of All SoCC

Synchronization=1,RadioEquipmentClock=1,NodeGroupSyncMember=1 (The CPRI link is down)

Before to delete the NGS MO, please check the next MOs

- Confirm if the GMSM is configured for NGS. Use the next command.
o RXMOP:MO=RXOTF-<#TG>;

This print shows us a node with NGS configured for GMS:

MO TFMODE SYNCSRC TFCOMPNEG TFCOMPPOS FSOFFSET TFAUTO

RXOTF-753 SA DEFAULT 0 NO

This print shows us a node without NGS configured for GMS:

MO TFMODE SYNCSRC TFCOMPNEG TFCOMPPOS FSOFFSET TFAUTO

RXOTF-93 SA DEFAULT OFF NO

- Confirm that the LTE radios are not as mixedmode, use the next command
o get . sharedw

The next print shows us the LTE radios as mixedmode.

FieldReplaceableUnit=RU-1-1 isSharedWithExternalMe true
FieldReplaceableUnit=RU-1-2 isSharedWithExternalMe true
FieldReplaceableUnit=RU-1-3 isSharedWithExternalMe true
FieldReplaceableUnit=RU-1-4 isSharedWithExternalMe true
FieldReplaceableUnit=RU-1-5 isSharedWithExternalMe true
FieldReplaceableUnit=RU-1-6 isSharedWithExternalMe true
 - Confirm that the LTE radios are not sharing with GSM or UMTS, comparing the serials. Use the
next command.
o RXMFP:MO=RXOTRX-<#TG>-0&-2&-4; ( with the TG and the TRX)

This is an example about radios sharing bwtween GSM and LTE

LTE node:
FieldReplaceableUnit=RU-1-2 D16H807007
FieldReplaceableUnit=RU-1-4 D16H821168
FieldReplaceableUnit=RU-1-6 D16H820988
GMS node:
RU RUREVISION RUSERIALNO
0 KRC 118 66/2 D16H807007
RU RUREVISION RUSERIALNO
0 KRC 118 66/2 D16H821168
RU RUREVISION RUSERIALNO
0 KRC 118 66/2 D16H820988

- In addition, in UMTS we can confirm if the node uses the NGS with the command st sync, if
the node uses a GPS reference, it’s probably that the NGS is not needed.
(UNLOCKED) 1 (ENABLED) Equipment=1,Subrack=1,Slot=1,PlugInUnit=1,TimingUnit=1,GpsSyncRef=1

So, we can delete de NGS on LTE nodes where NGS is not configured for UMTS or GSM