Expert Workshop E143 Chapter 8

Troubleshooting 800xA
Service Framework

Extended Automation System 800xA

© ABB Consult IT - 1
Revision C
 Troubleshooting
Service Framework Topics

„ Summary of Service
„ Service Framework
„ File set distribution
„ Affinity
„ Error Scenarios
© ABB Consult IT - 2
Revision C
 Troubleshooting
Service Framework List of Services

Service Process name Run in Type of

Redundancy
Alarm Manager AdvAeAlarmManager.exe AS Service/Service

Aspect Directory AfwADServer.exe AS Service/Service

File Set Distribution [Client] AfwFsdNode.exe All Nodes

File Set Distribution [Server] AfwFsdSrv.exe AS Service/Service

Lock Server AfwLockSrv.exe AS Service/Standby

Service Manager [Client] AfwSMClient.exe All Nodes

Service Manager [Server] AfwServiceManager.exe Server

Nodes
© ABB Consult IT - 3

Cross Reference Service AfwXRefSrv.exe AS Service/Service

Revision C
 Troubleshooting
Service Framework List of Services

Service Process name Run in Type of

Redundancy
Soft Alarms AdvAeSoftAlarms.exe AS

Event Collector AdvAeEventCollector.exe Usually CS Service/Standby

OPC DA AdvDSOPCConnector.exe CS Service/Service

Basic History AdvHtHistorySrv.exe CS Service/Service

Time Server AfwTimeSrv.exe CS Service/Standby

Time Server Handler AfwTimeSrvHdlr.exe All Nodes

Tray Icon Service AfwTrayIcon.exe All Nodes

© ABB Consult IT - 4
Revision C
 Troubleshooting
Service Framework Service Framework
Affinity can be configured
for service providers in
Service/Service relation

You must belong to

software developer group
Service to change this info.

Service Group

Defines the priority order of the

Service Provider Aspect Servers. The first one is
the Master Canditate

Service Provider

Service provider = task

Service Provider
which runs service.
© ABB Consult IT - 5

Depending on redundancy
you will have 2 or 3 service
Revision C

providers.
 Troubleshooting
Service Framework Redundancy

Aspect
Server 2

4. Client
3.
Aspect 1.
Server 1 2. 3. Aspect 5.
Master Server 3
Candidate
4.

Aspect directory services

1. Client makes a transaction to its Aspect Server (AS3)

2. AS3 sends transaction to Master Candidate (AS1)
3. AS1 sends the complete transaction to AS2 and a subset back to AS3
© ABB Consult IT - 6

4. When both AS2 and AS3 send a complete transaction AS1 replies
Revision C

with a commit transaction back to AS2 and AS3

5. AS3 sends commit back to Client transaction complete
 Troubleshooting
Service Framework States of Services

Init

InitializeState
ErrorState

ReadyState

SyncState MigrationState

BackgroundState ServiceState TerminationState

© ABB Consult IT - 7

Termination
Revision C
 Troubleshooting
Service Framework Commands on Services

Command Description

Cold Reset Cold Reset is the same as starting over with an

empty system. For the aspect directory it
means to remove all aspect directory data and
do a total synchronization.
New Session The New session command will start create a new
Session identifier. The command is necessary
to perform in some error situations. See the
support cases for details.
Reset Force synchronization, but only for changed data.

Run Go to the target state specified.

Suspend Makes possible to get a new sub set of commands

© ABB Consult IT - 8
Revision C
 Troubleshooting
Service Framework File Set Distribution
1.Client Node 1.Server Node
Components using FSD
1.SM Client 1.AfwFsdNode 1.AfwFsdSrv
•Graphics
1.CS-Lib
•Operator notes
•System synchronization 1.AfwFsdNodeControl 1.AfwFsdAccess

Priority determines
1.Client Process, 1.Disk I/O
the distribution
e.g. Workplace
1.Disk
I/O
1.Client Process,
e.g. Backup 1.NTFS
1.NTFS

1.AfwFsdNodeConfig 1.Aspect
, Aspect Directory Aspect Server
Admin Structure Inventory
Objects Priorities > 5 means
always copied to client
© ABB Consult IT - 9
Revision C
 Troubleshooting
Service Framework CS Lib
Client/Server Library is a support library that helps an application
communicate with another application over a socket based protocol.
Shows the connections to all
services for the specified
application

Shows the same information

© ABB Consult IT - 10

An example where a client is

Revision C

connected to different servers,

should be corrected by setting
up affinity
 Troubleshooting
Service Framework CS Lib Useful Operations

Operation Description
Clients Clients connected to the Service; IP and Port should be the same as in
AfwServiceMemTblViewer

Servers The port that the service is listening to, first hand information from windows,
AfwServiceMemTblViewer should show the same

Back End Shows the connection from other client to this server.

AddressCache All DNS queries done by CS-Lib; Unresolved => bad

Summary Summary of the service; lots of dead => bad

© ABB Consult IT - 11
Revision C
 Troubleshooting
Service Framework Service Manager and SMClient
Client Node Client Node
Listen to changes from Service
Shared Shared
Manager. Writes changes to Memory
Service
Client
Memory
Service
Client
Table Table
Shared Memory Table. Info written:
• Server node name and address
SMClient SMClient
• Port number

• Id’s of related objects in the Service Structure

Servers are found in:

HKEY_LOCAL_MACHINE\SOFTWARE\ABB\AFW\Systems\<system
id>\$Servers

Server Node Server Node

Service Service
Starts Stops and supervises Manager
Service
Manager
Service
services. Information about
services to start in
Shared Shared
SMDATA.dat SMClient Memory SMClient Memory
Table Table
© ABB Consult IT - 12
Revision C
 Troubleshooting
Service Framework
Service Connection with Affinity Configured

The nodes in the defined node group

gets the affinity defined for the node
group. The client will be connected to
the same machine for all applicable
services.
© ABB Consult IT - 13
Revision C
 Troubleshooting
Service Framework Behavior by a Restart

Shutdown of AS2 Æ
Switch over to Aspect
Server in 2nd Pos

AS2 is back the

concerning affinity
connection goes back
© ABB Consult IT - 14
Revision C
 Troubleshooting
Service Framework Caution When Removing Server

Removing a server with configuration wizard removes the server

everywhere, that means it is not present in the affinity configuration

Scenario:
1. Remove AS2
2. Add AS2 again and start of AS2
3. Stop of AS1 Æ clients have no more
connection to the system
© ABB Consult IT - 15

Conclusion: Be aware of the consequences when

Revision C

you are removing a node from the system!

 Troubleshooting
Service Framework Error Scenarios

„ How to check service status

„ Read only mode
„ Error state
„ Check of synchronization
„ Recover a node
© ABB Consult IT - 16
Revision C
 Troubleshooting
Service Framework How to Check the Service Status?
1. Add More columns e.g. Long message

2. Call up detail from context menu

3. Copy to Excel or Note pad for further

analyze

4. Use GUID search

© ABB Consult IT - 17
Revision C
 Troubleshooting
Service Framework
© ABB Consult IT - 18
Revision C
AfwServiceStatus
 Troubleshooting
Service Framework
Check Resources with Task Manager

Commit Charge Total

< Physical Total

Peak should never reach Limit

Æ Crash of processes
© ABB Consult IT - 19
Revision C
 Troubleshooting
Service Framework
Check Individual Processes with Task Manager

„ Add columns in the process tabs especially Virtual Memory size,

Handle count, Thread count, User Objects and GDI Objects
„ Take a screen shot of the process list in each node when the
system runs normal. Use this as a reference.
© ABB Consult IT - 20
Revision C
 Troubleshooting
Service Framework Read-only Mode

„ When does it happen?

„ What to do?
© ABB Consult IT - 21
Revision C
 Troubleshooting
Service Framework
© ABB Consult IT - 22
Revision C
Read-only Mode
 Troubleshooting
Service Framework Read-only Mode
„ How to bring AS2 and AS3 back?
„ Select the service provider for the Aspect directory
AS2 and AS3
„ Run the command Suspend, Cold Reset and Run in
a sequence
„ Aspect directory will now make a complete
synchronization
© ABB Consult IT - 23
Revision C
 Troubleshooting
Service Framework Cold Start of Aspect Directory

The suspend command brings up The cold reset command deletes

a new dialog, The status changes the MainDB.afwdb file on the
© ABB Consult IT - 24

from Service Æ ready node

Revision C
 Troubleshooting
Service Framework Cold Start of Aspect Directory

The Run commands makes a total

synchronization of the MainDB.afwdb files
© ABB Consult IT - 25
Revision C
 Troubleshooting
Service Framework Error State in 1 out of 2 Redundancy
Objects are changed after
communication failure.

If the “wrong” AD service is in

error state after a restart,
© ABB Consult IT - 26

Restart this one first, make a

cold start of the other one.
Revision C
 Troubleshooting
Service Framework
Aspect Directory Synchronization Analyze Bad Case

Afwsysinfo –csd
shows checksum
differences between
aspect directories
© ABB Consult IT - 27

Determine if it is normal or repair action should be executed.

Revision C
 Troubleshooting
Service Framework
Aspect Directory Synchronization Analyze Good Case

Afwsysinfo –csd
shows checksum
differences between
aspect directories,
good case shows
also statistic about
the number of
objects
© ABB Consult IT - 28
Revision C
 Troubleshooting
Service Framework Recover Node
„ Determine node type
„ Example for a Redundant Aspect Server main actions
„ Remove the failed node from the Domain.
„ Repair the failed node and restore the failed node
„ Disconnect and connect the node to the domain
„ Remove system registration and aspect directory files by using the
Configuration Wizard
„ Connect the node from Configuration Wizard
„ The aspect server will now synchronize again with the main aspect
server
„ Example for a Connectivity Server AC 800M
„ The same as for Redundant Aspect server +
„ Restore *.cfg file, OPC Server Panel
© ABB Consult IT - 29

„ Restore History from file system if backed up

Revision C
 Troubleshooting
Service Framework
© ABB Consult IT - 30
Revision C
System Registration and Aspect Directory Files
 Troubleshooting
Service Framework
© ABB Consult IT - 31
Revision C
Connect node to system again
 Troubleshooting
Service Framework
© ABB Consult IT - 32
Revision C
ABB Logo
Maintenenance Expert workshop

Exercise 8.1 System 800xA Troubleshooting Service Framework

8.1.1 Description
This exercise deals with trouble shooting the Service Framework. In a redundant
configuration of the 800xA system.

8.1.2 Objectives
Upon completion of this exercise you will be able to:
• Understand how affinity works.
• Find out how the startup works.

8.1.3 Reference documentation

• Documentation in workshop binder.
• Administration and Security

1/4
 E143-08 Exercise 8.1 - RevB.doc

8.1.4 Exercise Steps

1. Check the connection to services for the client with the Service Connection
Viewer.
2. Stop the server, which the client is connected to. If the client is connected to
different services stop the service with the Aspect Directory Service connection.
What happens?
Check the license status. What measurements do you need to take if the server is
broken for a longer time?

3. Start the server again what happens to the connections?

4. Make an affinity configuration.

2/4
Maintenenance Expert workshop

5. Use the service connection viewer in order to check the connection of client. Is the
client connected according to the affinity configuration?

6. Drag the client from the odd client node group to the even client node group.
Make reboot of the client. Inspect the connections with service connection viewer.
What happens?

7. Stop the second aspect server DCASCSx2 from configuration wizard. Check the
service connection viewer, did you get a switch over to first aspect server?

3/4
 E143-08 Exercise 8.1 - RevB.doc

8. Remove the stopped Aspect Server from the configuration wizard of the running
Aspect Server. Add the server again. Wait until server is up and running. Make a
reboot of client. Why hasn’t the client connected to the original server as shown in
bullet 6?

9. Remove all nodes in the Affinity configuration. Shutdown the Aspect Server
DCASCSx1. Do you get any connection from the client after a reboot of the
client? What happens? How can you correct it?

4/4
Maintenenance Expert workshop

Exercise 8.2 System 800xA Troubleshooting 1 oof 2 Redundancy

8.2.1 Description
This exercise deals with trouble shooting 1 oof 2 redundancy. The student will be
aware how the 1 oof 2 redundancy works in a system where the network is lost.

8.2.2 Objectives
Upon completion of this exercise you will be able to:
• Recover from a situation where you have lost the network in a 1 oof 2
redundancy configuration.

8.2.3 Reference documentation

• Documentation in workshop binder.
• Administration and Security

1/2
 E143-08 Exercise 8.2 - RevA.doc

8.2.4 Exercise Steps

1. Stop both network by removing the power from the switches.
2. On DCASCSx1 add a generic object in the Functional Structure. On DCASCSx2
add an other object in the Functional structure.
3. Start up the switches again. What happens to the services?

4. You want to keep the data on the machine which is in error state. What can you
do?

5. Get the Aspect Directory service in service state on both machines. Before you do
the cold init on the Aspect Directory service open up a windows explorer and
navigate to the \OperateITData\AspDir directory. Start the System Event list on
the Service object as well. Check the synchronization messages.

2/2