Mission2-General ECE Troubleshooting - Email - LG
Mission2-General ECE Troubleshooting - Email - LG
Speakers:
Joshua Raja, CX
Alex Artamonov, CX
Jose Adolio, CX
1|Page
Learning Objectives
This lab guide is for enterprise voice and contact center engineers interested in understanding and
troubleshooting Enterprise Chat and Email (ECE).
An explanation of the components of ECE, the new ECE 12.6 features, and how both email and chat activities
move through the system. This is focused on the PCCE 12.6 integrated ECE, but the principles will apply to UCCE
integrated systems as well.
Pre-requisite knowledge
This is an intermediate session intended for engineers with prior Contact Center Enterprise (CCE) experience.
Prior experience with Enterprise Chat and Email is not required but will be helpful.
Disclaimer
This training document is to familiarize users with ECE 12.6 and PCCE 12.6. The design of the lab is not one which
would be supported in a production environment. While the features described do work, there are many other
features in ECE which may be needed in your environment. For design-related questions please contact your
representative at Cisco, or a Cisco partner.
2|Page
Lab Topology and Access
Lab Deployment
3|Page
Lab Addressing and Credentials -Add ECE Components
CVP Reporting
cvprep1 198.18.133.70 DCLOUD\Administrator C1sco12345
Server
Note: The lab pod is not configured and sized according to SRND production requirements. It was created using VM clones of a master pod
with the goal of hosting multiple identical pods. These labs are for instructional purposes only; please do not consider the deployment
model and allocated hardware resources as a reference for a production system.
4|Page
Server Connection
NOTE: There is a new application on the Windows 10 Workstation desktops that allows you to quickly connect to any of the servers
within your demonstration session. This is very useful if you need to access any virtual machines native interface.
Double click the mRemoteNG shortcut [ ] to open it and see the list of servers. Then click on the server’s
name on the left side panel to open it up.
5|Page
Lab 1: Attributes and Precision Queue
For this task you will access SPOG Interface and will go through the configuration of a new PQ
Username: [email protected]
Password: C1sco12345
Step 5. Configure the new Attribute based on the below settings and save
6|Page
Step 6. Go back to SPOG and select Users > Agents
7|Page
Task 2: Configure a new Precision Queue
Step 10. Create a new Precision Queue and set the following values and save
8|Page
9|Page
Lab 2: New Email Queue and Workflow change
Important:
Step 1. Step 1. Open your DCloud lab session and open WKST1
Username: [email protected]
Password: C1sco12345
Step 5. Under Service > Business Rules > Queues > New and make sure the following settings are configured
10 |
Page
11 |
Page
Task 2: Change Existing Workflow
Step 3. Select Inbound workflow and look for Email Workflow – Integrated
The options will show up and you will select Diagram tab
Step 4. Double click on the Queue node. A new screen will open and select the queue we created SummitQueue
12 |
Page
Lab 3: Test Agent Login and Change Status
Step 1. Log in to Finesse Desktop. On the WKST1, open Chrome Browser. Click Demo Links tab and select Finesse
Agent Desktop.
Step 2. On Finesse Agent Desktop, log in with user: sjeffers and click Next.
Step 3. Leave the Dcloud ADFS default password, and click Sign in.
13 |
Page
Step 4. Make sure the extension is 1080 and click Submit.
Step 5. On the Finesse Desktop, make sure the agent stays at Not Ready mode for Voice but change status of
Email to Ready
Whenever an agent is logged in, EAMS sends a MEDIA_LOGIN_REQ. The MEDIA_LOGIN_REQ logs the specified
agent into an Media Route Domain(MRD) (logs agent into all skills configured for that MRD and agent). When an
agent marks himself as available, the EAMS does send two more requests that indicate that the agent is
ROUTABLE or NOT ROUTABLE and READY or NOT READY, and provides client-defined agent information. The CTI
client must have specified the Application Path for the related MRD Peripheral pair in the Open Request
message or the log in is rejected.
14 |
Page
Use the search functionality to find the log file you want. Click into the search box and type EAMS
Locate the eg_log_cceece_EAMS-process.log file, right-click the file and choose Edit with Notepad++.
Search for MEDIA_LOGIN_REQ and we should see the EAMS sending the message to the CTI Server.
CTI Server should process the Media Login Request and if the configuration is correct, it should send a Media
Login Response. From EAMS we will see the below message
2022-11-02 01:51:43.225 GMT+0000 <@> INFO <@> [110:Thread-7] <@> ProcessId:11008 <@> PID:1 <@>
UID:12 <@> UserSessionId: <@> ClientIP: <@> com.ipcc.listener.arm.ARMLogger <@> <@> Received
MEDIA_LOGIN_RESP -> 0 0 0 8 0 0 0 -104 0 0 31 65 0 0 0 0 <@>
Finally, when the Agent becomes Available (Ready) for Email, EAMS sends the event MAKE_AGENT_READY_IND
to CTI Server
2022-11-02 01:56:58.081 GMT+0000 <@> INFO <@> [4876:RMI TCP Connection(3484)-198.18.135.29] <@>
ProcessId:11008 <@> PID:1 <@> UID:12 <@> UserSessionId: <@> ClientIP: <@>
com.ipcc.message.IPCCMessage <@> logOutgoingMessageObj() <@> Outgoing message-
MAKE_AGENT_READY_IND - ARMMakeAgentReadyInd{invokeId=8038, icmAgentId=5172, mrdID=5006,
routable=1, messageFixedPartSize=14, messageFloatingPartSize=0} <@>
2022-11-02 01:56:58.082 GMT+0000 <@> INFO <@> [4876:RMI TCP Connection(3484)-198.18.135.29] <@>
ProcessId:11008 <@> PID:1 <@> UID:1135 <@> UserSessionId:2707459e-fe0d-421b-a365-1cedaf3280d2 <@>
15 |
Page
ClientIP:198.18.133.75 <@> com.ipcc.listener.arm.ARMLogger <@> <@> Sending MAKE_AGENT_READY_IND ->
0 0 0 14 0 0 0 -99 0 0 31 102 0 0 19 -114 0 0 20 52 0 1 <@>
2022-11-02 01:56:58.082 GMT+0000 <@> INFO <@> [4876:RMI TCP Connection(3484)-198.18.135.29] <@>
ProcessId:11008 <@> PID:1 <@> UID:1135 <@> UserSessionId:2707459e-fe0d-421b-a365-1cedaf3280d2 <@>
ClientIP:198.18.133.75 <@> com.ipcc.listener.AgentAvailabilityStatusHandler <@> <@> PRINT_STATE after
sending MAKE_AGENT_READY_IND to ARM: IcmAgentID, MrdId, STATE = 5172 5006
This concludes the configuration of a new Precision Queue and Skill for Agent Email.
16 |
Page
Lab 4: Email Flow Troubleshooting
Now that you have completed configuring a new queue for email setup in SPOG, we need to send emails to our
agent. After doing some testing it is reported that emails are not being delivered. Let see what is happening.
We will divide the flow of the email interaction into three different legs. These legs are Ingress Leg, Treatment
Leg and Agent Leg. In this lab you will understand how the services works with each other and how can we
troubleshoot this situation
To send emails to the agent, we will be using the Demo Website from dcloud. For that follow these steps
Step 2. Open a new tab -> Demo Links -> Demo Website
17 |
Page
Step 4. Enter the required details and for email you can use your personal or any other email that you have
access to.
In a high overview the Ingress Leg is the customer sending an email, received by your email server and ECE
Services Server monitoring each new email that arrives to the mailboxes
Services
Server
Database
Server
Step 5. Once you send the email, we need to Confirm ECE can Reach the eMail Server
One of the quickest ways to check if emails are coming in is to directly query the database. We may still need to
use the log files, but we should take advantage of all tools at our disposal.
Make sure the connection settings are as shown in the screenshot below and click on the Connect button.
18 |
Page
Click on the New Query button at the top to start writing our query.
Copy and the query below and paste it into the query editor. Execute this by clicking Execute or by pressing F5
or Ctrl-E on the keyboard.
Scroll over in the results to the WHEN_CREATED and SUBJECT fields. Do you see your email subject in the
results?
19 |
Page
Once the email disappears from the mail server, ECE will insert it into the database with ACTIVITY_STATUS=3000
and ACTIVITY_SUB_STATUS=3100. This translates to Ready for Inbound Workflow. We can validate this by
querying the eGActiveDB in SQL Server Management Studio with the same query that we used previously.
Because this STATUS/SUB_STATUS is such a transient condition, it can be difficult to see in a fully functional ECE
system. Remember, the next step is for the Workflow Engine to pick up the email and begin processing it.
Below is a picture of the query above from a similar example in the lab. If you’re very quick after sending a new
email, you may see results similar to the screenshot below.
We have included an all the status and sub status meanings at the end of this document in the appendix.
By default, the Retriever polls the mail server every 30 seconds so we should see information on what is
happening in the current retriever logs. Let’s do that now. We’ll need to be on the ECE server.
Use the search functionality to find the log file you want. Click into the search box and type rx.
Locate the eg_log_ECE_rx-process.log file, right-click the file and choose Edit with Notepad++.
20 |
Page
In Notepad++, click on Search -> Find from the Menu.
In the “Find what” box type Processing alias:, then click the Find All in Current Document button.
This will show all the lines which contain this string at the bottom of the Notepad++ window. Here, we can
quickly see how the retriever is working. (Scroll down to the bottom of the search results window and scroll
right.)
2022-11-01 02:41:41.278 GMT+0000 <@> INFO <@> [88:RxInstance id : 999] <@> ProcessId:10836 <@> PID:1
<@> UID:12 <@> UserSessionId: <@> ClientIP: <@> com.egain.mail.module.retriever.service.RxProcess <@>
retrieveMailsForHostRecur<999> <@> Processing alias: [email protected] <@>
Note that this message is at INFO level, which is why we needed to increase the trace level previously.
The eMails are now being pulled from the eMail Server. This lab exercise will explain ECE’s Email Treatment leg,
including ECE Workflows and CCE Routing Scripts, and how to diagnose and troubleshoot issues within them.
MR PG CCE
Router
Services
Server
21 |
Page
Let’s re-run the query that we used at the end of the Ingress leg and look at that email we sent again. Notice the
QUEUE_NAME. Are we sure this is the right email? Look over at the SUBJECT, it shows “Summit_Test”, this is
ours. Now look at the ACTIVITY_STATUS and ACTIVITY_SUB_STATUS, they are 4000/4100, this means “Ready for
Internal Assignment.” This means it came in but didn’t go where we expected it to. Make sure you remember
the ACTIVITY_ID, this is going to be needed to continue troubleshooting.
The first thing we will need to validate is the ECE Start Workflow
Step 3. Select Inbound workflow and look for Start Workflow – Standard
Step 4. The options will show up and you will select Diagram tab
Remember the order of the workflow, the system defined Start Workflow – Standard runs before any user
created workflows run. Review this workflow briefly. Nothing is wrong here, but you should know what the
default version looks like in case a customer makes changes. Another item of note, the route to Default
exception queue is NOT what is causing our identified issue. Rather, eMails only go to Default exception queue
if they meet one of the rules in the Handling Bounce Backs node. If you look back to the database query results
and scroll over to ACTIVITY_SUB_TYPE, you’ll see that the value is a 1. Handling Bounce Backs will set the
ACTIVITY_SUB_TYPE to either 4 or 5. See the appendix for details on the values for ACTIVITY_TYPE and
ACTIVITY_SUB_TYPE.
We now know that the standard workflow did not cause our issue, let’s look at the workflow that will route
things into UCCE.
22 |
Page
We need to be sure it is pointing to the right Queue we setup previously. If it is not pointing to the right queue,
you will see routing issues.
Step 3. Select Inbound workflow and look for Email Workflow – Integrated
Step 4. The options will show up and you will select Diagram tab
This means that ECE is now good, and we can move our focus to the CCE side.
Once the email has been pulled into ECE from the eMail server and the Start Workflow – Standard and the user
created Start Workflow has run, the activity will be in ACTIVITY_SUB_STATUS=4100. Next, the EAAS process will
send a NEW_TASK message to UCCE Once this happens, the ACTIVITY_SUB_STATUS=4105. The ScriptSelector
passed in the NEW_TASK message will be the script that UCCE will run in order to route the task. The Treatment
Leg of the call is complete once the email is queued in the script.
Step 1. Open dcloud and remote desktop to WKST1, then open mRemoteNG and look for the AW-HDS-DDS.
Once the server opens, look for the CCE Script Editor in the Administration Tools
Step 2. Open the CumulusEmail from File -> Open in the Script Editor.
Step 3. Start monitor the script by clicking the monitor button . All counters should show 0.
23 |
Page
Step 4. Switch back over to ECE and look at our query again.
Notice that the ACTIVITY_SUB_STATUS=4100, not 4105. This indicates that ECE never sent a NEW_TASK to
UCCE. Let’s see why this is happening.
Step 5. Switch over to the ECE server, then open Windows Explorer and navigate to C:\Cisco\eService_RT\logs\
folder. Click into the search box and type EAAS.
Step 6. Right-click on the eg_log_ECE_EAAS-process.log file and choose Edit with Notepad++.
24 |
Page
Step 7. Scroll down to the bottom of this file. Does anything here look out of place? Shouldn’t we have
something in this log file? After all, ECE is supposed to be running and connected to UCCE. Let’s look at the
services status in ECE
Step 8. Go back to SPOG and go to Overview > Email and Chat. Change to Partition view
Step 9. Select the Services tab and on the left pane look for Unified CCE, what is the status? Ahh, now we see our
issue. The EAAS Process is stopped. To resolve this situation, make sure to start the process.
Step 10. Make sure to also check the status of the EAMS process and if it is stopped, let’s start it
Step 11. Remember that for every process, there could be multiple instances. Next, we need to be sure that the
instances are started as they do not start as part of the process start. From SPOG change the view to Instances
25 |
Page
Step 12. As we can see both Instances of EAAS and EAMS are stopped even though we already started the
Process. We need to start the EAAS and EAMS instances before continuing and make sure they show up as
running
Notice though that there are two instances under EAMS, one named EAMS-instance and the other named
CUCM_PG. Which one should we start? While you can tell by opening the Edit screen for each, the way that
ECE handles the EAMS-Instance is that the instance which is copied when a new agent PG is added. You will
need to configure and start the instance which is named the same as your Peripheral name in Configuration
Manager.
Step 13. Now that the services are started, if we look back at the Script Editor, SUCCESS!! You should now see 1
new route request into ECE.
Step 14. To see what this looks like in the logs, switch back over to the ECE server and open the
eg_log_ECE_EAAS_process.txt file in Notepad++.
26 |
Page
Step 15. Use Find in Notepad++ and search for the text NEW_TASK.
Another way to search for this is to use the ACTIVITY_ID. Search the file again for, activityId=3053. If your
ACTIVITY_ID is different, substitute your ID. Finally, because eGain does not always write the activity in this
exact format, you can search for just the ID. This method is particularly useful when you use an editor like
Notepad++ which can search across many files very quickly. At the end of the lab guide are some search
examples that we use to track things in TAC.
2022-11-01 03:41:07.973 GMT+0000 <@> INFO <@> [111:Thread-9] <@> ProcessId:11972 <@> PID:1 <@>
UID:12 <@> UserSessionId: <@> ClientIP: <@> com.ipcc.mr.request.CentralRequestRepository <@>
addActivity() <@> addActivity() - New activity added to the CentralRequestRepository. activityId=3053,
activitySubType=1, EAASInstanceId=888 <@>
2022-11-01 03:41:07.984 GMT+0000 <@> INFO <@> [123:mr-request-executor::-0] <@> ProcessId:11972 <@>
PID:1 <@> UID:12 <@> UserSessionId: <@> ClientIP: <@> com.ipcc.arm.MRLogger <@> logOutgoingMessage()
<@> MSG_TYP_NEW_TASK -> 0 0 0 -128 0 0 4 76 0 0 0 1 0 0 0 1 0 0 19 -115 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -
1 -1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 99 10 69 67 69 95 69 109 97 105 108 0 13 4 74 111 115 101 14 17 106
97 100 111 108 105 111 64 99 105 115 99 111 46 99 111 109 15 0 16 11 83 117 109 109 105 116 95 84 101 115
116 82 26 117 115 101 114 46 101 99 101 46 97 99 116 105 118 105 116 121 46 105 100 0 51 48 53 51 0 <@>
2022-11-01 03:41:08.105 GMT+0000 <@> INFO <@> [123:mr-request-executor::-0] <@> ProcessId:11972 <@>
PID:1 <@> UID:12 <@> UserSessionId: <@> ClientIP: <@> com.ipcc.mr.request.NewTaskProcessor <@>
processNewTaskMsg <@> New task for the activity sent sucessfully. Staus is: true . activityId: 3053 <@>
2022-11-01 03:41:08.107 GMT+0000 <@> INFO <@> [123:mr-request-executor::-0] <@> ProcessId:11972 <@>
PID:1 <@> UID:12 <@> UserSessionId: <@> ClientIP: <@> com.ipcc.mr.RequestExecutorCmd <@>
RequestExecutorCmd() <@> New task message processing completed with Status=truefor activityId=3053.
EAASInstanceId=888 <@>
2022-11-01 03:55:26.400 GMT+0000 <@> INFO <@> [119:Thread-14] <@> ProcessId:11972 <@> PID:1 <@>
UID:12 <@> UserSessionId: <@> ClientIP: <@> com.ipcc.arm.MRLogger <@> logIncomingMessage() <@>
MSG_TYP_DO_THIS_WITH_TASK -> 0 0 1 126 0 0 4 77 0 0 0 1 0 0 0 1 0 2 89 -42 0 0 13 81 0 0 0 1 0 0 20 50 -1 -1 -
1 -1 0 0 20 52 0 0 19 -115 0 0 0 1 0 0 19 -100 107 4 49 49 51 53 102 0 101 0 13 4 74 111 115 101 14 17 106 97
100 111 108 105 111 64 99 105 115 99 111 46 99 111 109 15 0 16 11 83 117 109 109 105 116 95 84 101 115 116
17 0 18 19 99 99 95 67 117 115 116 111 109 101 114 73 100 61 48 48 48 48 59 19 40 69 109 97 105 108 61 109
105 99 104 97 101 108 46 108 105 116 116 108 101 102 111 111 116 64 100 99 108 111 117 100 46 99 105 115
27 |
Page
99 111 46 99 111 20 18 77 111 98 105 108 101 61 50 49 50 55 56 57 50 48 48 49 59 21 0 22 0 82 26 117 115 101
114 46 101 99 101 46 97 99 116 105 118 105 116 121 46 105 100 0 51 48 53 51 0 82 80 117 115 101 114 46 67
120 83 117 114 118 101 121 73 110 102 111 0 65 71 61 53 49 55 50 124 83 71 61 53 49 55 48 124 80 81 61 53 48
50 48 124 65 71 84 61 53 48 49 52 124 83 73 68 61 81 67 111 110 116 97 99 116 32 67 101 110 116 101 114 32
70 101 101 100 98 97 99 107 0 82 87 80 79 68 46 73 68 0 99 99 95 67 117 115 116 111 109 101 114 73 100 61 48
48 48 48 59 69 109 97 105 108 61 109 105 99 104 97 101 108 46 108 105 116 116 108 101 102 111 111 116 64
100 99 108 111 117 100 46 99 105 115 99 111 46 99 111 109 59 77 111 98 105 108 101 61 50 49 50 55 56 57 50
48 48 49 59 0 <@>
Our email routing issues are now resolved, and we can route emails to our agent. Task 3 will complete the
process to route our Email to the agent.
MR PG CCE
Router
Finesse
Agent
First, let’s recap what has happened so far to make this email appear in the Script Editor Monitor mode:
ECE Retriever process pulled message off eMail Server and parsed into ECE database and assigned an
ACTIVITY_ID.
ECE Start Workflow added the Activity to Case (either new or existing) and checked for hard and soft
undeliverables.
ECE Inbound Workflow sent Auto-Acknowledgement mail to sender then routed incoming email to an ECE
Queue, which in turn maps to a CCE Dialed Number/Script Selector.
ECE EAAS sent a NEW_TASK message to CCE MR PG PIM requesting routing instructions.
MR PG PIM then translates this message as a NEW_CALL message to CCE Router. The CCE Router will try and
find the CCE Routing Script scheduled for the ScriptSelector specified in the message and start routing the email
following the script logic. Here is where the Email treatment leg finished.
We verified that the email is currently in the Queue to Skill Group node in our routing script.
28 |
Page
Push Mode – Route eMail to Agent
Push based routing is the standard way to have eMails sent to agents. Once the Router identifies an available
agent, it will send a CONNECT message to MR PG. This will be sent to the ECE EAAS process as a
DO_THIS_WITH_TASK message. ECE will then show the email to the agent in the gadget.
Step 1. The Agent Logged into Finesse Desktop and Ready for Email. Within just a few seconds, you should see
the email arrive in the Agent’s inbox. DO NOT CLICK ON IT!! We will look at this further in the next section.
Notice that you can see the CASE and ACTIVITY IDs both when the email is presented to the agent
Step 2. Switch over to the ECE server and run the query that we’ve been using throughout this lab. We are going
to use this several times throughout this section.
As a pro-tip, you can add the following just before the ORDER BY clause, WHERE ACTIVITY_ID=3053 (replace
3053 with the Activity ID from your email.)
Right now, the Activity should be in ACTIVITY_STATUS=5000 and ACTIVITY_SUB_STATUS=5100. This indicates
that the activity has been OFFERED to an agent, but it has not yet been read. For reporting purposes, CCE has
not yet started counting Handle Time for the task.
Step 3. Now, switch back over to Agent sjeffers and click on the eMail to accept it. Switch back over to ECE once
you have done this. Execute our query again and look at the new ACTIVITY_STATUS and ACTIVITY_SUB_STATUS.
29 |
Page
You should now see these as 5000/5900 respectively. ACTIVITY_SUB_STATUS=5900 indicates that the activity is
In Progress. This means the agent has Started the email and is actively working it. It is at this point that CCE
starts the Handle Time for the task.
Step 4. Now, switch back over to Agent sjeffers, compose a reply (be creative ☺), then press Send & Complete to
send this eMail. If you are quick, you can switch back over to ECE, then execute the query a few times and you’ll
see the email go through the following ACTIVITY_STATUS and ACTIVITY_SUB_STATUS states.
Step 5. When your activity gets to 9000/9100, then your email has been sent to the customer and you are
complete.
1000 New
1900 In progress
2900 In progress
30 |
Page
3000 Workflow
3800 Error
3900 Progress
4000 Assignment
4200 In progress
4900 Error
5000 Assigned
5100 New
5200 Pending
5300 Wrap up
5800 Error
5900 In progress
7800 Error
7900 In progress
31 |
Page
ACTIVITY_STATUS Description ACTIVITY_SUB_STATUS Description
9000 Completed
9100 Done
9200 Abandoned
1 Email
1 General
2 Web form
3 Secure
4 Permanent Undeliverable
5 Temporary Undeliverable
6 Reply
7 Forward
8 Compose
9 Auto reply
10 Auto acknowledge
11 Group reply
12 Redirect
13 Undispatch
14 Supervisory accept
15 Supervisory reject
16 Supervisory reattempt
17 Chat Transcript
ACTIVITY_MODE Description
100 Inbound
200 Outbound
500 None
32 |
Page