https://ivr-tester.sketchingdev.co.

uk/#/

When you are developing functionalities related to IVR (Interactive Voice Response) workflows,
the manual testing process seems nice the first three times. However, when fixing or adding new
settings starts taking more time, one remembers why programmers hate talking on the phone and
prefer to do everything through e-mails. So, here is a solution that makes automation possible.

The need, the motivation, and the tools

The need and motivation

We all agree that a repetitive task, which requires too much time, drains the energy from the doer.
So, we need an automatic way to test longer or tedious IVR workflows and use this time and
energy for other tasks.

The tool

SIPp is an Open Source testing tool based on SIP (Session Initiation Protocol) protocol generally
used for stress tests. It can emulate call flows in both directions, from UAC (User Agent Client)
to UAS (User Agent Server) or UAS to UAC. A powerful advantage of this tool is that it can
send audio or video RTP (Real-Time Transport Protocol) traffic through RTP echo and
RTP/pcap replay. The tools uses XML files to generate custom calls scenarios in which you can
register a SIP user to either call another user or interact with an IVR.

Context

In order to move along this post, we need to be acquainted with the concepts of SIP messages
sequence. We do not aim to fully explain how PBX and SIP protocols work, but it is important to
mention that SIP is a protocol designed for signaling and controlling multimedia sessions. SIP is
not a transport protocol: it only sends text messages based on HTTP elements and uses
complementary SDP (Session Description Protocol) and RTP to transmit media streams.​
For interacting with IVR, we must be able to emulate DTMF (Dual-tone multi-frequency
signaling) signals. With DTMF, each key pressed on your phone or softphone generates tones,
which are made up of adding two specific frequencies.
There are two ways to do that:

●​ By capturing RTP traffic and only isolating the packets we need, or

●​ By creating audio files and converting them to RTP packets with a tool, for example
wav2rtp.

Here is an example of RTP packets capture of DTMF signal for key 1:

Wireshark capture for DTMF 1

First packet detail

 Last packet detail

By using this, it becomes more flexible to generate a specific sequence of tones or capture all
individual tones and reproduce a sequence tone by tone.

Basic IVR test

Getting ready
Step 1: Install SIPp

We need the last stable version of SIPp:

tar -xvzf sipp-xxx.tar

# cd sipp-xxx
# autoreconf -ivf
# ./configure –with-sctp –with-pcap –with-openssl
# make && make install

Common dependencies required are ncurses, libpcap, libnet and lksctp-tools.

Step 2: Configurations

First, a SIP user should be able to register on your PBX:

Also, we need an extension for an IVR:

Step 3: Workflow Scenario

In order to test the IVR:

1.​ We need to be registered on PBX to make calls.

2.​ We call to 111 extension and wait for 3 seconds to press a key.
3.​ We send 1 DTMF capture to PBX with Play PCAP. This simulates having pressed key 1.
4.​ Wait for a few seconds and send Bye to end the call.
5.​ Unregister SIP user from PBX and then end the test.
<?xml version=”1.0″ encoding=”ISO-8859-1″?>
<scenario name=”ivr_test”>
<send retrans=”500″>
<![CDATA[
REGISTER sip:[field1] SIP/2.0
Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
From: <sip:[field0]@[field1]>;tag=[call_number]
To: <sip:[field0]@[field1]>
Call-ID: [call_id]
CSeq: [cseq] REGISTER
Contact: <sip:[field0]@[local_ip]:[local_port];transport=[transport]>
Max-Forwards: 10
Expires: 120
User-Agent: SIPp/Intraway
Content-Length: 0
]]>
</send>
<recv response=”100″ optional=”true”/>
<recv response=”401″ auth=”true” optional=”false”/>
<send retrans=”500″>
<![CDATA[
REGISTER sip:[field1] SIP/2.0
Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
From: <sip:[field0]@[field1]>;tag=[call_number]
To: <sip:[field0]@[field1]>
Call-ID: [call_id]
CSeq: [cseq] REGISTER
Contact: <sip:[field0]@[local_ip]:[local_port];transport=[transport]>;expires=3600
Max-Forwards: 10
Expires: 3600
[field2]
User-Agent: SIPp/Intraway
Content-Length: 0
]]>
</send>

<!– OK –>
<recv response=”100″ optional=”true”/>
<recv response=”200″ rtd=”true” />

<send retrans=”500″ start_rtd=”true”>

<![CDATA[

INVITE sip:[field3]@[remote_ip]:[remote_port];transport=[transport] SIP/2.0

Via: SIP/2.0/[transport] [field1]:[local_port];branch=[branch]
From: <sip:[field0]@[field1]>;tag=[call_number]
To: <sip:[field3]@[remote_ip]:[remote_port]>
[last_Call-ID:]
CSeq: [cseq] INVITE
Contact: [field0] <sip:[field0]@[field1]:[local_port]>
Max-Forwards: 70
[field2]
Content-Type: application/sdp
Content-Length: [len]

v=0
o=intraway 0 0 IN IP[local_ip_type] [local_ip]
s=intraway
c=IN IP[local_ip_type] [local_ip]
t=0 0
m=audio [media_port] RTP/AVP 8
a=rtpmap:8 PCMA/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-11,16

]]>
</send>

<!– ERROR –>

<recv response=”401″ auth=”true” rtd=”true” rrs=”true” optional=”false” />
<send retrans=”500″ start_rtd=”true”>
<![CDATA[
INVITE sip:[field3]@[remote_ip]:[remote_port];transport=[transport] SIP/2.0
Via: SIP/2.0/[transport] [field1]:[local_port];branch=[branch]
From: <sip:[field0]@[field1];transport=[transport]>;tag=[call_number]
To: <sip:[field3]@[remote_ip]:[remote_port];transport=[transport]>
[last_Call-ID:]
CSeq: [cseq] INVITE
Contact: [field0] <sip:[field0]@[field1]:[local_port]>
Max-Forwards: 70
Content-Type: application/sdp
[field2]
Content-Length: [len]

v=0
o=intraway 0 0 IN IP[local_ip_type] [local_ip]
s=intraway
c=IN IP[local_ip_type] [local_ip]
t=0 0
m=audio [media_port] RTP/AVP 8
a=rtpmap:8 PCMA/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-11,16

]]>
</send>

<!– TRYING AND RINGING –>

<recv response=”100″ rtd=”true” optional=”true” />
<recv response=”180″ rtd=”true” optional=”true” />

<recv response=”200″ rtd=”true” rrs=”true” />

<send>
<![CDATA[

ACK [next_url] SIP/2.0

[last_Via:]​
[last_From:]​
[last_To:]​
[last_Call-ID:]
CSeq: [cseq] ACK
Max-Forwards: 70
Subject: IVR Testing flow
Content-Length: 0
[routes]

]]>
</send>

<pause milliseconds=”4000″ />

<nop>
<action>
<exec play_pcap_audio=”pcap/1.pcap”/>
</action>
</nop>

<pause milliseconds=”5000″ />

<send retrans=”500″ start_rtd=”true”>

<![CDATA[
BYE [next_url] SIP/2.0
[last_Via:]​
[last_From:]​
[last_To:]​
[last_Call-ID:]
CSeq: [cseq] BYE
Max-Forwards: 70
Subject: IVR Testing flow
Content-Length: 0
[routes]
]]>
</send>

<!– OK –>
<recv response=”200″ rtd=”true” crlf=”true” />

<pause milliseconds=”1000″ />

<label id=”unregister” />

<send retrans=”500″>
<![CDATA[
REGISTER sip:[remote_ip] SIP/2.0
Via: SIP/2.0/[transport] [field1]:[local_port];branch=[branch]
From: [field0] <sip:[field0]@[field1]>;tag=[call_number]
To: [field0] <sip:[field0]@[field1]>
Call-ID: [call_id]
CSeq: [cseq] REGISTER
Contact: [field0] <sip:[field0]@[field1]:[local_port]>;expires=0
Expires: 0
[field2]
Max-Forwards: 70
Content-Length: 0
]]>
</send>

<recv optional=”true” response=”100″/>

<recv response=”401″ auth=”true” optional=”true” next=”unregister”/>
<!– asterisk –>
<recv response=”200″ />

<!– response time repartition table (ms) –>

<ResponseTimeRepartition value=”10, 20, 30, 40, 50, 100, 150, 200″/>

<!– call length repartition table (ms) –>

<CallLengthRepartition value=”10, 50, 100, 500, 1000, 5000, 10000″/>

</scenario>

Step 4: External configuration

SIPp can inject values into the scenario using CSV files and it allows you to execute the same
scenario with different users or call different extensions. In this case, we configure the test user
12345.

Replace local_ip for your IP address for testing.

Start your engine

Step 1: Run SIPp test

To run the test we need to execute SIPp as follow:

sudo sipp -i <local_ip> -sf ivr_test.xml -inf users.csv -aa -m 1 -l 1 <pbx_ip>

●​ -i: Use this IP for Contact:, Via: and From: message headers.
●​ -sf: Indicates scenario filename to be used on execution.
●​ -inf: Injects values from an external CSV file during calls into the scenarios.
●​ -aa: Enables automatic 200 OK answer for INFO, UPDATE and NOTIFY messages.
●​ -m: The number of processed calls to end the execution.
●​ -l: Sets the maximum number of simultaneous calls.

If we want to view all SIP messages involved during the test, we only need to
add -trace_msg before <pbx_ip>.
In some OS, superuser privileges are required to create or bind raw socket to send RTP packets.

Step 2: Wait for the results

From the Asterisk PBX side we can see the execution:

From the SIPp side we see:

Conclusion
As we can see, creating an automatic test for IVR is not a complex task. All we need to know is a
clear call flow and then create a specific scenario. A scenario is just a XML file, though there are
some graphical tools which can improve its user experience by turning similar ones into diagram
flows, such as the ones used for software development.

Getting DTMF signals is not a problem: you can obtain all tones separately – which implies a
more flexible use in other scenarios, or capture an entire input for a specific test. For example,
when activating TV services using a smartcard ID, we can either send each tone isolated or as a
whole capture.

SIPp also allow us to know how complex the flow call is. We can see how long a call lasts or
how many inputs were needed to perform an action, helping us to come up with better ways to
improve the flow.

R feature testing confirms that the IVR system does what it claims to
do and is consistent with the design, typically a Voice User Interface
(VUI) or Visio flowchart. For example, when a customer calls to get
assistance with a particular problem, they're presented with an IVR
system that tells them, through menu options, to say a specific
number that corresponds to their inquiry. When a complete “dialog
traversal” is executed, every possible scenario or route through the
IVR is run. If an IVR menu only offers options 1, 2, and 3, a
comprehensive test also confirms that the IVR responds appropriately
if the caller enters 4 through 9, 0, *, #, or even no input at all.
Comprehensive feature testing is a requirement for any new IVR
application and many companies also validate every possibility
whenever they make any changes in their IVR system. An automated
feature testing process is far more effective and efficient than manual
testing. Automated testing enables more frequent application releases
that are fully vetted before customer traffic hits them.

What is IVR Experience Testing?

Experience testing refers to 24/7 testing of the Customer Experience
being delivered by IVR systems. Essentially, it's an automated secret
shopper. Test calls are made at regular intervals to ensure everything
is working correctly all the time. Testing tools like HeartBeat™
experience testing perform ongoing monitoring of the availability,
functionality, and performance of the IVR system and its supporting
infrastructure. Critical customer-facing IVR system application
functionality should be tested every 5 or 10 minutes with alerts
generated immediately when issues are identified.

What is IVR Spike Testing?

Spike testing is similar to stress testing but more intense. Spike testing
is when a burst of traffic is launched to the contact center that is above
and beyond what it is designed to handle. This identifies how the
system deals with a big spike in traffic as might happen to a
high-availability architecture if there are problems in the network, or if
a server goes down. Companies want to find out if their system will fall
over, gracefully degrade, or continue humming along. It often identifies
ripple effects, and is critical in IP environments.

What is IVR Regression Testing?

IVR regression testing is a type of analysis that checks if IVR systems
are performing as expected following the addition of a third-party
application or integration. All too often, updates and integrations can
prompt an issue in an IVR system, causing it to work differently and
thereby affecting the customer experience. Regression testing
provides businesses with critical information about the current status
of their IVR system architecture to ensure that end users continue to
have the best possible experience.

IVR testing scenarios

When it comes to speech recognition technology, there's always the
chance of glitches, even though the technology itself has come a long
way in the last few years. Voice call quality, fraud, call routing, accent
and pronunciation, call security and workflow are all areas that IVR
testing can address.

Verification:

The chance of fraud is ever-present in the digital world, which is why

it's vital to test that the IVR application is not compromised or
vulnerable. The IVR application will always verify a caller by asking
security questions like date of birth, secret words known only to the
user, and other relevant information to verify the caller's identity.

Call transfer or routing:

Speech recognition can only do so much, and not all caller inquiries or
problems can be resolved by an IVR system. When a call needs to be
transferred to an agent or a different department, it's important to test
whether it's going to the right area. For example, credit card
payments, and details are security sensitive, and need to be handled
in the correct area.

Call retry option:

Sometimes a caller is not able to recognize or follow a message or

prompt. The IVR system should have the facility to repeat the prompt
a predetermined number of times before disconnecting or transferring
the caller to an agent.

Accent and pronunciation:

When a caller first interacts with an IVR or speech recognition system,

the system should identify accent markers and automatically adjust to
the right accent profile.

Correct language:
In this global market, it's important that IVR automation encompasses
a broad selection of languages, and can be tested properly to ensure
that they work with all users.

IVR workflow:

In-depth testing is needed for all end flows in the system, and it's
crucial to validate that all prompts are related to the correct flow. All
too often a customer is misdirected and ends up hearing prompts
unrelated to their initial inquiry. For example, if a caller has lost a credit
card or wants to report a debit or credit card stolen, this is
time-sensitive and serious. The IVR needs to play the prompt related
to that issue so that it can be resolved quickly, or risk serious
ramifications.

Why you should use automatic

IVR testing
Carrying out IVR system tests manually just isn't practical in the real
world. Recruiting thousands of volunteers or even paying people to
stress test their system would be difficult to co-ordinate, time
consuming and cost-prohibitive. This is why it makes far more sense
to use automatic IVR testing tools like Prognosis
Heartbeat and Prognosis StressTest. The great thing about a software
application is that it can make one IVR call, or as many as required to
carry out the test. These tools scale infinitely, and reduce the need for
human labor, while providing accurate analytics and valuable IVR
system information. The software application can call up your services
as often as you need, 24 hours a day, 7 days a week, every day of the
year.
With the right tools in place, automatic testing helps to give IVR
systems managers peace of mind, with feedback confirming that every
voice call is being dealt with as it should, and that the customer IVR
experience is the best it can be.