An Experience Report

Solving a live performance problem in a telephony-based speech application

By M Umamaheswaran ([email protected]) &

M V S Raghunath ([email protected])

Project ONVOXAPD, CAPS (PCC- Bangalore)

Client OnMobile Systems
Technology Telephony, Java, Oracle 8i, Nuance Speech Recognition System

Introduction

Solving performance issues at production site is a challenging exercise, more so, when it involves
a speech recognition system with mobile clients where there are complex and dynamic factors
like noise and accent come into force. The key challenges included reproducing the load related
issue in our simulated test environment, the small window provided to fix the issues at a live site
and waiting for the next load spike to monitor the system.

In this document we have outlined the major performance concerns faced by the client, our
approach to fix the problems and the lessons learned from this experience.

Client Profile

OnMobile Systems Inc. is the California, USA based company, originally spun off from Infosys
Technologies Limited (old name is onscan). The flagship product of OnMobile is a multimodal
platform, MMP2500. The OnMobile platform allows wireless subscribers to access and navigate
through an extensive portfolio of location-sensitive applications through the integration of voice,
text and graphics.

Brief Description of the Business

MMP2500 platform enables the carriers and MVNOs (Mobile Virtual Network Operat ors) to
deploy a range of services like voice, SMS and location sensitive applications. It offers flexible
billing mechanism for each of the applications along with the seamless navigation across various
modes (voice, SMS and WAP) maintaining user context and other related information.

Orange India has offered its Mumbai circle for the field trial of MMP2500. Infosys and OnMobile
were involved in development of MMP2500 platform and a suite of applications (Horoscope,
Stocks, News and Cricket along with a main menu and profile management application).

Orange has positioned the above applications as “Premium” category value-added services for its
2.3 lakhs strong subscriber base.

Brief Description of System Architecture

MMP2500 has a highly scalable and distributed architecture. All components (except the
telephony interface unit of the telephony agent module) were java programs.

Management Server: Takes care of the management of all other components along with user
management and configuration management activities.

Telephony Agent: This module interfaces with the telephony switch and runs the business logic
of the application
Resource Server: This module manages the various resources required for a speech application
(like recognition server, license manager, resource manager and compilation server)

Database server: Oracle 8i was used as database to store user and application related
information.

Problem Faced

Various problems were faced at different stages of the deployment.

1. Problem: Load tests failed for 30 users on a single telephony agent whereas the
estimated capacity was 45 users for the given hardware setup.

Solution: Micro-level study of the telephony agent process revealed that there were too
many instances of the application objects created frequently as new calls landed on the
telephony server and destroyed when the calls ended. Here we introduced the concept of
object pooling for the application objects. That is, telephony server will always maintain a
pool of (say ‘m’) application objects for each application. When new calls land on the
telephony server, an application object from the pool will be retrieved and used by the
thread(which is handling the call). When the call ends the object will be released back to
the pool.

Further study into the dynamics of the usage-pattern of the application revealed that not
all sub-modules of the application were used by each call. For example, when users
enter a cricket application, they might just visit the “one-day” module and end the call
without even visiting “test-match” sub-module. So there is no need to create “test-match”
module object in the beginning itself, the creation can be delayed till actually the user
requests it.

With the above changes we were able to reach the full capacity of 45 simultaneous calls.

2. Problem: After deployment we found that the CPU of the database server went to 100%
usage when we reached just 40 users (20 calls from each of the two telephony servers)

Solution: This was most challenging problem because we were never able to simulate it
in our lab for a long time. The research to solve this problem spanned across 2 months
and many critical components were analyzed threadbare. Some of them are listed below:

a) The configuration information that we set in Management Server is supposed to

be communicated to Telephony Server. Some of the configuration information
like database connection pool size (set to 30 connections) stored in the
Management Server was not communicated to the telephony server. As a result
the default value was used (5 database connections). The default value was
obviously insufficient to take the load.

b) Method A fires a query AQ1 then calls method B that fires BQ1. On return to
method A, another query AQ2 is fired from method A.
Here method A gets a connection Acon from the DBConnectionPool and
without releasing the connection in method A, method B is called. Here method B
again gets a connection Bcon from DBConnectionPool and fires a query and
releases the connection BCon. On return to method A, another query is fired and
then Acon is released. So at one point of time (while firing the query BQ1) the
same thread is hogging two connections from the connection pool. This leads to
the creation of several Oracle processes and the denial of new connections once
 the number of processes exceeds a preset limit. Introducing a wrapper around
the ConnectionPoolManager that gives the same connection object to all
connection requests from the same “Thread” solved this problem.

c) There were a couple of instances where the job done by two queries could be
done by a single call to a stored procedure, thereby reducing the number of
database interactions.

d) We used “performance analyzing queries” on the oracle stat tables to find out
which table had too many inefficient reads and further research revealed the
efficient means of fetching the data.

e) Finally the real bug that took the cake was a simple and traditional indexing
problem in one of the database tables.
Simply put, there was a table with huge number of records (3 lakhs). It had a
composite primary key (say A, B, C in that order). There was a program firing a
“select” query which had simply B in the “where ” clause. And there was no index
created on B (or B, C or B, A or B, C, A or B, A, C). So the query went for “table
scan” each time the query was fired. 3 Lakhs records are obviously too big to go
for table scan. When we simply created an index for that table on column B, the
performance improved dramatically.

Lessons Learnt:

• Evaluate the w “ eight” of each query in each program. By weight we mean, the
number of times the query is fired in each session, number of records in that
table. According to the weight check whether “heavy” queries have enough
“indexes” to help the database server.
• Do a thorough code review of all critical modules, preferably by somebody
outside the module.
• Make sure that a QA test plan handles even test cases that check the
configuration details of the product.
• Try to understand the usage pattern of the users and rethink about timing of
creation of various java objects.
• Check the efficiency of the traditional connection pool management.
• Search for scope to push maximum database work to the backend (in the form of
stored procedures and triggers) instead of doing the same from the client
program

Team involved in solving the Problem

The report will remain unfinished without mentioning the team members involved in
solving this problem.

• Sunil.B.K. (Infosys) Kamesh Babu (OnMobile)

• Sreenivasan.B.P. (Infosys) Joji Varghese (OnMobile)
• Sreejith Chandrasekaran (Infosys) Purush Yeluripatti (OnMobile)
• Consultancy from InPTC Satish Gopalakrishnan (OnMobile)