SAP MANUAL TESTING

HOW TO TEST LESS, FASTER, AND BETTER

- 1 -
CONTENTS

Introduction .......................................................................................................................... 2
Why is Automated Testing Such a Challenge for SAP ERP? ................................................................... 2
SAP Manual Testing is Underserved .............................................................................................. 3
Deciding What to TEST: Cost vs. Risk ......................................................................................... 4
Creating Test Scenarios for Scripted Testing ............................................................................... 4
Unscripted Exploratory Testing ............................................................................................. 5
Capacity of Business Key Users ................................................................................................ 5
Documenting Tests Performed ................................................................................................. 5
Managing the Manual Testing Process ........................................................................................ 5
Optimizing the Manual Testing Process .......................................................................................... 6
Automated Test Scoping of SAP-Driven Changes ........................................................................... 6
Accelerated Creation of Test Scenarios ...................................................................................... 6
Accelerated Execution of Manual Testing ................................................................................... 6
Accelerated documentation of test runs ..................................................................................... 7
Designed and Built for Business Users ........................................................................................ 7
Test Management ................................................................................................................ 7




















- 2 -
INTRODUCTION

SAP ERP systems must evolve at the speed of business change. It is therefore imperative for SAP ERP managers
to adapt their SAP landscapes in a timely manner to address business needs such as process changes, rollouts to
new sites, and regulatory updates. In addition, they must cater to the maintenance needs of their SAP systems
and implement SAP support packs and enhancement package upgrades on a regular basis.

SAP ERP systems are highly complex, built on hundreds of millions of lines of standard code and customizations.
This means that even a small change to the system could trigger a series of inadvertent consequences. To
prevent production failures, companies regularly perform a series of functional test cycles which can be
either manually or automatically executed.

Industry papers discussing approaches to test automation have been available for well over a decade. Little has
been discussed, however, about the applicability of test automation to SAP ERP. This document describes the
specific challenges of SAP ERP functional testing, when test automation is inappropriate, and how to perform
manual testing faster, better, and cheaper in these cases.


WHY IS AUTOMATED TESTING SUCH A CHALLENGE FOR SAP ERP?

A survey performed by Panaya in 2012 of 100 SAP-run enterprises revealed that even the most mature testing
organizations perform only 15% to 25% of their functional regression testing using test automation tools; the
remainder is manual. There are three main reasons, inherent to SAP ERP, which explain why test automation
coverage is relatively low:

(i) SAP ERP functionality is heavily data driven. SAP business logic is dependent on and functions
differently with change in the input data; this translates to changes in screen flow, screen content
and screen format that must be factored into the test automation script.
(ii) For business processes that change frequently, the test automation scripts must be continually
kept up to date. A challenging and costly proposition.
(iii) Preparing the required test data for test automation is also challenging due to SAPs complex data
structure and the various dependencies that exist in an SAP business process. Think about an
automatic test script for a purchase-to-pay process that requires five different material types for
full testing coverage. Maintaining integrity of the data required to support the end-to-end process
for the five material types is a daunting task.












- 3 -
SAP MANUAL TESTING IS UNDERSERVED

Since its virtually impossible for a company to develop automation test scripts to cover every possible SAP
business scenario, manual testing is heavily relied upon to ensure that all the needed functionality is covered.
Yet manual testing is severely underserved in terms of tools that increase productivity, enforce processes, and
provide control.

Performing manual tests requires an intimate understanding of the business processes to be tested, and so, it
is typically the functional experts in IT or key business users (sometimes referred to as super users) that are
tasked with performing manual testing. They do their best to test, report defects, report progress, and
document test runs, on top of their other daily duties. This is all performed manually, with tools such as Word
and Excel.

In organizations where a dedicated test manager is allocated to the SAP team, management of the testing
cycle is performed by the test manager. In the majority of SAP shops, however, there isnt a dedicated test
manager; the test managers role is performed by project managers, in the scope of their respective projects,
as an added duty.

Test management includes defining the testing strategy, identifying the tests to be performed, assigning tests
to the manual testers, chasing the testers for their individual status reports and consolidating them to a big-
picture testing progress and coverage status, managing defects, ensuring that testers perform the testing on
time, and more! Approximately two thirds of SAP shops use general productivity software like spreadsheets in
support of these test management activities. It is little surprise, therefore, that they lack visibility and control
over the testing cycle, and put too much time and effort into managing the testing activities.



- 4 -
DECIDING WHAT TO TEST: COST VS. RISK

The first pain point comes early in the testing cycle: determining what, exactly, to test. Following a technical
or functional change in a massive application like SAP ERP, its a challenge to determine where exactly in the
system testing should be focused.

Customers have two options. One is to try to test every possible SAP transaction that may be affected to
minimize risk of production failures. That, of course, increases testing time and cost. The other alternative is
to essentially guess where the greatest impact will be and test those areas. The second option comes with
lower costs but much more risk. In practice, most organizations seek a happy medium between the two
extremes, but the costly nature of manual testing makes many lean toward less testing.

Finding a method to test more functionality within a strict timeline is one of the biggest challenges associated
with implementing changes in the ERP application, so finding a cost effective resolution is extremely valuable.



















CREATING TEST SCENARIOS FOR SCRIPTED TESTING

Test scenarios describe the steps needed to test a given function or process and the anticipated results of every
step. In theory, this should work well: each manual tester has a written script telling him or her what to do,
how to do it, and what results to expect. But in practice, this is far from the case.

Typically written in Word or Excel, test scripts contain varying levels of detail, with no standardization as to
what should and should not be included. Its a time-consuming process, as each script can take from 30 minutes
to several hours to create. And theres no guarantee that others will be able to reliably follow the script;
whats clear to whoever wrote the script may not be to those charged with following it.



- 5 -
UNSCRIPTED EXPLORATORY TESTING

In some cases, test scenarios dont exist at all. Instead, organizations practice a form of ad-hoc testing, known
as exploratory testing, where manual testers are tasked to perform a test as they see fit and report results.

There are two main use cases for exploratory testing:
1. The company doesnt have pre-defined scripts and there is no time to create them. Therefore, manual
testers run a test cycle based on their knowledge of the way the application is used. While this reduces
test preparation time, it adds risk that the business process will not be tested adequately.
2. The QA team or functional testers create test scripts and conduct tests based on these scripts. At the
User Acceptance Test phase the test manager doesnt want the key user to repeat the exact same
scripts, because they were successfully tested already. Furthermore, the test manager needs to ensure
the application is tested exactly as the business users will use it in their daily work, so he asks the user
to simply do their day-to-day tasks on the QA system and check that everything works properly. This
method provides an additional assurance for the quality of the application.

Problems may arise when exploratory testing uncovers issues that developers must address. With no record of
what, exactly, the tester did, its difficult for the developers to accurately recreate problems that need fixing.
Whats more, theres no knowledge transfer of the testing process from the business user to IT to enable IT to
create test scenarios for future use.

CAPACITY OF BUSINESS KEY USERS

Testing takes the business key users away from their pressing day jobs and limits their ability to fulfil their
other duties, such as supporting end-users, delivering training, defining processes, or liaising with IT regarding
functional needs. These key users are typically among the most experienced and highest paid in the user
organization and have very little time to spare. Since the key users are forced to fit test activities around their
normal workload, testing often has to wait.

DOCUMENTING TESTS PERFORMED

At the same time, these time-strapped key users are often required to collect evidence to prove whether or
not these tests were successful. This is especially true for organizations subject to government regulations,
such as Sarbanes Oxley (SOX), U.S. Food and Drug Administration (FDA) rules, and others. Users must collect
detailed proof of what was tested, by whom, and how. It can be a painstaking, time-consuming process for
testers to constantly take screen shots, then copy and paste them into a document to show which tests were
performed. This process can take at least 20 to 30 minutes per test, which translates into several days of non-
productive work per test cycle per key user

MANAGING THE MANUAL TESTING PROCESS

Finally, when testing is manual, the testing manager has no real-time visibility into the process. He can
distribute detailed instructions to testers but has no way of knowing if, when, and how they are conducting the
tests. As a result, he cant unequivocally know the level of progress being made and the precise issues which
have cropped up. When testers are located at multiple sites, the challenge becomes all the greater, forcing the
manager to resort to emails, spreadsheets and phone calls to keep tabs on testers.


- 6 -
OPTIMIZING THE MANUAL TESTING PROCESS

So its clear that manual testing will always be required; theres simply no way around having experienced
users verify that the SAP system functions as it should. However, many aspects of the testing process can and
should be accelerated and managed. These include:

Determining what to test (test scoping)
Creating detailed test scenarios
Executing manual tests
Reporting defects that can be easily reproduced
Documenting test run evidence
Tracking test run progress

In short, manual testers require better tools to make the process more efficient, accurate and less painful.
Testing managers need visibility and control over manual testing cycles. This visibility will enable organizations
to conduct more testing in the same time window and hence, find more problems before the code goes into
production. Increasing testing efficiency also has the effect of driving down the cost of testing and potentially
speeding up project implementation, thus increasing business agility.


SAP IT teams seeking a solution to accelerate the different facets of the manual testing process should
evaluate manual testing solutions on the basis of the following attributes:


1. AUTOMATED TEST SCOPING OF SAP-DRIVEN CHANGES
Determining which functionality of the SAP ERP to test following a major change especially when
implementing SAP-driven changes, such as support packages, enhancement packages or business functions is
crucial to minimizing the risk and cost of change. Analyzing the production usage of your ERP transactions
combined with an impact analysis of the changes on your system can yield a risk-based approach to test
prioritization that can reduce testing efforts by as much as 70%.

2. ACCELERATED CREATION OF TEST SCENARIOS
Writing test scenarios is a painstaking process but its also one thats well suited to automation. Your testing
tool should capture the business process to be tested while a functional expert or key user navigates the ERP
application. The testing tool should translate the steps into structured human-readable test scenarios,
including documentation of expected results for every step of the scenario. Youre essentially capturing the
knowledge of that key user and making it available for all others to emulate in their own testing. At the same
time, you ensure the scripts are highly accurate but take far less time to create.

3. ACCELERATED EXECUTION OF MANUAL TESTING
Similarly, your testing solution should ease the burden on testers by offloading many manual tasks associated
with testing, such as keying the right data into the transaction fields and comparing the results to those
expected, to ensure accuracy.





- 7 -
4. ACCELERATED DOCUMENTATION OF TEST RUNS
Another aspect of test execution is the process of capturing and documenting test evidence for every test run.
Seek a testing solution that enables you to determine exactly which process the tester followed, what the
results were and how those results compare to what was expected. This is a crucial step in terms of providing
proof of testing for auditors, as well as tracing production failures to the gaps in your test scenarios. It also
relieves testers of the tedious process of manually documenting test results.

5. ACCELERATED REPORTING OF DEFECTS
The ability to effectively report functional failure is crucial in enabling IT address issues uncovered by testing.
Developers should get an accurate, screen-by-screen and keystroke-by-keystroke depiction of how the tester
reached a problem point, so they can recreate and fix the problem. This is exactly the kind of data that testers
struggle to capture and communicate accurately.

6. DESIGNED AND BUILT FOR BUSINESS USERS
Remember your key business users will ultimately have to use the testing tool as part of their daily routine.
They are already pressed for time; the last thing they want is to spend time learning how to use a testing tool
that they never wanted to use in the first place. So they need seamless access to the testing tool from the ERP
application. It should require minimal, if any, training to perform such testing activities as: viewing the
assigned test tasks, executing the test tasks, testing in collaboration with other testers to complete a full
business process, and reporting defects and progress. The tool should not require complex installation of client-
side software, and must be intuitive and simple to use, without interfering with the non-testing tasks of the key
user.

7. TEST MANAGEMENT
Finally, your manual testing solution should provide you with ongoing visibility and governance over the end-to-
end testing process. You should be able to plan test projects, assign tasks to all manual testers, determine how
close the testers are to completing their assigned tasks, and receive alerts when problems arise or steps are
completed. The solution should also enable you to track a distributed multi-site testing operation.

8. SIMPLE SET-UP, MINIMAL ONGOING MAINTENANCE
You need a solution that doesnt require cumbersome software installations, frequent patching or painful
upgrade processes on the manual testers client machines. It should take IT less than a day to set up the full
solution.

KEEP UP THE PACE

The pace of business change isnt likely to slow down, so you cant expect the SAP ERP testing burden to
diminish either. Organizations that take the initiative to optimize their manual testing processes will ease the
burden on manual testers, enabling them to spend less time testing while allowing IT to complete more
projects. This ultimately empowers the company to become more agile and competitive.

Learn how Panaya can help you optimize your SAP testing processes. Visit us at: www.panaya.com.




- 8 -
DISCLAIMER AND TRADEMARK NOTICES

This report is provided by Panaya Ltd. It is completely independent of and not affiliated with Oracle or SAP AG.
Oracle is a registered trademark of Oracle. Oracle and other Oracle products and services mentioned herein as
well as their respective logos are trademarks or registered trademarks of Oracle. SAP is a registered trademark of
SAP AG. SAP and other SAP products and services mentioned herein as well as their respective logos are
trademarks or registered trademarks of SAP AG in Germany and in several other countries. All other product and
service names mentioned are the trademarks of their respective companies.

DISCLAIMER OF WARRANTY
Panaya Ltd. makes no representation or warranties, either express or implied by or with respect to anything in
this document, and shall not be liable for any implied warranties of merchantability or fitness for a particular
purpose or for any indirect special or consequential damages.

COPYRIGHT NOTICE
No part of this publication may be reproduced, stored in a retrieval system or transmitted, in any form or by any
means, photocopying, recording or otherwise, without prior written consent of Panaya Ltd. No patent liability is
assumed with respect to the use of the information contained herein. While every precaution has been taken in
the preparation of this publication, Panaya Ltd. assumes no responsibility for errors or omissions. This publication
is subject to change without notice . 2013 Panaya Ltd. All rights reserved.

ABOUT PANAYA
Panaya SaaS Solutions empower companies that use SAP or Oracle to reduce up to 70% of their upgrade and
testing risk and effort. Panaya simulates the upcoming upgrade, automatically pinpointing which custom
programs will break as a result of the upgrade and automatically fixing most of these problems.

Panayas testing solutions dramatically expedite ERP testing and eliminate the need for manual test script
maintenance. Seamlessly capturing business knowledge in the background, as users work with the ERP
applications, Panaya automatically generates plain-English test scripts that are rapidly executed and continually
self-adjust based on test results.