O

Whitepaper

operational
acceptance testing
Business continuity assurance

Trusted partner for your Digital Journey

 Summary
Operational Acceptance Testing (OAT) Acceptance Testing Irrespective of the delivery model em-
Table of contents Table of figures is the penultimate phase of Acceptance
Testing in a Software Testing Life cycle
Formal testing with respect to user needs,
requirements, and business processes con-
ployed for the project (waterfall, agile,
devops), 3 aspects need to be addressed:
and is the last defence line between a ducted to determine whether or not a system
software development project and de- satisfies the acceptance criteria and to enable 1. Conformance of the asset built with
03 01 ployment of software on production.
FFor decedes, Operational Acceptance has
the user, customers or other authorized entity to
determine whether or not to accept the system.
the requirements and specifications
2. Operate and support the asset
Summary Main Activities and Operational been undermined and misunderstood. Where once deployed to production
User Acceptance has been written about Operational Acceptance Testing
Acceptance Testing stakeholders and hailed as a “final phase in testing before Operational testing in the acceptance
3. Ensure adherence with Business KPI,
IT SLA and business continuity.
production”. User Acceptance is but one side of test phase, typically performed in a (sim-
04 the coin, Operational Acceptance is the other. ulated) operational environment by op- To address these questions, the project
Introduction 02 The International Software Testing
erations and/or systems administration
staff focusing on operational aspects,
team needs to perform the Operational
aspects of acceptance testing and eval-
Quality attribute evaluation Qualifications Board (ISTQB) defines
‘Acceptance Testing’ and ‘OAT’ as:
E.g. recoverability, resource-behavior, in-
stallability and technical compliance.
uation commonly known as Operational
Acceptance Testing (OAT). This whitepaper
06 evaluates the quality characteristics associ-

Scope of Operational Acceptance 03 ated with operational acceptance test scope

from the perspective of the newly released
Testing Main test phases of Operational software testing standard ISO 29119.

Acceptance Testing Types

14
Facets of Operational Acceptance 04
Testing The future of Operational
Acceptance Testing
15
Conclusion 05
Automated Network Testing
19
References & Annexes 06
AtoS Bridge & its Services

For Reference please contact: 07

Sriini Vaasudevan Paladin – Business Process Oriented
Test Lead & Operations Manager/
Product Owner – Telco CRM
Monitoring
Eemsgolaan 7, 9727 DW Gron-
ingen, Netherlands
+31 (0)683624587 08
[email protected]
Opscode Chef

09
Puppet Labs

Operational Acceptance Testing version 1.0 2 3 Operational Acceptance Testing

Introduction
Initially it used to be a practice for testing to This evolution in Information, Technology, a new understanding Operational Acceptance The present paper gives an overview of how
be part of the Software Development Lifecycle Infrastructure, Library provoked the fact that Testing (OAT) came on board. By 2010 the soft- to use ISO 25000 (ISO/IEC JTC 1/SC 7 Software
(SDLC) wherein the development teams did In a project, when the project teams completed the devel- Business continuity cannot be only assured by ware quality industry published the ISO 25000 and Systems Engineering, 2010) to scope out
a test analysis on their own product leading purely functional quality assurance, this created SQuaRE [REF-5] series of standards that outlined OAT systematically and how to apply test meth-
to impartially within their own team. Through
opment of the software, it was released and handed over to a need for introducing various non functional the scope of OAT, in respect to the evaluation ods successfully by having application owners
experience and growing sophistication in the operation team and the application owner. lt immediately aspects borne out of the ITIL processes (Infor- of quality requirements, at a framework and and operation teams involve other stakeholders
the software, software units evolved multiple mation Technology Infrastructure Library). This strategic level. In 2013 the Testing industry like architects, developers, or infrastructures.
phases for testing to improve the quality of
became a part of the business processes as it was launched new paradigm signaled a clear evolution from published the ISO 29119 Software Testing series
the end products produced. During the 1980’s into the production environments. Consequently, known and a purely functional scope to a more holistic of standards, which enabled effective and
Paul Rook designed the V-Model methodology scope of acceptance testing – encompassing efficient testing by qualified/quantified means to
to improve the overall efficiency and effec-
unknown defects of the software will directly impact business both functional and non-functional aspects support the delivery of reliable, robust IT assets.
tiveness in software development processes. continuity and potentially cause damage to a greater or lesser to ensure business continuity. After this new
By this time, the differing test phases had paradigm of acceptance had been accepted
grown to include: Unit Testing, Component
extent. In addition, responsibility typically is transferred from by the development and testing communities,
Testing, System Integration Testing (SIT), the development units to two stakeholders:
System Test (ST) and Acceptance Testing.

References to ‘acceptance testing’ were (then) Application Owner

Figure 1 : Main Activities and OAT stakeholders
understood and interpreted as to mean busi-
ness or User Acceptance Testing (UAT).
As a result of this interpretation, UAT was The single point of contact for business units concerning the
often perceived to be the last, or one of
the last, lines of defense between a soft-
operation of dedicated applications. These units are the
ware development and its implementa- internal part of line organization managing the maintenance,
tion into the production environment.
But this misconception started back-
and constitute the interface to operation teams.
ration Teams
firing. Hereby below a SOC.
Ope
Operation Team

An internal or external team deploying and operating software Standard

following well-defined processes, e.g. tailored by Information, Transition Operation
Technology, Infrastructure, Library. These units are equipped
with system and application utilities for managing the opera-
tion, they will contact the application owners if bigger changes Crisis
should be necessary to guarantee business continuity. Operation

Application Owners

Analysis Design Implemetation Functional Testing Operation

Operational Acceptance testing

Operational Acceptance Testing 4 5 Operational Acceptance Testing

Scope of Operational
Acceptance Testing
In traditional sequential development method,
the Operational Acceptance testing (OAT) is like-
Delivery Models
ly to be executed near the end of the software The relevance of the different types of OAT is
development life cycle. This would enable the The essentiality of OAT can be achieved through the process of incorporating it in the delivery mod- derived from the individual artefacts and their
test to be executed on a like or near-production els. In the growing evolution of software development, the delivery models have matured. From the quality attributes. The major test types are allo-
instance of the asset that is to be implement- primitive Waterfall to Agile which is rationalized now in the growing rage to go Devops. cated, and decisions about test execution follow
ed. This contributes to the idea of Operational Below a definition scale of all 3 and if OAT is coherent with them. from the risk evaluation - the scope of OAT is
Readiness and Assurance. Let’s define them : ultimately defined by selecting columns from
Waterfall or V Model - The waterfall model is a sequential design process, used in software devel- the table shown in Figure 3 and substantiation
»» Operational Readiness is the process of opment processes, in which progress is seen as flowing steadily downwards (like a waterfall) through of these types in coherence with the delivery
preparing the future asset owner, and the the phases of conception, initiation, analysis, design, construction, testing, production/implementation models is explained below as part of section 3.1.
support team, so that, at the time of imple- and maintenance.
mentation/cutover, they are fully ready to as-
sume ownership and operation of the asset. Agile Model - Agile development model is also an incremental model. Software is developed in incre- Figure 2 : Quality attribute evaluation
»» Assurance, in this context, refers to the act mental, rapid cycles. This results in small incremental releases with each release building on previous
of re-assuring the project and organization functionality. Each release is thoroughly tested to ensure software quality is maintained. It is used for
stakeholders that their (iterative) developing time critical applications.
asset, and their organization, is in a state of Operational Code Rehearsal Installation Framework/ SLA/OLA Load/ Security Backup & Failover Recovery
operational readiness – or by which to pro- Documentation Analysis Testing Testing Platform Monitoring Performance testing Restore Testing Testing
Dev Ops - A combination of Development & Operations – is a software development methodology Review Upgrade test Testing Testing
vide a measure of assurance that the asset which looks to integrate all the software development functions from development to operations Testing
will be ready in the adhered time frame. within the same cycle.
Transition x x x x x
OAT subsumes all test activities performed
by application owners and operation Figure 2: Software Model Process Evolution Standard x x x x x
teams to arrive at the acceptance deci-
sion to operate the software under agreed
Crisis x x x x x x x x
Service Level Agreements and Opera- Functional Suitability x x x x x x x x x x
tion Level Agreements (SLAs / OLAs). Process Evolution
These agreements provide measurable Continuous
DevOps Performance Efficiency x x x x x x
criteria which - if they are fulfilled - im- Delivery
plicitly ensure business continuity. Continuous
Compatability x x
Operation capability covers three groups of Integration
tasks: Agile Usability x x x
»» Transition: This designates the successful Waterfall
Reliability
deployment of software into the produc- x x x x x x
tion environment using existing system
management functionality. The production Security x x x x
environment could be a single server or
thousands of workstations in agencies. Maintainability x x x x x x
Software deployment includes software,
database updates, and reorganization but Portability x x x x
also fallback mechanisms in case of failures.
The operation team has to be trained
and equipped with the respective tools.
»» Standard Operation: This designates suc-
cessful business execution and monitoring
on both the system and the application level.
lt includes user support (helpdesk for user
authorization and incident management)
and operation in case of non-critical faults,
E.g. switching to a mirror system in case
of failure analysis on the main system.
»» Crisis Operation: System instability
causes failures and downtimes. Operation
teams must be able to reset the system
and its components into defined states
and continue stable standard operation.
Downtimes have to be as short as possi-
ble, and reset must be achievable without
data or transaction loss and/or damage.

Operational Acceptance Testing 6 7 Operational Acceptance Testing

Types of Operational
Acceptance Testing
Operational Documentation Review Rehearsal testing
Definition / Objective All Documents that are necessary to operate the system. Definition / Objective Unlike User Acceptance testing / E2E which integrates business functions. OAT integrates all functions and stakeholders of a
(e.g. architecture overviews, design documentation, operational docs) are identified, and they are checked for completeness, production system. A rehearsal or staging environment testing is necessary for bigger changes in production environments,
availability and accessibility to the relevant people. Transition, Standard and Crisis mode are addressed. especially when they concern many stakeholders. Objective is to avoid the risks of failures in the process chain and longer
system downtimes shall be minimized or avoided.
Input All prerequisites to implementation. Few examples are:
»» IBR (Infrastructure Blue prints) Input All prerequisites to implementation. Few examples are:
»» Implementation Plan »» Fully tested software with the test release recommendation implementation plan
»» Installation Manual and Application Production Manual »» Installation Manual and Application Production Manual
»» Architecture Overview »» Business Continuity / Disaster recovery plans.
»» Business Continuity/Disaster recovery plans Test Approach »» The main fundamentals in this phase are to ascertain that the test cases of functional change when executed ensure
Test Approach The task of the OAT tester is to ensure that these documents are available finalized and in sufficient quality to enable smooth business continuity.
operation. The documents must be handed over and accepted by the teams handling the production support, helpdesk, »» 2 sets are needed, Roll-forward and Roll-back scenarios.
disaster recovery, and business continuity. The handover must be documented. Operation teams must be included as early »» The test team in close conglomeration with operations has to execute the relevant test cases that adhere to various factors
as possible to obtain their valuable input about the documentation required for operating the system. This way, transparency based on the project being implemented.
about the completeness of documents is achieved early on in the development process, and there will be sufficient time left »» In case of Business projects, new functionality introduced is the main focus.
for countermeasures if documents are missing or lacking in quality. »» In case of LCM projects, all major modules including technical behavior of the system needs to be scrutinized.
Delivery Model Waterfall Model - This type of OAT is specifically designed for Waterfall Model. There is a higher cost involved in following the Delivery Model Waterfall Model - This type of OAT is carried out rigorously to make sure that the operational requirements are handled. This
overall handover to operations. is usually difficult in waterfall as the collaboration with operations is not so strong yielding to slower time to market and higher
Agile or Devops Model - This type of OAT is usually not followed rigorously in this model. But this can be easily covered by costs.
the close cooperation in the scrum teams wherein project, business and operations are involved. The costs involved are less
as no proper handover is needed in this approach. Agile or Devops Model - This type of OAT is seamlessly integrated in the agile/devops methodology due to the involvement
of project, business and operations in the same team. Due to this integration, it is more effectively executed with faster time
Risk of Not Executing If the operational documentation review is omitted or performed to late, testing will start without the final documentation to market and reduced costs in this model.
available which reduces the effectiveness of the tests.
As a result, an increased number of issues may be raised, causing delays in the timeline. Risk of Not Executing Implementation itself is at risk
For operation teams, not having the correct documentation can affect their ability to maintain, support and recover the sys- Business continuity at risk
tems. IT adherence of SLA is at risk
Major leakage of incidents is also possible.

Code Analysis
Definition/ Objective Code Analysis is a tool based automated and/or manual analysis of the source code with respect to dedicated metrics and Installation Testing
quality indicators. This type of test is to ensure transparency about the maintainability and stability of software. Applying
correction and refactoring increases handling of incidents needing hot fixes with little effort in terms of regression testing as Definition/ Objective This test activity ensures that the application installs and de-installs correctly on all intended platforms and configurations.
possible. Objective is to ensure correctness, completeness and successful integration into system management functionality for follow-
ing : Installation, De-installation, Fallback, Upgrade, Patch.
Input Source code of implemented software.
Coding and architecture guidelines. Input Target systems with different configurations (e.g. created from a base image), as well as the specification of changes to the
Test Approach The task of the OAT tester is to do the following: operating system (e.g. registry entries), and a list of additional packages that need to be installed.
»» Selecting the relevant code and tool set-up Test Approach The following aspects must be checked to ensure correct installation and de-installation:
»» Executing code analysis »» The specified registry entries and number of files available need to be verified after the installation is finished.
»» Assessing results and thresholds concerning quality attributes »» The application must use disk space as specified in the documentation in order to avoid problems with insufficient space
»» Analyzing trends with respect to former analysis on the hard disk.
»» Performing benchmarking with respect to standards »» If applicable, the installation over old(er) version(s) must be tested as well the installer must correctly detect and remove
»» Deciding on acceptance or rejection or update old resources.
Delivery Model Waterfall Model - This type of OAT is handled as part of unit testing in the waterfall model. Fully coherent and necessary »» The occurrence of installation breaks has to be tested in each installer step, as well as breaks due to other system events
phase. (e.g. network failure, insufficient resources). The application and the operating system must be in a defined and consistent
Agile or Devops Model - This type of OAT as a principle is also carried forward to Agile or Devops model. Static analysis is a state after every installation break possible.
quality technique where an automated tool checks for defects in the code, often looking for types of problems such as securi- »» Shared resources may have to be installed or updated during installation, and while these processes are performed,
ty defects or coding style issues (to name a few). An accelerator developed within Atos for this purpose is CAST tooling is used conflicts with other applications must be avoided. In terms of uninstallation, the system should be cleaned from shared
for code inspection, assigning mixed teams to increase the bandwidth at transition points (e.g. developers involved in testing, resources that are not used any more. This will result in higher performance and increased security for the whole system.
testers with operations go-live), agile, lean and integrated methods (devops, agile testing), continuous innovation and transfor- »» Since the installation routine is an integral part of the application and a potentially complicated software process, it is sub-
mation of the IT and process landscape, systematic capture of knowledge and systematic reuse of solutions across domains. ject to the regulations of ISO 25010.
Risk of Not Executing High risk of side effects besides functionality defects Delivery Model Waterfall Model - This type of OAT is poorly handled due to the handing over process from a previous phase to the next. Each
Greater effort required in regression and Rehearsal testing phase is executed with a set team. Nevertheless it is part of the process followed in this model.
Agile or Devops Model - This type of OAT is usually well covered in this model due to the team collaboration and clarity of
requirements. Continuous deployment and testing are highly adhered as part of this type of testing.
Risk of Not Executing May result in conflicts with other applications
Compromise or even broken systems
Low user acceptance even-though the software itself has been tested successfully and works as designed.
Number of manual effort, cost and conflicts in larger systems
Operational Acceptance Testing 8 9 Operational Acceptance Testing
 Framework / Platform Upgrade Testing Load / Performance Testing
Definition / Objective This type of testing comprises test activities that ensure successful exchange or upgrade of central components like run time Definition / Objective Performance testing is a technique used to ascertain the parameters of the asset in terms of responsiveness, effectiveness
environments, database systems or standard software versions. and stability under various workloads. This process involves quantitative tests performed to measure and report on the load,
Objective is to obtain proof of correct functionality, sufficient performance or fulfillment or other quality requirements. stress, endurance, volume and capacity threshold limits of the asset. Performance testing measures the quality attributes of
the system, such as scalability, capacity and resource utilization.
Input IBR (Infrastructure Blue prints)
Functionality in chain that needs regression testing Input All prerequisites to implementation. Few examples are:
IBR (Infrastructure Blue prints)
Test Approach 2 possible approaches for introducing central components: Architecture Overview
»» The first approach would be to set up central components as productive within the development system, i.e. central Application production manual highlighting the measures.
components would move parallel to the application software along the test stages towards a release date according to a Requirements for Performance and Load
common release plan. Testing would start implicitly with developer tests.
»» The second approach would be to test changing a central component in a production-like maintenance landscape. In this Test Approach »» Collecting test requirements from an operational point of view
case, a dedicated regression test would be performed parallel to production. Central components would be released for »» Integrating requirements into the Performance/Load model
both operation and development. »» Preparing the test environment and test data
»» Based on the approach the steps would be : »» Executing and analyzing the test
1. Deriving relevant applications from impact analysis »» Defining mitigation scenarios for performance risks
2. Selecting regression tests on the basis of risk assessment »» Deciding on acceptance or rejection
3. Performing regression tests (including job processing) Delivery Model Waterfall Model - Development life cycle times in waterfall model are slow. Performance testing is usually started very late
–– Parallel to development in this life cycle and the test engineers face the pressure of finishing on time. Another performance related issue is reliably
–– In a dedicated maintenance environment using data from a limited nonproduction environment to precisely predict how the system can perform in a more robust
4. Deciding on acceptance or rejection environment.
Delivery Model Waterfall Model - This type of OAT is followed in this model. The main drawback is the integration across teams. Usually this Agile or Devops Model - Agile development breaks through the barriers of traditional waterfall model to software develop-
type of testing needs test strategy to cover the various requirements to be testing in the old and new platform. Often this is a ment to provide better value faster and improve the ROI (return on investment). Performance testing that usually takes place
slow process in waterfall methodology. at the end of the lifecycle in a waterfall model will move to the beginning in an Agile methodology, including the analysis and
Agile or Devops Model - This type of OAT is followed in this model. The main advantage is the integration across teams. designing.
Usually this type of testing needs test strategy to cover the various requirements to be testing in the old and new platform. At The performance of an application is directly proportional to its design and hence should be addressed at an early stage of
times multiple scrum teams can be setup to cover the requirement of testing across platforms. the development lifecycle. Performance testing is considered throughout the Agile process that is from the release planning
stage onwards.
Risk of Not Executing Incompatible software platform
System downtimes Risk of Not Executing Performance issues many go unnoticed after going live
Non-functional issues affecting business continuity Business continuity interruption
Missing fallbacks SLA/OLA adherence hampered
Data defects
Very crucial for Multi-vendor environment and cloud computing

SLA / OLA Monitoring Testing

Definition / Objective This test type examines the implemented monitoring functionality in order to measure the service and operation level. Ob-
jective is to derive if the monitoring functionality is complete, correct and operable in order to derive the right service and
operation level.
Input Business Continuity Checks
KPI parameters
Thresholds and warning requirements
Test Approach Select relevant SLA/OLA
Deriving Monitoring scenarios to estimate service levels
Integrating scenarios into test scenarios of other test types
Execute tests and calculating service levels from monitoring
Delivery Model Waterfall Model - This type of OAT is not specifically designed for Waterfall Model. There is a higher Cost involved in following
the overall handover to Operations.
Agile or Devops Model - This type of OAT is usually followed rigorously in this model. But this can be easily covered by the
close cooperation in the scrum teams wherein project, business and operations are involved. The costs involved are less as
no proper handover is needed in this approach.

Operational Acceptance Testing 10 11 Operational Acceptance Testing

 Security Testing Backup & Restore Testing
Definition / Objective ISO defines this as a “type of testing conducted to evaluate the degree to which an asset, and associated data and information, Definition / Objective Backup and restore testing focuses on the quality of the implemented backup and restore strategy. This strategy may not
are protected so that unauthorized person or systems cannot use, read or modify them, and authorized persons or systems only have an impact on the requirements for software development but also on SUs that have to be fulfilled in operation. In
are not denied access to them.”[REF-4] an expanded test execution, the test objective of a backup includes all the resources, ranging from hardware to software and
It is a technique used to ascertain if the asset protects the data and maintains the functionalities as intended; in respect to documentation, people and processes.
authentication, authorization, availability, confidentiality, integrity and non-repudiation.
Input Working test environment and operation process
Input Risk and Vulnerability Analysis document
Test Approach Backup and restore testing can be executed in a use-case scenario based on well-defined test data and test environments.
Test documentation
In general, a test will comprise the following steps:
Configuration files
»» Quantifying or setting up the initial well-defined testing artefacts
Test Approach For all relevant input channels: »» Backing up existing testing artefacts
»» Check that trash data is handled carefully (fuzzing data). i.e. ensure that the CIA concept (Confidentiality, Integrity, »» Deleting the original artefacts
Availability) is still met »» Restoring artefacts from backup
»» Check that the overall system works properly when system components are switched off (e.g. firewall sniffer. proxies). »» Comparing original artefacts with restored ones.
For all components: »» If applicable, performing a roll-forward and checking again.
»» Check that all test-motivated bypasses (e.g. test users, test data) are deleted for all configuration items
Delivery Model Waterfall Model - This type of OAT is not specifically designed for Waterfall Model. There is a higher cost involved in following
»» Check that all sensitive data are handled according to security guidelines
the overall handover to operations.
»» Check that no debugging information is shown and error messages are customized (information disclosure)
Agile or Devops Model - This type of OAT is usually followed rigorously in this model. But this can be easily covered by the
»» Check that standard credentials are changed to individual and secure credentials
close cooperation in the scrum teams wherein project, business and operations are involved. The costs involved are less as
»» Check the configuration, switch off unused components and close unused ports (firewall)
no proper handover is needed in this approach.
»» Check used certificates for validity.
OAT Test Environment If backup and restore functionality is available, testing can in principle be executed parallel to early functional testing. Howev-
Delivery Model Waterfall Model - During the testing phase of the Waterfall Model, the QC team have a full set of functional requirements to
er, since the tests will involve planned downtimes or phases of exclusive usage of environments. Moreover, this activity will
test. These requirements are to be gathered in the requirements specification phase. Assuming that security was integrated
require the following:
into the previous phases of the SDLC, the QC team will also have a set of security requirements to validate. Security require-
»» Representative test data
ments go through the same creative process as normal functional requirements. Disadvantages are findings from early secu-
»» Established backup infrastructure
rity reviews are often ignored as “theoretical” and costly to go backwards in the development timeline.
»» Established restore infrastructure.
Agile or Devops Model - This type of OAT is usually followed rigorously in this model. But this can be easily covered by the
close cooperation in the Scrum teams wherein Project, Business and Operations are involved. The Costs involved are less as Risk of Not Executing Risk of losing data in a restore situation impeding ability to perform disaster recovery
no proper handover is needed in this approach. Advantage of integrating security testing in this model are: Business continuity interruption
»» Unit tests include security mechanisms & integrate peer code reviews SLA/OLA adherence hampered
»» Test input validation by verifying behavior in edge cases
»» Test access control by verifying behavior from multiple roles.
Risk of Not Executing Security vulnerabilities
Privacy issues resulting leakage of customer data
System/ server hack
Failover Testing / Recovery Testing
Definition / Objective The objective of failover testing can be subdivided into two categories:
»» The degree of the quality of fault recognition (technical measures have to be implemented to detect the failure event e.g.
a heartbeat)
»» The efficiency and effectiveness of the automatic failover reaction in terms of reaction time and data loss.
Input IBR (Infrastructure Blue prints)
Installation Manual and Application Production Manual
Architecture Overview/ Failover plans
Test Approach The test case specification has to describe the measures taken to trigger the failure event. lt is not necessary to execute events
exactly as they happen in the real world since it can be sufficient to simulate them with technical equipment.
For instance:
»» Failure: Lost Network Connection
»» Failure: File system or hard disk failure.
Delivery Model Waterfall Model - Development life cycle times in waterfall model are slow. Failover testing is usually started very late in this
life cycle and the test engineers face the pressure of finishing on getting accurate results from performance testing in time.
The major disadvantages foreseen in this model are late start in the lifecycle, unclarity of requirements.
Agile or Devops Model - Agile development breaks through the barriers of traditional waterfall model to software develop-
ment to provide better value faster and improve the ROI (return on investment). The failover and recovery of an application is
addressed at an early stage of the development lifecycle.

Operational Acceptance Testing 12 13 Operational Acceptance Testing

Facets of Operational Conclusion
Acceptance Testing
The objective of OAT is to achieve the com-
mitment of a handover of the software to
Efficiency and The other aspect of efficiency needing con-
tinuous assurance of quality is effectiveness.
Current situation The next big shift In this age of instant gratification and ubiquitous
personal technology, changes to software or
the application owner and the operation effectiveness For effectiveness, a set of quality gates have & outlook – Automation a network are expected to be implemented
team. Acceptance is based on successful Efficiency in Operational Acceptance testing to be defined which collect all the information Operational Acceptance Testing (OAT) is a Apart from the general requirements outlined seamlessly. End-users now expect applica-
tests and known but manageable defects needs to be assured and some methodologies about the various operation tests and allow crucial part of the Software Development above, companies are also be influenced by tions and services to be fully functional upon
throughout the entire life cycle of the soft- that can serve this purpose are listed as below decisions to be made concerning acceptance. Life-Cycle (SDLC). OAT is where the underly- the implementation models used by cloud release. Users also expect a steady stream
ware development project. The testing is Below you can see the implementation of the ing software configurations and operational service providers. Gartner recommends that of software updates to be implemented in a
organized among all the stakeholders. 1. Early defect detection with Busi- efficiency principles as gates in the life cycle. support components come together. This enterprises should apply the concepts of cloud way that do not hinder the overall utility of
The application owner and the operation team ness Process Validation (BPV) process tests the implementation of structural computing to future datacenter and infrastruc- the tool. Hence, automating the OAT process
»» perform a test of their own which is and Requirements Validation or functional changes to a software or service ture investments in order to increase agility and is becoming increasingly unavoidable.
integrated into the overall test plan 2. Risk based testing approach within a functional or non-functional environ- efficiency. This, of course, affects operational
»» support the testing teams in 3. Traceability of testing requirements ment. In other words, Operational Acceptance models, and OAT will have to support business Developers have traditionally tried to address
achieving the test results 4. Regular Test Assurance for accuracy and Testing evaluates whether or not an application continuity in this growing market. Forrester this by writing run-time tests as they code,
»» check third-party results with respect to quality of test cases and executions can be deployed to a network according to estimates that cloud revenue will grow from $41 essentially implementing manual unit testing
acceptance criteria. 5. Test Automation including Mod- IT Infrastructure Library (ITIL) standards. OAT billion in 2011 to $241 billion in 2020 (Valenti- for infrastructure while building an application.
el-based testing techniques for in- determines if a software will operate the way no-DeVries, 2011). Today, OAT is already being While this can be effective, it adds addition-
Consequently, operational acceptance starts creased quality and efficiency it is designed to without disrupting the whole executed by a varíety of stakeholders. Target al time to the original development cycle,
early on in the project and is achieved after 6. Consultant/ Application owner and installation, network or business that uses it. groups are specialists like application owners which undoubtedly leads to cost increases.
successfully passing all the quality gates. Operations led test health check or operation units in large organizations, as
7. Service Virtualization Ultimately, OAT should focus on the resiliency, well as individuals in the private sector. Internal
recover ability, integrity, manageability and sup- departments and / or vendors need to be
portability of a software or network installation. addressed to establish sustainable processes.
There should also be separate testing processes OAT should follow a systematic approach
for performance, security and data loss/disas- so as to mitigate the risks of cloud providers
Figure 3 : Main test phases of OAT Types ter recovery, which are, themselves, specialty and outsourcing partners, because compa-
areas of huge importance in their own right. nies which offer services using these types
of sourcing will not notice any incidents but
Change Driven Risk Management (CDRM) their impact will be felt directly by the clients.
Low Intensity Activity
Analysis Design Implementation System Installation System Installation E2E Installation Operational Implementation is the process that determines exactly how
test Integration Test/ Acceptance much Operational Acceptance Testing is
High Intensity Activity Test UAT Test needed. This CDRM process will lead to the
Load/
appropriate risk assessment strategy for a
Performance Testing new installation project. As a result, the OAT
Figure 4 : The future of Operational Acceptance Testing
process will be more efficient and focused in
Operational identifying and addressing operational risks.
Documentation
Review
Code Analysis
The next big shift is underway...
Installation testing

Security testing

Framework/ Platform
upgrade testing
2000s The Internet Mobile, Social,
Backup & 1970-80s Mainframe 1990s Client/Server
1990s Client/Server 2010s
Restore testing Bid Data & Cloud
Rehearsal testing

Failover testing

Recovery testing

SLA/OLA
monitoring test
Some of the research done in this area has
Quality led to the following evolutions and standards
OAT-QG1

OAT-QG2

OAT-QG3

OAT-QG4

OAT-QG5

Gates:
to cope with the above mentioned require-
ments.

Operational Acceptance Testing 14 15 Operational Acceptance Testing

The efficiency of an automated network testing Thankfully, several tools have made their way Figure 6 : Atos Bridge & its services
tool can shrink testing and recovery time from to the market that offer varying levels of testing
weeks to days - or even hours, if using the right and management automation. All three offer
system. Automating operational testing also detailed reporting of testing results, making Leading Telecom Provider in NL
can save revenue in the long-term, by reducing ALM (Application Life-cycle Management) easier
both cost of development and cost of mainte- for administrators. These are highlighted below : 24*7 Bridge
nance. To that end, automating OAT has been
somewhat of a holy grail amongst develop-
ers, enterprises and network administrators.
Bridge 24*7 Operations
Operational Syetem Database Business
Figure 5 : Automated Network Testing Application Management Process Chain
Management Monitoring
Management Monitoring

Bridge support teams

Bridge Bridge Bridge Bridge
Application System Bridge Network Security
DBA

S
GP
Support Support Support Support

Bridge Integral Service Management Center

WiF Integral Integral Third
Video Conference Testing on i/3 4-Wire
Automated
G/4G Incident BKPI
Problem Change Party
Android, Windows, or Linux Devices LTE Control Reporting
/E ther Mobile Phones Management Management Management

Network
net G/4G Wireless
WiFi/3
ernet Bridge Service Management Center
Testing
th
LTE/E

Incident Problem Change Trend Reporting

Management Management Management Analysis Service

4G
/3G/
iFi t Bridge Support & Development - Architecture & Tooling
W ne
her
/Et
E Performance &
LT Paladin Suite EDPP Bridge Infra VPC/VDI
Military Radio Capacity Management Deployment
Radio w/PIT Development Development
Management
Data Testing on SmartPhone, Tablet, or PC
(TCP, UDP, HTTP, VoIP, FTP, DNS, SMS, Email)

Figure 7 : Paladin – Business Process Oriented Monitoring

The Bridge Solution – Atos »» Business Process-oriented Monitoring

»» Single point of contact (SPOC)
With its overall business process based view »» ISI ( infrastructure Service Integration )
on the landscape, the Atos Bridge provides the coordination towards 3rd parties
operational single-point of contact towards Atos’ »» Application Management
customers. The unique combination of skills »» System Management
and capabilities for this centralized solution, »» Security Management
one Operator provides Application, System and »» Database Management
Database management, gives the most efficient »» Backup and Restore services
and effective support possible. On the Atos »» Daily operations and Service
Bridge the landscape is monitored 24x7 and Work Request execution
any unexpected events are handled, regard- »» Incident, Problem and Change man-
less of the delivery party responsible or the agement (SMC on Bridge)
technology being used. The Bridge provides the »» Reporting services
following services:

Operational Acceptance Testing 16 17 Operational Acceptance Testing

 References
& Annexes
Opscode’s Chef is an open-source systems 1. APM Group Limited, HM Government and TSO. ITIL - Information Technology Infrastructure Library. High Wycombe, Buckinghamshire. [Online]
Figure 8 : Opscode Chef
framework tool specifically designed for auto- 2012.
mating cloud-based services. The framework http://www.itil-officialsite.com.
offers users a fully automated infrastructure. 2. International Software Testing Qualifications Board. ISTOB Glossary. [Online] March 2010.
Users write abstract definitions, which are then http://www.istqb.org/downloads/glossary.html.
used as source code to build unique parts Dev Qa Release Ops
3. ISO/IEC JTC 1/SC 7 Software and Systems Engineering. ISO/IEC 25000 Software Engineering - Software Product Quality
of the infrastructure. These definitions are Requirements and Evaluation (SQuaRE) - Guide to SQuaRE. [Online] 17/1212010.
then applied to individual servers. The tool is http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_deta il.htm?csnumber=35683.
a Configuration Management System (CMS) 4. Software Engineering Institute. Architecture Tradeoff Analysis Method. Carnegie Mellon University, Pittsburgh, PA. [Cited: 2012.]
built upon a system integration framework, http://www.sei.cmu.edu/architecture/tools/evaluate/atam.cfm.
allowing step-by-step automated testing for 5. Wikipedia contributors. FMEA - Failure Mode and Effects Analysis. [Online] 2012.
your network. The over-all functionality of the http//de.wikipedia.org/wiki/FMEA.
tool can even be expanded using downloaded 6. Operational Acceptance Testing – ScriptRock
plug-ins, known as Cookbooks. As an open- https://www.scriptrock.com/
source framework, it is easily expandable and Dev
Qa
Dev
Qa Prod
7. Atos MS Solutions - The Atos Bridge 1.4.pdf
adaptable for and by the community of users.
Opscode’s software runs a native version of
the Ruby on Rails programming language,
which means that users must be familiar Turnkey Solution for Continuous Delivery Best of breed Integrations for DevOps

with the code and its syntax requirements.

Puppet for Enterprise software is an IT automa-

Figure 9 : Puppet Labs
tion software that gives system administrators
the power to easily automate repetitive tasks
(including Operational Acceptance Testing). The
tool also quickly deploys critical applications and
proactively manages infrastructure changes,
whether executed on-premise or remotely from
the cloud. Any task along the way of managing PUPPET MODULES PUPPET DASHBOARD
your network can be automated using the
application. A model-based approach automates
repetitive tasks and simply eliminates configu-
ration drift. The software enforces the desired Configuration

state of your infrastructure that you define, Applications

Operating System
even as you work on other projects within the
application. The application uses its own unique
coding language, which will require users to DATABASE
be familiar with its syntax. Fortunately, this has
been designed to be as user-friendly as possible.
Reports

Operational Acceptance Testing 18 19 Operational Acceptance Testing

About Atos
Atos SE (Societas Europaea) is a leader in digital
services with pro forma annual revenue of
circa € 12 billion and circa 100,000 employees
in 72 countries. Serving a global client base,
the Group provides Consulting & Systems
Integration services, Managed Services & BPO,
Cloud operations, Big Data & Cyber-security
solutions, as well as transactional services
through Worldline, the European leader in the
payments and transactional services industry.
With its deep technology expertise and industry
knowledge, the Group works with clients across
different business sectors: Defense, Financial
Services, Health, Manufacturing, Media, Utilities,
Public sector, Retail, Telecommunications, and
Transportation.

Atos is focused on business technology that

powers progress and helps organizations to
create their firm of the future. The Group is the
Worldwide Information Technology Partner for
the Olympic & Paralympic Games and is listed
on the Euronext Paris market. Atos operates
under the brands Atos, Atos Consulting, Atos
Worldgrid, Bull, Canopy, Unify and Worldline.
Management

Atos, the Atos logo, Atos Consulting, Atos Worldgrid, Worldline, BlueKiwi, Bull, Canopy the Open Cloud Company, Unify, Yunano, Zero Email, Zero Email Certified and
atos.net The Zero Email Company are registered trademarks of the Atos group. June 2016 © 2016 Atos