ISACA

Systems Implementation Assurance –

Lessons Learned

February 2009
 Agenda – Lessons Learned

1. Project Phase 1- Planning / Mobilization

2. Project Phase 2 – Design / Blueprint

3. Project Phase 3 – Realization / Build & Test

4. Project Phase 4 - Pre Go-live / Deliver Phase

5. Project Phase 5 - Post Go-live / Maintenance Phase

6. Example Project Discussion Document

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Phase 1- Planning/Mobilization

Careful planning, particularly in the early stages of a project, is

necessary to coordinate activities and manage project risks effectively.
The depth and formality of project plans should be commensurate with
the characteristics and risks of a given project.

 Outline Project Plan

 Define Roles and Responsibilities

 Define Project Communication and Reporting Requirements
 Define Deliverables and Expectations – Involvement of all Key Players
 Outline Risk Acceptance - Manage Internal and External Risks
 Define Project oversight activities – Definition of Standards
 Define Tollgates and Requirements
 Define Budget and estimated Project Costs
 Define Project Change Procedures

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Phase 1 Planning/ Mobilization – Lessons Learned

 Putting a proper project governance structure in place with

sufficient "checks and balances".
 Proper Executive and Senior Management buy-in and
involvement in project and milestones reached
 Projects are often comprised of international teams and must
consider both cultural issues and compliance with local laws
and regulations
 Broader industry and business issues must be taken into
consideration

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Phase 1 Planning/Mobilization – Lessons Learned cont.

 Underlying Data Model Consideration (e.g. US GAAP versus

IFRS)
Downstream impact on support functions such as internal
audit and security administration
 Additional Considerations to be aware of during the planning
stage:
 41% of projects fail to meet management’s objectives
 Only 28% of project fulfill management's expectations
 Only 16% of IT projects hit all their targets
 50% of projects end up late or over budget

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Planning/Mobilization – Lessons Learned cont.

Reasons for project failure in the planning stage:

 Bad estimates
 Scope changes
 Change in environment
 Insufficient resources
 Change in strategy
 Imprecise goals/ Insufficient budget
 Poor communication
 Insufficient support
 Wrong project management
 Insufficient motivation
 Stakeholders not adequately defined
 Poor quality of deliverables

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Project Phase 2 - Design/Blueprint

The design phase involves converting the informational, functional, and network
requirements identified during the initiation and planning phases into unified design
specifications that developers use to script programs during the development
phase

 Application Control Standards

 Designing appropriate security, audit, and automated controls
 Standards should be in place to ensure end users, network
administrators, auditors, and security personnel are appropriately
involved during initial project phases.
 Application control standards enhance the security, integrity, and
reliability of automated systems by ensuring input, processed, and
output information is authorized, accurate, complete, and secure.
 Automated input controls help ensure employees accurately input
information, systems properly record input, and systems either reject,
or accept and record, input errors for later review and correction (e.g.
Check Digits, Completeness Checks, Duplication Checks, Validity
Checks, Reasonableness Checks, etc.)

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Project Phase 2 - Design/Blueprint cont.

 Processing Controls - Automated processing controls help ensure systems

accurately process and record information and either reject, or process and
record, errors for later review and correction.
• Batch Controls
• Error Reporting
• Transaction Logs
• Run-to Run Totals
• Sequence Checks

 Output Controls - Automated output controls help ensure systems securely

maintain and properly distribute processed information

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Phase 2 Design/Blueprint – Lessons Learned

 Avoid excessive customization - companies desire to

"re-invent the wheel"
 Many key controls are application driven (e.g. controls which
depend on system generated reports, configuration settings
such as for the three-way match in the procurement cycle)
 Effective process to prioritize all the business "wish-lists”
 Decision Making from “Middle Management” – Timely
Decisions

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Project Phase 3 - Realization/Build & Test

Development
Development standards should be in place to address the responsibilities of
application and system programmers. Application programmers are responsible for
developing and maintaining end-user application.

 Library Controls - Libraries are collections of stored documentation, programs,

and data. Program libraries include reusable program routines or modules stored
in source or object code formats.
 Automated Password Controls – Management should establish logical
access controls for all libraries or objects within libraries
 Automated Library Applications – When feasible, management should
implement automated library programs, which are available from
equipment manufacturers and software vendors

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Project Phase 3 - Realization/Build & Test – cont.

 Version Controls
 Software Documentation
 System Descriptions – System descriptions provide narrative
explanations of operating environments and the interrelated input,
processing, and output functions of integrated application systems
 System Documentation – System documentation includes system
flowcharts and models that identify the source and type of input
information, processing and control actions (automated and manual), and
the nature and location of output information.
 System File Layouts – System file layouts describe collections of related
records generated by individual processing applications
 Naming Convention - critical part of program documentation
 End-User Instructions – Organizations should establish end-user instructions
that describe how to use an application.

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Project Phase 3 - Realization/Build & Test

Build & Test

The testing phase requires organizations to complete various tests to ensure the
accuracy of programmed code, the inclusion of expected functionality, and the
interoperability of applications and other network components. Thorough testing is
critical to ensuring systems meet organizational and end-user requirements.

 Acceptance Testing – to assess the overall functionality and interoperability of

an application
 End-to-End Testing - to assess the interoperability of an application and other
system components such as databases, hardware, software, or communication
devices
 Functional Testing - to assess the operability of a program against predefined
requirements
 Integration Testing - to assess the interfaces of integrated software
components

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Project Phase 3 - Realization/Build & Test – cont.

 Parallel Testing - to compare the output of a new application against a

similar, often the original, application
 Regression Testing - to assess functionality after programmers make
code changes to previously tested applications
 Stress Testing - to assess the maximum limits of an application
 String Testing - to assess the functionality of related code modules
 System Testing - to assess the functionality of an entire system
 Unit Testing - to assess the functionality of small modules of code

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Phase 3 Realization/Build & Test – Lessons Learned

 Project streams reporting 99% completion of tasks which, if

subject to deeper analysis, does not hold water

 Incomplete testing which can have a devastating post go-live

impact when "too lightly" tested configurations fail and disrupt
the business

 Data conversion is a task which many times are under-

estimated

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Project Phase 4 - Pre Go-live/Deliver Phase

The implementation phase involves installing approved applications into

production environments.
Primary tasks include…
 announcing the implementation schedule,
 training end users, and
 installing the product.
Additionally, organizations should…
 input and verify data,
 configure and test system and security parameters

Management should circulate implementation schedules to all affected

parties and should notify users of any implementation responsibilities.

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Phase 4 Pre Go-live/Deliver Phase – Lessons Learned

 Training is a key area where projects tend to cut corners:

 Insufficient training can be disastrous for the morale of
users, acceptance of the new application and company
productivity which can seriously hamper the pre-go-live
promises of more efficient post go-live environment.

 Strong personalities, ego's, compensation structures and a

mentality of "nothing will stop us from going live on x-date" can
mean that pre-determined exit factors for the deliver phase
such as successfully completed testing and completed cut-over
activities can be compromised

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Project Phase 5 - Post Go-live/ Maintenance Phase

Management should…
 conduct post-implementation reviews at the end of a project to validate the
completion of project objectives and assess project management activities.
interview all personnel actively involved in the operational use of a product and
document and address any identified problems.
 analyze the effectiveness of project management activities by comparing,
among other things, planned and actual costs, benefits, and development times.
 document the results and present them to senior management.

The maintenance phase involves…

 making changes to hardware, software, and documentation to support its
operational effectiveness.
 making changes to improve a system’s performance, correct problems, enhance
security, or address user requirements.

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Phase 5 Post Go-live/Maintenance Phase – Lessons
Learned

 PwC was able to categorize post go-live issues in the

following 35 buckets, sorted by number of incidents, highest
number first:
 Locked user/UID validity date required resetting
 Abend related issues
 Report generation
 Authentication
 Batch processing/upload issues

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Phase 5 Post Go-live/Maintenance Phase – Lessons
Learned cont.

 Interface processing issues

 Transaction Processing issues - mostly FI, FI-AP, SD
 PO/EBP GR IR Processing issues
 Access - General
 SAP Mail/Inbox/Workflow Issues
 Process Chain Issues
 Authorization Issue
 Shopping Cart PTP
 Master Data issue

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Phase 5 Post Go-live/Maintenance Phase – Lessons
Learned cont.

 HR Transaction Processing Issue

 Non - PROD access issue - to DEV,QA etc
 ABAP Error
 Miscellaneous
 BW/BI/Related Reports Issues
 Cannot access ESS
 Missing Data/Unable to display issues

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Phase 5 Post Go-live/Maintenance Phase – Lessons
Learned cont.

 Backup Issues
 Project Systems/WBS Issue
 Data Entry / Update / Delete Request
 Runtime Error
 User error/Training Issue
 Extracting/Downloading Data from SAP
 SAP GUI Access Issues
 Financial Period End Consolidation

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Phase 5 Post Go-live/Maintenance Phase – Lessons
Learned cont.

 File error/File copy requests

 Network Issue
 Foreign language/Unicode
 MSS Data Display Issues
 Transport request / issues
 Operating System Issue

Systems Implementation Assurance – Lessons Learned

PricewaterhouseCoopers
 Draft

Independent Project Assurance

February 2009

SDLC Selection Framework • IT Process Maturity

Understanding Your Objectives
Draft

The company is making a significant investment to implement a single pricing, billing,

invoicing, accounts receivable and cash management and collection system, utilizing
SAP as the core technology. With Business Blueprint of Phase II of Project SAP
complete, Executive Management would like to gain the appropriate assurance that
the project achieves it’s stated objectives:

 Realize the tangible and intangible business benefits outlined in the business case with the
priority to increase customer satisfaction with billing and an enhanced ability to efficiently and
effectively launch new products and services in the future.
 Deliver the project on time, within budget, with agreed critical functionality for the business as
quickly as possible.
 Leverage standard SAP business process design and core infrastructure to reduce risk and cost.
 Provide a standard platform to allow for ease of integration and reporting.
 Deliver a compliant system that addresses key stakeholder requirements, including financial and
regulatory reporting requirements.

SDLC Selection Framework • IT Process Maturity

Issues on Your Mind
Draft

Issue Possible Area of Assurance

Data Quality • Review controls around data cleansing and conversion for billing
and customer master data.
• Billing data quality and accuracy
• Share independent perspective on data conversion activities and
• Customer master conversion/migration
provide recommendations throughout the process.
• Customer rate accuracy
• Assess key interfaces identified and controls supporting
• Interfacing of information to legacy systems completeness, accuracy, validity, and restricted access risks.

Customer First Focus • Review controls and system configurations associated with invoice
generation and shipment rating and provide recommendations
• Invoice Presentation Quality and Accuracy
related to validity, completeness, accuracy, efficiency, and
• Shipment Rating Timeframe evidence of duplication.
• Share independent perspective on good practices associated with
revenue cycle and billing/invoicing.

Financial Reporting • Share other client experiences regarding security, internal control
and risk management associated with SAP upgrade to ECC 6.0.
• Inaccurate Bad Debt Provision Calculation
• Provide independent perspective on technical strategy for cash
• Excessive Unapplied Cash Balance
application.
• Current system Upgrade
• Assess process to define key financial and management reporting
requirements and assess the effectiveness of the reporting
designed to meet these requirements.

SDLC Selection Framework • IT Process Maturity

 Draft

Project Assurance – A Suggested Approach

• Ongoing review of the project, control and business
outcomes focusing on the stated Project SAP business
objectives, risks, and priorities.
Project
Governance
• Provide Executive Management with ongoing project Project
assurance reporting. Management
Functional
Readiness

• We would work along side the project identifying potential Business

Case Technical
issues as early as possible and hence allowing Executive Project Readiness
Management adequate time to consider, and if necessary Outcomes
address such issues. This is critical if the independent
project assurance role is to add value to the project and Organizational
help assist in its successful outcome. To this end we Readiness
believe the independent assurance function should:
– Attend and provide input to key project team meetings Business Implementation
Benefits
– Provide a rolling progress report on issues identified Outcomes Methodology
through our work Realization Data
Plan Quality
– Brief key program stakeholders on the status of our work
and issues arising on a regular basis
Controls
Outcomes
Interfaces

Project
Structure
ITGCs

Business
Processes

SDLC Selection Framework • IT Process Maturity

 Our Value Proposition to the company
Draft

• Flexible, tailored approach to focus on management’s priorities for assurance regarding the achievement of Project SAP
objectives.
– Efforts embedded in and integrated with overall Project SAP approach with a focus on value-add
– “One touch” integration of effort with external audit requirements to minimize disruption to project and avoid
surprises
– Evaluate and leverage work performed by others (e.g., Parent Company Internal Audit, SAP, etc.)

• “Hub and Spoke” deployment of world class functional and technical capabilities from PwC to the project:
– SAP Risk Management, Security, and Control
– Transportation & Logistics
– Business Process
– Data Assurance
– Program/Project Management
– Internal Control and Financial Reporting

• Distinguished history of providing independent project assurance services to the company and the parent company.
– Experience navigating the Demand and Supply IT Model
– Invested in relationships throughout the service center and the company.
– Teams deployed alongside of the company in Houston, Scottsdale, and Plantation.

SDLC Selection Framework • IT Process Maturity

Integrating our Audit into Project SAP
Draft

Control Design/Gap Realization Testing Framework

Blueprint Go-live & support
Analysis
Our experience of large
TIMETABLE
Agreement of expected key implementations has
controls within the draft found that the proving of
documentation during the Business Process/ IT General Controls the system is complex and
Blueprint and Realization difficult to manage
phases of the project allows Management Reporting effectively. A key factor
maximum opportunity to are the controls around
correct any issues within the Testing Framework the remediation of issues
design. reported during the testing
Data Conversion/ Cleansing phase.

Management Reporting Security and Access Control

Many key business process Data Conversion

controls rely upon system and Cleansing
generated data. The
requirement to manipulate this
Data integrity is a key risk
data as part of its use adds
within any environment;
additional risk. Effective design
and implementation of system
Security and Access Control this risk is increased
during periods of
reports maximises process
As greater use of system based controls changes such as a
efficiency and reduces the audit
are built into the control environment, the system replacement.
risk.
reliance upon the proper allocation of
access increases. Getting this right from
day one both for business and support
users reduces the risk that gaps are
found post live that affect our strategy.

SDLC Selection Framework • IT Process Maturity

Example Workplan
Draft

Business Process Management Reporting Testing Framework Data Security/Access Controls

/IT General Controls Conversion/Cleansing
Assess process to define Ensure requirements for unit Review proposed SAP
Review proposed key financial and testing, integration testing, Review scope, access related controls
business process control management reporting system testing, UAT, approach, and for sensitive access (SA)
documentation requirements and interface and performance requirements for data and Segregation of
containing the following assess the effectiveness testing are adequately cleansing and Duties (SOD) rule set;
types of controls: of the reporting designed considered with a focus on conversion. role maintenance; and
configurable, reports, to meet these testing of key controls. user provisioning.
manual procedures, requirements. Assess whether an adequate Assess quality controls
automated, and testing monitoring system is within the conversion, Assess SAP user roles
interfaces. Baseline key custom in place. setup and cleansing against SA and SOD rule
reports used to support processes to ensure sets.
Evaluate key controls the operation of manual Assess coordination of data integrity.
over financial reporting controls for financial testing between business Walkthrough user
(selected by the reporting (completeness, and IT. Review controls over provisioning and role
company) for accuracy). the data cleansing and maintenance process.
completeness, accuracy, Review configuration conversion process.
validity, restricted management and change Assess existence of
access, efficiency, control strategy and plan. Review sample of data processes to manage
resilience, and evidence cleansing and access during
of duplication. Review sample of testing conversion results. implementation and
scenarios and results during early stages of live
Review of SAP screens focusing on consistency in Review strategy for operation.
to confirm settings of approach and compliance master data
configurable controls. with policy in relation to key maintenance.
controls.
Walkthrough of business
process controls to
confirm
existence/operation of
the automated and
manual controls.

Assess SAP ITGCs

SDLC Selection Framework • IT Process Maturity

 Draft

Questions

 Contact Information
– Peter Harries, Partner 213 – 356 – 6760
– Charles Lewis, Partner 602 – 364 – 8290
– Pablo Hernandez, Senior Manager 602 – 364 – 8064
– JJ Marais, Senior Manager 602 – 364 – 8232

SDLC Selection Framework • IT Process Maturity