Accenture Data Migration Assessment
Accenture Data Migration Assessment
DATE: 24.09.2015
Table of Content
1 Executive Summary ...................................................................................................... 2
1.1 Scope of Work ........................................................................................................ 2
1.2 Approach for Data Migration Assessment .............................................................. 2
1.3 Key Findings ........................................................................................................... 3
1.4 Consequences ........................................................................................................ 6
1.5 Recommendations .................................................................................................. 6
2 Scope of Work .............................................................................................................. 7
2.1 Methodology for Data Migration Assessment ......................................................... 7
2.2 Limitations .............................................................................................................. 8
3 Approach and Methodology .......................................................................................... 9
3.1 Approach ................................................................................................................ 9
3.2 Data Migration Fundamentals ................................................................................. 9
3.3 Methodology ......................................................................................................... 10
4 Chronology of Events ................................................................................................. 11
5 Findings and Consequences ...................................................................................... 13
5.1 Assessment of the EFI Data Migration Analysis Phase ........................................ 13
5.1.1 Key Findings .................................................................................................. 13
5.2 Assessment of the EFI Data Migration Design Phase .......................................... 15
5.2.1 Key Findings .................................................................................................. 15
5.3 Assessment of the EFI Data Migration Build ........................................................ 16
5.4 Assessment of the EFI Data Migration Testing ..................................................... 16
5.4.1 Key Findings .................................................................................................. 17
5.5 Assessment of the EFI Data Migration for Production .......................................... 19
5.5.1 Key Findings .................................................................................................. 19
6 Appendix..................................................................................................................... 21
6.1 Documents Examined ........................................................................................... 21
6.2 List of Meetings..................................................................................................... 24
1|Page
DATE: 24.09.2015
1 Executive Summary
1.1 Scope of Work
The scope of this Data Migration Assessment was to review the process that was
performed for the migration of claim and treatment data for the EFI Programme. It
therefore includes assessing key tasks in the analysis, design and testing of the data
migration. Accenture has reviewed the timeline, methodology and sequence of events
from the start of the data migration effort in 2009 until the EFI+DMI System went into
operation in September 2013.
This report is based on review of the documents listed in Appendix 6.1 from the data
migration process for the EFI Programme. The purpose of the review has not been to
consider who is responsible for the decisions taken during the project execution. The
report does therefore not include a legal review of the contractual obligations in the EFI
and DMI contracts and the report can thus not be used to conclude whether or to what
extent any of the parties involved in the project execution can be held legally responsible
for their involvement in the project.
The assessment is made based on our experience and on assumption that the
conclusions in the assessment are representative also for the data migration documents
that have not been reviewed.
The assessment has been conducted by examining key documents and conducting
interviews with key people involved in the data migration for the EFI Programme. The
documents reviewed and the interviews conducted are listed in Appendix 6.1.
The examination of documents started with the original requirements for data migration
and ended with a report describing the deficiencies in data in the EFI+DMI System after
go-live. Interviews were conducted in order to clarify facts in the documents reviewed.
Accenture’s Delivery Methodology for Data Migration was used as a reference for
generally accepted IT industry normal practices of data migration projects. For each stage
of the methodology, we have assessed the facts as evidenced by the documents reviewed
versus a normal practice approach.
Accenture did review the documents listed in Appendix 6.1, which cover:
2|Page
DATE: 24.09.2015
Not all the entry/exit criteria for the migration trials, as stated in the conversion plan,
were adhered to during the execution of the data migration trials
The end-to-end testing of EFI and DMI with the migrated data was not completed
Data Cleansing
Cleansing of source data with deficiencies prior to migration is a vital task according to the
methodology. Although there was a clean-up list defined and the data cleansing activity
was tracked, we found that:
Status green had been reported although there were a number of unfinished
activities listed
The data cleansing still did not prevent data with e.g. incorrect amounts from being
migrated
None of the data migration trials (PK1, PK2, PK3, PK4) were on a full scale (100%
of source system data) – the most extensive migration trial, PK4, was limited to
~49% the amount of data in the production migration
The summary status reports for each of the migration trials (PK1, PK3, PK4)
reported that the trials were completed, although there were listed remaining
defects that were still to be fixed
As the EFI Modtag Fordring (MF) component, as well as EFI+DMI, were under
development and testing during the migration trials, it is a fact that neither the
application used for migration (MF) nor the target System (EFI+DMI) were tested in
their final state with the migrated data. The testing was therefore inconclusive and
3|Page
DATE: 24.09.2015
invalid, as it was done on earlier versions of MF, EFI and DMI than the versions
used during the production conversion and production
Only limited testing of the migrated data was performed in a test phase (pilot)
between PK4 and go-live. The end-to-end testing of EFI with the migrated data that
was described in the conversion plan was not performed
Deployment / Go-live
A review of the summary status report from the last migration trial PK4, revealed that:
The scope of PK4 had been reduced and had not covered 100% of all data to be
converted. Only ~49% of source data was attempted converted in PK4
The exit criteria for PK4 had been reduced compared to those listed for PK2 in the
conversion plan. PK4 would not have passed the exit criteria if they had been used
as specified in the conversion plan
The summary status report from PK4 still concluded with a recommendation that
PK4 be approved and that the project could continue to the production migration,
“with prioritized risk reduction actions and defect corrections”
The steering committee of the EFI Programme made the go-live decision following
the PK4 summary status report recommendation
4|Page
DATE: 24.09.2015
Figure 1 Accenture Data Migration Methodology and highlighted shortcomings in the EFI migration
project.
In Summary
The diagram above illustrates the major tasks of data migration according to Accenture’s
Methodology for Data Migration. The callouts in yellow indicate where our findings from the
data migration of EFI and DMI deviated from the methodology’s approach.
The evidence is clear that
Less than half of the combined body of data in all source systems were ever
included in migration trials prior to the production conversion.
The validation of incoming data was not sufficient to prevent incorrect claims being
migrated
The migrated data was not tested end-to-end with the EFI+DMI system before the
production migration
These facts are the most important deficiencies of the data migration project and
constituted a high level of risk that the data migration would not be successful.
5|Page
DATE: 24.09.2015
1.4 Consequences
The consequences of the EFI data migration were:
Data was migrated into the EFI+DMI System that could not be processed correctly
by the system. As the EFI+DMI system had not been thoroughly tested with the
migrated data it could not be expected that it would be able to process the migrated
data correctly
The EFI+DMI System was designed to handle data without deficiencies and Modtag
Fordring did not include sufficient data validation to ensure that only correct and
complete data was entered into EFI. For instance claims were placed on treatments
where the claim type – treatment combination is not allowed
As of 2 years post go-live it has not been possible to enable much of the disabled
automated functionality originally intended in EFI, due to a variety of functionality
and data related issues which are still not fixed, e.g. incorrect expiration dates on
claims
1.5 Recommendations
In order to recover large portions of the affected data, with a view at one point to continue
operation with new systems, a data cleansing exercise should be performed to detect and
correct data issues introduced by the data migration or subsequent operation. The data
cleansing should be aligned with other functional and technical changes being planned,
and will require more focus on clear requirements and testing. The data cleansing will be
complex and lengthy, and is likely to be able to resolve only a portion of the known data
issues.
6|Page
DATE: 24.09.2015
2 Scope of Work
The scope of this data migration assessment was to review the process that was
performed for the migration of claims and treatments data for the EFI Programme. It
therefore includes assessing key tasks in the analysis, design and testing of the data
migration. Accenture has reviewed the timeline, methodology and sequence of events
from the start of the data migration effort in 2009 until the EFI+DMI System went into
operation in September 2013.
Started with the documents made available to us from the EFI Programme at the
start of the assessment
We have requested and obtained a number of additional documents based on
document references found in the first documents we reviewed
Our examination of documents has not been exhaustive, but focused on obtaining a
logical sequence of documents describing the EFI data migration from start to end
7|Page
DATE: 24.09.2015
To assess the production data migration that was done in Aug/Sep 2013 to migrate
claims and treatments from the source systems into EFI and DMI before EFI and
DMI were put into production in Sep 2013
To examine the critical tasks that were performed in preparing for the data
migration, including the validation of the data migration through testing
2.2 Limitations
The scope of this data migration assessment was to review the process that was
performed for the migration of claims and treatments data for the EFI Programme. It
therefore includes assessing key tasks in the analysis, design and testing of the data
migration. Accenture has reviewed the timeline, methodology and sequence of events
from the start of the data migration effort in 2009 until the EFI+DMI System went into
operation in September 2013.
This report is based on review of the documents listed in Appendix 6.1 from the data
migration process for the EFI Programme. The purpose of the review has not been to
consider who is responsible for the decisions taken during the project execution. The
report does therefore not include a legal review of the contractual obligations in the EFI
and DMI contracts and the report can thus not be used to conclude whether or to what
extent any of the parties involved in the project execution can be held legally responsible
for their involvement in the project.
The report is made based on our experience and assumption that the conclusions in the
report are representative also for the data migration documents that have not been
reviewed.
8|Page
DATE: 24.09.2015
The target system must first have completed testing with prepared (controlled) test
data, this is a prerequisite for the following step
Then the target system must be tested with the migrated data to validate that it is in
fact capable of processing the migrated data correctly
9|Page
DATE: 24.09.2015
The testing described in the previous paragraph is normally a test of all relevant business
scenarios from start to finish; normally referred to as “case based testing” or “end-to-end
testing”.
The term “data conversion” is often used instead of and interchangeably with “data
migration”.
3.3 Methodology
Accenture’s Delivery Methodology (ADM) for Data Migration is used as a reference for
generally accepted industry practices of data migration.
The methodology is organised into a five-step process with a standard set of tasks to be
performed.
E: Data Migration
E1: Go-live decision
E2: Final migration run
E3: Production controls, to confirm that the migration has been successful in the
production environment.
10 | P a g e
DATE: 24.09.2015
4 Chronology of Events
Based on our review of all the documentation and interviews listed in the Appendix we
have drawn the following high-level timeline for the data migration of the EFI Programme
from start to finish.
4 Migration Build
9 Production Migration
10 Data Validation
11 Go Live Decision
11 | P a g e
DATE: 24.09.2015
2 Migration Design
3 Migration Build
4 Migration Trial 1
5 Migration Trial 2
6 Migration Trial 3
7 Migration Trial 4
12 Production Migration
According to the Data Migration Methodology, the total data migration effort would run in
parallel with systems development for most of the duration of the project.
However, we normally would have recommended a more concentrated project effort to
reduce the timeline to e.g. 2 – 2½ years in order to reduce the risk of scope changes
during the migration.
The remainder of the document deals with the assessment of the data migration work that
was done, as evidenced by the documents reviewed and the interviews conducted. The
report has been structured according to the phases of the Data Migration Methodology.
12 | P a g e
DATE: 24.09.2015
Information in the documents listed above was compared to what is to be expected in the
analysis phase of a data migration project, based on ADM for data migration.
Our assessment focused on answering the following key questions:
13 | P a g e
DATE: 24.09.2015
Informatica portal, plus specified data formats that the source system owners were to
deliver data in to the migration team.
It is our evaluation that the detailing of the original requirements [1] were lacking the
hierarchical breakdown structure which is required for defining all the tests that are
necessary in order to validate the full data migration effort.
14 | P a g e
DATE: 24.09.2015
This section describes Accenture’s assessment of the design phase of the EFI data
migration project. We identified the following documents as the most relevant for the
assessment of the design phase:
Ref.# Category (used Document title
for sorting)
34 Production Produktionskonvertering afrapportering v0_8
conversion (00000002).docx
46 Production Bilag B - Tværgående konverteringsmangler (9573038_2)
conversion (2).docx
57 Accenture Technical Report.docx
deliverable
Table 2 Most Relevant Documents for Assessment of Design Phase
Information in the documents listed above was compared to what is to be expected in the
design phase of a data migration project, based on ADM for Data Migration.
Our assessment focused on answering the following key questions:
Did the data migration design documents include the necessary controls to secure
that only quality data was migrated – and all non-conforming data written to error
logs?
What data controls were designed into Modtag Fordring, the part of EFI through
which the vast majority of claims were migrated? (Modtag Fordring is the interface
through which claimants submit their claims to EFI+DMI)
15 | P a g e
DATE: 24.09.2015
been added to the Web service message to Modtag Fordring to also accept historic
claims.
Validation in MF is based on a number of things [see Technical Report]:
Accenture have shown in our Technical Report [57], that Modtag Fordring does not include
validations to ensure that only valid claims can enter the system. E.g. when MF was used
to load claims in the migration, claims were accepted and put on a treatment for which the
claim type was not justified. There is no sanity checking on key fields, e.g. to ensuring that
dates are in a proper time sequence for the claim to be in a valid state, and the amount of
debt is not sanity checked [46].
No specific comments.
This section describes Accenture’s assessment of the testing of the EFI data migration
project. We identified the following documents as the most relevant for the assessment of
the testing that was done:
Ref.# Category (used Document title
for sorting)
3 Strategy Konvereteringsstrategi v1.2.pdf
4 Plan Konverteringsplan EFI v09 CSC KONV
5 PK1 Prøvekonvertering 1 afrapportering 19012012
7 PK2 Opsamlingsskema for test PK2.docx
10 PK3 05 b KFL_Prøvprøvekonvertering 3 afrapportering v0_2
[DOK32828717] (10064235_1).DOCX
16 | P a g e
DATE: 24.09.2015
Information in the documents listed above was compared to what is to be expected in the
testing of a data migration project, based on ADM for Data Migration.
Our assessment focused on answering the following key questions:
Migration Trials
None of the migration trials PK1, PK2 (we have not seen this report, however it is stated
[10] that this was the case), PK3 and PK4 represented a complete data migration trial,
hence, there was no full migration trial performed prior to the production data migration.
Mapping rules for treatments were adjusted after each migration trial, also after PK4 [3].
In the final migration trial - PK4, the following was the result as stated in [17]:
~9.3 million out of ~19.1 million claims attempted to be migrated (49% of total
claims)
EFI rejected a significant number of claims, for multiple reasons including a lack of
AKR functionality (customers that could not be reconciled with CPR or CVR)
The balances in EFI and DMI did not reconcile, post migration
There was a high probability that EFI would automatically mail 600,000 debtors
immediately
Payment ability (BeBB) calculations were not usable
Assignment of automatic treatments were not usable
o As a direct result of this testing, most automated functionality was disabled
before go-live (the “pilotspor” was used instead)
A number of remaining defects were listed
17 | P a g e
DATE: 24.09.2015
The summary status reports for each of the migration trials (PK1, PK3, PK4) [5, 10, 17]
reported that the trials were completed, although there were listed remaining defects that
were still to be fixed. (The summary status report for PK2 has not been identified).
Retesting of fixed defects were not to be done before go-live. This is extremely risky
as the need to retest defect corrections is common practice in any testing effort
(regression test).
Reduction of the end-to-end testing was to be compensated by a pilot and “other
relevant activities”, but there is no description of what kind of activities that were
intended to compensate for the lack of sufficient testing.
The evidence is clear on the fact that less than half of the combined body of data in all
source systems were ever included in migration trials prior to the production migration.
That and the fact that the migrated data was not tested end-to-end in EFI+DMI are the
single most important deficiencies of the data migration project.
There is no way to predict how data will behave in a system if it has not been tested
sufficiently, i.e. through testing end-to-end business processes on an amount of data
which represents all the data variants that will come from the source systems.
18 | P a g e
DATE: 24.09.2015
Information in the documents listed above was compared to what is to be expected in the
production migration phase of a data migration project, based on ADM for Data migration.
Our assessment focused on answering the following key questions:
19 | P a g e
DATE: 24.09.2015
PK4 summary status report [17] it is observed that a complete migration trial of data from
all sources was not completed, because Udbetaling Danmark (UDK) was not participating.
Based on our examination of the trial migration reports from PK1, PK3 and PK4 [5, 10, 17]
and the summary status report from the production conversion [34] it is our observation
that:
None of the 4 migration trials tested a full load of data from all sources
The exceptions (data migration defects) listed in PK3 and PK4 were not managed to a
conclusion before moving on to the next trial migration. Overall status green was set for
each of the migration trials despite the fact that there were remaining deficiencies and
when the identified exceptions had been assigned to be fixed, without waiting till after the
exceptions had been fixed and retested.
Despite the above shortcomings, the status report from PK4 [17], Section 7.1.1.2:
concluded with: “It is recommended to finalize and approve migration trial 4 and it is
recommended that the production migration can be performed on the current basis plus
risk evaluated and prioritized defect corrections and changes” (our translation).
It is our evaluation that the conversion team should not have recommended to the steering
group of the EFI Programme that the data migration could proceed to the production
migration. Each of the deficiencies listed above should have mandated a clear no-go
decision, according to Accenture’s Data Migration Methodology.
Accenture has done profiling of the migrated data in EFI and DMI which shows that the
length of time since a payment was last received on migrated claims is so long that in our
opinion they are effectively not being collected.
It was the decision of the project’s steering group [28, 29] to move claims into EFI where it
was known that subsequent manual tasks were necessary in order to process the claims
correctly.
20 | P a g e
DATE: 24.09.2015
6 Appendix
6.1 Documents Examined
The following table lists the documentation examined for the data migration assessment.
All documents in the list have been examined and where it is especially relevant to
explicitly reference a document, and sections of a document, this has been done in the
text of the report.
21 | P a g e
DATE: 24.09.2015
22 | P a g e
DATE: 24.09.2015
Production
42 20121106 Verdensbillede.pptx
conversion
Production 14 b 2012 01 19 Teknisk konvertering referat eftermiddag
43
conversion [DOK32937719] (10064278_1).DOCX
Production 14 b Teknisk konverteringsproces rev maj12 [DOK32937725]
44
conversion (10064283_1).DOCX
Production
45 EFI DMI prodkonvertering afrapportering v01.pptx
conversion
Production Bilag B - Tværgående konverteringsmangler (9573038_2)
46
conversion (2).docx
Production
47 Dag til dag Cutover plan - EFI 10 06 13.xlsx
conversion
Production
48 Processer i lukkeperioden EFI (3).docx
conversion
Production
49 Kopi af ANH FKT oplæg uge 16og17 v3
conversion
Production
50 2013-04-13 FKT reduktion og færdiggørelse
conversion
Production 130926 SKMI0168S PA EFI-statusrapportering - SKAT
51
conversion systemmodernisering fase 2 v3 2
Production 130730 31juli_UdfordringerttilLøsningnu_ved
52
conversion samletchefgruppe_forretning_teknik_projekt-pb.xlsx
Production
53 130612 EFI Review - Mgmt præsentation - v1 0 - 130607
conversion
Production
54 121126 Imødegåelse af trusler mod EFI go-live i maj 2013pptx
conversion
Production 130503 Imødegåelse af trusler mod EFI go-live i september 2013
55
conversion 030513
Production
56 Bilag_2_EFI leverancer
conversion
Accenture
57 Technical Report.docx
deliverable
58 Other BEBB Kunde.docx
59 Other REV Diverse planer for EFI.msg
06 b Fordringer_analyse_konklusion_v1_41_kommenteret[1]
60 Other
[DOK33853557] (10064248_1).DOCX
61 Other 27 b liste_man_tina (DOK31482432) (10064392_1).XLSX
27 b
62 Other mapningstabel_henlaeggelser_v08_2013_06_21_efter_prod_kon
v (DOK31482431) (10064395_1).XLSX
63 Other 130610 Oplæg til workshop 110412 V4.pptx
131001 Copy of 130926 SKMI0168S PA EFI-statusrapportering -
64 Other
SKAT systemmodernisering fase 2 v3 2.xls
Ministry of
65 Bilag_2_EFI leverancer (00000002).pdf
Finance
Table 5 Documents Examined
23 | P a g e
DATE: 24.09.2015
The following interviews were conducted with stakeholders in the EFI data migration
project:
Interview 1
Interview 2
Interview 3
Interview 4
Table 6 List of Meetings
24 | P a g e