12/28/2016

DeduplicationofMasterDataduringlargeSAPImplementationProjects|SAPBlogs

GetStarted

Solutions

Partner

Community

Support

About

Training

Community

Developer

Blogs

DeduplicationofMasterData
duringlargeSAP
ImplementationProjects
March7,2014

| 195Views |
KathiravanSubramaniam
morebythisauthor

SAPDataServices

share
0

share
0

tweet

share
14

Follow

DeduplicationofMasterDataduring
largeSAPImplementationProjects

https://blogs.sap.com/2014/03/07/deduplicationofmasterdataduringlargesapimplementationprojects/

1/12

12/28/2016

DeduplicationofMasterDataduringlargeSAPImplementationProjects|SAPBlogs

Abstract
Introduction
Overviewofdeduplicationprocess
Advantagesofdeduplicationprocess
Executionofdeduplicationprocess
Initialcleansingofsourcedata
Sourcemasterdatacomparison
Rulesusedindeduplicationlogic
Masterdatadeduplicationtable
Nominationofleadingmasterdata
Newmasterdatacreationinthetargetsystem
Mappingofnonleadingmasterdatatoleadingmaster
data
Impactonthetransactionaldatamigration
Projectteamstructureforthededuplicationprojects
Pitfallsandthemitigationplanindeduplicationprojects
Conclusion

Abstract
DuringlargeSAPImplementationproject,whichhappensby
consolidatingoneormoreSAPandnonSAPsystems,thereisahigh
possibilityofsamematerialmasterandvendormastertoreappearin
thetargetsystemwithdifferentnamesanddetails.Incorrectmaster
dataleadstovariousissueslikeincorrectreporting.Thiswhitepaper
addressestheissueofduplicaterecordsandprovidessolutionon
howtoeliminatethem.Thiswhitepaperdelvesintotheadvantagesof
deduplication,explainstheprocessstepstoexecutededuplication
alongwithinformationonfieldswhichshouldbeusedtopickthe
duplicaterecords,suggeststhebestteamstructuretomanagede
duplicationprojectsandprovidesaguideexplainingthecommon

https://blogs.sap.com/2014/03/07/deduplicationofmasterdataduringlargesapimplementationprojects/

2/12

12/28/2016

DeduplicationofMasterDataduringlargeSAPImplementationProjects|SAPBlogs

pitfallsandthemitigationplanswhileexecutingdeduplication
projects.

Introduction
Performingmasterdatacleansingatsourcesystems(SAPandnon
SAPsystems)andattheintermediarystagepriortofinaldataupload
intothetargetsystemisamajoractivityinSAPimplementationand
rolloutprojects.Thisisakeyactivitywhileconsolidatingoneormore
ERPsystems.Theprimaryactivityinmasterdatacleansingisde
duplicationofmasterdata.Deduplicationofmasterdatareferstothe
processoffindingtheidenticalmasterdatawithinoracrossthe
sourcesystem(s)andeliminatingthembeforemigratingtothetarget
system.Masterdataheretypicallyreferstomaterialmasterdataand
vendormasterdata.
Thereareseveraltoolsusedindeduplicationprojects.The
capabilitiesofsuchtoolsarenotdiscussedinthiswhitepaper.
Theaimofthiswhitepaperistoprovidekeyinformationofthede
duplicationprocesswhichcouldbefollowedinSAProlloutand
implementationprojects.

Overviewofdeduplicationprocess
Thefollowingpicturedepictsthededuplicationprocess

Figure1:Overviewofdeduplicationprocess
Thededuplicationprocesscomprisesfollowingsteps:
https://blogs.sap.com/2014/03/07/deduplicationofmasterdataduringlargesapimplementationprojects/

3/12

12/28/2016

DeduplicationofMasterDataduringlargeSAPImplementationProjects|SAPBlogs

1.Initialcleansingofsourcedata
2.Sourcedatacomparisonbasedondeduplicationlogicspecific
tomasterdataandpreparationofdeduplicationreports
3.Nominationofleadingmasterdata
4.Mappingofnonleadingmasterdatatoleadingmasterdata
5.Newmasterdatacreationinthetargetsystem

Advantagesofdeduplicationprocess
Theadvantagesofexecutingdeduplicationprocessduring
rollout/implementationprojectsarelistedasfollows:
1.Duplicatematerialandvendormasterrecordsleadstoincorrect
andinconsistentreporting
2.Incorrectconsumptionandavailabilityinformationofmaterial
masterleadstoinaccuratematerialplanning
3.Deduplicationprocessadheredduringthestartofthe
rollout/implementationprojectsreducesthetimeandcostspent
inmanuallyidentifyingduplicateslater
4.Deduplicationprocessimprovestheoverallreliabilityof
materialandvendorreports&analysis
5.Removingduplicatevendormasterrecordshelpstomaintain
effectiveandconsistentcommunicationwithvendors
6.Consolidated,consistent,harmonizedandcleansedmaster
dataareprerequisitesforinnovationandgrowth

Executionofdeduplicationprocess
Deduplicationprocessgetsexecutedasdescribedinfollowingsteps:

Initialcleansingofsourcedata
Thescopefordeduplicationofmasterdatacomprisesallthemaster
datainthesourcesystemsexceptthosewhichwouldfallunderoneor
morebelowcriteria:
a.Masterdatawhicharedeletedorblockedatthehighestlevelin
theorganisationalstructure.Howevermasterdatawhichareblocked
atoneofthelowerorganisationalstructurelevelmightstillbeactive
andrelevantatanotherorganisationallevel,hencethosedatashould
stillbeconsideredfordeduplicationexercise.
https://blogs.sap.com/2014/03/07/deduplicationofmasterdataduringlargesapimplementationprojects/

4/12

12/28/2016

DeduplicationofMasterDataduringlargeSAPImplementationProjects|SAPBlogs

b.Vendormasterdatawhicharenotcreatedatcompanycode
levelbutcreatedonlyatpurchasingorganisationallevelfordifferent
purposes
c.Masterdatawhichcannotbemigratedtotargetsystemdueto
nonavailabilityofmappingvaluesonimportantfieldslikeunitof
measureinmaterialmaster
Theinitialcleansingofsourcemasterdataisimportantsincethis
woulddramaticallyreducethenumberofduplicatedgroupofmaster
records.Initialcleansingwouldincludebothenrichmentofkeymaster
dataandperformingcorrectionsinthemasterdata.Predominantly,
initialcleansingwillbeperformedonthedatawithinthesource
systemonly.
Fewexamplesofinitialcleansingofsourcemasterdataareas
follows:
a.Correctthematerialmasterwithdummytext
b.CorrectmaterialmasterwithmaterialtypeHERSduplicated
withthematerialtypeHIBEtowhichtheHERSmaterialmasteris
assignedto
c.Updatekeydetailsusedindeduplicationlogicwhichare
missinginthematerialandvendormasterdata
d.Checkandcorrectredundantpartnerfunctionscreatedin
vendormaster
Importantfieldswhicharefocusedduringinitialcleansingor
enrichmentwithinsourcesysteminmaterialmasterandvendor
masterareasfollows:
MaterialMaster:
a.Materialdescription
b.Unitofmeasure
c.Manufacturerdetails
d.UNSPSCcode
e.Vendorpartnumber
VendorMaster:
a.Vendorname

https://blogs.sap.com/2014/03/07/deduplicationofmasterdataduringlargesapimplementationprojects/

5/12

12/28/2016

DeduplicationofMasterDataduringlargeSAPImplementationProjects|SAPBlogs

b.Address
c.VATregistrationnumber
d.DUNS
e.Bankaccountnumber

Sourcemasterdatacomparison
Thisstepisthecoreofdeduplicationprocess.Afterthedatais
cleansedandenriched,itwouldbecomparedagainsteachother.The
resultofthiscomparisonwouldbethegroupingofsimilarmaster
data.ThesourcesystemforthemasterdatacouldbesingleERP
systemormultipleERPsystems.Therearetoolsavailablefor
comparingthemasterdataandcreatingthegroupsofsimilarmaster
data.
Thededuplicationtoolappliesthededuplicationlogicinorderto
identifysimilarmasterdataanddevelopsthemasterdatade
duplicationtable.

Rulesusedindeduplicationlogic
Thecriteriausedtodeterminethegroupofsimilarmasterdatawould
dependonmanyfactorsliketheavailabilityofdata,levelofinitial
cleansingdone,scopeofenrichmentperformedwithinsourcedata,
etc.,
Someoftherulesusedinmaterialmasterdeduplicationlogicinorder
toidentifythesimilargroupofmaterialmastersareasfollows:
a.Samemanufacturerandsamebaseunitofmeasure
b.SameUNSPSCcode
c.Samemanufacturerandsamevendorpartnumber
d.Similardescription
Predominantlythedetailswouldbeconcatenatedandtext
comparisonsareperformedinordertoarriveatthesimilarmaster
datagroups.
Likewise,someoftherulesusedinvendormasterdeduplicationlogic
wouldbeasfollows:
a.SameDUNSnumber
https://blogs.sap.com/2014/03/07/deduplicationofmasterdataduringlargesapimplementationprojects/

6/12

12/28/2016

DeduplicationofMasterDataduringlargeSAPImplementationProjects|SAPBlogs

b.Samebankdetails
c.Sametaxcode
d.Sameaddressdetailslikename,streetnumber,PObox,Postal
code,etc.,

Masterdatadeduplicationtable
Masterdatadeduplicationtableistheresultofinitialdatacleansing
activityandtheapplicationofdeduplicationlogicontheprecleansed
sourcedata.Deduplicationtoolhasthecapabilityinordertoidentify
groupofmasterrecordswithinsingleERPsystemoracrossmultiple
ERPsystemsbasedonthededuplicationlogic.
Simpleexampleofmasterdatadeduplicationtablewouldappearas
follows:
Table1MasterDataDeDuplicationTable
Group
number

Source
System

Material

Name
SAP

Material
Description

Manufacturer

MaterialA

BoltHydraulic

Mnfr1

MaterialB

Bolt

Mnfr1

MaterialC

Bolt,longOil

Mnfr1

System1
1

SAP
System1

NonSAP
system2

hydr

Intheaboveexamplethededuplicationlogichasworkedonthe
sourcesystemdataandgroupedthesethreematerialswhichareof
similarnature.

Nominationofleadingmasterdata
Oncethegroupingofsimilarmasterdatahasbeendone,thereis
needtoselectthematerialwhichshouldgetmigratedintothetarget
system.
Table2MasterDataDeDuplicationTableAppendedwith
NominationColumns

https://blogs.sap.com/2014/03/07/deduplicationofmasterdataduringlargesapimplementationprojects/

7/12

12/28/2016

DeduplicationofMasterDataduringlargeSAPImplementationProjects|SAPBlogs
Group

Source

number

System

Material

Material

Manufacturer

Description

part

Name
1

SAP

Vendor

number
MaterialA

BoltHydraulic

Mnfr1

9N4524

MaterialB

Bolt

Mnfr1

9N4524

MaterialC

Bolt,longOil

Mnfr1

9N4524

System1
1

SAP
System1

NonSAP
system2

hydr

IntheaboveexampleifMaterialAisidentifiedastheleadingmaterial
whichshouldbemigratedintothetargetsystemandtheothertwo,
MaterialBandMaterialCareidentifiedasnonleadingmaterials
andtheyareduplicates.Thenonleadingmaterialswhicharethe
duplicateswillnotbemigratedtothetargetsystem.Leadingmaterial
isalsoreferredasparentmaterialandnonleadingmaterialisalso
referredaschildmaterial.
Intheaboveexample,refertothecolumnVendorpartnumber.The
samepartnumberprovidedbythesamemanufacturerwascreatedin
twodifferentsystemsinthreedifferentways.Hencetextsearchlogic
likenormalizationofthetext(removingthespecialcharactersto
determinetheactualtext)shouldbeimplementedtodeterminethe
duplicates.
Theselectionofleading/nonleadingmaterialisamanualactivity
whichshouldbeguidedbyfewprinciplesasfollows:
Ifagroupcontainsmasterdatafromtwodifferentsystems,thenthere
isconflictofwhichsystemspecificmasterdataisgivenfirst
preferencetobeselectedasleadingmaterial.Herenormallythe
thumbruleistohavetheoldestsystemmasterdatawhichhasthe
updatedinformationtogetthefirstpreference.Theotherapproach
wouldbetoselectthemasterdatawhichhasmosttransactionaldata.
Thisissuebecomescomplex(duringtheselectionoftheleading
material)whenthedifferentsystemsareownedbydifferentinternal
organizations.Normallythededuplicationprocessshouldbecarried
outcentrallywithcentralcocoordinatortomitigateconflictsarisingout
ofselectingtheleadingmaterialinthegroups.

Newmasterdatacreationinthetargetsystem

https://blogs.sap.com/2014/03/07/deduplicationofmasterdataduringlargesapimplementationprojects/

8/12

12/28/2016

DeduplicationofMasterDataduringlargeSAPImplementationProjects|SAPBlogs

Duringthisprocessstepweshouldhavearrivedwithallthemanual
nominationsofidentifyingtheleadingandnonleadingmasterdata.
Thiswouldenableustosegregatetheleadingmaterialwhichwillget
migratedtothetargetsystem.Theleadingmaterialwhichwouldget
migratedtothetargetsystemwouldhavenewmaterialnumberinthe
targetsystemasperthetargetsystemmaterialnumber
nomenclature.

Mappingofnonleadingmasterdatatoleading
masterdata
Asafinalprocessstepwithinthededuplicationprocess,after
receivingthenewmaterialnumberinthetargetsystem,weshould
arriveatthemappingofleadingandnonleadingmasterdatanumber
(oldsourcesystemmaterialnumber)totheleadingmasterdata
number(newmaterialnumberinthetargetsystem)whichwould
appearasperaboveexampleinsection3.3asfollows:
Table3MasterDataMappingTable

Oldsourcesystemmaterial
number

Newmaterialnumberinthetargetsystem

MaterialA

NewMaterialA

MaterialB

NewMaterialA

MaterialC

NewMaterialA

TheOldsourcesystemmaterialnumbercontainsboththeleading
(parent)andthenonleading(child)masterdatanumber.
Newmaterialnumberinthetargetsystemcontainsthematerial
numberwhichiscreatedinthetargetsystem.
Thenonleading(child)materialandvendorinheritstheleading
(parent)materialandvendormasterdata.Certaindatalikebank
detailsofchildvendorwillbeconsolidatedtotheparentvendor.Child
vendorwillinheritparentvendorgeneraldata.
Allnonleading(child)vendorscompanycode/purchasingorg/plant
willbeextendedtotheparentvendorinthetargetsystem.

Impactonthetransactionaldatamigration
https://blogs.sap.com/2014/03/07/deduplicationofmasterdataduringlargesapimplementationprojects/

9/12

12/28/2016

DeduplicationofMasterDataduringlargeSAPImplementationProjects|SAPBlogs

Thenonleadingmasterdatawillnotbemigratedtothetarget
system.Thetransactionaldataofthenonleadingmasterdatawould
becreatedusingtheequivalentmappedleadingmasterdata.

Projectteamstructureforthede
duplicationprojects
Deduplicationprojectinvolveslotofcoordinationbetweendifferent
ownersofthesourcesystems.Usuallyinprojectsstaggeredacross
geographies,therewouldbeseparateteamresponsibleforeach
companycoderelevantmasterdata.Inallsuchscenarios,there
shouldbededupcoordinatorineachlocationwhoshouldliaisewith
otherdedupcoordinatorsandthecentraldedupcoordinator.The
masterdataorganisationshouldprovidethehighlevelgovernance.It
isbeneficialtopositioncentraldeduplicationcoordinatorcentrally
acrossgeographies.
Table4ProjectTeamStructure

Roles
Centraldeduplicationco
ordinator

Majorresponsibilities
1.Providetechnicalguidancefordoingtheleading/nonleading
masterdatanominations
2.Coordinateleading/nonleadingmasterdatanominations
3.Issueandscopemanagement
4.Leadershipactivities
5.Arrangerecurringmeetingstotrackprogress
6.Solveconflicts

Deduplicationcoordinatorin

1.Performtheleading/nonleadingmasterdatanominations

everycompanycode/

2.Participateinrecurringmeetings

system/geography
Masterdataorganisation

3.Ensuredataqualityinthesourcesystem

1.Governanceonmasterdata
2.Provideclarificationonthemasterdatadesign
3.Reviewdataqualityofsourcesystemandimplementrequired
structuralchanges

Pitfallsandthemitigationplaninde
duplicationprojects
https://blogs.sap.com/2014/03/07/deduplicationofmasterdataduringlargesapimplementationprojects/

10/12

12/28/2016

DeduplicationofMasterDataduringlargeSAPImplementationProjects|SAPBlogs

Thecommonpitfallsandthemitigationplansindeduplication
projectsareasfollows:
Table5PitfallsandMitigationPlans

Pitfalls

Mitigationplan

Duringinitialreviewstages,thereislikely

Asageneralguideline,theresourceestimateshouldbe

underestimationofresourcesneededto

basedon100masterdataaweekperperson.Thisisbased

reviewtheitemsandperformthe

onauthorsexperienceindeduplicationprojectandthisis

nominationforleading/nonleading

withcompleteanalysisincludinginvestigationofpurchasing

masterdata

history

Iftherearemultiplesystemowners,time

Theroleofcentralcoordinatorandtheauthorityshouldbe

takentoreachconsensusonnominating

moresothatconflictscouldbesettledeasily

theleadingmasterdatawashuge
Incorrectnominationsleadsto

Resourcesinvolvedindeduplicationprojectshouldhave

complexities,whenthegroupcontains

detailedknowledgeonmasterdataanddeduplication

masterdataacrosssystems

process

Incompletenominationsleadsto

Trackingmechanismtodeterminewhichmasterdata

complexities,,whenthegroupcontains

nominationispendingwithwhichteam

masterdataacrosssystems
Highrisk(or)highvalueitems

Highriskand(or)highvalueitemsshouldbeapproached
withcaution

Conclusion
Thiswhitepaperdiscussesthededuplicationprocessduringtheinitial
stagesofrollout/implementationprojects.Howeveroncetheparent
masterdataisidentifiedandthenewmasterdataarecreatedafter
eliminatingthechildduplicates,itisimperativetohavedefined
approachtoavoidduplicatesfurtherinthetargetsystem.Therecould
besinglesourceformasterdatacreationandchangesalongwith
effectiverules&processestopreventduplicatesatsource.

AlertModerator

1Comment
https://blogs.sap.com/2014/03/07/deduplicationofmasterdataduringlargesapimplementationprojects/

11/12