Matthew Lawler lawlermj1@gmail.

com DW Design and Data Model Example

DW Design and Data

Model Example

Matthew Lawler [email protected]

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
1 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

INTRODUCTION 3

CONTEXT 4

HIGH LEVEL DESIGN 8

SOURCE DESIGN 11

SAL DESIGN 22

CAL DESIGN 28

EUL DESIGN 32

OBIEE DESIGN 39

TESTING 40

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
2 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Introduction

Licence
As this is a generic software document, this will be covered by the 'Creative Commons Zero v1.0
Universal' CC0 licence.

Warranty
The author does not make any warranty, express or implied, that any statements in this document
are free of error, or are consistent with particular standard of merchantability, or they will meet the
requirements for any particular application or environment. They should not be relied on for solving
a problem whose incorrect solution could result in injury or loss of property. If you do use this
material in such a manner, it is at your own risk. The author disclaims all liability for direct or
consequential damage resulting from its use.

Purpose
The primary goal of this document is to provide an example of DW design with data models.

Audience
The primary audience of this document are any DW designers or data modellers. It is assumed that
the reader is familiar with DW and Kimball standards.

Assumptions
This is only an example, so any attempt to use this for other cases is limited.

Approach
The document is a normal design document. It begins with context and definitions. It then provides
high level requirements traceability, scope, high level design and environments. Then it examines
the source data model and design in details, as imposes constraints on the subsequent Kimball
reporting data model. Source data quality was an issue, so it is discussed at length. Another issue
was date casting. After that the SAL or landing zone design is covered, where most of the data
quality issues were resolved. Then the CAL or Inmon zone design is covered. Finally, the EUL or
reporting zone design is discussed. The EUL data model is a standard Kimball fact dimensional
model. The end user tool here was OBIEE, so its design and impact on the EUL is also covered.
Finally, there is a discussion of testing design.

Documents
Normally there would be a list of documents, covering requirements, other design documents,
mappings spreadsheets, visio data models, and resultant artefacts such as DDL, SQL, etc.

Tags
Business Intelligence ; Data Design ; Data Load ; Data Mapping ; Data Model ; Data Transformation ;
Data Vault ; Data Warehouse ; Database ; Database Design ; Design Pattern ; DW Appliance ; Extract
Load Transform - ELT ; Extract Transform Load - ETL ; Fact / Dimension ; Hierarchy ; Inmon ; Kimball ;
Massive Parallel Processing - MPP ; Master Data Management ; Metadata ; Netezza ; Oracle ; SQL ;
Standards ; Teradata ; Data Architect ; Data Architecture ;

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
3 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Context
Source System Context Diagram
This shows the flow of data and major components in the SCEM/BAM source system.

12. BAM Dashboard

Access

Supply
Co-ordination
Centre

11. BAM Dashboards

BAM Dashboard 14. Status Remedy

10. Events, Exceptions & Alerts

16. Tickets
13. Tickets
9. Alert Notifications Supply Chain Event
Management

15. Trend Reporting UDS

7. Supplier OBIEE
8. External Events
Reports Data Mart

Supplier
SOW 1. Oracle Inventory 5.Oracle
Events 3. Oracle AP 4. Returns OM Events
File 2. Oracle Purchasing
Events
Events Events

Account Order
Inventory Purchasing Quality Management
Payables

Oracle E-Business Suite

6. 3PL Upload

3PL

Terms
This is meant to highlight the key terms that are used in the BAM system.

Term Definition
Alert This represents the lowest level of violation. It is displayed as either ‘Medium’ severity
or as an ‘Amber’ Traffic Light. An alert will not raise a Remedy ticket. An Alert can exist
without an Exception, but not vice versa. The business prefers to use the term Alert as
the common or abstract term to cover both Alert and Exception. This will be
implemented in the data model. It is assumed that a green or unalerted event is simply
on without any alert.
BAM BAM is a module of the SCEM package. For the sake of simplicity, the source system
will be referred to as BAM within this document, as this is the actual data source.
EMEM The application name is EMEM or Event Monitoring and Exception Management. This

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
4 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

is based on the Oracle's SCEM package.

Event There are 5 event types. 4 are created from BAM, and are called EP2P, IP2P, MO and
MR. A 5th type is called MANUAL, and this is created in Remedy, but does not exist in
BAM, so will not have any BAM data.
Exception This represents the highest level of violation. It is displayed as either ‘High’ severity or
as a ‘Red’ Traffic Light. An exception may raise a Remedy ticket, provided ticket raising
has been configured for the particular violation type.
Tolerance These are defined in the EBS QA schema. Currently, these are based only 2 levels. The
Business would also like to define based on Item Number, Suppler, Demand Plan
Category, etc.
Violation This term means the logic that is used to trigger an Alert/Exception. It is also used by
the SI as an abstraction for both Alert and Exception.
Violation There are a series of 42 BAM violation types, along with 3 MANUAL violation types.
Type

Acronyms
These are standard acronyms in the BAM system. Some are Supply acronyms, and some are IT
acronyms. Many are used in column names.

Acronym Type Expansion

BO Alert Back Order
CI Alert Critical Item
DIS Alert Disposal
DV Alert Delivery Violation
GOR Alert Goods Over Receipted
GUR Alert Goods Under Receipted
IQ Alert Insufficient Quantity
LD Alert Late Dispatch
LEFS Alert Late ERMA from Supplier
LGS Alert Late Goods Shipment (Supplier > Delivery Partner)
LMD Alert Late Material Delivery
LMR Alert Late Material Request
LMRS Alert Late Material Request Supplier
LP Alert Late Pick
LPD Alert Late PO Detail
LR Alert Late Replacement
LS Alert Low Zero Safety Stock
LT Alert Late Triage
MH Alert Material Header
MOQ Alert Minimum Order Quantity
NBD Alert Need By Date
PDC Alert Promise Date Changes
PDE Alert Promise Date Exception
PLT Alert Purchasing Lead Time
PN Alert Part Number
POA Alert Purchase Order Approve
POD Alert PO Delivery
PRA Alert PR Approval
RC Alert Return Completion
RI Alert Response Indicator
D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
5 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Acronym Type Expansion

SLA Alert Service Level Agreement
SLT Alert Supplier Lead Time
SUP Alert Supplier Under Paid
DP Event Demand Plan
ISO Event Internal Sales Order
MO Event Move Order
PO Event Purchase Order
POH Event Purchase Order Header
POL Event Purchase Order Line
PR Event Purchase Requisition
PRH Event Purchase Requisition Header
PRL Event Purchase Requisition Line
RM Event Return Material
RMA Event Return Material Authorization
SO Event Sales Order
SOH Event Sales Order Header
SOL Event Sales Order Line
DDL IT Data Definition Language
DO IT Data Object
EDO IT External Data Object
ELT IT Extract, Load and Transform
ERD IT Entity Relationship Diagram
ETL IT Extract, Transform and Load
LTO IT Logical Transaction Object
RI IT Referential Integrity
RPD IT RePository Database (for OBIEE)
SCD1 IT Slowly Changing Dimension Type 1
SCD2 IT Slowly Changing Dimension Type 2
SI IT Systems Integrator (Accenture)
MO Measure Minimum Order
QOH Measure Quantity On Hand
UOM Measure Unit Of Measure
SCC Organisation Supply Chain Council
EIS Process Enterprise Inventory and Service-level Optimization
EMEM Process Event Monitoring and Exception Management
EP2P Process External Procure to Pay
ERMA Process External Return Material Authorization
IP2P Process Internal Procure to Pay
IRMA Process Internal Return Material Authorization
MI Process Material Issue
MM Process Materials Management
MR Process Material Request
MSCEM Process Materials Supply Chain Event Management
MT Process Material Transfer
P2P Process Procure to Pay
SCEM Process Supply Chain Event Management
SCM Process Supply Chain Management
BAM System Business Activity Monitor
EBS System E-Business Suite
EUL System End User Layer

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
6 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Acronym Type Expansion

OBIEE System Oracle Business Intelligence Enterprise Edition
SAL System Source Aligned Layer
UDS System Unified Data Store

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
7 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

High Level Design

Technical Requirements
These requirements are provided here, as they were not in the documents provided by the end
users. They are incorporated into the final requirements document.

RKey Type Sub Type Description

B1 Business See Business requires reporting over BAM and a small subset of
Requirements Remedy and EBS Demand Plan tables.
B2 Business Time Business requires trend reporting over time.
T1 Technical Low Build Cost The solution needs to be provided at the lowest possible
build cost.
T2 Technical Low The solution needs to be provided at the lowest possible
Maintenance ongoing cost.
Cost
T3 Technical Performance There needs to be adequate performance.
T4 Technical Security Only authorised staff can have access.
T5 Technical Tested All deliverables need to be fully tested.
T6 Technical Transparency The reporting user will need to understand where the data is
sourced from.
T7 Technical User control The reporting user will need as much control and autonomy
over their data as possible. The ability to change the look and
feel of reports is also important.

Technical Requirements Traceability

This matrix does not replace the business requirements matrix. Instead, it supplements it, as it
traces key design decisions back to the technical requirements above. It also nominates which layer
supports the requirement.

O Design Decision RKey Layer

1 A fact dimension model will support all of the requirements. B1 EUL
2 EUL will provide joins over SAL from BAM, and a small subset of Remedy and B1 EUL
EBS tables.
3 Time grouping over months and quarters will be available by joining to the EUL B2 EUL
D_DATE time dimension in views.
4 EUL will provide joins over SAL/CAL Base tables. There is no need for ETL from T1, EUL
SAL or CAL to EUL. T2
5 OBIEE will interface directly to the EUL views. T1, OBIEE
T2
6 Views will be preferred to using ETL, as it is lower cost to build and maintain. T1, EUL
T2
7 While the all SAL tables are currently randomly distributed, they all have T3 SAL
identified distribution keys.
8 Performance testing will be a step after build. If performance is found to be T3 EUL
inadequate, then selected tables may be redefined using distribution keys, or
view materialisation may be used.
9 Security will rely on OBIEE's inbuilt capability. All access to EUL views will be T4 OBIEE
through OBIEE only.
10 All layers will be tested. Test team engagement will begin in parallel with Build. T5 All

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
8 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

O Design Decision RKey Layer

11 Automated testing will be completed before external testing. T5 All
12 All SAL and CAL tables DDL will be generated from the source tables DDL. T6 SAL
13 Column view names will be based on BAM column physical names, converted T6 EUL
into Title Case. E.g. ACCOUNT_ID -> 'Account Id'.
14 The SAL to EUL mappings will be embedded in the views. These will also be T6 EUL
published for documentation.
15 Each view has the same granularity as the underlying SAL table. There will be T6, EUL
no need to aggregate EUL with GROUP BY views. T7
16 End users are familiar with BI tools, especially OBIEE. Therefore, the primary BI T7 OBIEE
tool will be OBIEE.
17 OBIEE is optimised for Kimball style reporting, so EUL will provide a fact B1, OBIEE
dimensional view over the SAL tables. T7
18 OBIEE to provide some graphics and some slicing and dicing capability. This will T7 OBIEE
ensure that users will retain control over the look and feel of reports.
19 OBIEE will be used to perform filtering and aggregations. This will ensure that T7 OBIEE
users will retain control over the reports, and can trace back to source when
alerts occur.

Scope
Element Item
Deletes The BAM data is very ephemeral, and is deleted when the alerts are resolved. The
users need to retain data which is a central need for the UDS load.
Granularity There is no need for summing by event attribute quantities. For example, by PO
Quantity. Summations will be based on event types, alert types only.
Frequency Daily loads are sufficient. There is a weekly reporting cycle.
Join All reporting will be supported from BAM + Remedy data, joined by the BAM Remedy
ticket # and Remedy ticket #.
OOS Oracle Quality Schema as a source. Instead, there is some Tolerance data already
within BAM, and this will made available to the users.
OOS SCEM as a source. This was formerly the UDS source, but it was decided to use BAM as
the UDS source instead, as BAM data was in table form, whereas SCEM uses a more
complex internal structure.
Source BAM for events and alerts
Source EBS for Demand Planning Category
Source Remedy for Ticket info

Scope: Drops
The project was implemented using an agile approach with multiple drops.

Drop Table
1 BIA_BA_EUL.D_BAM_ALERT_TYPE
1 BIA_BA_EUL.D_BAM_EVENT_DATA_EP2P
1 BIA_BA_EUL.D_BAM_EVENT_DATA_IP2P
1 BIA_BA_EUL.D_BAM_EVENT_DATA_MO
1 BIA_BA_EUL.D_BAM_EVENT_DATA_MR
1 BIA_BA_EUL.F_BAM_ALERT
1.5 BIA_BA_EUL.D_BAM_EVENT_PARTY_CUST
1.5 BIA_BA_EUL.D_BAM_EVENT_PARTY_SUPPLIER
D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
9 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Drop Table
1.5 BIA_BA_EUL.D_BAM_EVENT_TYPE
2.1 BIA_BA_EUL.D_BAM_DP_CATEGORY
2.1 BIA_BA_EUL.D_BAM_EVENT_DATA_REMEDY
2.1 BIA_BA_EUL.D_BAM_REMEDY_TICKET

High Level Design on a page

Environments
Database Env Type Purpose Database Schema
Oracle Test Source EBS04EE SOA_ORABAM
Oracle Test UDS Load EBS04EE EIM_IS_BAM
Oracle Prod Source EBS01PR SOA_ORABAM
Oracle Prod UDS Load EBS01PR EIM_IS_BAM
Netezza Dev SAL BIA01SB BIA_SA_SCEM
Netezza Dev EUL BIA01SB BIA_BA_EUL
Netezza Test SAL BIA02SB BIA_SA_SCEM
Netezza Test EUL BIA02SB BIA_BA_EUL
Netezza Prod SAL BIA01IT BIA_SA_SCEM
Netezza Prod EUL BIA01IT BIA_BA_EUL

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
10 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Source Design
All the BAM tables come from the EBS02PR database, and the schema SOA_ORABAM.

BAM Data Design Pattern

The data design pattern uses 5 kinds of tables for each Event Type; Alert Type, Event Snapshot,
EventAlertRemedyLink, Alert Snapshot and Current Summary. The yellow tables - Alert Type, Event
Snapshot and EventAlertRemedyLink - will be used to construct the EUL fact dimensional model.
The blue tables - Alert Snapshot and Current Summary – will not be used for EUL model, but will be
retained in the SAL for reference.

Table Type Definition

Alert Type This is a common table used across all event types.
EventAlertRemedyLink This is a table that links the Event to the Alert and to the Remedy ticket
table. It contains a row for each alert and exception. So a violation type
can trigger an alert and an exception for the same event type data. This is
shown as 2 rows in this table. The alert code does not have an ‘R’ suffix,
while the exception uses an ‘R’ suffix. E.g. SCC6 versus SCC6R.
Event Snapshot This is populated at regular intervals with the source data. The data is
often just replicated, as it does not change much from day to day. The best
key is the timestamp column, along with the event natural key. Event
tables are at a specific cardinality level, based on a specific set of EBS keys.
For example, the PO and PO line. Event tables have a number of Foreign
Key columns that link back to related EBS tables. Primarily, the snapshot
tables provide a primitive time variance.
Alert Snapshot This is also populated at regular intervals with BAM based Alert data. Most
of the data is also replicated daily, as the alert status does not change every
day. The best key is a timestamp column, along with the natural key, and
the alert code. This table also pivots out the different violations into traffic
light columns for display. This data just replicated the data in the
normalised EventAlertRemedyLink table, so it is not needed.
Current Summary This table combines the current values of events and alerts. There is some
column replication or overlap between the snapshot tables and the
summary table. However, the snapshot tables do have additional columns
that the summary table does not, as well as time variance.

BAM Data Quality

This source had extreme data quality issues, which had to be addressed in the SAL layer.

Issue Description Major?

Data Type Data lengths and data types in some cases are changed from the EBS Yes
source into the BAM target. This adds an additional level of
complexity. It was hoped that data types could be used to split
dimensions, but this is not possible. Some of the BAM data lengths
are much larger than the EBS source, which may impact the design.
Empty EP2P All of External (EP2P) or PO data is missing. Therefore, BIA has been Yes
Tables unable to verify that relationships are correct. The Artermis project
D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
11 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Issue Description Major?

will eventually replace this process. Nonetheless, the users still want
to report on this process, so best efforts will be made to determine
the keys for reporting. Some data is now available.
Empty Remedy All of the Remedy columns are also not populated as the Remedy Yes
Tables interface has not yet been turned on. Also, there is no Remedy
interface in development, so there was no data there either. The
Remedy interface was recently switched on, and then switched off, as
the categories were incorrect. Some data is now available.
Empty Summary All of Summary (EDO) tables are empty. These are event and alert Yes
Tables summary tables that are used for the views.
Referential BAM does not seem to be apply referential integrity consistently. This Yes
integrity was more so in Development, but still occurred in Production.
Development has the same data structure as Production, but the data
is different. It mainly applied to the snapshot tables. This issue will
be resolved by ELT or post load cleansing. However, it will persist in
the source system, causing inconsistent reporting.
MO Alerts For MO events, there is a significant mismatch between events and No
without Events the alerts. That is, there are many alerts that have no matching
events, and vice versa. For other event types (IP2P, EP2P and MR),
these mismatches do not occur. Fortunately, when the current
production data was examined, the invalid event rows have been
removed, so this will not be an ongoing issue.
Boilerplate The surrogate BAM keys are all called SysIterId. These columns have a No
value range in low numbers, starting from 1. These columns are
primary keys, but they are not used in any relationships. SysIterTrID
values are in high numbers starting from millions. The role of
SysIterTrID columns is unclear, but they do not appear to have any
business value. Another boilerplate column is _TRXKEY. When
populated this follows a GUID format. However, this column is not
populated often, and is not used in any relationships. So it also
cannot be used. The full list of boilerplate columns are: SysIterTrID,
SysIterID, SysIterTotalsChild, SysIterOwnerID, SysIterName,
SysIterLastModifiedBy, SysIterLastModified, SysIterCreated,
SysIterACL. The BAM boilerplate columns are unusable for reporting.
Denormalisation The structure of BAM is already highly denormalised, as it is a No
reporting application. The data on the snapshot tables all have the
same cardinality, as they are just flattened copies of EBS data. This
means that they cannot easily be used to split out the columns into
groups representing the original EBS tables.
Lack of There was very little BAM system documentation. This amounted to a No
Documentation couple of emails. The original table set provided by Accenture was
only 16 tables, and 5 views. Profiling revealed that this was
incomplete, so 22 more were identified, for a total of 38 tables. This
lack of documentation and many empty tables mean that BIA can only
provide reporting artefacts on a best efforts basis.
Naming The design pattern of the BAM tables is consistent, but the naming No
Standards standards are not, making it hard to understand. For example, the
terms exception, violation and alert were used interchangeably in the
data design. This may have been caused by multiple designers. See
the Data Design Pattern example. See below a typical example. In

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
12 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Issue Description Major?

this case, the parent table _InternalP2P_DO has a key called _PR_Line.
However, in the child tables, this is called either _PRLLINENUM or
_Line_Num, which is not self-evident. This had to be discovered
manually and then verified by profiling.
Relationships A considerable amount of time was spent in discovering and profiling No
the relationships between the source tables. No joins in the BAM
views, so these were not helpful in relation discovery. BIA is confident
that the correct relationships have been found.

Column Name Example

Parent Table Parent Column Child Table Child Column
_InternalP2P_DO _PR_Line _EIS_AlertResponse _PRLLINENUM
_InternalP2P_DO _PR_Line _EIS_OrderManagementResponse _PRLLINENUM
_InternalP2P_DO _PR_Line _InternalP2P_Alert_DO _Line_Num

EBS to BAM Event Type Mappings

EBS Event Type BAM Event Type
PO EP2P
PR IP2P
MO MO
SO MO
ISO MO
MR MR

BAM Source Table List

A few definitions were provided by the development team. The remained of the definitions were
created as a result of the profiling. EUL means that this table will be used in EUL views. RI means
that this table has referential integrity issues, as discussed in the Data Quality section above.

Table Table Row Colum Definition EU R

Type Coun n L I
t Count
EP2P _EP2P_EP2PLTO 0 94 This is the standard DO created as part of 1 0
Response BAM application. This DO holds all
events coming from BAW (Supplier
SOWs). This data object holds events as
they come into BAM. They are not
consolidated.
EP2P _Ext_P2P_Alert_ 0 24 This is the data object behind the Alert 1 0
List_DO2 Report (Dashboard - Procure to Pay
External) that users can view. It holds all
alerts generated against the incoming
events from BAW (Supplier SOWs).
EXC EMEM_REMEDY 68 14 Remedy Interface Table. A mapping 1 0
_MAPPING rulebook that defines the ALERT_CODE
and relates the EMEM_EXCEPTION_ID to
D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
13 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Table Table Row Colum Definition EU R

Type Coun n L I
t Count
a table and column.
EXC EMEM_TOLERA 42 22 Remedy Interface Table. Defines 1 0
NCE violation types aka
EMEM_EXCEPTION_ID.
IP2P _EIS_OrderMana 9352 81 This is the standard DO created as part of 1 1
gementRespons BAM application. This DO holds all
e events related to Internal Procure to Pay
coming from EBS. This data object holds
events as they come into BAM. They are
not consolidated.
IP2P _InternalP2P_Al 480 24 This is the data object behind the Alert 1 1
ert_DO Report (Dashboard - Procure to Pay
Internal) that users can view. It holds all
alerts generated against the incoming
events related to Internal Procure to Pay
coming from EBS.
MO _MSCEM_LTORe 4026 133 This is the standard DO created as part of 1 1
sponseBAM 4 BAM application. This DO holds all
events related to Material Management
coming from EBS. This data object holds
events as they come into BAM. They are
not consolidated.
MO MOMovement_ 2105 28 This is the data object behind the Alert 1 1
Alert_DO 7 Report (Dashboard \226\8364\8220
Material Transfer and Material Issue)
that users can view. It holds all alerts
generated against the incoming events
related to Material Management coming
from EBS.
MR _MRAlertRespon 80 29 This is the data object behind the Alert 1 1
seBAM_DO Report (Dashboard \226\8364\8220
Reverse Logistics) that users can view. It
holds all alerts generated against the
incoming events related to Reverse
Logistics coming from EBS.
MR _MRSCEM_MRL 236 177 This is the data object behind the main 1 1
TOResponseBA Report (Dashboard \226\8364\8220
M Reverse Logistics) that users can view. It
holds all events related to Reverse
Logistics coming from EBS consolidated
at the IRMA Number and IRMA line level.
EP2P _EP2P_EP2PAler 0 30 This is the standard DO created as part of 0 0
tResponse BAM application. It holds all alerts
generated against the incoming events
from BAW (Supplier SOWs).
EP2P _EP2P_GUR_Viol 0 15 Traffic Light for Purchase Orders. 0 0
ation_DO
EP2P _EP2P_LGS_Viol 0 15 Traffic Light for Purchase Orders. 0 0

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
14 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Table Table Row Colum Definition EU R

Type Coun n L I
t Count
ation_DO
EP2P _EP2P_LMD_Vio 0 15 Traffic Light for Purchase Orders. 0 0
lation_DO
EP2P _EP2P_LP_Violat 0 15 Traffic Light for Purchase Orders. 0 0
ion_DO
EP2P _EP2P_OrderList 0 89 This is the data object behind the main 0 0
Report (Dashboard - Procure to Pay
External) that users can view. It holds all
events coming from BAW (Supplier
SOWs) consolidated at the Customer PO
Number and Line number level.
EP2P _EP2P_PLT_Viol 0 15 Traffic Light for Purchase Orders. 0 0
ation
EP2P _EP2P_SLT_Viola 0 15 Traffic Light for Purchase Orders. 0 0
tion
IP2P _EIS_AlertRespo 472 88 This is the standard DO created as part of 0 1
nse BAM application. It holds all alerts
generated against the incoming events
related to Internal Procure to Pay coming
from EBS.
IP2P _InternalP2P_D 1244 76 This is the data object behind the main 0 1
O Report (Dashboard - Procure to Pay
Internal) that users can view. It holds all
events related to Internal Procure to Pay
coming from EBS consolidated at the
Purchase Requisition and Line number
level.
MO _MM_BO_VIOLA 444 16 Traffic Light for (Internal) Sales Orders or 0 0
TION_DO Move Orders.
MO _MM_CI_VIOLA 7 16 Traffic Light for (Internal) Sales Orders or 0 0
TION_DO Move Orders.
MO _MM_DV_VIOLA 9 16 Traffic Light for (Internal) Sales Orders or 0 0
TION_DO Move Orders.
MO _MM_IQ_VIOLA 2563 16 Traffic Light for (Internal) Sales Orders or 0 0
TION_DO Move Orders.
MO _MM_LD_VIOLA 5102 16 Traffic Light for (Internal) Sales Orders or 0 0
TION_DO Move Orders.
MO _MM_LP_VIOLA 5796 16 Traffic Light for (Internal) Sales Orders or 0 0
TION_DO Move Orders.
MO _MM_LS_VIOLA 667 16 Traffic Light for (Internal) Sales Orders or 0 0
TION_DO Move Orders.
MO _MM_MH_VIOL 138 16 Traffic Light for (Internal) Sales Orders or 0 0
ATION_DO Move Orders.
MO _MM_POD_VIOL 1375 16 Traffic Light for (Internal) Sales Orders or 0 0
ATION_DO Move Orders.
MO _MOSCEMEX_M 8586 159 This is the data object behind the main 0 0
OHLLTO_DO Report (Dashboard \226\8364\8220
Material Transfer and Material Issue)

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
15 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Table Table Row Colum Definition EU R

Type Coun n L I
t Count
that users can view. It holds all events
related to Material Management coming
from EBS consolidated at the Order and
Order line level. Order number could be
a Sales Order or an Internal Sales order
or Move Orders.
MO _MSCEM_AlertR 1974 32 This is the standard DO created as part of 0 1
esponseBAM 6 BAM application. It holds all alerts
generated against the incoming events
related to Material Management coming
from EBS.
MR _MRSCEM_MRA 176 31 This is the standard DO created as part of 0 1
lertResponse SCEM application. It holds all alerts
generated against the incoming events
related to Reverse Logistics coming from
EBS.
MR _MRSCEM_MRL 783 164 This is the standard DO created as part of 0 1
TOResponse SCEM application. This DO holds all
events related to Reverse Logistics
coming from EBS. This data object holds
events as they come into BAM. They are
not consolidated.
Summar _EP2P_EDO 0 16 EMEM Summary Table. 0 0
y
Summar _IP2P_EDO 0 16 EMEM Summary Table. 0 0
y
Summar _MI_EDO 0 16 EMEM Summary Table. 0 0
y
Summar _MT_EDO 0 16 EMEM Summary Table. 0 0
y
Summar _RM_EDO 0 16 EMEM Summary Table. 0 0
y
ZTotal 1609 0 0
Altogether the complete source has 38 tables, containing 1,609 columns. Many tables are empty.
All other tables in the source schema are not required for reporting. A tables and keys data model is
shown for completeness, but most tables will not be used.

10 tables will be used for EUL. These contain 626 columns. 6 of these tables have referential
integrity issues. 2 of these tables are empty, but are included as they are part of the scope.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
16 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

BAM Source Tables and Keys Data Model

Doc Title: BAM.EBS02PR.SOA_ORABAM Page Name: Data Model CAL Keys
EBS02PR
SOA_ORABAM
CAL Data Model
With Table and Key names

By Matthew Lawler

Last Edit: 12/04/2015 By: Matthew Lawler Filename: BAM.EBS02PR.SOA_ORABAM.vsd

The visio file has more detailed A3 versions of the data models, if required.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
17 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

BAM Source Tables and Columns Data Model (used)

Doc Title: BAM.EBS02PR.SOA_ORABAM Page Name: Data Model CAL Columns

EBS02PR
SOA_ORABAM
CAL Data Model
With Table and Column names

By Matthew Lawler

Last Edit: 12/04/2015 By: Matthew Lawler Filename: BAM.EBS02PR.SOA_ORABAM.vsd

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
18 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

BAM Source Tables and Keys Data Model (All)

Doc Title: BAM.EBS02PR.SOA_ORABAM Page Name: Data Model BAM All Keys

EBS02PR
BAM Subject Area
With Table and Key names only

By Matthew Lawler

Last Edit: 1/04/2015 By: Matthew Lawler Filename: BAM.EBS02PR.SOA_ORABAM.vsd

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
19 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Remedy
The Remedy tables will be the source for a Remedy ticket dimension. They will also be a source for a
Remedy alert fact that can be unioned with the other alert fact source tables. The Remedy alert fact
will be filtered based on Supply Tenancy. It also has an ‘EMEM’ string as a prefix of the name.

The source Remedy tables are:

Schema Table name Table Description

BIA_SA_RMDY_HIST RMDY_FIELD_ENUM_VALUES Remedy ENUM table. This is used to
decode code values.
BIA_SA_RMDY_HIST RMDY_T2115__BASE Remedy Ticket Base table. This is the main
table.
BIA_SA_RMDY_HIST RMDY_T2115__C_C1000000151 Remedy Ticket Description table. This
contains longer description fields.

DP Category
DP Category or Demand Plan Category is a 3 level hierarchy contained within an EBS Item Master
Flex field. This data is contained in a single source column, using a full stop as a delimiter. It is keyed
by Item Number, which will need to be on the fact table. The flex field data will be pivoted out into
4 columns, for Item Number, and the 3 DP Category levels. The correct organisation level for the flex
field is ‘MASTER’.

The correct source for this was the tables EBS_MTL_ITEM_CATEGORIES, EBS_MTL_CATEGORIES_B
and EBS_MTL_SYSTEM_ITEMS_B. This could not be determined until we had received snapshots of
the actual screens used to define this code. Once this was available, the internal EBS design was
discoverable. See the appendix for these snapshots.

Schema Table Description

BIA_SA_EBS_HIST EBS_MTL_CATEGORIES_B EBS categories table. All the values used in
categories.
BIA_SA_EBS_HIST EBS_MTL_SYSTEM_ITEMS_B EBS Inventory Item table. All items are in this
table.
BIA_SA_EBS_HIST EBS_MTL_ITEM_CATEGORIES Many to many join between categories and
items.

Tolerance
This tolerance data is available in the Alert type dimension, sourced from BAM directly.

Date Casting
The sources used different ways to represent dates. BAM had taken actual dates from EBS and cast
them as strings into VARCHARs, so these needed to be transformed back to be usable by reporting
users. Remedy used UNIX time or Epoch to save dates. These cast functions were applied where
ever needed. 52 BAM columns and 25 Remedy columns were cast. No EBS date columns were
required in the solution. Note that the pattern handles null values as well. This works in Netezza.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
20 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Date Source Cast Type

Type
Char BAM COALESCE( CASE WHEN SQLEXT.REGEXP_LIKE ( SUBSTR (
YYYY- DISPOSALTRANSACTIONDATE ,1,10), '^(19|20)\d\d[- /.](0[1-9]|1[012])[-
MM-DD /.](0[1-9]|[12][0-9]|3[01])$') THEN TO_DATE(SUBSTR ( <date column>
,1,10), 'YYYY-MM-DD') ELSE TO_DATE('1970-01-01', 'YYYY-MM-DD') END ,
TO_DATE('1970-01-01', 'YYYY-MM-DD'))
Unix Remedy COALESCE( TO_TIMESTAMP( TO_CHAR(TO_DATE('19700101000000',
(Epoch) 'YYYYMMDDHH24MISS')+(( <date column> - 18000)
/(60*60*24)),'YYYYMMDDHH24MISS'), 'YYYYMMDDHH24MISS') ,
TO_TIMESTAMP('1970-01-01', 'YYYY-MM-DD'))

Using IS_DELETED_YN
This became an issue that needed to be clarified. The IS_DELETED_YN flag must be used differently
depending on the source. The IS_DELETED_YN flag is used as a redundant flag to indicate
TRANSACTION_TYPE_C = 'D'.

Source Used Reason

BAM No The BAM source did not capture true table deletes, and the purpose of loading
into UDS was to retain artificially or falsely deleted rows. So this flag was ignored.
Remedy Yes The Remedy source identifies true deletes, so this flag is useful, and was used to
filter out deleted rows.
EBS Yes The EBS source identifies true deletes, so this flag is useful, and was used to filter
out deleted rows.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
21 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

SAL Design
SAL Design Overview
Question Answer
Performance The ETL that loaded the SAL tables does not use distribution keys. This means that
there may be a performance hit, but hopefully the low volumes will mean that this
will not be an issue. Nonetheless the distribution keys are known, and can be
applied later if ETL remediation is required. Performance issues were encountered,
but these were resolved in the cleansing and EUL.
DP Category The ITEM_NUMBER_ID will be the key to join to this dimension.
Remedy The Remedy ticket numbers are stored in the four Event Alert Remedy Link tables.
They will join to INCIDENT_NUMBER_ID in the SAL Remedy table.
Transparency As the SAL tables are one to one mappings from the BAM source tables, refer to the
source metadata as this will be the same for SAL tables.
True Keys The SAL model shows the source tables and their true keys.
Referential The BAM source system does not enforce Referential Integrity (RI) well, as is
integrity evident from the production data. This will be resolved by cleansing the data,
which will discard duplicate data.
Time Variance The ETL that loaded the SAL tables does not use the true keys. This means that
they are in effect snapshot tables, and that time variant views will be needed to
present keys and values correctly to the EUL layer.
SCD1 Invariant (I_*) tables will be used to filter out all but the most recent data, which
will be used for SCD1 dimension.
Data Quality To resolve the many data quality issues, data quality tables called Nub (N_*) tables
will be created. These will de-duplicate the data, add in default values, cast dates
correctly, change column lengths correctly and discard boilerplate columns.
Surrogate Surrogate keys for the Facts and Dimension will be generated using 2 approaches.
Keys Firstly, for small dimensions, the keys will use the hash function. For large
dimensions, surrogate key lookup (N_*_SK and N_*_DK) tables will be created.

BAM to SAL Table Mappings

Many of the tables in the BIA_SA_SCEM schema are not needed for reporting. They are either ETL
metadata or unneeded tables that replicate data in the following list. The relationships between
these tables are identical to the source system relationships already shown above.

BAM Source Table Name SAL Table Name Used in EUL?

_EIS_OrderManagementResponse SCEM_EIS_ORDERMANAGEMENTRESPONSE 1
EMEM_REMEDY_MAPPING SCEM_EMEM_REMEDY_MAPPING 1
EMEM_TOLERANCE SCEM_EMEM_TOLERANCE 1
_EP2P_EP2PLTOResponse SCEM_EP2P_EP2PLTORESPONSE 1
_Ext_P2P_Alert_List_DO2 SCEM_EXT_P2P_ALERT_LIST_DO2 1
_InternalP2P_Alert_DO SCEM_INTERNALP2P_ALERT_DO 1
MOMovement_Alert_DO SCEM_MOMOVEMENT_ALERT_DO 1
_MRAlertResponseBAM_DO SCEM_MRALERTRESPONSEBAM_DO 1
_MRSCEM_MRLTOResponse SCEM_MRSCEM_MRLTORESPONSE 1
_MSCEM_LTOResponseBAM SCEM_MSCEM_LTORESPONSEBAM 1

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
22 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

BAM Source Table Name SAL Table Name Used in EUL?

_EIS_AlertResponse SCEM_EIS_ALERTRESPONSE
_EP2P_EDO SCEM_EP2P_EDO
_EP2P_EP2PAlertResponse SCEM_EP2P_EP2PALERTRESPONSE
_EP2P_GUR_Violation_DO SCEM_EP2P_GUR_VIOLATION_DO
_EP2P_LGS_Violation_DO SCEM_EP2P_LGS_VIOLATION_DO
_EP2P_LMD_Violation_DO SCEM_EP2P_LMD_VIOLATION_DO
_EP2P_LP_Violation_DO SCEM_EP2P_LP_VIOLATION_DO
_EP2P_OrderList SCEM_EP2P_ORDERLIST
_EP2P_PLT_Violation SCEM_EP2P_PLT_VIOLATION
_EP2P_SLT_Violation SCEM_EP2P_SLT_VIOLATION
_InternalP2P_DO SCEM_INTERNALP2P_DO
_IP2P_EDO SCEM_IP2P_EDO
_MI_EDO SCEM_MI_EDO
_MM_BO_VIOLATION_DO SCEM_MM_BO_VIOLATION_DO
_MM_CI_VIOLATION_DO SCEM_MM_CI_VIOLATION_DO
_MM_DV_VIOLATION_DO SCEM_MM_DV_VIOLATION_DO
_MM_IQ_VIOLATION_DO SCEM_MM_IQ_VIOLATION_DO
_MM_LD_VIOLATION_DO SCEM_MM_LD_VIOLATION_DO
_MM_LP_VIOLATION_DO SCEM_MM_LP_VIOLATION_DO
_MM_LS_VIOLATION_DO SCEM_MM_LS_VIOLATION_DO
_MM_MH_VIOLATION_DO SCEM_MM_MH_VIOLATION_DO
_MM_POD_VIOLATION_DO SCEM_MM_POD_VIOLATION_DO
_MOSCEMEX_MOHLLTO_DO SCEM_MOSCEMEX_MOHLLTO_DO
_MRSCEM_MRAlertResponse SCEM_MRSCEM_MRALERTRESPONSE
_MRSCEM_MRLTOResponseBAM SCEM_MRSCEM_MRLTORESPONSEBAM
_MSCEM_AlertResponseBAM SCEM_MSCEM_ALERTRESPONSEBAM
_MT_EDO SCEM_MT_EDO
_RM_EDO SCEM_RM_EDO

SAL Tables and Keys Data Model

This model shows the 10 BIA_SA_SCEM tables that will be needed for the EUL.
Doc Title: BAM.EBS02PR.SOA_ORABAM Page Name: Data Model SAL Keys

BIA01SB
BIA_SA_SCEM
SAL Data Model
With Table and Key names

By Matthew Lawler

Last Edit: 15/06/2015 By: Matthew Lawler Filename: BAM.EBS02PR.SOA_ORABAM.vsd

The model above shows the expanded keys in the link tables, discovered during later profiling.

Data Quality Tables

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
23 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

These are additional helper tables added to the SAL to eliminate data quality issues.

Data Quality Table Name Data Quality Definition

N_BAM_ALERT_SK Surrogate Key generation table for the Fact table
N_BAM_ALERT_DK Surrogate Key generation table for the event data dimensions on
the Fact table
I_BAM_EVENT_DATA_EP2P Invariant table, with non-current rows filtered out, but columns
still in source order, with source names.
I_BAM_EVENT_DATA_IP2P Invariant table, with non-current rows filtered out, but columns
still in source order, with source names.
I_BAM_EVENT_DATA_MO Invariant table, with non-current rows filtered out, but columns
still in source order, with source names.
I_BAM_EVENT_DATA_MR Invariant table, with non-current rows filtered out, but columns
still in source order, with source names.
N_BAM_EVENT_DATA_EP2P Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.
N_BAM_EVENT_DATA_IP2P Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.
N_BAM_EVENT_DATA_MO Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.
N_BAM_EVENT_DATA_MR Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.
N_BAM_ALERT_EP2P Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.
N_BAM_ALERT_IP2P Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.
N_BAM_ALERT_MO Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.
N_BAM_ALERT_MR Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.
I_BAM_EVENT_DATA_EP2P Invariant table, with non-current rows filtered out, but columns
still in source order, with source names.
I_BAM_EVENT_DATA_IP2P Invariant table, with non-current rows filtered out, but columns
still in source order, with source names.
I_BAM_EVENT_DATA_MO Invariant table, with non-current rows filtered out, but columns
still in source order, with source names.
I_BAM_EVENT_DATA_MR Invariant table, with non-current rows filtered out, but columns
still in source order, with source names.
I_BAM_DP_CATEGORY Invariant table, with non-current rows filtered out, but columns
still in source order, with source names.
I_BAM_EVENT_DATA_REMEDY Invariant table, with non-current rows filtered out, but columns
still in source order, with source names.
I_BAM_REMEDY_TICKET Invariant table, with non-current rows filtered out, but columns
still in source order, with source names.
K_BAM_ALERT_EP2P Nubbed or cleansed source table, with distribution key added.
K_BAM_ALERT_IP2P Nubbed or cleansed source table, with distribution key added.
K_BAM_ALERT_MO Nubbed or cleansed source table, with distribution key added.
K_BAM_ALERT_MR Nubbed or cleansed source table, with distribution key added.
K_BAM_ALERT_REMEDY Nubbed or cleansed source table, with distribution key added.
K_BAM_ALERT_SK Nubbed or cleansed source table, with distribution key added.
K_BAM_EVENT_DATA_EP2P Nubbed or cleansed source table, with distribution key added.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
24 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Data Quality Table Name Data Quality Definition

K_BAM_EVENT_DATA_IP2P Nubbed or cleansed source table, with distribution key added.
K_BAM_EVENT_DATA_MO Nubbed or cleansed source table, with distribution key added.
K_BAM_EVENT_DATA_MR Nubbed or cleansed source table, with distribution key added.
K_BAM_EVENT_DATA_REMEDY Nubbed or cleansed source table, with distribution key added.
K_BAM_REMEDY_TICKET Nubbed or cleansed source table, with distribution key added.
N_BAM_ALERT_REMEDY Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.
N_BAM_DP_CATEGORY Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.
N_BAM_EVENT_DATA_REMEDY Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.
N_BAM_REMEDY_TICKET Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.
N_BAM_REMEDY_TICKET_JOIN Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.
N_BAM_TICKETS Nubbed source table, without boilerplate columns, deduplicated,
added defaults, converted dates, fixed column lengths.

Data Default Rules

IF column is Character && is Key = “-1”

IF column is Character && not is Key = "N/A"

IF column is Number && is Key = -1

IF column is Number && not is Key = 0

Remedy
Remedy based dimensions, D_BAM_REMEDY_TICKET and D_BAM_EVENT_DATA_REMEDY are based
on the source joins below. All the columns in RMDY_T2115_BASE are included in the
D_BAM_REMEDY_TICKET dimension. The RMDY_FILED_ENUM_VALUEs is used to expand 6 code
columns. The RMDY_T2115__C_C1000000151 is used to provide the ticket description information.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
25 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

DP Category
The correct source for this are the tables: EBS_MTL_SYSTEM_ITEMS_B, EBS_MTL_ITEM_CATEGORIES
and EBS_MTL_CATEGORIES_B. The original join within EBS is below. The UDS version is contained
within the final deliverable.

SELECT DISTINCT MSIB.SEGMENT1 ITEM_ID, MCB.SEGMENT1, MCB.SEGMENT2, MCB.SEGMENT3,

MSIB.DESCRIPTION

FROM INV.MTL_CATEGORIES_B MCB

INNER JOIN INV.MTL_ITEM_CATEGORIES MIC

ON MIC.CATEGORY_ID = MCB.CATEGORY_ID

INNER JOIN INV.MTL_SYSTEM_ITEMS_B MSIB

ON MIC.INVENTORY_ITEM_ID = MSIB.INVENTORY_ITEM_ID

AND MIC.ORGANIZATION_ID = MSIB.ORGANIZATION_ID

WHERE MIC.CATEGORY_SET_ID = 1100000061 -- 'DP Category'

AND MCB.STRUCTURE_ID = 50348 -- 'DP Category'

AND MIC.ORGANIZATION_ID = 86

Post SAL Load Proc: F_BAM_ALERT_P()

All joins across all source tables are part of the procedure called F_BAM_ALERT_P(). This will contain
the SQL needed to populate the intermediate data quality tables N_*, K_*, I_*, etc., as well as
refresh the Netezza views. This is the master for the ELT approach.

As Datastage captures SQL errors, a hard error (255 error) is needed from any PROC. Otherwise, it
will not detect the error and it will not fail the job. Therefore, remove the clause “EXCEPTION WHEN
OTHERS THEN” from the PROC, and allow all errors. This also works in practice when testing, as
Aginity handles the SQL error as well.

Implementation Script
No Description
1 Log in as BIA_SA_SCEM, and Select BIA01IT database
2 Create the following Procedure: BIA_BA_EUL.Z_DROP_OBJECT_IF_EXISTS_P.SQL (In Aginity,
select Query, and then Execute as a Single Batch, for each file. )
3 Create the following Procedure: BIA_SA_SCEM.F_BAM_ALERT_P.SQL (In Aginity, select Query,
and then Execute as a Single Batch, for each file.)
4 Execute the deploy script called 01_DEPLOY.BIA01IT.BIA_SA_SCEM.F_BAM_ALERT_P.SQL
5 Execute the post load proc by CALL BIA_SA_SCEM.F_BAM_ALERT_P()
6 If there are unresolvable issues, then the rollback script is
02_ROLLBACK.BIA01IT.BIA_SA_SCEM.F_BAM_ALERT_P.SQL

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
26 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Operations: Trouble Shooting Procedures for Known Issues

Determine whether the error occurs in the Datastage SAL load step, or in Post Datastage
F_BAM_ALERT_P() PROC. This should be indicated by the error email.

If Datastage fails
In this case, refer to the standard Datastage trouble shooting procedure.

If the Post Datastage PROC called F_BAM_ALERT_P() fails

The F_BAM_ALERT_P() PROC uses the tables loaded by the Datastage job, and populates a series of
intermediate SAL tables and EUL tables.

In all cases, rerun the F_BAM_ALERT_P() PROC manually, at least once. Note that this PROC is
completely rerunnable, so it can be rerun as many times as required.

The PROC normally returns a 'F_BAM_ALERT_P SUCCEEDED' output text string that will contain start
and stop times for each inserted table.

If the following text string 'F_BAM_ALERT_P SUCCEEDED' DOES NOT appear at the end of the return
value, then there has been an error in this PROC.

The last successful command will be at the end of the file, and indicated by the following text string
'F_BAM_ALERT_P executed for: ' <tableName>. The next command will be the one that failed.
Search the PROC to determine which table load the PROC failed at. Determine if there is some
environmental cause for this error. If so, fix the error, and then rerun the PROC.

If the cause of the error is unclear, and rerunning does not return a successful outcome, then
capture the output text string and forward it to the development team for resolution.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
27 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

CAL Design
CAL Design Overview
The CAL was not directly needed as there were no cross schema integration requirements. It was
cleaner to design all joins within a single SAL schema, before exposing the results into the EUL.
These design patterns were tested in the build phase. The goal was to optimise for the end
reporting user, using the integrated capability of the Netezza and OBIEE environments. They are
recorded as design decisions here, in order to make clear the choices and considerations behind
each decision.

To use physical tables or views

physical tables views
positive 3 times higher query less state to manage; works for small tables (< 50,000
performance rows); useful for prototyping
negative more state and tables to is up to 3 times slower than a query in a physical table; does
manage not scale above 50,000

Decision: Physical tables are preferable to views for performance reasons, except for small tables (<
50,000 rows), in a Netezza database

To generate EUL distribution keys or not

generate not generate
positive exploits Netezza primary performance feature, no additional analysis to determine
up to 100 times performance improvement best EUL Distribution Key
negative requires analysis to determine a common EUL For non-small tables (> 50,000 rows)
Distribution Key across all non-small tables; performance is so poor as to make
forces tools like OBIEE to join on 2 columns, the data unusable ; unable to use SAL
logical dimension key, and the distribution key.; Distribution Keys, as the EUL Fact/Dim
Additional tables are needed to generate the data model will have a different
distribution key, as well as other surrogate keys. structure to the SAL data model.

Decision: Generate EUL distribution keys except for trivial cases

To create a script to drop Netezza Tables, Views and Procedures or not

drop script no drop script
positive provides a failsafe DROP This is really a DBA proc, and should have been built by
script that can be the Vendor.
repeatedly called in a proc

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
28 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

negative Extra work needed. Unable to rerun scripts or procedures when dropping and
creating tables. Unlike Oracle, Netezza generates an error
when a DROP is used on a table that does not exist.

Decision: Build the script

This kind of missing Netezza Database infrastructure, there needs to be a common script available
and visible across all Netezza instances.

Make cleansing, key allocation and re-ordering as separate composable

steps or one integrated step

separate composable steps one integrated step

positive easier to reason about changes; able to none or minimum intermediate state tables;
improve performance ok for simpler sources
negative multiple intermediate state tables; large, harder to resolve bugs, and unentangle
refresh script performance issues

Decision: Use separate composable steps for large dimensions, and use the integrated for simpler,
smaller cases

So the pattern is to use a series of composable steps that convert the raw SAL data into a form that
can be used for reporting.

1. Convert raw SAL into Nub N_* data

2. Add Distribution keys so N_* -> K_*

3. Filter out non-current rows to give Invariant data K_* -> I_*

4. Move data into EUL as Facts of Dims I_* -> F_*, D_*

Raw Source -> Nub

Extract SAL source data into Nub Tables (N_*) in order to discard boilerplate columns, de-duplicate
rows, add defaults values (e.g. N/A for nulls), convert types (e.g. Text -> Dates), fix column lengths,
etc. Note that this step does not include name changes.

Nub -> Key

Add distribution Key to Nubbed data, which is critical for adequate Netezza performance. Note that
this step does not include name changes.

Key -> Invariant

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
29 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Filter out non-Current rows, to provide Invariant or SCD1 data. Note that this step can also be used
to provide Time Variant data.

Some conversion of raw timestamps into time periods (Effective from and Effective to) can be done
here.

Invariant -> Dimension

Re-order and rename the source columns.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
30 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Doc Title: BAM.DOAP Page Name: ELT Flow

SAL Table
SAL Table SAL Table

For each source table, create

Nub by
For all source tables that For all source tables that
- discarding boilerplate
comprise the Distribution key, comprise the Fact surrogate
- de-duplication
union the tables, and generate key, union the tables, and
- add defaults
a surrogate Distribution key generate a surrogate Fact key
- convert dates
- fix columns lengths

N_*_DK (Nub Distribution N_*_SK (Nub Surrogate Key

N_* (Nub Table)
Key Table) Table)

For each Nub table, To the Fact Surrogate table,

- add Distribution Key - add Distribution Key

K_* (Key Table) K_*_SK (Fact Base Table)

For each Key table, either filter

non-current rows, or convert to
Time Variant design

I_* (Invariant),
T_* (Time Variant)

For the Fact table,

For each Variant table, - use fact base
- reorder columns - combine K_*, I_* tables
- rename columns - reorder columns
- rename columns

D_* (Dimension) F_* (Fact)

Last Edit: 15/06/2015 By: Matthew Lawler Filename: BAM.EBS02PR.SOA_ORABAM.vsd

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
31 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

EUL Design
EUL Design Overview
Question Answer
All Any tables starting with D_* are dimensions, and with F_* are facts. All tables are
derived from the SAL tables already defined, unless otherwise stated. Note that the
SAL snapshot tables are not used in the EUL model. They should still be retained in SAL
if the business needs to later access these tables.
Dimension Dimension for Date is D_DATE. This is the current EUL D_DATE dimension will be used.
Dimension Dimension for Demand Plan Product hierarchy will be taken from the EBS SAL.
Dimension Dimension for Event data is based on each of the Snapshot tables.
Dimension Dimension for Event Party is derived from the BAM SAL only, to be able to report on
Suppliers, etc. This is not a conformed dimension, as it is simply extracted from the
BAM SAL tables.
Dimension Dimension for Event types is manually defined as one of the 5 events.
Dimension Dimension for Alert types is derived from the EMEM_REMEDY reference tables.
Dimension Dimension for Remedy will be taken from the Remedy SAL.
Fact The Fact table will be based on a union over all the Event Alert Remedy Link tables.
This will allow integration across all event and alert types, using a single fact table. A
default value will be added for non-applicable events.

To union Event Types as a single Fact, or have multiple Facts?

Single Fact based on Event Types Multiple Event Type Facts
Positive Single Fact simpler for users, as no union required. Simpler views, which will be less
Allows reporting across event types, by date, alert prone to breaking. Easier to use
type and ticket # Able to combine non BAM Manual under OBIEE, as the number of
Remedy facts columns will be lower.
Negative More complex views, which may be more fragile. More difficult to build queries
Many dummy FKs for unshared keys. Possible across event types. Users may
double counting of dummy rows. Possibly difficult need to know how to use the SQL
to use with 700 odd columns in integrated data union statement.
model.
Decision: Single Fact Table only

The following design is based on the single fact design choice.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
32 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

EUL Table/View List

Altogether, there are 13 dimensions and 1 fact. These use about 700 columns.

Name Definition Source base tables Cols

D_BAM_ALERT_TYPE Has all alert Join of EMEM_TOLERANCE and 17
types EMEM_REMEDY_MAPPING, with
RMDY_T2115__BASE and
RMDY_FIELD_ENUM_VALUES
D_BAM_DP_CATEGORY TBA EBS_MTL_SYSTEM_ITEMS, 4
EBS_MTL_ITEM_CATEGORIES and
EBS_MTL_CATEGORIES_B
D_BAM_EVENT_DATA_EP2P EBS PO _EP2P_EP2PLTOResponse 94
D_BAM_EVENT_DATA_IP2P EBS PR _EIS_OrderManagementResponse 81
D_BAM_EVENT_DATA_MO EBS MO, _MSCEM_LTOResponseBAM 133
SO Events
D_BAM_EVENT_DATA_MR EBS MR _MRSCEM_MRLTOResponse 164
D_BAM_EVENT_DATA_REMEDY Remedy RMDY_T2115__BASE only 34
key
D_BAM_EVENT_PARTY_CUST Customer _EP2P_EP2PLTOResponse, 1
specific to _MSCEM_LTOResponseBAM,
BAM. _MRSCEM_MRLTOResponse
D_BAM_EVENT_PARTY_SUPPLIER Supplier _EP2P_EP2PLTOResponse, 1
specific to _EIS_OrderManagementResponse,
BAM. _MRSCEM_MRLTOResponse
D_BAM_EVENT_TYPE A degenerate dimension. IP2P, EP2P, MR, 1
MO or MANUAL.
D_BAM_REMEDY_TICKET Ticket data RMDY_T2115__BASE, 34
RMDY_FIELD_ENUM_VALUES and
RMDY_T2115__C_C1000000151
D_DATE Date None. 66
F_BAM_ALERT Fact keys Union of _MRAlertResponseBAM_DO, 27
_InternalP2P_Alert_DO,
MOMovement_Alert_DO,
_Ext_P2P_Alert_List_DO2 and remedy.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
33 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

EUL Table/View List Specification

These are parameters to drive the generation of the objects. All dimensions will be SCD1.

Name Filter Columns Order Columns SCD? Generate?

D_BAM_ALERT No No 1 Yes
D_BAM_ALERT_TYPE No No 1 Yes
D_BAM_DP_CATEGORY No No 1 Yes
D_BAM_EVENT_DATA_EP2P Active Cardinality 1 Yes
D_BAM_EVENT_DATA_IP2P Active Cardinality 1 Yes
D_BAM_EVENT_DATA_MO Active Cardinality 1 Yes
D_BAM_EVENT_DATA_MR Active Cardinality 1 Yes
D_BAM_EVENT_DATA_REMEDY No No 1 Yes
D_BAM_EVENT_PARTY_CUST No No 1 Yes
D_BAM_EVENT_PARTY_SUPPLIER No No 1 Yes
D_BAM_EVENT_TYPE No No 1 Yes
D_BAM_REMEDY_TICKET No No 1 Yes
D_DATE No No 1 No
F_BAM_ALERT No No 1 Yes

SAL to EUL Table Mappings

This shows the source table for each of the main EUL target tables. Naturally this is just the tip of
the iceberg. All column to column mappings are in the spreadsheet document.

Source Schema Source Table Target Target Table

Schema
BIA_SA_RMDY_ RMDY_T2115__BASE BIA_SA_SC N_BAM_EVENT_DATA_REM
HIST EM EDY
BIA_SA_RMDY_ RMDY_T2115__BASE BIA_SA_SC N_BAM_REMEDY_TICKET
HIST EM
BIA_SA_SCEM I_BAM_EVENT_DATA_EP2P BIA_BA_EU D_BAM_EVENT_DATA_EP2
L P
BIA_SA_SCEM I_BAM_EVENT_DATA_IP2P BIA_BA_EU D_BAM_EVENT_DATA_IP2P
L
BIA_SA_SCEM I_BAM_EVENT_DATA_MO BIA_BA_EU D_BAM_EVENT_DATA_MO
L
BIA_SA_SCEM I_BAM_EVENT_DATA_MR BIA_BA_EU D_BAM_EVENT_DATA_MR
L
BIA_SA_SCEM I_BAM_EVENT_DATA_REMEDY BIA_BA_EU D_BAM_EVENT_DATA_REM
L EDY
BIA_SA_SCEM N_BAM_ALERT_EP2P BIA_BA_EU F_BAM_ALERT
L
BIA_SA_SCEM N_BAM_ALERT_EP2P BIA_SA_SC N_BAM_ALERT_DK
EM
BIA_SA_SCEM N_BAM_ALERT_EP2P BIA_SA_SC N_BAM_ALERT_SK
EM
BIA_SA_SCEM N_BAM_ALERT_EP2P BIA_SA_SC N_BAM_TICKETS
EM
BIA_SA_SCEM N_BAM_ALERT_IP2P BIA_BA_EU F_BAM_ALERT
D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
34 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Source Schema Source Table Target Target Table

Schema
L
BIA_SA_SCEM N_BAM_ALERT_IP2P BIA_SA_SC N_BAM_ALERT_DK
EM
BIA_SA_SCEM N_BAM_ALERT_IP2P BIA_SA_SC N_BAM_ALERT_SK
EM
BIA_SA_SCEM N_BAM_ALERT_IP2P BIA_SA_SC N_BAM_TICKETS
EM
BIA_SA_SCEM N_BAM_ALERT_MO BIA_BA_EU F_BAM_ALERT
L
BIA_SA_SCEM N_BAM_ALERT_MO BIA_SA_SC N_BAM_ALERT_DK
EM
BIA_SA_SCEM N_BAM_ALERT_MO BIA_SA_SC N_BAM_ALERT_SK
EM
BIA_SA_SCEM N_BAM_ALERT_MO BIA_SA_SC N_BAM_TICKETS
EM
BIA_SA_SCEM N_BAM_ALERT_MR BIA_BA_EU F_BAM_ALERT
L
BIA_SA_SCEM N_BAM_ALERT_MR BIA_SA_SC N_BAM_ALERT_DK
EM
BIA_SA_SCEM N_BAM_ALERT_MR BIA_SA_SC N_BAM_ALERT_SK
EM
BIA_SA_SCEM N_BAM_ALERT_MR BIA_SA_SC N_BAM_TICKETS
EM
BIA_SA_SCEM N_BAM_ALERT_REMEDY BIA_SA_SC N_BAM_ALERT_DK
EM
BIA_SA_SCEM N_BAM_ALERT_REMEDY BIA_SA_SC N_BAM_ALERT_SK
EM
BIA_SA_SCEM SCEM_EIS_ORDERMANAGEMENTRE BIA_BA_EU D_BAM_EVENT_PARTY_SU
SPONSE L PPLIER
BIA_SA_SCEM SCEM_EMEM_REMEDY_MAPPING BIA_BA_EU D_BAM_EVENT_TYPE
L
BIA_SA_SCEM SCEM_EP2P_EP2PLTORESPONSE BIA_BA_EU D_BAM_EVENT_PARTY_CU
L ST
BIA_SA_SCEM SCEM_EP2P_EP2PLTORESPONSE BIA_BA_EU D_BAM_EVENT_PARTY_SU
L PPLIER
BIA_SA_SCEM SCEM_MRSCEM_MRLTORESPONSE BIA_BA_EU D_BAM_EVENT_PARTY_SU
L PPLIER
BIA_SA_SCEM SCEM_MSCEM_LTORESPONSEBAM BIA_BA_EU D_BAM_EVENT_PARTY_CU
L ST

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
35 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

SAL to EUL mapping data flow

This diagram shows the SAL tables, and how N_*, K_* and I_* tables are used to resolve data quality
issues. The final fact and dimension tables are shown in red below.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
36 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

EUL Tables and Keys Data Model (ERD) Upper Case

Doc Title: BAM.EBS02PR.SOA_ORABAM Page Name: Data Model EUL Keys

BIA01SB
BIA_BA_EUL
EUL Data Model
With Table and Key names

By Matthew Lawler
Last Edit: 20/05/2015 By: Matthew Lawler Filename: BAM.EBS02PR.SOA_ORABAM.vsd

This shows all tables and keys of the final model. The diagram below shows all columns based on
BAM sourced data. The Remedy and DP category columns will be added for drop 2. Primary and
foreign keys are in bold case, all other columns are unbolded.

SAL to EUL Tables and Keys Mappings

All mappings are in the XLS document. Please refer to it for these details.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
37 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

BAM EUL Tables and Keys Data Model (ERD)

Doc Title: BAM.EBS02PR.SOA_ORABAM Page Name: Data Model EUL Columns UC

BIA01SB
BIA_BA_EUL
EUL Data Model
With Table and Column names

By Matthew Lawler

Last Edit: 11/05/2015 By: Matthew Lawler Filename: BAM.EBS02PR.SOA_ORABAM.vsd

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
38 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

OBIEE Design
OBIEE Design Overview
Question Answer
Kimball OBIEE needs a clearly defined fact table grain. This is provided in the EUL model.
Kimball The business model mapping layer works best with a properly formed Kimball star
schema. All group bys needed will be defined in the CAL and EUL layers.
Security User access can be managed directly in OBIEE.
Single An RPD cannot have unconnected tables. Therefore, this forms a natural way to split
RPD RPD subject area releases. However, in this case, as there is only one fact table, a single
RPD will be deployed. Modified RPDs will be redeployed as a unit.
Key OBIEE requires that the dimensions have single key. When the true dimension key is
Structure made up of compound keys, these may be concatenated into a single key, using a
unique separator to prevent inadvertent key duplication. Alternately the key can be
created using a lookup table..
Folder Folder size should contain between 10 and 20 columns only. Larger folders impact
size usability and comprehensibility. Also users can’t deselect columns in OBEE. Instead,
rather they select columns. Therefore, large dimensions will be organised in sets of
folders, that follow some logical pattern.
User OBIEE folders contain shared dimensions, and shared facts. The first dimension in the
Control dimension folder will be D_DATE, followed by all other dimensions.
User The presentation layer supports simple WHERE clause, and simple aggregations.
Control
Data Only standard Netezza data types can be used. In particular, do not use BIGINT, as
Types OBIEE does not recognise this data type. All surrogate keys in OBIEE must be integers.
The standard Netezza data type for this should be NUMERIC (19, 0).
Table OBIEE provides some automatic quality checks, including that the row count must equal
counts the distinct count of the primary key.
SCD2 Use the standard Kimball surrogate key design pattern to represent SCD2. OBIEE does
not allow BETWEEN joins.
No Nulls OBIEE assumes that all columns have values, so null values must be filled with defaults.
Keys While an additional dimension key does not fit OBIEE's standard join pattern, the
additional distribution key join is essential to get acceptable Netezza performance.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
39 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

Testing
Overview
System Testing (ST) can start when the EUL data is in place. ST will focus on verifying transformation
such as below. UAT can start when the OBIEE RPD is in place. UAT will focus on the business logic.

Testable statements
A testable statement is some system property that will remain unchanged when some
transformation is applied to some system object.

Property Transformation Object Status

All mapped ETL from Source Compare current SAL views to source data state Completed
rows to SAL immediately after load, and before any source
updates.
All relation ETL from Source All joins in source generate the same counts as Completed
counts to SAL the equivalent joins in SAL, immediately after ETL,
and before any source updates.
All mapped Views from SAL to Any identity mappings (that is, there is no data
rows EUL change from source to target.)
Row count Views from SAL to All row counts of SAL base tables should equal the
EUL row counts of EUL views, without any WHERE
filters.
All mapped Views from EUL to Any identity mappings (that is, there is no data
rows OBIEE change from source to target.)
Row count Views from EUL to All row counts of EUL base views should equal the
OBIEE row counts of OBIEE views, without any WHERE
filters.
PK distinct = For any key (especially surrogate keys), the count
row count of the primary key = table row count.

Deployment Tests
These are the actual invariants or tests performed each time new code was deployed.

User Test Type Test Desc Expected

Value
BIA_SA_SCEM TableExist checks for required tables list of 37
tables; see
code Units
BIA_SA_SCEM ProcExist checks for required procedures list of 1
procs; see
code Units
BIA_BA_EUL TableExist checks for required tables list of 13
tables; see
code Units
BIA_BA_EUL ProcExist checks for required procedures list of 1
procs; see

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
40 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

User Test Type Test Desc Expected

Value
code Units
BIA_BA_EUL AnyRows Each dim and fact has some rows, except EP2P 1
BIA_BA_EUL AllPKUnique The PK of each dim and fact is unique 0
BIA_BA_EUL AllLeftJoins All dimension left joins to the fact has same row 0
count.
BIA_BA_EUL Date Casting 2 Date columns in each Dim TRUE
BIA_BA_EUL Green Alerts That green alerts are now populated 1
BIA_BA_EUL RealNubPKUn Each Event and Alert Nub real PK is unique 0
ique
BIA_BA_EUL EachAlertHas All SAL alerts must have 1 event. 0
Event
BIA_BA_EUL EachEventHa All SAL events must have at least 1 alert. 0
sAlert
BIA_BA_EUL NoDefault There must never be a default for EVENT_TYPE_EX or 1
ALERT_TYPE_EX.
BIA_BA_EUL AtLeastOneV There must be at least one instance of each value in 1
alue each fact key.
BIA_BA_EUL NoMissingDi Fact with inner joins to each dimension has same Row 0
mSurrogateK count as fact using surrogate keys.
eys
BIA_BA_EUL RealDimPKUn The real PK of each dim table is unique (all cols 0
ique already not null in dims)
BIA_BA_EUL NoMissingDi Fact with left joins on all dimensions has same RC as 0
mRealKeysLef table without joins on real keys
tJoin
BIA_BA_EUL NoMissingDi table with left joins on all dimensions has same RC as 0
mRealKeysIn table without joins on real keys
nerJoin
BIA_BA_EUL NoMissingDi Dim left join each dimension to the fact table on real 0
mParentReal keys. There should be no events without an alert.
KeysLeftJoin
BIA_BA_EUL NoMissingDi Dim left join each dimension to the fact table on 0
mParentSurr surrogate keys. There should be no events without an
ogateKeysLef alert
tJoin

There is no need to do SCD2 testing for BAM.

Firstly, while the DAF has the capability of detecting SCD2 in the source, testing SCD2 is pointless in
the case of the BAM source.

The reason is that the BAM munges all rows together and generates a false, synthetic, surrogate key,
completely unrelated to the actual underlying business data. The synthetic key is used to detect
SCD2 in the DAF ETL.

So, whenever EBS data changes are propagated through to BAM, BAM creates a NEW row, and does
not update a current row. With a new key each time, the DAF ETL will never detect an update, and a
TRANSACTION_TYPE_C = ‘U’ cannot occur. Therefore, the only possible TRANSACTION_TYPE_C

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
41 of 42
Matthew Lawler [email protected] DW Design and Data Model Example

values will be 'I' or 'D'. D occurs whenever a BAM row is deleted, which occurs for housekeeping
reasons. This D event has no business value, and is just a consequence of the BAM data
administration. Therefore, all BAM data will now be filtered for 'I' TRANSACTION_TYPE_C only.

Secondly, one the key tasks of the N_* tables is to reduce the tables, so that true business key is
actually unique on these tables. So, it is also really important when generating test rows to not use
the same business keys as previous rows.

The example below shows this:

select LTOTIMESTAMP, CUSTPONO, CUSTPOLINENO, count(*) as CT from

BIA_SA_SCEM.SCEM_EP2P_EP2PLTORESPONSE group by LTOTIMESTAMP, CUSTPONO,
CUSTPOLINENO ORDER by CT Desc

There are 10 duplicates here, which is to be expected, due to the duplicate rows on the BAM source.
Clearly, this is also test data. So to fix it, the data is filtered onto the N_* table.

select LTOTIMESTAMP, CUSTPONO, CUSTPOLINENO, count(*) as CT from

BIA_SA_SCEM.N_BAM_EVENT_DATA_EP2P group by LTOTIMESTAMP, CUSTPONO, CUSTPOLINENO
ORDER by CT Desc

This still produced one duplicate, which is an error. The reason is that there are 2 test case rows
were entered with the same key, but with different values. This situation could not occur in reality,
as each time a new event is detected, it must have a new timestamp by definition. The true keys for
all tables are all defined in the spreadsheet.

D:\D\Documents\DW Me\0 Publish\DW Design and Data Model Example.docx February 13, 2018
42 of 42