The arguments put forth by him in favor of a library data warehouse are

• Data from several heterogenous data sources (MS Excel spreadsheets, MS Access CSVfile, etc.) can
be extracted and brought together in a data warehouse.
• Even when DIIT expands into several branches in multiple cities, it still can have one ware-house to
support the information needs of the institution.

• Data anomalies can be corrected through an ETL package.

• Missing or incomplete records can be detected and duly corrected.

• Uniformity can be maintained over each attribute of a table.

• Data can be conveniently retrieved for analysis and generating reports (like the report on spending
requested above).

• Fact-based decision making can be easily supported by a data warehouse.

• Ad hoc queries can be easily supported

DATA MART
DATA MART
GOALS of DATA WAREHOUSE
BI-PROCESS
What are the problems faced in Data
Integration?
Challenges in Data Integration
Data Mapping
• Identification of data relationships
• Consolidation of multiple database in single database
• Data transformation between data source and data destination
DATA MAPPING
• The process of creating data element mapping between two distinct
data models
• It is used as the first step towards a wide variety of data integration
tasks
 Data transformation between data source & data destination
 Identification of data relationships
 Discovery of hidden data
 Consolidation of multiple database into single database
Some Transformation Types
Selecting only certain columns to load
Translating a few coded values
Encoding some free –form values
Deriving a new calculated value
Joining together data derived from multiple sources
Summarizing multiple rows data
Splitting a column into multiple column
Ralph Kimball’s Approach Vs W.H. Inmon’s
Approach
• Kimball : A data warehouse is made up of all data marts in the
enterprise – Bottom –up Approach . Faster , cheaper and less complex
• W.H . Inmon : DW is subject oriented, non-volatile integrated, time-
variant collection of data – Top-down approach . Expensive and
complex . Achieves “ Single Version of truth” for lrge organisations
and worth the investment.
Data Integration Technologies
• Data Interchange :
Structured transmission of data between 2 or more organizations
electronically –or between two trading partners not email.

Object Request Brokering :

Middleware software that allow making calls from one another and
also in-process data transformation .
 Modelling Techniques

Entity Relationship Modelling

Logical Design Technique

Focuses on reducing data redundancy

Contributes in initial stages of constructing

DW

Problem of creating huge number of tables

with dozens of joins –Massive spider web of
joins between tables
Steps To Draw ER Model
Identify
Entities

Identify
Relation
between
Entities

Identify Key
attribute

Identify
other
relevant
attributes

Draw ER
diagram

Review with
Business
Users
Problems posed by ER Modeling
• End Users finds it difficult to comprehend and traverse

• Lack of software to query general ER Model

• ER modeling cannot be used for DW where performance access and

ad hoc queries are required to be done
Dimensional Modeling
• Logical design techniques that focuses on presenting data in standard
format for end –user consumption
• Based on Schema –Star or Snowflake
• Consists of one large table and number of small dimension tables
• Fact table has multipart primary key and each table has a single part
primary key
 Basics of Data Integration
(Extraction Transformation
Loading)
 Difference Between ER Modeling and
Dimensional Modeling
ER Modeling Dimensional Modeling
Optimized for Transactional Data Optimized for Query ability and performance

Eliminates redundant data Does not eliminate redundant data where appropriate

Highly normalized Aggregates most of the attributes of a dimension into a

single entity

It is a complex maze with hundreds of entities It has logical grouped set in schemas

Used for Transactional Systems Used for Analytical Systems

It is split as per the entities It is split as per dimensions and facts

Data Quality
What is data quality?
In simple terms, data quality tells us how reliable a particular set of
data is and whether or not it will be good enough for a user to employ
in decision-making. This quality is often measured by degrees.
Data Quality Dimensions
There are six primary, or core, dimensions to data quality. These are
the metrics analysts use to determine the data’s viability and its
usefulness to the people who need it.

ACCURACY,
COMPLETENESS,
VALIDITY,
CONSISTENCY,
TIMELINESS,
Accuracy

• The data must conform to actual, real-world scenarios and reflect

real-world objects and events. Analysts should use verifiable sources
to confirm the measure of accuracy, determined by how close the
values jibe with the verified correct information sources
E.g. Address of the customer in the database is the real address
Age of the patient is accurate as in hospital database
Completeness

• Completeness measures the data's ability to deliver all the

mandatory values that are available successfully.

E.g. Data of all students at University is available

Data of all the patients of a hospital are available
Dat of all employees
Consistency

• Data consistency describes the data’s uniformity as it moves across

applications and networks and when it comes from multiple sources.
Consistency also means that the same datasets stored in different
locations should be the same and not conflict.
• E.g. Customer cancelled and surrendered his credit card and yet
status reads due
• Employee left but his email id is active
Timeliness

Timely data is information that is readily available whenever it’s

needed. This dimension also covers keeping the data current; data
should undergo real-time updates to ensure that it is always available
and accessible.
• Eg Airlines need to provide timely flight information to passengers
• Enterprise to publish timely quarterly results
Uniqueness

• Uniqueness means that no duplications or redundant information

are overlapping across all the datasets. No record in the dataset
exists multiple times. Analysts use data cleansing and deduplication
to help address a low uniqueness score.
• For example, when reviewing customer data, you should expect that
each customer has a unique customer ID.
Validity

• Data must be collected according to the organization’s defined

business rules and parameters. The information should also conform
to the correct, accepted formats, and all dataset values should fall
within the proper range.
• Formatting usually includes metadata, such as valid data types, ranges,
patterns, and more.
 • Data profiling is the process of
examining, analyzing, and
creating useful summaries of
data. The process yields a high-
level overview which aids in the
discovery of data quality issues,
Data Profiling risks, and overall trends. Data
profiling produces critical
insights into data that companies
can then leverage to their
advantage.
Two types of Data Profiling
Data Quality Profiling
Database Profiling
Data Quality Profiling

• Analysing the data from a data source or a

database against the business requirements
• Enables to identify issues in data quality
• Analysis may be represented as
• Summaries : Counts and Percentages on
completeness of the dataset , uniqueness
of columns etc .
Details : Involves lists containing
information on data records or data
problems in individual records
Database Profiling

Procedures of Analysis of a database wrt its schema ,

relationship between tables , columns used , keys of the tables
etc

Database Profiling is done initially followed by data quality

profiling
When is it conducted?
At the discovery /
requirements
gathering phase

• As soon as source data

system are identified
• Business requirements laid
out
• Data quality profiling needs
to be done to avoid
correction and re-work
Just before the Dimensional
Modelling Process

• Intensive data Profiling is done

• More database Profiling
• Analysis of schema designs for DW
• Aims to identify the best method to
convert the source data to the
dimensional model
 To identify possible errors that
may creep in during ETL

During ETL
Helps in identifying what data
Package to extract and what filters to
apply
Design

More Data quality Profiling

Domino’s data avalanche

• With almost 14,000 locations, Domino’s was already the largest pizza
company in the world by 2015. But when the company launched
its AnyWare ordering system, it was suddenly faced with an avalanche of
data. Users could now place orders through virtually any type of device or
app, including smart watches, TVs, car entertainment systems, and social
media platforms.
• That meant Domino’s had data coming at it from all sides. By putting
reliable data profiling to work, Domino’s now collects and analyzes
data from all of the company’s point of sales systems in order to streamline
analysis and improve data quality. As a result, Domino’s has gained deeper
insights into its customer base, enhanced its fraud detection processes,
boosted operational efficiency, and increased sales.