UNIT II

1.1 What is Data Warehouse?

Data Warehouse Introduction

A data warehouse is a collection of data marts representing historical data from

different operations in the company. This data is stored in a structure optimized
for querying and data analysis as a data warehouse. Table design, dimensions
and organization should be consistent throughout a data warehouse so that
reports or queries across the data warehouse are consistent. A data warehouse
can also be viewed as a database for historical data from different functions
within a company.

The term Data Warehouse was coined by Bill Inmon in 1990, which he
defined in the following way: "A warehouse is a subject-oriented, integrated,
time-variant and non-volatile collection of data in support of management's
decision making process". He defined the terms in the sentence as follows:

Subject Oriented: Data that gives information about a particular subject

instead of about a company's ongoing operations.

Integrated: Data that is gathered into the data warehouse from a variety of
sources and merged into a coherent whole.

Time-variant: All data in the data warehouse is identified with a particular time
period.

Non-volatile: Data is stable in a data warehouse. More data is added but data is
never removed.

This enables management to gain a consistent picture of the business. It is a

single, complete and consistent store of data obtained from a variety of different
sources made available to end users in what they can understand and
use in a business context. It can be
 Used for decision Support
 Used to manage and control business
 Used by managers and end-users to understand the business and make
judgments
Data Warehousing is an architectural construct of information systems that
provides users with current and historical decision support information that is
hard to access or present in traditional operational data stores.

Other important terminology

Enterprise Data warehouse: It collects all information about subjects

(customers, products, sales, assets, personnel) that span the entire
organization

Data Mart: Departmental subsets that focus on selected subjects. A data mart is
a segment of a data warehouse that can provide data for reporting and analysis
on a section, unit, department or operation in the company, e.g. sales, payroll,
production. Data marts are sometimes complete individual data warehouses
which are usually smaller than the corporate data warehouse.

Decision Support System (DSS):Information technology to help the

knowledge worker (executive, manager, and analyst) makes faster & better
decisions

Drill-down: Traversing the summarization levels from highly summarized data

to the underlying current or old detail

Metadata: Data about data. Containing location and description of

warehouse system components: names, definition, structure…

Benefits of data warehousing

  Data warehouses are designed to perform well with aggregate queries
running on large amounts of data.
 The structure of data warehouses is easier for end users to navigate,
understand and query against unlike the relational databases
primarily designed to handle lots of transactions.
 Data warehouses enable queries that cut across different segments of
a company's operation. E.g. production data could be compared
against inventory data even if they were originally stored in different
databases with different structures.
 Queries that would be complex in very normalized databases could be
easier to build and maintain in data warehouses, decreasing the
workload on transaction systems.
 Data warehousing is an efficient way to manage and report on data
that is from a variety of sources, non uniform and scattered
throughout a company.
 Data warehousing is an efficient way to manage demand for lots of
information from lots of users.
 Data warehousing provides the capability to analyze large amounts of
historical data for nuggets of wisdom that can provide an organization
with competitive advantage.

Operational and informational Data

• Operational Data:
 Focusing on transactional function such as bank card
withdrawals and deposits
 Detailed
 Updateable
 Reflects current data
• Informational Data:
 Focusing on providing answers to problems posed
by decision makers
 Summarized
 Non updateable
Data Warehouse Characteristics

• A data warehouse can be viewed as an information system with the

following attributes: – It is a database designed for analytical tasks
– It‘s content is periodically updated
– It contains current and historical data to provide a historical perspective of
information

Operational data store (ODS)

• ODS is an architecture concept to support day-to-day operational decision

support and contains current value data propagated from operational
applications
• ODS is subject-oriented, similar to a classic definition of
a Data warehouse • ODS is integrated

However:

ODS DATA WAREHOUSE

 Volatile Non volatile
Very current data Current and historical data
Detailed data Pre calculated summaries

1.1.1 Differences between Operational Database Systems and Data

Warehouses

Features of OLTP and OLAP

The major distinguishing features between OLTP and OLAP are summarized as
follows.

1. Users and system orientation: An OLTP system is customer-oriented and is used

for transaction and query processing by clerks, clients, and information technology
professionals. An OLAP system is market-oriented and is used for data analysis by
knowledge workers, including managers, executives, and analysts.

2. Data contents: An OLTP system manages current data that, typically, are too
detailed to be easily used for decision making. An OLAP system manages large
amounts of historical data, provides facilities for summarization and aggregation, and
stores and manages information at different levels of granularity. These features make
the data easier for use in informed decision making.

3. Database design: An OLTP system usually adopts an entity-relationship (ER) data

model and an application oriented database design. An OLAP system typically adopts
either a star or snowflake model and a subject-oriented database design.

4. View: An OLTP system focuses mainly on the current data within an enterprise or
department, without referring to historical data or data in different organizations. In
contrast, an OLAP system often spans multiple versions of a database schema. OLAP
systems also deal with information that originates from different organizations,
integrating information from many data stores. Because of their huge volume, OLAP
data are stored on multiple storage media.

5. Access patterns: The access patterns of an OLTP system consist mainly of short,
atomic transactions. Such a system requires concurrency control and recovery
mechanisms. However, accesses to OLAP systems are mostly read-only operations
although many could be complex queries.

Comparison between OLTP and OLAP systems.

1.2 A Multidimensional Data Model.

The most popular data model for data warehouses is a multidimensional model.
This model can exist in the form of a star schema, a snowflake schema, or a fact
constellation schema. Let's have a look at each of these schema types.
1.2.1 From Tables and Spreadsheets to Data Cubes

1.2.2 Stars,Snowflakes,and Fact Constellations:

Schemas for Multidimensional Databases

 Star schema: The star schema is a modeling paradigm in which the data
warehouse contains (1) a large central table (fact table), and (2) a set of smaller
attendant tables (dimension tables), one for each dimension. The schema graph
resembles a starburst, with the dimension tables displayed in a radial pattern
around the central fact table.

Figure Star schema of a data warehouse for sales.

 Snowflake schema: The snowflake schema is a variant of the star schema
model, where some dimension tables are normalized, thereby further splitting
the data into additional tables. The resulting schema graph forms a shape
similar to a snowflake. The major difference between the snowflake and star
schema models is that the dimension tables of the snowflake model may be
kept in normalized form. Such a table is easy to maintain and also saves storage
space because a large dimension table can be extremely large when the
dimensional structure is included as columns.

Figure Snowflake schema of a data warehouse for sales.

  Fact constellation: Sophisticated applications may require multiple fact tables
to share dimension tables. This kind of schema can be viewed as a collection
of stars, and hence is called a galaxy schema or a fact constellation.

Figure Fact constellation schema of a data warehouse for sales

and shipping.

1.2.3 Example for Defining Star, Snowflake, and Fact Constellation

Schemas

A Data Mining Query Language, DMQL: Language Primitives

 Cube Definition (Fact Table)
define cube <cube_name> [<dimension_list>]: <measure_list>
 Dimension Definition (Dimension Table)
define dimension <dimension_name> as (<attribute_or_subdimension_list>)
 Special Case (Shared Dimension Tables)
 First time as “cube definition”
 define dimension <dimension_name> as
<dimension_name_first_time> in cube <cube_name_first_time>
Defining a Star Schema in DMQL
define cube sales_star [time, item, branch, location]:
dollars_sold = sum(sales_in_dollars), avg_sales = avg(sales_in_dollars), units_sold =
count(*)
define dimension time as (time_key, day, day_of_week, month, quarter, year)
define dimension item as (item_key, item_name, brand, type, supplier_type)
define dimension branch as (branch_key, branch_name, branch_type)
define dimension location as (location_key, street, city, province_or_state, country)

Defining a Snowflake Schema in DMQL

define cube sales_snowflake [time, item, branch, location]:
dollars_sold = sum(sales_in_dollars), avg_sales = avg(sales_in_dollars), units_sold =
count(*)
define dimension time as (time_key, day, day_of_week, month, quarter, year)
define dimension item as (item_key, item_name, brand, type, supplier(supplier_key,
supplier_type))
define dimension branch as (branch_key, branch_name, branch_type)
define dimension location as (location_key, street, city(city_key, province_or_state,
country))

Defining a Fact Constellation in DMQL

define cube sales [time, item, branch, location]:
dollars_sold = sum(sales_in_dollars), avg_sales = avg(sales_in_dollars), units_sold =
count(*)
define dimension time as (time_key, day, day_of_week, month, quarter, year)
define dimension item as (item_key, item_name, brand, type, supplier_type)
define dimension branch as (branch_key, branch_name, branch_type)
define dimension location as (location_key, street, city, province_or_state, country)
define cube shipping [time, item, shipper, from_location, to_location]:
dollar_cost = sum(cost_in_dollars), unit_shipped = count(*)
define dimension time as time in cube sales
define dimension item as item in cube sales
define dimension shipper as (shipper_key, shipper_name, location as location in cube
sales, shipper_type)
define dimension from_location as location in cube sales
define dimension to_location as location in cube sales
1.2.4 Measures: Three Categories
Measure: a function evaluated on aggregated data corresponding to given dimension-
value pairs.
Measures can be:
 distributive: if the measure can be calculated in a distributive manner.
 E.g., count(), sum(), min(), max().
 algebraic: if it can be computed from arguments obtained by applying
distributive aggregate functions.
 E.g., avg()=sum()/count(), min_N(), standard_deviation().
 holistic: if it is not algebraic.
 E.g., median(), mode(), rank().

1.2.5 A Concept Hierarchy

A Concept hierarchy defines a sequence of mappings from a set of low level Concepts
to higher level, more general Concepts. Concept hierarchies allow data to be handled
at varying levels of abstraction

1.2.6 OLAP operations on multidimensional data.

1. Roll-up: The roll-up operation performs aggregation on a data cube, either by

climbing-up a concept hierarchy for a dimension or by dimension reduction. Figure
shows the result of a roll-up operation performed on the central cube by climbing up
the concept hierarchy for location. This hierarchy was defined as the total order street
< city < province or state <country.

2. Drill-down: Drill-down is the reverse of roll-up. It navigates from less detailed data
to more detailed data. Drill-down can be realized by either stepping-down a concept
hierarchy for a dimension or introducing additional dimensions. Figure shows the
result of a drill-down operation performed on the central cube by stepping down a
concept hierarchy for time defined as day < month < quarter < year. Drill- down occurs
by descending the time hierarchy from the level of quarter to the more detailed level
of month.

3. Slice and dice: The slice operation performs a selection on one dimension of the
given cube, resulting in a sub cube. Figure shows a slice operation where the sales data
are selected from the central cube for the dimension time using the criteria
time=”Q2". The dice operation defines a sub cube by performing a selection on two
or more dimensions.

4. Pivot (rotate): Pivot is a visualization operation which rotates the data axes in view
in order to provide an alternative presentation of the data. Figure shows a pivot
operation where the item and location axes in a 2-D slice are rotated.
Figure: Examples of typical OLAP operations on multidimensional data.
1.3 Data warehouse architecture
1.3.1 Steps for the Design and Construction of Data Warehouse
This subsection presents a business analysis framework for data
warehouse design. The basic steps involved in the design process are also
described.

The Design of a Data Warehouse: A Business Analysis Framework

Four different views regarding the design of a data warehouse must be
considered: the top-down view, the data source view, the data warehouse
view, the business query view.

 The top-down view allows the selection of relevant information

necessary for the data warehouse.
 The data source view exposes the information being captured,
stored and managed by operational systems.
 The data warehouse view includes fact tables and dimension tables
 Finally the business query view is the Perspective of data in the
data warehouse from the viewpoint of the end user.

1.3.2 Three-tier Data warehouse architecture

 The bottom tier is ware-house database server which is almost always a
relational database system. The middle tier is an OLAP server which is typically
implemented using either (1) a Relational OLAP (ROLAP) model, (2) a
Multidimensional OLAP (MOLAP) model. The top tier is a client, which contains
query and reporting tools, analysis tools, and/or data mining tools (e.g., trend analysis,
prediction, and so on).

From the architecture point of view, there are three data warehouse models: the
enterprise warehouse, the data mart, and the virtual warehouse.

 Enterprise warehouse: An enterprise warehouse collects all of the information

about subjects spanning the entire organization. It provides corporate-wide data
integration, usually from one or more operational systems or external
information providers, and is cross-functional in scope. It typically contains
detailed data as well as summarized data, and can range in size from a few
gigabytes to hundreds of gigabytes, terabytes, or beyond.
 Data mart: A data mart contains a subset of corporate-wide data that is of value
to a specific group of users. The scope is connected to specific, selected
subjects. For example, a marketing data mart may connect its subjects to
customer, item, and sales. The data contained in data marts tend to be
summarized. Depending on the source of data, data marts can be categorized
into the following two classes:

(i).Independent data marts are sourced from data captured from one or more
operational systems or external information providers, or from data generated
locally within a particular department or geographic area.

(ii).Dependent data marts are sourced directly from enterprise data

warehouses.

 Virtual warehouse: A virtual warehouse is a set of views over operational

databases. For efficient query processing, only some of the possible summary
views may be materialized. A virtual warehouse is easy to build but requires
excess capacity on operational database servers.
 Figure: A recommended approach for data warehouse development.

1.3.3 Data warehouse Back-End Tools and Utilities

The ETL (Extract Transformation Load) process
In this section we will discussed about the 4 major process of the data warehouse.
They are extract (data from the operational systems and bring it to the data
warehouse), transform (the data into internal format and structure of the data
warehouse), cleanse (to make sure it is of sufficient quality to be used for decision
making) and load (cleanse data is put into the data warehouse).
The four processes from extraction through loading often referred collectively as
Data Staging.

EXTRACT
Some of the data elements in the operational database can be reasonably be expected
to be useful in the decision making, but others are of less value for that purpose. For
this reason, it is necessary to extract the relevant data from the operational database
before bringing into the data warehouse. Many commercial tools are available to help
with the extraction process. Data Junction is one of the commercial products. The
user of one of these tools typically has an easy-to-use windowed interface by which to
specify the following:

(i) Which files and tables are to be accessed in the source database?
(ii) Which fields are to be extracted from them? This is often done
internally by SQL Select statement.
(iii) What are those to be called in the resulting database?
 (iv) What is the target machine and database format of the output?
(v) On what schedule should the extraction process be repeated?

TRANSFORM

The operational databases developed can be based on any set of priorities, which keeps
changing with the requirements. Therefore those who develop data warehouse based
on these databases are typically faced with inconsistency among their data sources.
Transformation process deals with rectifying any inconsistency (if any).

One of the most common transformation issues is ‘Attribute Naming Inconsistency’.

It is common for the given data element to be referred to by different data names in
different databases. Employee Name may be EMP_NAME in one database, ENAME
in the other. Thus one set of Data Names are picked and used consistently in the data
warehouse. Once all the data elements have right names, they must be converted to
common formats. The conversion may encompass the following:

(i) Characters must be converted ASCII to EBCDIC or vise versa.

(ii) Mixed Text may be converted to all uppercase for consistency.
(iii) Numerical data must be converted in to a common format.
(iv) Data Format has to be standardized.
(v) Measurement may have to convert. (Rs/ $)
(vi) Coded data (Male/ Female, M/F) must be converted into a common
format.
All these transformation activities are automated and many commercial products are
available to perform the tasks. DataMAPPER from Applied Database Technologies
is one such comprehensive tool.

CLEANSING
Information quality is the key consideration in determining the value of the
information. The developer of the data warehouse is not usually in a position to change
the quality of its underlying historic data, though a data warehousing project can put
spotlight on the data quality issues and lead to improvements for the future. It is,
therefore, usually necessary to go through the data entered into the data warehouse and
make it as error free as possible. This process is known as Data Cleansing.

Data Cleansing must deal with many types of possible errors. These include missing
data and incorrect data at one source; inconsistent data and conflicting data when two
or more source are involved. There are several algorithms followed to clean the data,
which will be discussed in the coming lecture notes.

LOADING
Loading often implies physical movement of the data from the computer(s) storing the
source database(s) to that which will store the data warehouse database, assuming it is
different. This takes place immediately after the extraction phase. The most common
channel for data movement is a high-speed communication link. Ex: Oracle Warehouse
Builder is the API from Oracle, which provides the features to perform the ETL task
on Oracle Data Warehouse.

Data cleaning problems

This section classifies the major data quality problems to be solved by data cleaning
and data transformation. As we will see, these problems are closely related and should
thus be treated in a uniform way. Data transformations [26] are needed to support any
changes in the structure, representation or content of data. These transformations
become necessary in many situations, e.g., to deal with schema evolution, migrating a
legacy system to a new information system, or when multiple data sources are to be
integrated. As shown in Fig. 2 we roughly distinguish between single-source and multi-
source problems and between schema- and instance-related problems. Schema-level
problems of course are also reflected in the instances; they can be addressed at the
schema level by an improved schema design (schema evolution), schema translation
and schema integration. Instance-level problems, on the other hand, refer to errors and
inconsistencies in the actual data contents which are not visible at the schema level.
They are the primary focus of data cleaning. Fig. 2 also indicates some typical problems
for the various cases. While not shown in Fig. 2, the single-source problems occur (with
increased likelihood) in the multi-source case, too, besides specific multi-source
problems.
Single-source problems
The data quality of a source largely depends on the degree to which it is governed by
schema and integrity constraints controlling permissible data values. For sources
without schema, such as files, there are few restrictions on what data can be entered
and stored, giving rise to a high probability of errors and inconsistencies. Database
systems, on the other hand, enforce restrictions of a specific data model (e.g., the
relational approach requires simple attribute values, referential integrity, etc.) as well
as application-specific integrity constraints. Schema-related data quality problems thus
occur because of the lack of appropriate model-specific or application-specific
integrity constraints, e.g., due to data model limitations or poor schema design, or
because only a few integrity constraints were defined to limit the overhead for integrity
control. Instance-specific problems relate to errors and inconsistencies that cannot be
prevented at the schema level (e.g., misspellings).
For both schema- and instance-level problems we can differentiate different problem
scopes: attribute (field), record, record type and source; examples for the various cases
are shown in Tables 1 and 2. Note that uniqueness constraints specified at the schema
level do not prevent duplicated instances, e.g., if information on the same real world
entity is entered twice with different attribute values (see example in Table 2).

Multi-source problems
The problems present in single sources are aggravated when multiple sources need to
be integrated. Each source may contain dirty data and the data in the sources may be
represented differently, overlap or contradict. This is because the sources are typically
developed, deployed and maintained independently to serve specific needs. This results
in a large degree of heterogeneity w.r.t. data management systems, data models,
schema designs and the actual data.

At the schema level, data model and schema design differences are to be
addressed by the steps of schema translation and schema integration, respectively. The
main problems w.r.t. schema design are naming and structural conflicts. Naming
conflicts arise when the same name is used for different objects (homonyms) or
different names are used for the same object (synonyms). Structural conflicts occur in
many variations and refer to different representations of the same object in different
sources, e.g., attribute vs. table representation, different component structure, different
data types, different integrity constraints, etc. In addition to schema-level
conflicts, many conflicts appear only at the instance level (data conflicts). All problems
from the single-source case can occur with different representations in different
sources (e.g., duplicated records, contradicting records,…). Furthermore, even when
there are the same attribute names and data types, there may be different value
representations (e.g., for marital status) or different interpretation of the values (e.g.,
measurement units Dollar vs. Euro) across sources. Moreover, information in the
sources may be provided at different aggregation levels (e.g., sales per product vs. sales
per product group) or refer to different points in time (e.g. current sales as of yesterday
for source 1 vs. as of last week for source 2).

A main problem for cleaning data from multiple sources is to identify

overlapping data, in particular matching records referring to the same real-world entity
(e.g., customer). This problem is also referred to as the object identity problem,
duplicate elimination or the merge/purge problem. Frequently, the information is only
partially redundant and the sources may complement each other by providing
additional information about an entity. Thus duplicate information should be purged
out and complementing information should be consolidated and merged in order to
achieve a consistent view of real world entities.

The two sources in the example of Fig. 3 are both in relational format but exhibit
schema and data conflicts. At the schema level, there are name conflicts (synonyms
Customer/Client, Cid/Cno, Sex/Gender) and structural conflicts (different
representations for names and addresses). At the instance level, we note that there are
different gender representations (“0”/”1” vs. “F”/”M”) and presumably a duplicate
record (Kristen Smith). The latter observation also reveals that while Cid/Cno are both
source-specific identifiers, their contents are not comparable between the sources;
different numbers (11/493) may refer to the same person while different persons can
have the same number (24). Solving these problems requires both schema integration
and data cleaning; the third table shows a possible solution. Note that the schema
conflicts should be resolved first to allow data cleaning, in particular detection of
duplicates based on a uniform representation of names and addresses, and matching of
the Gender/Sex values.

Data cleaning approaches

In general, data cleaning involves several phases
Data analysis: In order to detect which kinds of errors and inconsistencies are to
be removed, a detailed
data analysis is required. In addition to a manual inspection of the data or data
samples, analysis programs should be used to gain metadata about the data
properties and detect data quality problems.

Definition of transformation workflow and mapping rules: Depending on the

number of data sources ,their degree of heterogeneity and the “dirtyness” of the
data, a large number of data transformation and cleaning steps may have to be
executed. Sometime, a schema translation is used to map sources to a common
data model; for data warehouses, typically a relational representation is used.
Early data cleaning steps can correct single-source instance problems and
prepare the data for integration. Later steps deal with schema/data integration
and cleaning multi-source instance problems, e.g., duplicates.
For data warehousing, the control and data flow for these transformation and
cleaning steps should be specified within a workflow that defines the ETL process
(Fig. 1).

The schema-related data transformations as well as the cleaning steps

should be specified by a declarative query and mapping language as far as
possible, to enable automatic generation of the transformation code. In addition,
it should be possible to invoke user-written cleaning code and special purpose
tools during a data transformation workflow. The transformation steps may
request user feedback on data instances for which they have no built-in cleaning
logic.

Verification: The correctness and effectiveness of a transformation workflow

and the transformation definitions should be tested and evaluated, e.g., on a
sample or copy of the source data, to improve the definitions if necessary.
Multiple iterations of the analysis, design and verification steps may be needed,
e.g., since some errors only become apparent after applying some
transformations.

Transformation: Execution of the transformation steps either by running the

ETL workflow for loading and refreshing a data warehouse or during answering
queries on multiple sources.

Backflow of cleaned data: After (single-source) errors are removed, the

cleaned data should also replace the dirty data in the original sources in order to
give legacy applications the improved data too and to avoid redoing the cleaning
work for future data extractions. For data warehousing, the cleaned data is
available from the data staging area (Fig. 1).

Data analysis
Metadata reflected in schemas is typically insufficient to assess the data quality of
a source, especially if only a few integrity constraints are enforced. It is thus
important to analyse the actual instances to obtain real (reengineered) metadata
on data characteristics or unusual value patterns. This metadata helps finding
data quality problems. Moreover, it can effectively contribute to identify attribute
correspondences between source schemas (schema matching), based on which
automatic data transformations can be derived.
There are two related approaches for data analysis, data profiling and data
mining. Data profiling focuses on the instance analysis of individual attributes. It
derives information such as the data type, length, value range, discrete values and
their frequency, variance, uniqueness, occurrence of null values, typical string
pattern (e.g., for phone numbers), etc., providing an exact view of various quality
aspects of the attribute.
 Table 3 shows examples of how this metadata can help detecting data
quality problems.

1.3.4 Metadata repository

Metadata are data about data. When used in a data warehouse, metadata are the data
that define warehouse objects. Metadata are created for the data names and definitions
of the given warehouse. Additional metadata are created and captured for time
stamping any extracted data, the source of the extracted data, and missing fields that
have been added by data cleaning or integration processes. A metadata repository
should contain:

 A description of the structure of the data warehouse. This includes the

warehouse schema, view, dimensions, hierarchies, and derived data definitions,
as well as data mart locations and contents;
 Operational metadata, which include data lineage (history of migrated data and
the sequence of transformations applied to it), currency of data (active,
archived, or purged), and monitoring information (warehouse usage statistics,
error reports, and audit trails);
 the algorithms used for summarization, which include measure and dimension
definition algorithms, data on granularity, partitions, subject areas, aggregation,
summarization, and predefined queries and reports;
 The mapping from the operational environment to the data warehouse, which
includes source databases and their contents, gateway descriptions, data
partitions, data extraction, cleaning, transformation rules and defaults, data
refresh and purging rules, and security (user authorization and access control).
  Data related to system performance, which include indices and profiles that
improve data access and retrieval performance, in addition to rules for the
timing and scheduling of refresh, update, and replication cycles; and
 Business metadata, which include business terms and definitions, data
ownership information, and charging policies.

1.3.5 Types of OLAP Servers: ROLAP versus MOLAP versus HOLAP

1. Relational OLAP (ROLAP)

 Use relational or extended-relational DBMS to store and manage

warehouse data and OLAP middle ware to support missing pieces

 Include optimization of DBMS backend, implementation of

aggregation navigation logic, and additional tools and services

 greater scalability

2. Multidimensional OLAP (MOLAP)

 Array-based multidimensional storage engine (sparse matrix

techniques)

 fast indexing to pre-computed summarized data

3. Hybrid OLAP (HOLAP)

 User flexibility, e.g., low level: relational, high-level: array

4. Specialized SQL servers

 specialized support for SQL queries over star/snowflake schemas

1.4 Data Warehouse Implementation

1.4.1 Efficient Computation of Data Cubes

Data cube can be viewed as a lattice of cuboids

  The bottom-most cuboid is the base cuboid
 The top-most cuboid (apex) contains only one cell
 How many cuboids in an n-dimensional cube with L levels?

Materialization of data cube

 Materialize every (cuboid) (full materialization), none (no materialization), or

some (partial materialization)
 Selection of which cuboids to materialize
 Based on size, sharing, access frequency, etc.
Cube Operation

 Cube definition and computation in DMQL

define cube sales[item, city, year]: sum(sales_in_dollars)

compute cube sales

 Transform it into a SQL-like language (with a new operator cube by,

introduced by Gray et al.’96)
SELECT item, city, year, SUM (amount)

FROM SALES

CUBE BY item, city, year

 Need compute the following Group-Bys

(date, product, customer),

(date,product),(date, customer), (product, customer),

(date), (product), (customer)

()
Cube Computation: ROLAP-Based Method

 Efficient cube computation methods

o ROLAP-based cubing algorithms (Agarwal et al’96)
o Array-based cubing algorithm (Zhao et al’97)
o Bottom-up computation method (Bayer & Ramarkrishnan’99)
 ROLAP-based cubing algorithms
o Sorting, hashing, and grouping operations are applied to the dimension
attributes in order to reorder and cluster related tuples
 Grouping is performed on some sub aggregates as a “partial grouping
step”
 Aggregates may be computed from previously computed aggregates,
rather than from the base fact table

Multi-way Array Aggregation for Cube

 Computation
 Partition arrays into chunks (a small sub cube which fits in memory).
 Compressed sparse array addressing: (chunk_id, offset)
 Compute aggregates in “multiway” by visiting cube cells in the order
which minimizes the # of times to visit each cell, and reduces memory
access and storage cost.
1.4.2 Indexing OLAP data
The bitmap indexing method is popular in OLAP products because it allows quick
searching in data cubes.

The bitmap index is an alternative representation of the record ID (RID) list.

In the bitmap index for a given attribute, there is a distinct bit vector, By, for each value
v in the domain of the attribute. If the domain of a given attribute consists of n values,
then n bits are needed for each entry in the bitmap index

The join indexing method gained popularity from its use in relational database
query processing. Traditional indexing maps the value in a given column to a list of
rows having that value. In contrast, join indexing registers the joinable rows of two
relations from a relational database. For example, if two relations R(RID;A) and S(B;
SID) join on the attributes A and B, then the join index record contains the pair (RID;
SID), where RID and SID are record identifiers from the R and S relations,
respectively.

1.4.3 Efficient processing of OLAP queries

1. Determine which operations should be performed on the available cuboids. This

involves transforming any selection, projection, roll-up (group-by) and drill-down
operations specified in the query into corresponding SQL and/or OLAP operations. For
example, slicing and dicing of a data cube may correspond to selection and/or
projection operations on a materialized cuboid.
2. Determine to which materialized cuboid(s) the relevant operations should be
applied. This involves identifying all of the materialized cuboids that may potentially
be used to answer the query.

1.5 From Data Warehousing to Data mining

1.5.1 Data Warehouse Usage:

Three kinds of data warehouse applications

1. Information processing

 supports querying, basic statistical analysis, and reporting

using crosstabs, tables, charts and graphs

2. Analytical processing

 multidimensional analysis of data warehouse data

 supports basic OLAP operations, slice-dice, drilling, pivoting

3. Data mining

 knowledge discovery from hidden patterns

 supports associations, constructing analytical models,

performing classification and prediction, and presenting the
mining results using visualization tools.

 Differences among the three tasks

Note:

From On-Line Analytical Processing to On Line Analytical Mining (OLAM)

called from data warehousing to data mining
1.5.2 From on-line analytical processing to on-line analytical mining.
On-Line Analytical Mining (OLAM) (also called OLAP mining), which
integrates on-line analytical processing (OLAP) with data mining and mining
knowledge in multidimensional databases, is particularly important for the following
reasons.

1. High quality of data in data warehouses.

Most data mining tools need to work on integrated, consistent, and cleaned
data, which requires costly data cleaning, data transformation and data integration as
preprocessing steps. A data warehouse constructed by such preprocessing serves as a
valuable source of high quality data for OLAP as well as for data mining.

2. Available information processing infrastructure surrounding data

warehouses.

Comprehensive information processing and data analysis infrastructures have

been or will be systematically constructed surrounding data warehouses, which include
accessing, integration, consolidation, and transformation of multiple, heterogeneous
databases, ODBC/OLEDB connections, Web-accessing and service facilities,
reporting and OLAP analysis tools.

3. OLAP-based exploratory data analysis.

Effective data mining needs exploratory data analysis. A user will often want
to traverse through a database, select portions of relevant data, analyze them at different
granularities, and present knowledge/results in different forms. On-line analytical
mining provides facilities for data mining on different subsets of data and at different
levels of abstraction, by drilling, pivoting, filtering, dicing and slicing on a data cube
and on some intermediate data mining results.

4. On-line selection of data mining functions.

By integrating OLAP with multiple data mining functions, on-line analytical

mining provides users with the exibility to select desired data mining functions and
swap data mining tasks dynamically.
Architecture for on-line analytical mining

An OLAM engine performs analytical mining in data cubes in a similar manner

as an OLAP engine performs on-line analytical processing. An integrated OLAM and
OLAP architecture is shown in Figure, where the OLAM and OLAP engines both
accept users' on-line queries via a User GUI API and work with the data cube in the
data analysis via a Cube API.

A metadata directory is used to guide the access of the data cube. The data cube
can be constructed by accessing and/or integrating multiple databases and/or by
filtering a data warehouse via a Database API which may support OLEDB or ODBC
connections. Since an OLAM engine may perform multiple data mining tasks, such as
concept description, association, classification, prediction, clustering, time-series
analysis ,etc., it usually consists of multiple, integrated data mining modules and is
more sophisticated than an OLAP engine.

Figure: An integrated OLAM and OLAP architecture.