Data Mining and

Business Intelligence
1st Lecture

Iraklis Varlamis
1
About the course

•Lectures: Monday 5:00-7:30

(+visits to the lab at the 2nd floor)

•Office: 5.1

•Email:[email protected]
•Eclass
• http://eclass.hua.gr/courses/DIT161/

2
Reading

• Tan, Steinbach & Kumar. Introduction to data

mining, Addison-Wesley.
http://www-users.cs.umn.edu/~kumar/dmbook/index.php

• Jiawei Han, Micheline Kamber, and Jian Pei.

Data mining – Concepts and techniques (3rd
edition)
These presentations are based on the book and
on the presentations of Jiawei Han
http://www.cs.uiuc.edu/~hanj/

3
Course Outline

Organize and Manage Data

• 1st: Basic concepts of DM & BI. Solutions and Architectures. The
stages of DM process. Application examples.
• 2nd: The process of Data Mining. Classification techniques and
algorithms. Evaluation techniques and metrics.
• 3rd: Data preparation. Dimensionality reduction. Regression.
Core Data Mining Techniques
• 4th: Classification techniques. Practical application of classification
techniques (Lab)
...
• 9th: Clustering techniques and algorithms. Evaluation metrics.
Cluster description.
• 10th: Association rules extraction. Techniques and algorithms.
• 11th: Data warehouses. Data quality. Cubes and multidimensional
data analysis. Concept hierarchies and data projection in
dimensions.

4
Course Outline

Case studies
•5th: Introduction to Graph/Network Mining
•6th: Measuring networks and random graph model.
• 7th: A graph processing library
• 8th: Social Recommender systems
...
• 12th: Presentation of assignments

5
Grading system

• What is graded:

– Final written exam: 60%

– Group assignment: 40% [compulsory]
• Interim report
• Final presentation and documentation

• Written exams: with lecture notes and open

books

6
Definitions

7
Definition and concepts

• Business Intelligence (BI) refers to applications and

technologies accessing the appropriate data and
information in order to make the correct business decision
at the correct moment.

• Two types of BI Systems:

– Those that provide data analysis tools
• Multidimensional data analysis (or online analytical
processing)
• Data mining
• Decision support systems
– Those that provide information in structured format
• Dashboards

8
9
Multidimensional Data Analysis

• Multidimensional analysis provides users with an

excellent view of what is happening or what has
happened.

• Allows users to analyze data in such a way that

they can quickly answer business questions

• To accomplish this multidimensional analysis

tools allow users to “slice and dice” the data in
any desired way.

10
Data mining

• Searching for valuable business information in a

large database or data warehouse
• Data mining performs two basic operations:
– Predicting trends and behaviors
– Identifying previously unknown patterns and
relationships
• Data mining: The process that combines
techniques from statistics, artificial intelligence
and machine learning in order to process data
and extract implicit, non-obvious, interesting and
potentially useful knowledge that can support
decision making

11
Decision support systems

• Decision support systems

• DSS capabilities
– Sensitivity analysis
– What-if analysis
– Goal-seeking analysis

12
Digital Dashboards

• Dashboards:
– Provide rapid access to timely information.
– Provide direct access to management reports.
– Are very user-friendly and supported by graphics.

13
The management cockpit

• A strategic management room that enables top-level

decision makers to pilot their businesses better
• The environment encourages more efficient management
meetings and boosts team performance via effective
communication
• Key performance indicators and information relating to
critical success factors are displayed graphically on the
walls of the meeting room
• External information can be easily imported to the room to
allow competitive analysis

14
From Data to Knowledge

15
Evolution of sciences

• Before 1600, empirical science

• 1600-1950, theoretical science
– Dominated by theoretical models, which often motivate
experiments for better understanding the world
• 1950-1990, computational science
– We try to understand complex mathematical models through
simulation
• 1990-now, data science
– Abundance of data
– Ability to process huge data sets
– Data resources interconnection (through internet)
– The needs for collection, management, querying and
visualization of data increase to the volume of data

16
The Data Gap

• Usually information is hidden in data. This

information is not obvious
• Analysts need weeks to locate it through
Hypothesis testing
• Most of the data are never analysed
• We have the data! Now what?
…and we don’t know what to do with
them.

17
Why do we need data analysts

• Explosive data growth (from terabytes to

petabytes)
• Automated data collection
• Abundant sources
– Businesses : web, e-transactions, stock market
– Sciences: sensors, bioinformatics, scientific
experiments
– Society: news, digital cameras, social networks
“We are drowning in Data but starving for
Knowledge”. John Naisbit in “Megatrends”.
• We need automated analysis of large data sets

18
What else is data mining

• Knowledge Discovery in Databases

• Knowledge extraction
• Data and models analysis
• Data surveying over time and in varying degrees
of detail
• Collection of information and creation of business
intelligence

• NOT data search

• NOT query processing
• NOT a smart system that reacts to the rule base

19
 Business intelligence (ΒΙ)

Source: http://decision-quality.com/ 20
 Stages of BI creation

Collection Storage Analysis Delivery

“Business Intelligence-The Missing Link.” http://www.ittoolbox.com/peer/bi.pdf, Viewed 4/12/2006. 21
 It is not simple
Increasing potential
to support
business decisions End User
Decision
Making

Data Presentation Business

Analyst
Visualization Techniques
Data Mining Data
Information Discovery Analyst

Data Exploration
Statistical Summary, Querying, and Reporting

Data Preprocessing/Integration, Data Warehouses

DBA
Data Sources
Paper, Files, Web documents, Scientific experiments, Database Systems

22
What is Business Intelligence?

1. Tables and charts

– Tables (scorecards) present the key performance indicators
(e.g. ROI)
– Charts (dashboards) present the performance in a condensed
and simple “dials and gauges” format

http://www.microstrategy.com/Solutions/5Styles/
23
What is Business Intelligence?

1. Tables and charts

2. Company reports
– Extended reports, adapted to the needs of each user
group

24
What is Business Intelligence?

1. Tables and charts

2. Company reports
3. Data analytics (OLAP: On-line Analytical Processing)
– Associates data subsets (e.g. temporal data,
customer data, income data) in a multidimensional
analysis (“cube” analysis)
– Selectively provides access to the initial (raw) data

25
What is Business Intelligence?

1. Tables and charts

2. Company reports
3. Data analytics (OLAP: On-line Analytical Processing)
4. Composite analysis and prediction
– Training and evaluation of data mining techniques to
past data.
– Application to new data and prognostics
(i.e.predictions on how data will evolve, what-if
scenarios, etc.)

26
What is Business Intelligence?

1. Tables and charts

2. Company reports
3. Data analytics (OLAP: On-line Analytical Processing)
4. Composite analysis and prediction
5. Alerts
– Automatic generation of reports and notifications for
troubles and opportunities

27
 Market interest

• Income from BI software development

• Companies that
develop BI software

•http://apandre.wordpress.com/market/ 28
Scientific interest – Big Data

• Volume
– Scalable algorithms
• Variety
– Multidimensional data, e.g. microarray DNA data
contain a few 10K features,
– Spatial, spatiotemporal data, time-series data
– Web data, multimedia
– Graphs and hypergraphs in social networks
• Velocity
– Data streams, sensor data
• Veracity
– We are not always sure about data accuracy (e.g. GPS
data)
VALUE

29
Data mining for BI

30
 Data mining steps

Pattern Evaluation

Data Mining

Task-relevant Data

Data Warehouse Selection

Data Cleaning

Data Integration

Databases
31
Main principles

• Select useful data

• Determine the format of the extracted
knowledge (rules, predicted values,
groupings etc)
• Captures the domain knowledge (e.g. in
conceptual hierarchies or ontologies)
• Determine metrics for the evaluation of
found patterns (simplicity, certainty, utility,
innovation)
• Visualized (interactive, abstract, choose
depending on the knowledge format)
32
Example - Bank

• Business objective: Give residential loans that

can be paid back
• Existing knowledge:
– Clients with children studying use these loans to pay
tuition fees
– Customers with variable income use these loans to
offset their income
• Lots of data:
– Large stores that continuously collect data
from multiple active sources (data warehouse)

33
Sampling

• Choose a part of Customer Data that have

been granted a loan in the past
– Some paid it back
– Others not

34
Pattern mining

• Find the rules that predict whether a customer will

be able to pay the loan back
IF (Salary < 40k) and
(numChildren > 0) and
(ageChild1 > 18 and ageChild1 < 22)
THEN YES
• Group customers and describe each group with
its predominant characteristics
– Among the many groups that have no special meaning,
we find a group of customers that take a loan using their
payroll or savings account

35
 Common architecture for DW and DM
Mining query Mining result Layer4
User Interface
User GUI API
Layer3
OLAM OLAP
Engine Engine OLAP/OLAM

Data Cube API

Layer2
MDDB
MDDB
Meta Data

Filtering&Integration Database API Filtering

Layer1
Data cleaning Data
Databases Data
Data integration Warehouse Repository
36
Discussion

• Briefly describe the following, emphasizing

on the data they contain, their differences
and their possible applications:
– object-relational databases,
– spatial databases,
– text databases,
– multimedia databases,
– World Wide Web

37
Discussion

• Describe the following data mining

concepts
– association and correlation analysis,
classification, prediction, clustering, data
evolution analysis, characterization,
discrimination.
• What are the scientific challenges from
data mining in:
– Data streams, spatio-temporal data,
bio-infromatics?

38
Data mining tasks (1)

• Characterization and Discrimination

– Generalize, summarize, compare and contrast features of my
data, e.g. collecting data from meteorological sensors, how
can I distinguish dry from wet areas of the country?
• Association, correlation vs causation
– Correlation does not imply causation. When ice cream sales increase,
the number of drowning increases too
• Classification and prediction
– Models (functions) that describe and distinguish classes or concepts
for future prediction
e.g. prediction of unknown or missing values Δημιουργία μοντέλων
(συναρτήσεων) που περιγράφουν και διακρίνουν κατηγορίες ή έννοιες
για μελλοντική πρόβλεψη
– Classification of countries based on climate

39
Data mining tasks (2)

• Cluster analysis
– Grouped samples in new unknown groups , e.g. group homes
which are for sale and study the characteristics of groups
– Aim to maximize the similarity within groups and the diversity
between groups
• Outlier analysis
– Exceptional samples have completely different behavior from
all other samples
– It is noise , error or exception?
• Trend analysis
– Trends and variations: e.g. regression analysis (finding a
function that describes the data, find data that deviate far from
this)
– Mining sequential patterns : e.g. searching for cameras 🡪
searching for a memory card
– periodicity analysis
40
Discussion

• Give an example of data mining

usefulness in a business (or a sector in
general) that you are familiar with
• Describe the data, knowledge to be
produced, domain knowledge, evaluation
measures standards, visualizations

41
Market analysis

• Where are the data:

– transactions with credit cards, coupons, customer
complaints, market research
• Targeted advertising
– find groups of customers with common characteristics:
interests, income, buying habits
– Define buyers patterns (in time)
• Cross market analysis: correlated products
(diapers, beer), bundle sales
• Analysis of customer needs
– What are the best products for each group of customers
– What factors attract new customers

42