SHAIK BILAL AHMED +91-8143786956

Introduction Data Mining Tasks Classification & Evaluation Clustering Application Examples

More data is generated:

Web, text, images Business transactions, calls, ... Scientific data: astronomy, biology, etc

More data is captured:

Storage technology faster and cheaper DBMS can handle bigger DB

Winter Corp. 2005 Commercial Database Survey:

1. 2. 3.

Max Planck Inst. for Meteorology , 222 TB Yahoo ~ 100 TB (Largest Data Warehouse) AT&T ~ 94 TB

www.wintercorp.com/VLDB/2005_TopTen_Survey/TopTenWinners_2005.asp

In 2 years (2003 to 2005), the size of the largest database TRIPLED!

Twice as much information was created in 2002 as in 1999 (~30% growth rate) Other growth rate estimates even higher Very little data will ever be looked at by a human

Knowledge Discovery is NEEDED to make sense and use of data.

Knowledge Discovery in Data is the non-trivial process of identifying

valid novel potentially useful and ultimately understandable patterns in data. from Advances in Knowledge Discovery and Data Mining, Fayyad, Piatetsky-Shapiro, Smyth, and Uthurusamy, (Chapter 1), AAAI/MIT Press 1996

Machine Learning

Visualization

Data Mining and Knowledge Discovery

Statistics

Databases

Statistics:

Machine learning

more theory-based more focused on testing hypotheses more heuristic focused on improving performance of a learning agent also looks at real-time learning and robotics areas not part of data mining integrates theory and heuristics focus on the entire process of knowledge discovery, including data cleaning, learning, and integration and visualization of results

Data Mining and Knowledge Discovery

Distinctions are fuzzy

Monitoring

see www.crisp-dm.org for more information

Continuous monitoring and improvement is an addition to CRISP

10

Data Fishing, Data Dredging: 1960

used by statisticians (as bad name) used in DB community, business

Data Mining :1990 -

Knowledge Discovery in Databases (1989-)

used by AI, Machine Learning Community also Data Archaeology, Information Harvesting, Information Discovery, Knowledge Extraction, ...

Currently: Data Mining and Knowledge Discovery are used interchangeably

11

Instance (also Item or Record):

an example, described by a number of attributes, e.g. a day can be described by temperature, humidity and cloud status
measuring aspects of the Instance, e.g. temperature

Attribute or Field

Class (Label)

grouping of instances, e.g. days good for playing

13

Classification: predicting an item class Clustering: finding clusters in data Associations: e.g. A & B & C occur frequently Visualization: to facilitate human discovery Summarization: describing a group
Deviation Detection: finding changes Estimation: predicting a continuous value Link Analysis: finding relationships

14

Learn a method for predicting the instance class from pre-labeled (classified) instances Many approaches: Statistics, Decision Trees, Neural Networks, ...

15

Find natural grouping of instances given un-labeled data

16

Transactions
TID Produce 1 MILK, BREAD, EGGS 2 BREAD, SUGAR 3 BREAD, CEREAL 4 MILK, BREAD, SUGAR 5 MILK, CEREAL 6 BREAD, CEREAL 7 MILK, CEREAL 8 MILK, BREAD, CEREAL, EGGS 9 MILK, BREAD, CEREAL

Frequent Itemsets: Milk, Bread (4) Bread, Cereal (3) Milk, Bread, Cereal (2)

Rules: Milk => Bread (66%)

17

Visualizing the data to facilitate human discovery

Presenting the discovered results in a visually "nice" way

18

 Describe features of the selected group Use natural language and graphics Usually in Combination with Deviation detection or other methods

Average length of stay in this study area rose 45.7 percent, from 4.3 days to 6.2 days, because ...
19

Find true patterns and avoid overfitting

(finding seemingly signifcant but really random patterns due to searching too many possibilites)
20

Learn a method for predicting the instance class from pre-labeled (classified) instances Many approaches: Regression, Decision Trees, Bayesian, Neural Networks, ...

Given a set of points from classes what is the class of new point ?
22

 Linear Regression
w0 + w1 x + w2 y >= 0

Regression computes wi from data to minimize squared error to fit the data

Not flexible enough

23

Any regression technique can be used for classification

Training: perform a regression for each class, setting the output to 1 for training instances that belong to class, and 0 for those that dont Prediction: predict class corresponding to model with largest output value (membership value)

For linear regression this is known as multi-response linear regression

24

if X > 5 then blue else if Y > 3 then blue else if X > 2 then green else blue

25

An internal node is a test on an attribute. A branch represents an outcome of the test, e.g., Color=red. A leaf node represents a class label or class label distribution. At each node, one attribute is chosen to split training examples into distinct classes as much as possible A new instance is classified by following a matching path to a leaf node.

26

Outlook sunny sunny overcast rain rain

Temperature hot hot hot mild cool

Humidity high high high high normal

Windy false true false false false

Play? No No Yes Yes Yes

rain
overcast sunny sunny

cool
cool mild cool

normal
normal high normal

true
true false false

No
Yes No Yes

Note: Outlook is the Forecast, no relation to Microsoft email program

rain
sunny overcast overcast rain

mild
mild mild hot mild

normal
normal high normal high

false
true true false true

Yes
Yes Yes Yes No
27

Outlook
sunny overcast Humidity high normal Yes rain

Windy true false

No

Yes

No

Yes
28

 Can select more complex regions Can be more accurate Also can overfit the data find patterns in random noise

29

Nave Bayes Rules Support Vector Machines Genetic Algorithms See www.KDnuggets.com/software/

30

No model is uniformly the best Dimensions for Comparison

speed of training speed of model application noise tolerance explanation ability

Best Results: Hybrid, Integrated models

32

Train Run Noise Can Use time Time Toler Prior ance Knowledge Decision fast fast poor no Trees Rules med fast poor no Neural slow Networks Bayesian slow fast fast good no good yes

Accuracy Underon Customer standable Modelling medium medium good good medium good poor good

A hybrid method will have higher accuracy

33

How predictive is the model we learned? Error on the training data is not a good indicator of performance on future data

The new data will probably not be exactly the same as the training data!

Overfitting fitting the training data too precisely - usually leads to poor results on new data

34

Possible evaluation measures:

Classification Accuracy Total cost/benefit when different errors involve different costs Lift and ROC curves Error in numeric predictions

How reliable are the predicted results ?

35

Natural performance measure for classification problems: error rate

Success: instances class is predicted correctly Error: instances class is predicted incorrectly Error rate: proportion of errors made over the whole set of instances you can find patterns even in random data

Training set error rate: is way too optimistic!

36

If many (>1000) examples are available, including >100 examples from each class

A simple evaluation will give useful results

Randomly split data into training and test sets (usually 2/3 for train, 1/3 for test)

Build a classifier using the train set and evaluate it using the test set

37

THE PAST Results Known

+ + +

Training set

Data

Testing set

38

THE PAST Results Known

+ + +

Training set

Data

Model Builder

Testing set

39

Results Known
+ + +

Training set

Data

Model Builder
Evaluate
Predictions
Y N

Testing set

+ + -

40

Sometimes, classes have very unequal frequency

Attrition prediction: 97% stay, 3% attrite (in a month) medical diagnosis: 90% healthy, 10% disease eCommerce: 99% dont buy, 1% buy Security: >99.99% of Americans are not terrorists

Similar situation with multiple classes Majority class classifier can be 97% correct, but useless

41

If we have two classes that are very unbalanced, then how can we evaluate our classifier method?

42

With two classes, a good approach is to build BALANCED train and test sets, and train model on a balanced set

randomly select desired number of minority class instances add equal number of randomly selected majority class

How do we generalize balancing to multiple classes?

43

Generalize balancing to multiple classes

Ensure that each class is represented with approximately equal proportions in train and test

44

It is important that the test data is not used in any way to create the classifier Some learning schemes operate in two stages:

Stage 1: builds the basic structure Stage 2: optimizes parameter settings

The test data cant be used for parameter tuning! Proper procedure uses three sets: training data, validation data, and test data

Validation data is used to optimize parameters

45

Once evaluation is complete, all the data can be used to build the final classifier Generally, the larger the training data the better the classifier (but returns diminish) The larger the test data the more accurate the error estimate

46

Results Known
+ + +

Training set

Model Builder

Data

Evaluate

Model Builder

Predictions
+ + + - Final Evaluation + 47

Validation set

Final Test Set

Final Model

Cross-validation avoids overlapping test sets

First step: data is split into k subsets of equal size Second step: each subset in turn is used for testing and the remainder for training

This is called k-fold cross-validation Often the subsets are stratified before the crossvalidation is performed The error estimates are averaged to yield an overall error estimate

48

Cross-validation example:
Break up data into groups of the same size Hold aside one group for testing and use the rest to build model

Test

Repeat

49 49

Standard method for evaluation: stratified tenfold cross-validation Why ten? Extensive experiments have shown that this is the best choice to get an accurate estimate Stratification reduces the estimates variance Even better: repeated stratified cross-validation

E.g. ten-fold cross-validation is repeated ten times and results are averaged (reduces the variance)

50

Find most likely prospects to contact Not everybody needs to be contacted Number of targets is usually much smaller than number of prospects Typical Applications

retailers, catalogues, direct mail (and e-mail) customer acquisition, cross-sell, attrition prediction ...

51

Accuracy on the entire dataset is not the right measure Approach

develop a target model score all prospects and rank them by decreasing score select top P% of prospects for action

How do we decide what is the best subset of prospects ?

52

Use a model to assign score to each customer Sort customers by decreasing score Expect more targets (hits) near the top of the list No 1 Score Target CustID Age 0.97 Y 1746 3 hits in top 5% of the list

2 3 4 5
99 100

0.95 0.94 0.93 0.92

0.11 0.06

N Y Y N
N N

1024 2478 3820 4897

2734 2422
53

If there 15 targets overall, then top 5 has 3/15=20% of targets

Definition: CPH(P,M) = % of all targets in the first P% of the list scored by model M CPH frequently called Gains

100 90 80 70 60 50 40 30 20 10 0
5 15 25 35 45 55 65 75 85 95

5% of random list have 5% of targets

Cumulative % Hits

Random

Pct list

54

100 90 80 70 60 50 40 30 20 10 0
5 15 25 35 45 55 65 75 85 95

5% of random list have 5% of targets,

but 5% of model ranked list have 21% of targets CPH(5%,model)=21%.

55

Cumulative % Hits

Random Model

Pct list

Lift(P,M) = CPH(P,M) / P
Lift (at 5%)
= 21% / 5% = 4.2 better than random
4.5 4 3.5 3 2.5 2 1.5 1 Lift

Note: Some authors 0.5 use Lift for what 0 we call CPH.

15

25

35

45

55

65

75

85

P -- percent of the list

56

95

Lift helps us decide which models are better If cost/benefit values are not available or changing, we can use Lift to select a better model. Model with the higher Lift curve will generally be better

57

Unsupervised learning: Finds natural grouping of instances given un-labeled data

59

Many different method and algorithms:

For numeric and/or symbolic data Deterministic vs. probabilistic Exclusive vs. overlapping Hierarchical vs. flat Top-down vs. bottom-up

60

Manual inspection Benchmarking on existing labels Cluster quality measures

distance measures high similarity within a cluster, low across clusters

61

Simplest case: one numeric attribute A

Distance(X,Y) = A(X) A(Y)

Distance(X,Y) = Euclidean distance between X,Y

Several numeric attributes:

Nominal attributes: distance is set to 1 if values are different, 0 if they are equal Are all attributes equally important?

Weighting the attributes might be necessary

62

Works with numeric data only 1) Pick a number (K) of cluster centers (at random) 2) Assign every item to its nearest cluster center (e.g. using Euclidean distance) 3) Move each cluster center to the mean of its assigned items 4) Repeat steps 2,3 until convergence (change in cluster assignments less than a threshold)

63

1
c1

Y Pick 3 initial cluster centers (randomly) c2

c3

X
64

c1

Y
Assign each point to the closest cluster center

c2

c3

X
65

c1

c1

Y Move each cluster center to the mean of each cluster c2

c2 c3 c3

X
66

Reassign points Y closest to a different new cluster center Q: Which points are reassigned? c2

c1

c3

X
67

Reassign points Y closest to a different new cluster center Q: Which points are reassigned? c2

c1

c3

X
68

1 c1

Y A: three points with animation

c2 3 2 c3

X
69

c1

Y re-compute cluster means

c2

c3

X
70

c1

move cluster centers to cluster means

c2 c3

X
71

require knowledge-based decisions have a changing environment have sub-optimal current methods have accessible, sufficient, and relevant data provides high payoff for the right decisions!

73

Advertising Bioinformatics Customer Relationship Management (CRM) Database Marketing Fraud Detection eCommerce Health Care Investment/Securities Manufacturing, Process Control Sports and Entertainment Telecommunications Web

74

Before Google, web search engines used mainly keywords on a page results were easily subject to manipulation Google's early success was partly due to its algorithm which uses mainly links to the page Google founders Sergey Brin and Larry Page were students at Stanford in 1990s Their research in databases and data mining led to Google
75

Leukemia: Acute Lymphoblastic (ALL) vs Acute Myeloid (AML), Golub et al, Science, v.286, 1999

72 examples (38 train, 34 test), about 7,000 genes AML

ALL

Visually similar, but genetically very different

Best Model: 97% accuracy, 1 error (sample suspected mislabelled)

76

New and better molecular diagnostics

Jan 11, 2005: FDA approved Roche Diagnostic AmpliChip, based on Affymetrix technology
few new drugs, large pipeline, Partially depends on genetic signature finding and refining biological pathways

New molecular targets for therapy

Improved treatment outcome

Fundamental Biological Discovery

Personalized medicine ?!

77

Most major direct marketing companies are using modeling and data mining Most financial companies are using customer modeling Modeling is easier than changing customer behaviour Example

Verizon Wireless reduced customer attrition rate from 2% to 1.5%, saving many millions of $

78

Amazon.com recommendations

if you bought (viewed) X, you are likely to buy Y

Netflix
If you liked "Monty Python and the Holy Grail", you get a recommendation for "This is Spinal Tap"

Comparison shopping

Froogle, mySimon, Yahoo Shopping,

79

Credit Card Fraud Detection

over 20 Million credit cards protected by Neural networks (Fair, Isaac)

Securities Fraud Detection

NASDAQ KDD system

AT&T, Bell Atlantic, British Telecom/MCI

Phone fraud detection

80

TIA: Terrorism (formerly Total) Information Awareness Program

TIA program closed by Congress in 2003 because of privacy concerns

However, in 2006 we learn that NSA is analyzing US domestic call info to find potential terrorists

Invasion of Privacy or Needed Intelligence?

81

Data Mining will be ineffective - generate millions of false positives and invade privacy First, can data mining be effective?

82

Criticism: Databases have 5% errors, so analyzing 100 million suspects will generate 5 million false positives Reality: Analytical models correlate many items of information to reduce false positives. Example: Identify one biased coin from 1,000.
After one throw of each coin, we cannot After 30 throws, one biased coin will stand out with high probability. Can identify 19 biased coins out of 100 million with sufficient number of throws

83

Can find unusual patterns in the network structure

84

Data Mining is just one additional tool to help analysts Combining multiple models and link analysis can reduce false positives Today there are millions of false positives with manual analysis Analytic technology has the potential to reduce the current high rate of false positives

85

Data Mining looks for patterns, not people! Technical solutions can limit privacy invasion
Replacing sensitive personal data with anon. ID Give randomized outputs Multi-party computation distributed data

Bayardo & Srikant, Technological Solutions for Protecting Privacy, IEEE Computer, Sep 2003

86

Over-inflated expectations Growing acceptance and mainstreaming

rising expectations

Disappointment
1990 1998

Performance Expectations

2000

2002

2005
87

Data Mining and Knowledge Discovery are needed to deal with the flood of data Knowledge Discovery is a process ! Avoid overfitting (finding random patterns by searching too many possibilities)

88