Introduction to Data

Mining

Lecture 1: Definition, History, Importance, Applications

M. Usman Sarwar
Date: 18/03/2025
 What is Data Mining?

Definition: Data Mining is the process of discovering patterns,

correlations, and useful insights from large datasets.

Also known as Knowledge Discovery in Databases (KDD).

Utilizes techniques from statistics, machine learning, and

database management.
 1960s-1980s: Development of
databases and data management
systems.

History of 1990s: Evolution of machine

Data Mining learning and statistical techniques.

2000s-Present: Big Data, AI, and

cloud computing have
revolutionized data mining
applications.
 Importance of Data Mining

Used for fraud

Helps in decision-
detection, market
making and predictive
analysis, and risk
analytics.
management.

Enhances customer
Essential for AI-driven
relationship
applications.
management (CRM).
 Business: Market basket analysis, customer
segmentation

Healthcare: Disease prediction, drug discovery

Applications
of Data Finance: Fraud detection, credit scoring

Mining
Social Media: Sentiment analysis,
recommendation systems

Science: Astronomy, genomics

 KDD: The overall process of discovering
knowledge from data (includes data cleaning,
integration, selection, etc.)

Data Mining
vs. Knowledge Data Mining: A step in the KDD process
Discovery in focused on extracting patterns

Databases
KDD Steps:
Data Pattern Knowledge
Data Cleaning Data Integration Data Selection Data Mining
Transformation Evaluation Presentation
 KDD Process Example
• Dataset: Customer Purchase Behavior
Customer_ID Age Income Purchase_Amount Category
• KDD Steps and Example Output
1 25 30000 200 Electronics
o Selection → Extract relevant features (Age, Income,
2 40 50000 350 Clothing
Purchase_Amount)
o Preprocessing → Handle missing values, remove duplicates.
3 30 45000 120 Grocery o Transformation → Normalize Income and Purchase_Amount.
4 22 27000 400 Electronics o Data Mining → Apply clustering to find customer segments.
o Interpretation → Identify high-spending customer groups.
5 35 60000 150 Grocery
• Example Output(Clusters Identified):
o Cluster 1: Young, low-income, high spenders (Electronics)
o Cluster 2: Middle-aged, high-income, moderate spenders
(Grocery, Clothing)
M. Usman Sarwar(Experienced Data consultant) 9
 Data Mining vs. Machine Learning vs.
Statistics

Feature Data Mining Machine Learning Statistics

Goal Extract knowledge Learn patterns and Analyze and
from data make predictions summarize data

Approach Uses rules and Uses models and Uses probability

patterns algorithms and inference
Example Association rule Neural networks, Hypothesis testing
mining SVM
 Classification: Assigning labels to data (e.g., spam
detection)

Clustering: Grouping similar data (e.g., customer

segmentation)

Data Mining Association: Finding relationships between

Tasks variables (e.g., market basket analysis)

Outlier Detection: Identifying anomalies (e.g., fraud

detection)

Regression: Predicting continuous values (e.g.,

house prices)
 Data Quality: Missing, noisy, or inconsistent data

Scalability: Handling large datasets efficiently

Challenges in High Dimensionality: Curse of dimensionality

Data Mining
Privacy and Security: Protecting sensitive
information

Interpretability: Making results understandable to

users
 Privacy Concerns: Unauthorized use of personal
data

Bias and Fairness: Ensuring algorithms are

Ethical and unbiased

Privacy Issues Transparency: Making data mining processes

in Data understandable

Mining Regulations: GDPR, HIPAA, etc.

Case Study: Example of a data mining privacy

breach
Introduction to Data Mining Processes

• Data mining is a structured process, not just an algorithm.

• Two popular methodologies: CRISP-DM and SEMMA.
• Importance of following a structured approach for successful data
mining projects.
 What is CRISP-DM?
• A widely-used methodology for data
mining projects.
CRISP-DM: • Flexible and non-proprietary.
Cross-Industry
Phases of CRISP-DM:
Standard
• Business Understanding
Process for • Data Understanding
Data Mining • Data Preparation
• Modeling
• Evaluation
• Deployment
 Business Understanding: Define project goals and
requirements.

Data Understanding: Collect and explore data.

CRISP-DM Data Preparation: Clean, transform, and preprocess

data.
Phases in
Detail Modeling: Select and apply data mining techniques.

Evaluation: Assess model performance and results.

Deployment: Implement the model in the real world.

 CRISP-DM Example
• Business Understanding → Predict loan approval based on credit
Approved
Customer_ID Age Credit_Score Loan_Amount score, age, and loan amount.
(Y/N)
1 25 700 10000 Yes • Data Understanding → Analyze distributions of credit scores and
2 40 650 20000 No
loan approvals.

3 30 750 5000 Yes • Data Preparation → Handle missing values, scale numerical
4 22 620 15000 No
features.
5 35 720 12000 Yes • Modeling → Apply Decision Tree Classifier.
• Evaluation → Accuracy: 85%, Confusion Matrix:
• Deployment → Deploy model for loan approval automation.
• Example Output:
A decision rule from the model:
• If Credit_Score > 700 → Approve Loan.
• If Credit_Score < 650 → Reject Loan.
 • What is SEMMA?
o A methodology developed by SAS for
data mining.

SEMMA: Sample,
o Focuses on the technical aspects of data
mining.

Explore, Modify, • Steps in SEMMA:

o Sample: Extract a representative dataset.

Model, Assess o Explore: Analyze data for patterns and

anomalies.
o Modify: Preprocess and transform data.
o Model: Apply data mining algorithms.
o Assess: Evaluate model performance.
 SEMMA Example
• Dataset: Fraud Detection in Transactions
Fraudulent
Transaction_ID Amount Location Time_of_Day
(Y/N) • SEMMA Steps and Example Output
1001 500 New York Night No o Sample → Extract 5000 transactions for model training.
1002 1500 California Evening Yes
o Explore → Identify outliers in high-value transactions.
1003 200 Texas Morning No
o Modify → Create new features (e.g., suspicious
1004 2500 Florida Night Yes
transaction flag).
1005 700 Texas Afternoon No
o Model → Train Logistic Regression to predict fraud.
o Assess → Model Precision: 90%, Recall: 85%.
• Example Output:
Transactions flagged as fraudulent:
o Transaction 1002 (California, $1500, Evening)
o Transaction 1004 (Florida, $2500, Night)
 CRISP-DM vs. SEMMA

CRISP-DM: SEMMA: When to use which?

Focuses on both business and Focuses on the technical CRISP-DM for large, business-
technical aspects. process. driven projects.
More comprehensive and Easier to implement for smaller SEMMA for quick, technical-
widely adopted. projects. focused projects.
M. Usman Sarwar(Experienced Data consultant) 24
 Data mining tools help extract,
process, and analyze large
datasets.

Introduction They vary in capabilities, from

to Data data preprocessing to
visualization and model building.
Mining Tools

Choosing the right tool depends

on the specific task and dataset
requirements.
 Open-source tools: Free and community-
supported.

Commercial tools: Paid tools with

Categories of enterprise support.

Data Mining
Tools Programming-based tools: Require
coding knowledge.

GUI-based tools: User-friendly, drag-and-

drop interfaces.
Popular Open-Source Data Mining Tools

Java-based, user-friendly GUI.

WEKA (Waikato Environment
Supports machine learning, data
for Knowledge Analysis) preprocessing, and visualization.

No-code and low-code options.

RapidMiner Used for ETL (Extract, Transform, Load),
modeling, and evaluation.
 • Orange

Popular Open-
o Python-based, visual programming tool.
o Great for beginners and interactive

Source Data Mining

analysis.
• KNIME (Konstanz Information Miner)

Tools o Data integration and analytics platform.

o Used for big data processing and ML
workflows.
 Most popular for data
Python (Pandas, science and ML.

Scikit-learn, Extensive libraries for

preprocessing,
TensorFlow, PyTorch) visualization, and
modeling.

Popular Widely used for

Programming- R (caret, ggplot2, statistical analysis and

visualization.
Random Forest) Great for academic

Based Tools
and research use.

Used for querying

SQL (Structured large datasets.

Query Language) Essential for database-

driven data mining.
Commercial Data Mining Tools

Advanced analytics platform for enterprises.

SAS Enterprise Miner Suitable for large-scale data mining projects.

User-friendly interface with automated

IBM SPSS Modeler modeling.
Supports predictive analytics and ML.

Cloud-based machine learning services.

Microsoft Azure Machine
Provides automated ML and deep learning
Learning capabilities.
Big Data and Cloud-Based Data Mining Tools

Distributed computing framework for big data.

Apache Hadoop Handles large-scale data storage and processing.

Faster alternative to Hadoop for big data analytics.

Apache Spark Supports ML algorithms and streaming data
processing.

Cloud-based data warehouse for analytics.

Google BigQuery Suitable for real-time big data processing.
 • Considerations:

Choosing the Right o Data size and complexity.

o Ease of use and learning curve.
Tool o Community support and documentation.
o Integration with existing workflows.
 Data Mining extracts patterns
and insights from large datasets.

Summary It plays a crucial role in various

industries.

Ethical considerations are

essential for responsible data
mining.
 CRISP-DM and SEMMA are two popular
methodologies for data mining.

CRISP-DM is more comprehensive, while SEMMA is

technical-focused.

Tools like WEKA, RapidMiner, and Python libraries are

essential for data mining.
Summary(continued)
A wide range of tools are available for data mining,
from open-source to commercial solutions.

Choosing the right tool depends on project needs

and expertise level.

Cloud and big data solutions are gaining popularity

for large-scale applications.
 • https://oleg-dubetcky.medium.com/project-
management-for-data-science-kdd-semma-

Useful Links
and-crisp-dm-fe9d03d3ab6c
• https://www.geeksforgeeks.org/kdd-process-
in-data-mining/