Chapter 2.

Getting to
Know Your Data
HUI-YIN CHANG (張彙音)

1
Types of Data Sets
Record
◦ Relational records
◦ Data matrix, e.g., numerical matrix, crosstabs
◦ Document data: text documents: term-

timeout

season
coach

game
score
team

ball

lost
pla

wi
n
y
frequency vector
◦ Transaction data
Document 1 3 0 5 0 2 6 0 2 0 2
Graph and network
◦ World Wide Web Document 2 0 7 0 2 1 0 0 3 0 0
◦ Social or information networks
Document 3 0 1 0 0 1 2 2 0 3 0
◦ Molecular Structures

Ordered
TID Items
◦ Video data: sequence of images
◦ Temporal data: time-series 1 Bread, Coke, Milk
◦ Sequential Data: transaction sequences 2 Beer, Bread
◦ Genetic sequence data
3 Beer, Coke, Diaper, Milk
Spatial, image and multimedia 4 Beer, Bread, Diaper, Milk
◦ Spatial data: maps 5 Coke, Diaper, Milk
◦ Image data:
◦ Video data:
2
Data Objects
Data sets are made up of data objects.

A data object represents an entity.

◦ Examples:
◦ sales database: customers, store items, sales
◦ medical database: patients, treatments
◦ university database: students, professors, courses

Also called samples , examples, instances, data points, objects, tuples.

Data objects are described by attributes.

Database rows -> data objects; columns ->attributes.

3
Attributes
Attribute (or dimensions, features, variables): a data field,
representing a characteristic or feature of a data object.
◦ E.g., customer _ID, name, address
Types:
◦ Nominal
◦ Binary
◦ Numeric: quantitative
◦ Interval-scaled
◦ Ratio-scaled

4
Attribute Types
Nominal (詞性): categories, states, or “names of things”
◦ Hair_color = {auburn, black, blond, brown, grey, red, white}
◦ marital status, occupation, ID numbers, zip codes
Binary
◦ Nominal attribute with only 2 states (0 and 1)
◦ Symmetric binary: both outcomes equally important
◦ e.g., gender
◦ Asymmetric binary: outcomes not equally important.
◦ e.g., medical test (positive vs. negative)
◦ convention (習慣): assign 1 to most important outcome (e.g., HIV positive)
Ordinal
◦ Values have a meaningful order (ranking) but magnitude between successive values is not
known.
◦ Size = {small, medium, large}, grades, army rankings

5
Attribute Types: Numeric
Quantity (integer or real-valued)

Interval
◦ Measured on a scale of equal-sized units
◦ Can be compared and quantified the difference between values
◦ Values have order
◦ E.g., temperature in C˚or F˚, calendar dates
◦ No true zero-point

Ratio
◦ Inherent zero-point
◦ We can speak of values as being an order of magnitude larger than the unit of
measurement (10 K˚ is twice as high as 5 K˚).
◦ e.g., temperature in Kelvin, length, counts, monetary quantities

6
Discrete vs. Continuous Attributes
Discrete Attribute
◦ Has only a finite or countably infinite set of values
◦ E.g., zip codes, profession, or the set of words in a collection of documents

◦ Sometimes, represented as integer variables

◦ Note: Binary attributes are a special case of discrete attributes
Continuous Attribute
◦ Has real numbers as attribute values
◦ E.g., temperature, height, or weight

◦ Practically, real values can only be measured and represented

using a finite number of digits
◦ Continuous attributes are typically represented as floating-point
variables

7
Basic Statistical Descriptions of Data
Motivation
◦ To better understand the data: central tendency, variation and
spread
Data dispersion characteristics
◦ median, max, min, quantiles, outliers, variance, etc.

Numerical dimensions correspond to sorted intervals

◦ Data dispersion: analyzed with multiple granularities of
precision
◦ Boxplot or quantile analysis on sorted intervals

Dispersion analysis on computed measures

◦ Folding measures into numerical dimensions
◦ Boxplot or quantile analysis on the transformed cube

8
Measuring the Central Tendency
Mean (algebraic measure) (sample vs. population): 1 n
x =  xi =  x
Note: n is sample size and N is population size. N
n i =1
◦ Weighted arithmetic mean: n

◦ Trimmed mean: chopping extreme values w x i i

x= i =1
n

w
Median:
i
◦ Middle value if odd number of values, or average of the middle two i =1
values otherwise
◦ Estimated by interpolation (for grouped data):

𝑛/2 − (σ 𝑓𝑟𝑒𝑞)𝑙
𝑚𝑒𝑑𝑖𝑎𝑛 = 𝐿1 + ( )𝑤𝑖𝑑𝑡ℎ
𝑓𝑟𝑒𝑞𝑚𝑒𝑑𝑖𝑎𝑛
Mode
◦ Value that occurs most frequently in the data
◦ Unimodal, bimodal, trimodal
◦ Empirical formula: mean − mode = 3  (mean − median)
111-120 2

9
Symmetric vs. Skewed Data
symmetric
Median, mean and mode of symmetric,
positively and negatively skewed data

positively skewed negatively skewed

10
Measuring the Dispersion of Data
⚫ Range
⚫ Quantiles (分位數)
⚫ Quartiles (四分衛數)
⚫ Percentiles
⚫ Interquartile range

0th quartile = 0th quantile = 0th percentile = min負無窮遠點

1st quartile = .25th quantile = 25th percentile = lower quartile下四分位數（中點以下，又取中點）
2nd quartile = .5th quantile = 50th percentile = median中點/中位數
3rd quartile = .75th quantile = 75th percentile = upper quartile上四分位數（中點以上，又取中點）
4th quartile = 1st quantile = 100th percentile = max無窮遠點
(source: https://www.yamab2b.com/why/3499712.html)

11
Measuring the Dispersion of Data
Quartiles, outliers and boxplots
◦ Quartiles: Q1 (25th percentile), Q3 (75th percentile)
◦ Inter-quartile range: IQR = Q3 – Q1
◦ Five number summary: min, Q1, median, Q3, max
◦ Boxplot: ends of the box are the quartiles; median is marked; add whiskers, and plot
outliers individually
◦ Outlier: usually, a value higher/lower than 1.5 x IQR Question:
What’s the difference between
Variance and standard deviation (sample: s, population: σ) variance and standard deviation?
(變異數和標準差有何不同?)
◦ Variance: (algebraic, scalable computation)
1 n 1 n 2 1 n 2
 [ xi − ( xi ) ]
n n
1 1
s = ( xi − x ) =  =  −  =  xi −  2
2 2 2 2 2
( x )
n − 1 i =1 n − 1 i =1 n i =1 N i =1
i
N i =1

◦ Standard deviation s (or σ) is the square root of variance s2 (or σ2)

12
 Measuring the Dispersion of Data

Ex.
Height: 170, 170, 170
Mean: 170
Variation: 0

Height: 160, 170, 180

Mean: 170
Variation: 200/3

Source: https://www.scribbr.com/statistics/standard-deviation/ 13
Measuring the Dispersion of Data

Different standard deviation

14
Boxplot Analysis
Five-number summary of a distribution
◦ Minimum, Q1, Median, Q3, Maximum

Boxplot
◦ Data is represented with a box
◦ The ends of the box are at the first and third quartiles,
i.e., the height of the box is IQR
◦ The median is marked by a line within the box
◦ Whiskers: two lines outside the box extended to
Minimum and Maximum
◦ Outliers: points beyond a specified outlier threshold,
plotted individually

15
Visualization of Data Dispersion: 3-D Boxplots

16
Properties of Normal Distribution Curve

The normal (distribution) curve

◦ From μ–σ to μ+σ: contains about 68% of the measurements (μ:
mean, σ: standard deviation)
◦ From μ–2σ to μ+2σ: contains about 95% of it
◦ From μ–3σ to μ+3σ: contains about 99.7% of it

17
 Graphic Displays of Basic Statistical Descriptions
Boxplot: graphic display of five-number summary

Histogram: x-axis are values, y-axis represents frequencies

Quantile plot: each value xi is paired with fi indicating that approximately 100 fi
% of data are  xi

Quantile-quantile (q-q) plot: graphs the quantiles of one univariant distribution

against the corresponding quantiles of another

Scatter plot: each pair of values is a pair of coordinates and plotted as points in
the plane

18
Histogram Analysis
Histogram: Graph display of tabulated
frequencies, shown as bars 40

It shows what proportion of cases fall into 35

each of several categories 30

Differs from a bar chart in that it is the area25

of the bar that denotes the value, not the 20
height as in bar charts, a crucial distinction 15
when the categories are not of uniform
10
width
5
The categories are usually specified as non-
0
overlapping intervals of some variable. The 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000
categories (bars) must be adjacent

19
Quantile Plot
Displays all of the data (allowing the user to assess both the overall
behavior and unusual occurrences)
Plots quantile information
◦ For a data xi data sorted in increasing order, fi indicates that
approximately 100 fi% of the data are below or equal to the
value xi

20
 Example of Q-Q plot
Example points:
20, 23, 7, 1, 15, 29, 24, 13, 19, 12, 32, 6, 11, 18

Sort:
1, 6, 7, 11, 12, 13, 15, 18, 19, 20, 23, 24, 29, 32

Use 7 as an example:
-> 3/(14+1)
常態分布中,第 20% 名的數值是 -0.8416
繪製 x = -0.8416, y = 7

How about “24”?

-> 12/(14+1) = 0.8
常態分布中,第 80% 名的數值是0.8416212
qnorm(0.8)

Source: https://haosquare.com/normal-distribution-qqplot/ 21
 Example of Q-Q plot

Source: https://haosquare.com/normal-distribution-qqplot/ 22
 Example of Q-Q plot

Source: https://haosquare.com/normal-distribution-qqplot/ 23
Quantile-Quantile (Q-Q) Plot
Graphs the quantiles of one univariate distribution against the corresponding
quantiles of another
View: Is there is a shift in going from one distribution to another?
Example shows unit price of items sold at Branch 1 vs. Branch 2 for each quantile.
Unit prices of items sold at Branch 1 tend to be lower than those at Branch 2.

24
Scatter plot
Provides a first look at bivariate data to see clusters of points,
outliers, etc
Each pair of values is treated as a pair of coordinates and plotted
as points in the plane

25
Positively and Negatively Correlated Data

The left half fragment is positively

correlated

The right half is negative correlated

26
Uncorrelated Data

27
Similarity and Dissimilarity
Similarity
◦ Numerical measure of how alike two data objects are
◦ Value is higher when objects are more alike
◦ Often falls in the range [0,1]
Dissimilarity (e.g., distance)
◦ Numerical measure of how different two data objects are
◦ Lower when objects are more alike
◦ Minimum dissimilarity is often 0
◦ Upper limit varies
Proximity (接近) refers to a similarity or dissimilarity

28
 Data Matrix and Dissimilarity Matrix
Data matrix
◦ n data points with p  x11 ... x1f ... x1p 
dimensions  
 ... ... ... ... ... 
◦ Two modes x ... xif ... xip 
 i1 
 ... ... ... ... ... 
x ... xnf ... xnp 
 n1 
Dissimilarity matrix
 0 
◦ n data points, but registers  d(2,1) 
only the distance  0 
 d(3,1) d ( 3,2) 0 
◦ A triangular matrix  
◦ Single mode  : : : 
d ( n,1) d ( n,2) ... ... 0

29
Proximity Measure for Nominal Attributes

Can take 2 or more states, e.g., red, yellow, blue, green (generalization of a
binary attribute)

Method 1: Simple matching

◦ m: # of matches, p: total # of variables

d (i, j) = p −
p
m

Method 2: Use a large number of binary attributes

◦ creating a new binary attribute for each of the M nominal states

30
 Example for Nominal Attributes
Object Identifier Test-1 Test-2 Test-3
1 Code A Excellent 45
2 Code B Fair 22
3 Code C Good 64
4 Code A Excellent 28

Question:
- What is the attribute type of Test-1, Test-2, and Test-3?
- Compute the dissimilarity of objects based on Test-1

𝑝−𝑚 1−0 0
𝑠𝑖𝑚 𝑖, 𝑗
𝑑 𝑖, 𝑗 = 𝑑 2,1 = 1 0 𝑚
𝑝 1 = 1 − 𝑑 𝑖, 𝑗 =
1 1 0 𝑝
, where m is the number of 0 1 1 0
matches, and p is the total number
of attributes describing the objects.
31
Proximity Measure for Binary Attributes
Object j
A contingency table for binary data
Object i

Distance measure for symmetric binary

variables:

Distance measure for asymmetric binary

variables:

Jaccard coefficient (similarity measure for

asymmetric binary variables):

◼ Note: Jaccard coefficient is the same as “coherence”:

32
Dissimilarity between Binary Variables
Example
Name Gender Fever Cough Test-1 Test-2 Test-3 Test-4
Jack M Y 1 N 0 P 1 N 0 N 0 N 0
Mary F Y 1 N 0 P 1 N 0 P 1 N 0
Jim M Y 1 P 1 N 0 N 0 N 0 N 0

◦ Gender is a symmetric attribute Jack

◦ The remaining attributes are asymmetric binary
◦ Let the values Y and P be 1, and the value N 0 1 0

0+1 1 1 1
d ( jack , mary ) = = 0.33 Jim
2+ 0+1
1+1 0 1 3
d ( jack , jim ) = = 0.67
1+1+1
1+ 2
d ( jim , mary ) = = 0.75
1+1+ 2

33
Standardizing Numeric Data
x
z=  − 
Z-score:
◦ X: raw score to be standardized, μ: mean of the population, σ: standard
deviation
◦ the distance between the raw score and the population mean in units of the
standard deviation
◦ negative when the raw score is below the mean, “+” when above

An alternative way: Calculate the mean absolute deviation

s f = 1n (| x1 f − m f | + | x2 f − m f | +...+ | xnf − m f |)

mf = 1 xif − m f
n (x1 f + x2 f + ... + xnf )
where
zif =
.

sf
◦ standardized measure (z-score):

Using mean absolute deviation is more robust than using standard deviation
34
Example:
Data Matrix and Dissimilarity Matrix

Data Matrix
point attribute1 attribute2
x1 1 2
x2 3 5
x3 2 0
x4 4 5

Dissimilarity Matrix
(with Euclidean Distance)
x1 x2 x3 x4
x1 0
x2 3.61 0
x3 5.1 5.1 0
x4 4.24 1 5.39 0

35
Distance on Numeric Data: Minkowski Distance
Minkowski distance: A popular distance measure

where i = (xi1, xi2, …, xip) and j = (xj1, xj2, …, xjp) are two p-
dimensional data objects, and h is the order (the distance so
defined is also called L-h norm)
Properties
◦ d(i, j) > 0 if i ≠ j, and d(i, i) = 0 (Positive definiteness)
◦ d(i, j) = d(j, i) (Symmetry)
◦ d(i, j)  d(i, k) + d(k, j) (Triangle Inequality)
A distance that satisfies these properties is a metric (公制)

36
Special Cases of Minkowski Distance
h = 1: Manhattan (city block, L1 norm) distance
◦ E.g., the Hamming distance: the number of bits that are different
between two binary vectors

d (i, j) =| x − x | + | x − x | +...+ | x − x |
i1 j1 i2 j 2 ip jp

h = 2: (L2 norm) Euclidean distance

d (i, j) = (| x − x |2 + | x − x |2 +...+ | x − x |2 )
i1 j1 i2 j 2 ip jp

h → . “supremum” (Lmax norm, L norm) distance.

◦ This is the maximum difference between any component (attribute)
of the vectors

37
Example: Minkowski Distance
Dissimilarity Matrices
point attribute 1 attribute 2 Manhattan (L1)
x1 1 2
L x1 x2 x3 x4
x2 3 5 x1 0
x3 2 0 x2 5 0
x4 4 5 x3 3 6 0
x4 6 1 7 0
Euclidean (L2)
L2 x1 x2 x3 x4
x1 0
x2 3.61 0
x3 2.24 5.1 0
x4 4.24 1 5.39 0

Supremum
L x1 x2 x3 x4
x1 0
x2 3 0
x3 2 5 0
x4 3 1 5 0
38
 Ordinal Variables

An ordinal variable can be discrete or continuous

Order is important, e.g., rank
Can be treated like interval-scaled
◦ replace xif by their rank rif {1,..., M f }
◦ map the range of each variable onto [0, 1] by replacing i-th
object in the f-th variable by
rif −1
zif =
M f −1
◦ compute the dissimilarity using methods for interval-scaled
variables

39
 Example for Ordinal Type
Object Identifier Test-1 Test-2 Convert to rank Test-3
1 Code A Excellent 3 -> 1 45
2 Code B Fair 1 -> 0 22
3 Code C Good 2 -> 0.5 64
4 Code A Excellent 3 -> 1 28

Compute the dissimilarity of objects based on Test-2

0
𝑟𝑖𝑓 − 1 3−1 𝑠𝑖𝑚 𝑖, 𝑗
𝑧𝑖𝑓 = 𝑧1 = d(i,j)= 1 0 = 1 − 𝑑 𝑖, 𝑗
𝑀𝑓 − 1 3−1 0.5 0.5 0
0 1.0 0.5 0
, where 𝑟𝑖𝑓 is the rank of the ith object in the
fth attribute, and 𝑀𝑓 is the number of possible
states that an ordinal attribute can have.

40
Attributes of Mixed Type
A database may contain all attribute types
◦ Nominal, symmetric binary, asymmetric binary, numeric, ordinal
One may use a weighted formula to combine their effects

 pf = 1 ij( f ) dij( f )
d (i, j) =
 pf = 1 ij( f )
◦ f is binary or nominal:
dij(f) = 0 if xif = xjf , or dij(f) = 1 otherwise

◦ f is numeric: use the normalized distance

◦ f is ordinal
◦ Compute ranks rif and
zif =
r −1
if
◦ Treat zif as interval-scaled
M −1 f

41
 Example for Mixed Type
Object Identifier Test-1 Test-2 Convert to rank Test-3 Normalization
1 Code A Excellent 3 45
2 Code B Fair 1 22
3 Code C Good 2 64
4 Code A Excellent 3 28

Compute the dissimilarity of objects based on Test-2

0 0 0
𝑑 𝑇𝑒𝑠𝑡−1 𝑖, 𝑗 = 1 0 𝑑 𝑇𝑒𝑠𝑡−2 𝑖, 𝑗 = 1 0 𝑑 𝑇𝑒𝑠𝑡−3 𝑖, 𝑗 = 0.55 0
1 1 0 0.5 0.5 0 0.45 1.00 0
0 1 1 0 0 1.0 0.5 0 0.40 0.14 0.86 0

(𝑓) |𝑥𝑖𝑓 − 𝑥𝑗𝑓 | |22 − 45|

𝑑𝑖𝑗 = 𝑑 2,1 = ~0.55
𝑚𝑎𝑥ℎ 𝑥ℎ𝑓 − 𝑚𝑖𝑛ℎ 𝑥ℎ𝑓 64 − 22
42
 Example for Mixed Type
0 0 0
𝑑 𝑇𝑒𝑠𝑡−1 𝑖, 𝑗 = 1 0 𝑑 𝑇𝑒𝑠𝑡−2 𝑖, 𝑗 = 1 0 𝑑 𝑇𝑒𝑠𝑡−3 𝑖, 𝑗 = 0.55 0
1 1 0 0.5 0.5 0 0.45 1.00 0
0 1 1 0 0 1.0 0.5 0 0.40 0.14 0.86 0

1 1 +1 1 +1 0.55
𝑑 2,1 = = 0.85
3
(𝑓) (𝑓)
σ𝑝𝑓=1 𝛿𝑖𝑗 𝑑𝑖𝑗 1 1 + 1 0.5 + 1 0.45
𝑑(𝑖, 𝑗) = (𝑓) 𝑑 3,1 = = 0.65
σ𝑝𝑓=1 𝛿𝑖𝑗 3
1 0 + 1 0 + 1 0.40
𝑑 4,1 = = 0.1333
3
(𝑓)
, where the indicator 𝛿𝑖𝑗 = 0 if either (1) 𝑥𝑖𝑓 or 𝑥𝑗𝑓 is missing or (2) 𝑥𝑖𝑓 = 𝑥𝑗𝑓 = 0
(𝑓)
and attribute f is asymmetric binary; otherwise, 𝛿𝑖𝑗 = 1

43
Cosine Similarity
A document can be represented by thousands of attributes, each recording the
frequency of a particular word (such as keywords) or phrase in the document.

Other vector objects: gene features in micro-arrays, …

Applications: information retrieval, biologic taxonomy, gene feature mapping, ...
Cosine measure: If d1 and d2 are two vectors (e.g., term-frequency vectors), then
cos(d1, d2) = (d1 • d2) /||d1|| ||d2|| ,
where • indicates vector dot product, ||d||: the length of vector d

44
Example: Cosine Similarity
cos(d1, d2) = (d1 • d2) /||d1|| ||d2|| ,
where • indicates vector dot product, ||d|: the length of vector d

Ex: Find the similarity between documents 1 and 2.

d1 = (5, 0, 3, 0, 2, 0, 0, 2, 0, 0)
d2 = (3, 0, 2, 0, 1, 1, 0, 1, 0, 1)

d1•d2 = 5*3+0*0+3*2+0*0+2*1+0*1+0*1+2*1+0*0+0*1 = 25
||d1||= (5*5+0*0+3*3+0*0+2*2+0*0+0*0+2*2+0*0+0*0)0.5=(42)0.5 = 6.481
||d2||= (3*3+0*0+2*2+0*0+1*1+1*1+0*0+1*1+0*0+1*1)0.5=(17)0.5 = 4.12
cos(d1, d2 ) = 0.94

45
Summary
Data attribute types: nominal, binary, ordinal, interval-scaled, ratio-scaled

Many types of data sets, e.g., numerical, text, graph, Web, image.

Gain insight into the data by:

◦ Basic statistical data description: central tendency, dispersion, graphical
displays
◦ Data visualization: map data onto graphical primitives
◦ Measure data similarity

Above steps are the beginning of data preprocessing.

Many methods have been developed but still an active area of research.

46
Thanks for Your Attention
Q&A

47