Lecture #2: Data

aka STAT121A, AC209A, CSCIE-109A

CS109A Introduction to Data Science

Pavlos Protopapas, Kevin Rader and Chris Tanner

1
Lecture Outline: Data, Summaries, and Visuals

• What are Data?

• Data Exploration
• Descriptive Statistics
• Visualizations

• An Example

Reading: Ch. 1 in An Introduction to Statistical Learning (ISLR)

CS109A, PROTOPAPAS, RADER, TANNER 2

What are Data?

CS109A, PROTOPAPAS, RADER, TANNER 3

The Data Science Process
Recall the data science process.
Ask an interesting question

Get the Data

Explore the Data

Model the Data

Communicate/Visualize the
Results
Today we will begin introducing the data collection and data
exploration steps.

CS109A, PROTOPAPAS, RADER, TANNER 4

What are data?

“A datum is a single measurement of something on a scale that is

understandable to both the recorder and the reader. Data are multiple
such measurements.”
Claim: everything is (can be) data!

CS109A, PROTOPAPAS, RADER, TANNER 5

Where do data come from?
• Internal sources: already collected by or is part of the overall data
collection of you organization.
For example: business-centric data that is available in the organization
data base to record day to day operations; scientific or experimental
data.
• Existing External Sources: available in ready to read format from an
outside source for free or for a fee.
For example: public government databases, stock market data, Yelp
reviews, [your favorite sport]-reference.
• External Sources Requiring Collection Efforts: available from
external source but acquisition requires special processing.
For example: data appearing only in print form, or data on websites.

CS109A, PROTOPAPAS, RADER, TANNER 6

Ways to gather online data
How to get data generated, published or hosted online:
• API (Application Programming Interface): using a prebuilt set
of functions developed by a company to access their services.
Often pay to use. For example: Google Map API, Facebook API,
Twitter API
• RSS (Rich Site Summary): summarizes frequently updated online
content in standard format. Free to read if the site has one. For
example: news-related sites, blogs
• Web scraping: using software, scripts or by-hand extracting data
from what is displayed on a page or what is contained in the
HTML file (often in tables).

CS109A, PROTOPAPAS, RADER, TANNER 7

Web scraping
• Why do it? Older government or smaller news sites might not have
APIs for accessing data, or publish RSS feeds or have databases for
download. Or, you don’t want to pay to use the API or the database.
• How do you do it? See HW1 (beautifulsoup)
• Should you do it?
– You just want to explore: Are you violating their terms of service?
Privacy concerns for website and their clients?
– You want to publish your analysis or product: Do they have an API or
fee that you are bypassing? Are they willing to share this data? Are
you violating their terms of service? Are there privacy concerns?

CS109A, PROTOPAPAS, RADER, TANNER 8

Types of data
What kind of values are in your data (data types)?

Simple or atomic:
• Numeric: integers, floats
• Boolean: binary or true false values
• Strings: sequence of symbols

CS109A, PROTOPAPAS, RADER, TANNER 9

Data types
What kind of values are in your data (data types)? Compound,
composed of a bunch of atomic types:
• Date and time: compound value with a specific structure
• Lists: a list is a sequence of values
• Dictionaries: A dictionary is a collection of key-value pairs, a pair
of values x : y where x is usually a string called the key
representing the “name” of the entry, and y is a value of any type.
Example: Student record: what are x and y?
• First: Kevin
• Last: Rader
• Classes: [CS-109A, STAT139]
CS109A, PROTOPAPAS, RADER, TANNER 10
Data storage
How is your data represented and stored (data format)?
• Tabular Data: a dataset that is a two-dimensional table, where
each row typically represents a single data record, and each column
represents one type of measurement (csv, dat, xlsx, etc.).
• Structured Data: each data record is presented in a form of a
[possibly complex and multi-tiered] dictionary (json, xml, etc.)
• Semistructured Data: not all records are represented by the same
set of keys or some data records are not represented using the key-
value pair structure.

CS109A, PROTOPAPAS, RADER, TANNER 11

Tabular Data
In tabular data, we expect each record or observation to represent a set of
measurements of a single object or event. We’ve ‘seen’ this already in Lecture 1:

CS109A, PROTOPAPAS, RADER, TANNER 12

Tabular Data

Each type of measurement is called a variable or an attribute of the data (e.g.

seq_id, status and duration are variables or attributes). The number of
attributes is called the dimension. These are often called features.
We expect each table to contain a set of records or observations of the same kind
of object or event (e.g. our table above contains observations of rides/checkouts).
CS109A, PROTOPAPAS, RADER, TANNER 13
Types of Data

We’ll see later that it’s important to distinguish between classes of variables
or attributes based on the type of values they can take on.
• Quantitative variable: is numerical and can be either:
• discrete - a finite number of values are possible in any bounded
interval. For example: “Number of siblings” is a discrete variable
• continuous - an infinite number of values are possible in any
bounded interval. For example: “Height” is a continuous variable
• Categorical variable: no inherent order among the values For example:
“What kind of pet you have” is a categorical variable

CS109A, PROTOPAPAS, RADER, TANNER 14

Common Issues

Common issues with data:

• Missing values: how do we fill in?
• Wrong values: how can we detect and correct?
• Messy format
• Not usable: the data cannot answer the question posed

CS109A, PROTOPAPAS, RADER, TANNER 15

Messy Data

The following is a table accounting for the number of produce

deliveries over a weekend.
What are the variables in this dataset? What object or event are we
measuring?

What’s the issue? How do we fix it?

CS109A, PROTOPAPAS, RADER, TANNER 16

Messy Data

We’re measuring individual deliveries; the variables are Time, Day,

Number of Produce.

Problem: each column header represents a single value rather than a

variable. Row headers are “hiding” the Day variable. The values of the
variable, “Number of Produce”, is not recorded in a single column.

CS109A, PROTOPAPAS, RADER, TANNER 17

Fixing Messy Data

We need to reorganize the information to make explicit the event we’re

observing and the variables associated to this event.

CS109A, PROTOPAPAS, RADER, TANNER 18

Tabular = Happy Pavlos 

Common causes of messiness are:

• Column headers are values, not variable names
• Variables are stored in both rows and columns
• Multiple variables are stored in one column/entry
• Multiple types of experimental units stored in same table
In general, we want each file to correspond to a dataset, each column
to represent a single variable and each row to represent a single
observation.
We want to tabularize the data. This makes Python happy.

CS109A, PROTOPAPAS, RADER, TANNER 19

Data Exploration: Descriptive Statistics

CS109A, PROTOPAPAS, RADER, TANNER 20

Basics of Sampling
Population versus sample:
• A population is the entire set of objects or events under study.
Population can be hypothetical “all students” or all students in this class.
• A sample is a “representative” subset of the objects or events under
study. Needed because it’s impossible or intractable to obtain or compute
with population data.
Biases in samples:
• Selection bias: some subjects or records are more likely to be selected
• Volunteer/nonresponse bias: subjects or records who are not easily
available are not represented
Examples?

CS109A, PROTOPAPAS, RADER, TANNER 21

Sample mean

The mean of a set of n observations of a variable is denoted and is defined

as: 𝑥 +𝑥 +…+𝑥 𝑛
1
= ∑ 𝑥𝑖
1 2 𝑛
𝑥=
𝑛 𝑛 𝑖=1

The mean describes what a “typical” sample value looks like, or where is
the “center” of the distribution of the data.
Key theme: there is always uncertainty involved when calculating a sample
mean to estimate a population mean.
CS109A, PROTOPAPAS, RADER, TANNER 22
Sample median

The median of a set of n number of observations in a sample, ordered by

value, of a variable is is defined by

Example (already in order):

Ages: 17, 19, 21, 22, 23, 23, 23, 38
Median = (22+23)/2 = 22.5
The median also describes what a typical observation looks like, or where is
the center of the distribution of the sample of observations.

CS109A, PROTOPAPAS, RADER, TANNER 23

Mean vs. Median

The mean is sensitive to extreme values (outliers)

CS109A, PROTOPAPAS, RADER, TANNER 24

Mean, median, and skewness

The mean is sensitive to outliers:

The above distribution is called right-skewed since the mean is greater

than the median. Note: skewness often “follows the longer tail”.
CS109A, PROTOPAPAS, RADER, TANNER 25
Computational time

How hard (in terms of algorithmic complexity) is it to calculate

• the mean?
at most O(n)
• the median?
at most O(n) or O(n log n)

Note: Practicality of implementation should be considered!

CS109A, PROTOPAPAS, RADER, TANNER 26

Regarding Categorical Variables...

For categorical variables, neither mean or median make sense. Why?

The mode might be a better way to find the most “representative”

value.

CS109A, PROTOPAPAS, RADER, TANNER 27

Measures of Spread: Range

The spread of a sample of observations measures how well the mean or

median describes the sample.
One way to measure spread of a sample of observations is via the
range.

Range = Maximum Value - Minimum Value

CS109A, PROTOPAPAS, RADER, TANNER 28

Measures of Spread: Variance

The (sample) variance, denoted s2, measures how much on average the
sample values deviate from the mean:
𝑛
1
2
𝑠= ∑
𝑛−1 𝑖=1
2
¿𝑥 𝑖 − 𝑥∨¿ ¿
Note: the term measures the amount by which each deviates from the
mean . Squaring these deviations means that s2 is sensitive to extreme
values (outliers).
Note: s2 doesn’t have the same units as the :(
What does a variance of 1,008 mean? Or 0.0001?

CS109A, PROTOPAPAS, RADER, TANNER 29

Measures of Spread: Standard Deviation

The (sample) standard deviation, denoted s, is the square root of

the variance

√
𝑛
1
𝑠= √ 𝑠 = ∑
2 2
¿ 𝑥 𝑖 − 𝑥∨¿ ¿
𝑛−1 𝑖=1
Note: s does have the same units as the . Phew!

CS109A, PROTOPAPAS, RADER, TANNER 30

Data Exploration: Visualizations

CS109A, PROTOPAPAS, RADER, TANNER 31

Anscombe’s Data

The following four data sets comprise the Anscombe’s Quartet; all four
sets of data have identical simple summary statistics.

CS109A, PROTOPAPAS, RADER, TANNER 32

Anscombe’s Data (cont.)

Summary statistics clearly don’t tell the story of how they differ. But a
picture can be worth a thousand words:

CS109A, PROTOPAPAS, RADER, TANNER 33

More Visualization Motivation

If I tell you that the average score for Homework 0 was: 7.64/15 =
50.9% last year, what does that suggest?

And what does the graph suggest?

CS109A, PROTOPAPAS, RADER, TANNER 34

More Visualization Motivation

Visualizations help us to analyze and explore the data. They help to:

• Identify hidden patterns and trends

• Formulate/test hypotheses
• Communicate any modeling results
– Present information and ideas succinctly
– Provide evidence and support
– Influence and persuade
• Determine the next step in analysis/modeling

CS109A, PROTOPAPAS, RADER, TANNER 35

Types of Visualizations
What do you want your visualization to show about your data?

• Distribution: how a variable or variables in the dataset distribute

over a range of possible values.
• Relationship: how the values of multiple variables in the dataset
relate
• Composition: how the dataset breaks down into subgroups
• Comparison: how trends in multiple variable or datasets compare

CS109A, PROTOPAPAS, RADER, TANNER 36

Histograms to visualize distribution

A histogram is a way to visualize how 1-dimensional data is distributed

across certain values.

Note: Trends in histograms are sensitive to number of bins.

CS109A, PROTOPAPAS, RADER, TANNER 37

Pie chart for a categorical variable?
A pie chart is a way to visualize the static composition (aka, distribution) of a
variable (or single group).

Pie charts are often frowned upon (and bar charts are used instead). Why?

CS109A, PROTOPAPAS, RADER, TANNER 38

Scatter plots to visualize relationships
A scatter plot is a way to visualize the relationship between two
different attributes of multi-dimensional data.

CS109A, PROTOPAPAS, RADER, TANNER 39

Stacked area graph to show trend over time
A stacked area graph is a way to visualize the composition of a group
as it changes over time (or some other quantitative variable). This shows
the relationship of a categorical variable (AgeGroup) to a quantitative
variable (year).

CS109A, PROTOPAPAS, RADER, TANNER 40

Multiple histograms
Plotting multiple histograms (and kernel density estimates of the
distribution, here) on the same axes is a way to visualize how different
variables compare (or how a variable differs over specific groups).

CS109A, PROTOPAPAS, RADER, TANNER 41

Boxplots
A boxplot is a simplified visualization to compare a quantitative variable
across groups. It highlights the range, quartiles, median and any outliers
present in a data set.

CS109A, PROTOPAPAS, RADER, TANNER 42

[Not] Anything is possible!

Often your dataset seem too complex to visualize:

• Data is too high dimensional (how do you plot 100 variables on the
same set of axes?)
• Some variables are categorical (how do you plot values like Cat or
No?)

CS109A, PROTOPAPAS, RADER, TANNER 43

More dimensions not always better

When the data is high dimensional, a scatter plot of all data attributes can
be impossible or unhelpful

CS109A, PROTOPAPAS, RADER, TANNER 44

Reducing complexity
Relationships may be easier to spot by producing multiple plots of lower
dimensionality.

CS109A, PROTOPAPAS, RADER, TANNER 45

Reducing complexity

For 3D data, color coding a categorical attribute can be “effective”

This visualizes a set of Iris

measurements. The
variables are: petal length,
sepal length, Iris type
(setosa, versicolor, virginica).

Except when it’s not effective.

What could be a better choice?
CS109A, PROTOPAPAS, RADER, TANNER 46
3D can work

For 3D data, a quantitative attribute can be encoded by size in a bubble

chart.

The above visualizes a set of consumer products. The variables are:

revenue, consumer rating, product type and product cost.
CS109A, PROTOPAPAS, RADER, TANNER 47
An Example: The Data Science Process

CS109A, PROTOPAPAS, RADER, TANNER 48

What?

The Data Science Process is similar to the scientific process - one of

observation, model building, analysis, and conclusion:
Ask an interesting question

Get the Data

Explore the Data

Model the Data

Communicate/Visualize the
Results
Note: This process is by no means linear!
CS109A, PROTOPAPAS, RADER, TANNER 49
Analyzing Hubway Data

Introduction: Hubway is metro-Boston’s public bike share program, with more

than 1600 bikes at 160+ stations across the Greater Boston area. Hubway is owned
by four municipalities in the area.

By 2016, Hubway operated 185 stations and 1750 bicycles, with 5 million ride
since launching in 2011.

The Data: In April 2017, Hubway held a Data Visualization Challenge at the
Microsoft NERD Center in Cambridge, releasing 5 years of trip data.

The Question: What does the data tell us about the ride share program?

CS109A, PROTOPAPAS, RADER, TANNER 50

The Data Exploration/Question Refinement Cycle

Our original question: ‘What does the data tell us about the ride share program?’
is a reasonable slogan to promote a hackathon. It is not good for guiding scientific
investigation.

Before we can refine the question, we have to look at the data!

Based on the data, what kind of questions can we ask?

CS109A, PROTOPAPAS, RADER, TANNER 51

The Data Exploration/Question Refinement Cycle

Who? Who’s using the bikes?

Refine into specific hypotheses:

• More men or more women?
• Older or younger people?
• Subscribers or one time users?

CS109A, PROTOPAPAS, RADER, TANNER 52

The Data Exploration/Question Refinement Cycle

Where? Where are bikes being checked out?

Refine into specific hypotheses:

• More in Boston than Cambridge?
• More in commercial or residential?
• More around tourist attractions?

Sometimes the data is given to you in pieces and must be merged!

CS109A, PROTOPAPAS, RADER, TANNER 53

The Data Exploration/Question Refinement Cycle

When? When are the bikes being checked out?

Refine into specific hypotheses:

• More during the weekend than on the weekdays?
• More during rush hour?
• More during the summer than the fall?

Sometimes the feature you want to explore doesn’t exist in the data, and must
be engineered!

CS109A, PROTOPAPAS, RADER, TANNER 54

The Data Exploration/Question Refinement Cycle

Why? For what reasons/activities are people

checking out bikes?

Refine into specific hypotheses:

• More bikes are used for recreation than commute?
• More bikes are used for touristic purposes?
• Bikes are use to bypass traffic?

Do we have the data to answer these questions with reasonable certainty?

What data do we need to collect in order to answer these questions?

CS109A, PROTOPAPAS, RADER, TANNER 55

The Data Exploration/Question Refinement Cycle

How? Questions that combine variables.

• How does user demographics impact the duration the bikes are being used? Or where
they are being checked out?
• How does weather or traffic conditions impact bike usage?
• How do the characteristics of the station location affect the number of bikes being
checked out?

How questions are about modeling relationships between different variables.

CS109A, PROTOPAPAS, RADER, TANNER 56

Inspirations for Data Viz/Exploration

So how well did we do in formulating

creative hypotheses and manipulating
the data for answers?

Check out the winners of the Hubway

Challenge:

http://hubwaydatachallenge.org

CS109A, PROTOPAPAS, RADER, TANNER 57