Pandas Reference Sheet

POWERED BY THE SCIENTISTS AT THE DATA INCUBATOR

Loading/exporting a data set Examining the data

path_to_file: string indicating the path to the file, df.head(n)—returns first n rows
e.g., ‘data/results.csv’
df.tail(n)—returns last n rows
df = pd.read_csv(path_to_file)—read a CSV file df.describe()—returns summary statistics for each
df = pd.read_excel(path_to_file)—read an Excel file numerical column

df = pd.read_html(path_to_file)—parses HTML to find df[‘State’].unique()—returns unique values for the

all tables column

df.to_csv(path_to_file)—creates CSV of the data frame df.columns—returns column names

df.shape—returns the number of rows and columns

Selecting and filtering Statistical operations

can be applied to both data frames and series/column
SELECTING COLUMNS
df[‘State’]—selects ‘State’ column df[‘Population’].sum()—sum of all values of a column
df[[‘State’, ‘Population’]]—selects ‘State’ and df.sum()—sum for all numerical columns
‘Population’ column df.mean()—mean
df.std()—standard deviation
SELECTING BY LABEL
df.min()— minimum value
df.loc[‘a’]—selects row by index label
df.count()—count of values, excludes missing values
df.loc[‘a’, ‘State’]—selects single value of row ‘a’ and
column ‘State’ df.max()—maximum value
df[‘Population’].apply(func)—apply func to each
SELECTING BY POSITION value of column
df.iloc[0]—selects rows in position 0
df.iloc[0, 0]—selects single value by position at row 0 and
column 0
Data cleaning and modifications
FILTERING
df[‘State’].isnull()—returns True/False for rows with
df[df[‘Population’] > 20000000]]—filter out rows not missing values
meeting the condition
df.dropna(axis=0)—drop rows containing missing values
df.query(“Population > 20000000”)—filter out rows
df.dropna(axis=1)—drop columns containing missing
not meeting the condition
values
df.fillna(0)—fill in missing values, here filled with 0
State Capital Population df.sort_values(‘Population’, ascending=True)
—sort rows by a column’s values
a Texas Austin 28700000
df.set_index(‘State’)—changes index to a specified
column
b New York Albany 19540000
df.reset_index()—makes the current index a column

c Washington Olympia 7536000 df.rename(columns={‘Population’=’Pop.’})

—renames columns
Example data frame
 © 2019 Pragmatic Institute, LLC
Grouping and aggregation
grouped = df.groupby(by=’col1’)—create grouped by object
grouped[‘col2’].mean()—mean value of ‘col2’ for each group
grouped.agg({‘col2’: np.mean, ‘col3’: [np.mean, np.std]})—apply different functions to different columns
grouped.apply(func)—apply func to each group

col1 col2 col3

col1 col2 col3

col1 col2 col3

col1 col2 col3

Merging data frames

There are several ways to merge two data frames, depending on the value of method. The resulting indices are integers starting with zero.

df1.merge(df2, how=method, on=’State’)

State Capital Population State Highest Point

+
a Texas Austin 28700000 x Washington Mount Rainier

b New York Albany 19540000 y New York Mount Marcy

c Washington Olympia 7536000 z Nebraska Panorama Point

Data frame df1 Data frame df2

State Capital Population Highest Point State Capital Population Highest Point

0 Texas Austin 28700000 NaN 0 New York Albany 19540000 Mount Marcy

1 New York Albany 19540000 Mount Marcy 1 Washington Olympia 7536000 Mount Rainier

2 Washington Olympia 7536000 Mount Rainier how=‘inner’

how=‘left’ State Capital Population Highest Point

State Capital Population Highest Point 0 Texas Austin 28700000 NaN

0 New York Albany 19540000 Mount Marcy 1 New York Albany 19540000 Mount Marcy

1 Washington Olympia 7536000 Mount Rainier 2 Washington Olympia 7536000 Mount Rainier

2 Nebraska NaN NaN Panorama Point 3 Nebraska NaN NaN Panorama Point

how=‘right’ how=‘outer’

Register or learn more about other courses in our data curriculum by visiting pragmaticinstitute.com/data-science or calling 480.515.1411.
 Reference Sheet
The data Splitting into training data and test data
Your data needs to be contained in a two-dimensional feature matrix and, in the case of from sklearn.model_selection import train_test_split
supervised learning, a one-dimensional label vector. The data has to be numeric (NumPy X_train, X_test, y_train, y_test = train_test_split(X, y)
array, SciPy sparse matrix, pandas DataFrame).

Predictors: supervised learning

Transformers: preprocessing the data
EXAMPLE
EXAMPLE
ex_predictor = ExamplePredictor()—creates a new instance
ex_transf = ExampleTransformer()—creates a new instance
ex_predictor.fit(X_train, y_train)—fits model on training data
ex_transf.fit(X_train)—fits transformer on training data
y_pred = ex_predictor.predict(X_train)—predicts on training data
transf_X = ex_transf.transform(X_train)—transforms training data
y_pred_probs = ex_predictor.predict_proba(X_train)—classifiers
transf_X_test = ex_transf.transform(X_test)—transforms test data only, predicts class probabilities on training data

STANDARDIZE FEATURES (ZERO MEAN, UNIT VARIANCE) LINEAR REGRESSION

from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LinearRegression
scaler = StandardScaler()
lr = LinearRegression()

SCALE EACH FEATURE BY ITS MAX ABS VALUE

DECISION TREE REGRESSION MODEL
from sklearn.preprocessing import MaxAbsScaler
from sklearn.tree import DecisionTreeRegressor
max_scaler = MaxAbsScaler()
tree = DecisionTreeRegressor(max_depth=n)
GENERATE POLYNOMIAL FEATURES
RANDOM FOREST REGRESSION MODEL
from sklearn.preprocessing import PolynomialFeatures
poly_transform = PolynomialFeatures(degree=n) from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor()
ONE-HOT ENCODE CATEGORICAL FEATURES
from sklearn.preprocessing import OneHotEncoder LOGISTIC REGRESSION
ohe = OneHotEncoder() from sklearn.linear_model import LogisticRegression
logr = LogisticRegression()
PRINCIPAL COMPONENT ANALYSIS
from sklearn.decomposition import PCA RANDOM FOREST CLASSIFICATION MODEL
pca = PCA(n_components=n)
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier()
Predictors: unsupervised learning Pipeline

© 2019 Pragmatic Institute, LLC

EXAMPLE
EXAMPLE
from sklearn.pipeline import Pipeline
ex_predictor = ExamplePredictor()—creates a new instance
pipe = Pipeline([(‘feature scaling’, StandardScaler()),
ex_predictor.fit(X_train)—fits model on training data
(‘linear regression’, LinearRegression())])
y_pred = ex_predictor.predict(X_train)—predicts on training data
pipe.fit(X_train, y_train)—fits model on training data

K-MEANS CLUSTERING y_pred = pipe.predict(X_train)—predicts on training data

from sklearn.cluster import KMeans y_pred_test = pipe.predict(X_test)—predicts on test data

km = KMeans(n_clusters=n)
scaler = pipe.named_steps[‘feature scaling’]
lr = pipe.named_steps[‘linear regression’]
Evaluating model performance
from sklearn import metrics Feature union
EXAMPLE
REGRESSION METRICS
from sklearn.pipeline import FeatureUnion
metrics.mean_absolute_error(y_true, y_pred)—Mean absolute error
union = FeatureUnion([(‘transf_1’, ExampleTransformer1()),
metrics.mean_squared_error(y_true, y_pred)—Mean squared error (‘transf_2’, ExampleTransformer2())])
metrics.r2_score(y_true, y_pred)—R2 score union.fit(X_train)—fits on training data
X_transf = union.transform(X_train)—transforms training data
CLASSIFICATION METRICS
metrics.accuracy_score(y_true, y_pred)—Accuracy score
metrics.precision_score(y_true, y_pred)—Precision score Transforming only some features/columns
metrics.recall_score(y_true, y_pred)—Recall score EXAMPLE
metrics.classification_report(y_true, y_pred)—Classification report from sklearn.compose import ColumnTransformer
metrics.roc_auc_score(y_true, y_pred_probs)—ROC AUC score example_transf = ColumnTransformer(
metrics.log_loss(y_true, y_pred_probs)—Cross-entropy loss [(transformer_name, transformer, columns_to_transform)])
example_transf.fit(X_train)
CLUSTERING METRICS X_transf = example_transf.transform(X_train)
metrics.silhouette_score(X_train, y_pred)—Silhouette score

CROSS-VALIDATION Optimizing hyperparameters

from sklearn.model_selection import cross_val_score from sklearn.grid_search import GridSearchCV
cross_val_score(lr, X_train, y_train, cv=5)
grid = GridSearchCV(estimator=DecisionTreeRegressor(),
param_grid={‘max_depth’: range(3, 10)})
grid.fit(X_train, y_train)
print(grid.best_estimator_)—estimator that was chosen by the search
print(grid.best_params_)—parameters that gave the best results

Register or learn more about other courses in our data curriculum by visiting pragmaticinstitute.com/data-science or calling 480.515.1411.
 Python Syntax REFERENCE SHEET
POWERED BY THE SCIENTISTS AT THE DATA INCUBATOR

SYNTAX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Creating variables Functions
Variables can be created by: Functions are a great way to group related lines of code
deg_C = 10.5 # This is a variable into a single unit that can be called upon. Here, we define
a function with two positional arguments a and b and one
A variable name can consist of letters, numbers and the keyword argument multiplier with a default value of
underscore character (_) but the variable name may not 1.
start with a number. Comments are created with a # and
are ignored by the Python interpreter. def subtract(a, b, multiplier=1):

“””
Subtract two numbers and scale the result.
”””
Common mathematical operations diff = multiplier * (a - b)
2 + 3 - addition
return diff
1 - 4 - subtraction
2 * 3 - multiplication
Now, we call the function.
4 / 3 - division
In [1]: subtract(1, 2, multiplier=2)
4 // 3 - floor division (round down)
Out [1]: -2
2 ** 3 - raise to the power
a += 1 - compute a + 1 and assign the result to a
a -= 1 - compute a - 1 and assign the result to a

Boolean logic
Common built-in functions These operations will return either True or
False,depending on the value of the two variables. They
print(temp_data) - print/display the value of temp_
are often used in conjunction with if/elif statements.
data
len(temp_data) - returns the number of values of the a < b - is a less than b
iterable a > b - is a greater than b
sum(temp_data) - returns sum of the values of the a <= b - is a less than or equal to to b
iterable a >= b - is a greater than or equal to b
min(temp_data) - returns the minimum value of the a == b - do a and b have the same value
iterable a != b - do a and b not have the same value
max(temp_data) - returns the maximum value of the a is b - is a the same object as b
iterable
sorted(temp_data) - returns a list of the sorted
values of temp_data
range(start, end, step) - returns an iterable from
start to end (exclusive) using a step size of step
(defaults to 1)
 © 2020 Pragmatic Institute, LLC
Loops if/elif/else blocks
Loops are a way to repeatedly execute a block of code. if/elif/else blocks let us control the behavior of
There are two types of loops: for and while loops. For our program based on conditions. For example, what
loops are used to loop through every value of an iterable, value to assign to a variable based on the value of
like a list or tuple. While loops are used to continually another variable. At a minimum, you need one condition
execute a block of code while a provided condition is still to test, using if. Multiple conditions can be tested
true. using multiple elif statements. The code in the else
block, which is optional, is run when none of the tested
for temp in temp_data: conditions are met.
print(temp)
if amount < 5:
count = 0 rate = 0.1
while count < 10: elif amount <= 5 and amount < 10:
print(count) rate = 0.2
count += 1 else:
rate = 0.25

DATA STRUCTURES . . . . . . . . . . . . . . . . . . . . . . . .
Strings Lists
Strings are a sequence of characters and are great when Lists are an ordered collection of Python objects. The
wanting to represent text. They’re created using either items of the lists do not have to be the same data type.
single or double quotes. They can be indexed but strings For example, you can store strings and integers inside
are immutable. Strings are iterables, iterating over each the same list. Lists are mutable; they can be altered after
of the characters. their creation. Since they are ordered, they can be indexed
by position. Note, Python uses zero indexing so the “first”
sentence = ‘The quick brown fox jumped over element is index by 0. Lists are created using square
the lazy dog.’ brackets [].

Common operations with strings and usage: temp_data = [10.5, 12.2, 5, 8.7, 1]

sentence.lower() - returns new string with all Common operations on lists and example usage:
characters in lowercase
sentence.upper() - returns a new string with all temp_data.append(2.5) - adds 2.5 to the end of the
characters in uppercase list
sentence.startswith(‘The’) - returns True or temp_data.sort() - sorts the elements of the list in
False if string starts with ‘The’ ascending order. Use reverse=True
sentence.endswith(‘?’) - returns True or False if to sort by descending order.
string ends with ‘?’ temp_data.remove(12.2) - removes the first
sentence.split() - returns a list resulting from occurrence of 12.2 from the list
splitting the string by a provided separator, defaults temp_data.pop() - remove and returns the last
to splitting by whitespace if no argument is passed. element of the list
sentence.strip() - returns a new string with leading temp_data[0] - access value at position 0
and trailing whitespace removed temp_data[:3] - access the first three values, positions
‘fox’ in sentence - returns True if ‘fox’ is present 0 to 3 (inclusive-exclusive)
in sentence. temp_data[-1] - access the the last element
‘taco ‘ + ‘cat’ - returns a new string from temp_data[1:4:2] - access values from position 1
concatenating the two strings (inclusive) and 4 (exclusive) with a step size of 2
f”My name is {name} and I’m {age} years len(temp_data) - returns the number of values in the
old.” - returns a string with the values of variables list
name and age substituted into {name} and {age}, sum(temp_data) - returns sum of the values of the list
respectively. min(temp_data) - returns the minimum value of the list
sentence.replace(“brown”, “red”) - replace max(temp_data) - returns the maximum value of the
every occurrence of “brown” with “red” list
len(sentence) - returns the number of characters of
the string

–2–
 © 2020 Pragmatic Institute, LLC
Tuples Dictionary
Tuples are similar to lists but they are immutable; they Dictionaries store data in key-value pairs. Values can
cannot be modified. As with lists, they can be indexed in a be indexed using the key associated with the value.
similar fashion. Tuples are created by using parentheses There’s no restriction in what can be values but keys
(). are restricted to immutable types. For example, strings,
numerics and tuples can be keys. Dictionaries are
array_shape = (100, 20) created using curly braces {} with the key and value pair
separated by a colon :. Iterating over a dictionary yields
the keys.

customer_data = {
Sets ‘name’: ‘Clarissa’,
‘account_id’: 100045,
Sets are a collection of unique values. They’re a great
‘account_balance’: 4515.76,
data structure to use when wanting to keep track of
‘open_account’: True
only unique values. The members of a set need to be
}
immutable. For example, lists are not allowed but tuples
are. A set can be created by passing an iterable to set or
customer_data[‘name’] - access value associated
directly using curly braces.
with ‘name’
even_numbers = set([x for x in range(100) customer_data[‘telephone’] = None - create new
if x % 2 == 0]) key-value pair ‘telephone’: None
squares = { 1, 1, 2, 4, 2, 9, 16, 25, 36, customer_data[‘telephone’] = ‘555-1234’ -
49, 64, 81, 100} update value of key ‘telephone’
del customer_data[‘telephone’] - delete key-
even_numbers.add(100) - add 100 to the set even_ value for ‘telephone’
numbers ‘age’ in customer_data - returns True/False if key
even_numbers.difference(squares) - returns a ‘age’ is in the dictionary
set that is the difference between even_numbers customer_data.get(‘age’, -1) - returns the value
and squares of key ‘age’ if it exists, returns None otherwise.
even_numbers.union({1, 3, 5, 7, 9}) - return a Optional second argument is returned instead of
set that is the union of the two sets None if key does not exist.
squares.intersection(even_numbers) - return set customer_data.keys() - returns an iterable over all
of common elements keys
1 in even_numbers - returns True or False if 1 is a customer_data.items() - returns an iterable over all
member of the set even_numbers key-value pairs
customer_data.values() - returns an iterable over
all values

* * *

–3–
 REFERENCE SHEET
POWERED BY THE SCIENTISTS AT THE DATA INCUBATOR

Query structure Sample tables

SELECT <expressions> transactions
FROM <tables>
WHERE <conditions>
customer_id items price
GROUP BY <columns> 27 5 12.00
HAVING <conditions>
ORDER BY <columns> 33 25 11.00
LIMIT <number> 60 150 9.00
Only SELECT and FROM are mandatory 60 250 9.00

LIMIT restricts the number of rows returned customers

id customer state
44 Amy CA
SELECT chooses what to get 60 Brian CA
27 Pat NY
SELECT customer_id, items*price AS total
FROM transactions; 51 Alex NULL
can select columns or expressions, such as product,
ratios, etc.

Rename cols or expressions with AS WHERE filters results

SELECT *
Get number of rows in table customers:
FROM customers
SELECT COUNT(*) FROM customers;
WHERE state = ‘CA’;
Get distinct elements in column state:
Select from a list:
SELECT DISTINCT state FROM customers;
WHERE customer IN (‘Amy’, ‘Pat’);
Number of distinct elements:
A pattern, % can be filled in with anything:
SELECT COUNT(DISTINCT state)
WHERE customer LIKE ‘A%’;
FROM customers;
Find missing values:
WHERE state IS NULL;

CASE allows if-like behavior Combine filters:

WHERE customer_id < 50
SELECT customer_id, AND state = ‘CA’;
CASE WHEN items < 10 THEN ‘few’
WHEN items > 10 AND items < 100 WHERE name = ‘Amy’
THEN ‘many’ OR NOT state = ‘CA’;
ELSE ‘lots’;
 © 2021 Pragmatic Institute, LLC
Create temporary tables using schemas JOIN combines tables
CREATE TEMP TABLE trans ( Use ON or WHERE to set matching condition. Use table
customer_id INTEGER, prefix if ambiguous.
items INTEGER,
price REAL SELECT customer, items, state
); FROM customers JOIN transactions
ON customer_id = id;
Add to the table:
INSERT INTO trans VALUES (27, 5, 12.00), SELECT customer, items, state
(33, 25, 11.00); FROM customers, transactions
WHERE customer_id = customers.id;
CREATE TEMP TABLE custs (
id INTEGER PRIMARY KEY, LEFT JOIN includes unmatched values from the first
customer TEXT NOT NULL, table.
state TEXT
); SELECT customer, items, state
FROM customers LEFT JOIN transactions
Or by saving a query ON customer_id = customers.id;

CREATE TEMP TABLE big AS RIGHT JOIN does the same for the second table.
SELECT * FROM transactions
WHERE items > 100; SELECT customer, items, state
FROM customers RIGHT JOIN transactions
Replace TEMP TABLE with TEMP VIEW to get a live- ON customer_id = id;
updated VIEW.
FULL JOIN includes all unmatched rows.

SELECT customer, items, state

FROM customers FULL JOIN transactions
GROUP BY aggregates ON customer_id = customers.id;

SELECT state, COUNT(*) AS number

FROM customers
GROUP BY state;
Subqueries allow more complex
Many options for aggregation: SUM, COUNT, AVG, etc. operations
Everything in the SELECT must be either in the GROUP SELECT customer, total
BY or in an aggregation. FROM customers JOIN
(SELECT customer_id,
HAVING is for conditioning after aggregation. SUM(items*price) AS total
FROM transactions
SELECT state, COUNT(*) AS number
GROUP BY customer_id) AS orders
FROM customers
ON customer_id = id;
GROUP BY state
HAVING COUNT(*) > 1;

–2–