More On Pandas
More On Pandas
DEPARTMENT- ACADEMIC
UNIT-1
Bachelor of Engineering (Computer Science &
Engineering)
Disruptive Technologies – I
Prepared By: Dr. Anuj Kumar Goel
Course Outcome
CO Title Level
Number
2
Python for Data Analysis
Overview of Python Libraries for Data
Tutorial Content Scientists
Descriptive statistics
Inferential statistics
4
Python Libraries for Data Science
Many popular Python toolboxes/libraries:
• NumPy
• SciPy
All these libraries are
• Pandas installed on the SCC
• SciKit-Learn
Visualization libraries
• matplotlib
• Seaborn
5
Python Libraries for Data Science
NumPy:
introduces objects for multidimensional arrays and matrices, as well as
functions that allow to easily perform advanced mathematical and statistical
operations on those objects
Link: http://www.numpy.org/
6
Python Libraries for Data Science
SciPy:
collection of algorithms for linear algebra, differential equations, numerical
integration, optimization, statistics and more
built on NumPy
Link: https://www.scipy.org/scipylib/
7
Python Libraries for Data Science
Pandas:
adds data structures and tools designed to work with table-like data (similar
to Series and Data Frames in R)
Link: http://pandas.pydata.org/
8
Python Libraries for Data Science
SciKit-Learn:
provides machine learning algorithms: classification, regression, clustering,
model validation etc.
Link: http://scikit-learn.org/
9
Python Libraries for Data Science
matplotlib:
python 2D plotting library which produces publication quality figures in a
variety of hardcopy formats
10
Python Libraries for Data Science
Seaborn:
based on matplotlib
Link: https://seaborn.pydata.org/
11
Start Jupyter nootebook
# On the Shared Computing Cluster
[scc1 ~] jupyter notebook
13
Reading data using pandas
In [ ]: #Read csv file
df = pd.read_csv("http://rcs.bu.edu/examples/python/data_analysis/Salaries.csv")
Note: The above command has many optional arguments to fine-tune the data import process.
pd.read_excel('myfile.xlsx',sheet_name='Sheet1', index_col=None,
na_values=['NA'])
pd.read_stata('myfile.dta')
pd.read_sas('myfile.sas7bdat')
pd.read_hdf('myfile.h5','df')
Figure 3. Reading data files in Pandas
14
Exploring data frames
In [3]: #List first 5 records
df.head()
Out[3]:
Can you guess how to view the last few records; Hint:
16
Data Frame data types
Pandas Type Native Python Type Description
object string The most general dtype. Will be
assigned to your column if column
has mixed types (numbers and
strings).
datetime64, timedelta[ns] N/A (but see the datetime module Values meant to hold time data.
in Python’s standard library) Look into these for time series
experiments.
Out[4]: dtype('int64')
df.attribute description
dtypes list the types of the columns
columns list the column names
axes list the row labels and column names
ndim number of dimensions
20
Data Frames methods
Unlike attributes, python methods have parenthesis.
All attributes and methods can be listed with a dir() function: dir(df)
df.method() description
head( [n] ), tail( [n] ) first/last n rows Table 3. Data frame
methods
describe() generate descriptive statistics (for numeric columns only)
What are the mean values of the first 50 records in the dataset? Hint: use
head() method to subset the first 50 records and then calculate the mean
22
Selecting a column in a Data Frame
Method 1: Subset the data frame using column name:
df['sex']
Note: there is an attribute rank for pandas data frames, so to select a column with a name
"rank" we should use method 1.
23
Hands-on exercises
Find how many values in the salary column (use count method);
24
Data Frames groupby method
Using "group by" method we can:
• Split the data into groups based on some criteria
• Calculate statistics (or apply a function) to each group
• Similar to dplyr() function in R
In [ ]: #Group data using rank
df_rank = df.groupby(['rank'])
In [ ]: #Calculate mean value for each numeric column per each group
df_rank.mean()
Once groupby object is create we can calculate various statistics for each group:
Note: If single brackets are used to specify the column (e.g. salary), then the output is Pandas Series object.
When double brackets are used the output is a Data Frame
29
Data Frames: Slicing
When selecting one column, it is possible to use single set of brackets, but the
resulting object will be a Series (not a DataFrame):
In [ ]: #Select column salary:
df['salary']
When we need to select more than one column and/or make the output to be a
DataFrame, we should use double brackets:
In [ ]: #Select column salary:
df[['rank','salary']]
If we need to select a range of rows, we can specify the range using ":"
Notice that the first row has a position 0, and the last value in the range is omitted:
So for 0:10 range the first 10 rows are returned with the positions starting with 0
and ending with 9
31
Data Frames: method loc
If we need to select a range of rows, using their labels we can use method loc:
Out[ ]:
If we need to select a range of rows and/or columns, using their positions we can
use method iloc:
In [ ]: #Select rows by their labels:
df_sub.iloc[10:20,[0, 3, 4, 5]]
Out[ ]:
34
Figure 11. selecting rows/column through position (summary)
Data Frames: Sorting
We can sort the data by a value in the column. By default the sorting will occur in
ascending order and a new data frame is return.
In [ ]: # Create a new data frame from the original sorted by the column Salary
df_sorted = df.sort_values( by ='service')
df_sorted.head()
Out[ ]:
Out[ ]:
Out[ ]:
39
Aggregation Functions in Pandas
Aggregation - computing a summary statistic about each group, i.e.
• compute group sums or means
• compute group sizes/counts
min, max
count, sum, prod
mean, median, mode, mad
std, var
40
Aggregation Functions in Pandas
agg() method are useful when multiple statistics are computed per column:
In [ ]: flights[['dep_delay','arr_delay']].agg(['min','mean','max'])
Out[ ]:
kurt kurtosis
In [ ]: %matplotlib inline
43
Graphics
description
distplot histogram
barplot estimate of central tendency for a numeric variable
violinplot similar to boxplot, also shows the probability density of the
data
jointplot Scatterplot
regplot Regression plot
pairplot Pairplot
boxplot boxplot
swarmplot categorical scatterplot
factorplot General categorical plot
The first one is mostly used for regular analysis using R style formulas, while scikit-learn is
more tailored for Machine Learning.
statsmodels:
• linear regressions
• ANOVA tests
• hypothesis testings
• many more ...
scikit-learn:
• kmeans
• support vector machines
• random forests
• many more ...
45
References
• http://www.numpy.org/
• https://www.scipy.org/scipylib/
• http://pandas.pydata.org/
• https://matplotlib.org/
46
THANK YOU
For queries
Email: [email protected]
47