School of Engineering
Ajeenkya D Y Patil University, Pune
MID-TERM EXAMINATION
REPORT
STUDENT NAME: BHOLA KAMBLE
URN: 2021-B-01082003A
COURSE NAME: STATISTICAL SCIENCE
SEMESTER: III
PROGRAM & SPECIALIZATION: BCA (DS)
ACADEMIC YEAR: 2022-23
Statistic in python
Data representation and interaction
Data as a table
The setting that we consider for statistical analysis is that of multiple observations or samples described by a set
of different attributes or features. The data can than be seen as a 2D table, or matrix, with columns giving the
different attributes of the data, and rows the observations. For instance, the data contained in
examples/brain_size.csv:
"";"Gender";"FSIQ";"VIQ";"PIQ";"Weight";"Height";"MRI_Count"
"1";"Female";133;132;124;"118";"64.5";816932
"2";"Male";140;150;124;".";"72.5";1001121
"3";"Male";139;123;150;"143";"73.3";1038437
"4";"Male";133;129;128;"172";"68.8";965353
"5";"Female";137;132;134;"147";"65.0";951545
The pandas data-frame
We will store and manipulate this data in a pandas.DataFrame, from the pandas module. It is the Python equivalent
of the spreadsheet table. It is different from a 2D numpy array as it has named columns, can contain a mixture of
different data types by column, and has elaborate selection and pivotal mechanisms.
Creating dataframes: reading data files or converting arrays
Separator
It is a CSV file, but the separator is “;”
Reading from a CSV file: Using the above CSV file that gives observations of brain size and weight and IQ
(Willerman et al. 1991), the data are a mixture of numerical and categorical values:
>>>>>> import pandas
>>> data = pandas.read_csv('examples/brain_size.csv', sep=';', na_values=".")
>>> data
Unnamed: 0 Gender FSIQ VIQ PIQ Weight Height MRI_Count
0 1 Female 133 132 124 118.0 64.5 816932
1 2 Male 140 150 124 NaN 72.5 1001121
2 3 Male 139 123 150 143.0 73.3 1038437
3 4 Male 133 129 128 172.0 68.8 965353
4 5 Female 137 132 134 147.0 65.0 951545
...
Missing values
The weight of the second individual is missing in the CSV file. If we don’t specify the missing value (NA = not
available) marker, we will not be able to do statistical analysis.
Creating from arrays: A pandas.DataFrame can also be seen as a dictionary of 1D ‘series’, eg arrays or lists. If
we have 3 numpy arrays:
>>>>>> import numpy as np
>>> t = np.linspace(-6, 6, 20)
>>> sin_t = np.sin(t)
>>> cos_t = np.cos(t)
We can expose them as a pandas.DataFrame:
>>>>>> pandas.DataFrame({'t': t, 'sin': sin_t, 'cos': cos_t})
t sin cos
0 -6.000000 0.279415 0.960170
1 -5.368421 0.792419 0.609977
2 -4.736842 0.999701 0.024451
3 -4.105263 0.821291 -0.570509
4 -3.473684 0.326021 -0.945363
5 -2.842105 -0.295030 -0.955488
6 -2.210526 -0.802257 -0.596979
7 -1.578947 -0.999967 -0.008151
8 -0.947368 -0.811882 0.583822
...
Other inputs: pandas can input data from SQL, excel files, or other formats. See the pandas documentation.
Manipulating data
data is a pandas.DataFrame, that resembles R’s dataframe:
>>>>>> data.shape # 40 rows and 8 columns
(40, 8)
>>> data.columns # It has columns
Index([u'Unnamed: 0', u'Gender', u'FSIQ', u'VIQ', u'PIQ', u'Weight', u'Height',
u'MRI_Count'], dtype='object')
>>> print(data['Gender']) # Columns can be addressed by name
0 Female
1 Male
2 Male
3 Male
4 Female
...
>>> # Simpler selector
>>> data[data['Gender'] == 'Female']['VIQ'].mean()
109.45
Note
For a quick view on a large dataframe, use its describe method: pandas.DataFrame.describe().
groupby: splitting a dataframe on values of categorical variables:
>>>>>> groupby_gender = data.groupby('Gender')
>>> for gender, value in groupby_gender['VIQ']:
... print((gender, value.mean()))
('Female', 109.45)
('Male', 115.25)
groupby_gender is a powerful object that exposes many operations on the resulting group of dataframes:
>>>>>> groupby_gender.mean()
Unnamed: 0 FSIQ VIQ PIQ Weight Height MRI_Count
Gender
Female 19.65 111.9 109.45 110.45 137.200000 65.765000 862654.6
Male 21.35 115.0 115.25 111.60 166.444444 71.431579 954855.4
Use tab-completion on groupby_gender to find more. Other common grouping functions are median, count (useful for
checking to see the amount of missing values in different subsets) or sum. Groupby evaluation is lazy, no work is done
until an aggregation function is applied.
Exercise
• What is the mean value for VIQ for the full population?
• How many males/females were included in this study?
• Hint use ‘tab completion’ to find out the methods that can be called, instead of ‘mean’ in the above
example.
• What is the average value of MRI counts expressed in log units, for males and females?
Note
groupby_gender.boxplot is used for the plots above (see this example).
Plotting data
Pandas comes with some plotting tools (pandas.tools.plotting, using matplotlib behind the scene) to
display statistics of the data in dataframes:
Scatter matrices:
>>>>>> from pandas.tools import plotting
>>> plotting.scatter_matrix(data[['Weight', 'Height', 'MRI_Count']])
>>>>>> plotting.scatter_matrix(data[['PIQ', 'VIQ', 'FSIQ']])
Two populations
The IQ metrics are bimodal, as if there are 2 sub-populations.
Exercise
Plot the scatter matrix for males only, and for females only. Do you think that the 2 sub-populations correspond
to gender?
Hypothesis testing: comparing two groups
For simple statistical tests, we will use the scipy.stats sub-module of scipy:
>>>>>> from scipy import stats
See also
Scipy is a vast library. For a quick summary to the whole library, see the scipy chapter.
Student’s t-test: the simplest statistical test
1-sample t-test: testing the value of a population mean
scipy.stats.ttest_1samp() tests if the population mean of data is likely to be equal to a given value
(technically if observations are drawn from a Gaussian distributions of given population mean). It returns the T
statistic, and the p-value (see the function’s help):
>>>>>> stats.ttest_1samp(data['VIQ'], 0)
Ttest_1sampResult(statistic=30.088099970..., pvalue=1.32891964...e-28)
With a p-value of 10^-28 we can claim that the population mean for the IQ (VIQ measure) is not 0.
2-sample t-test: testing for difference across populations
We have seen above that the mean VIQ in the male and female populations were different. To test if this is
significant, we do a 2-sample t-test with scipy.stats.ttest_ind():
>>>>>> female_viq = data[data['Gender'] == 'Female']['VIQ']
>>> male_viq = data[data['Gender'] == 'Male']['VIQ']
>>> stats.ttest_ind(female_viq, male_viq)
Ttest_indResult(statistic=-0.77261617232..., pvalue=0.4445287677858...)
