Exploring Correlation in Python

This article aims to give a better understanding of a very important technique of multivariate exploration. A correlation Matrix is basically a covariance matrix. Also known as the auto-covariance matrix, dispersion matrix, variance matrix, or variance-covariance matrix. It is a matrix in which the i-j position defines the correlation between the i^th and j^th parameter of the given data set. When the data points follow a roughly straight-line trend, the variables are said to have an approximately linear relationship. In some cases, the data points fall close to a straight line, but more often there is quite a bit of variability of the points around the straight-line trend. A summary measure called correlation describes the strength of the linear association.

Correlation in Python

Correlation summarizes the strength and direction of the linear (straight-line) association between two quantitative variables. Denoted by r, it takes values between -1 and +1. A positive value for r indicates a positive association, and a negative value for r indicates a negative association. The closer r is to 1 the closer the data points fall to a straight line, thus, the linear association is stronger. The closer r is to 0, making the linear association weaker. 

Correlation

Correlation is the statistical measure that defines to which extent two variables are linearly related to each other.  In statistics, correlation is defined by the Pearson Correlation formula :

r = \frac{\sum(x_i -\bar{x})(y_i -\bar{y})}{\sqrt{\sum(x_i -\bar{x})^{2}\sum(y_i -\bar{y})^{2}}}

where,

• r: Correlation coefficient 
• x_i     : i^th value first dataset X
• \bar{x}     : Mean of first dataset X
• y_i     : i^th value second dataset Y
• \bar{y}     : Mean of second dataset Y

Condition: The length of the dataset X and Y must be the same.

The  Correlation value can be positive, negative, or zeros.

Implementations with code:

Import the numpy library and define a custom dataset x and y of equal length:

# Import the numpy library
import numpy as np

# Define the dataset
x = np.array([1,3,5,7,8,9, 10, 15])
y = np.array([10, 20, 30, 40, 50, 60, 70, 80])

Define the correlation by applying the above formula:

def Pearson_correlation(X,Y):
    if len(X)==len(Y):
        Sum_xy = sum((X-X.mean())*(Y-Y.mean()))
        Sum_x_squared = sum((X-X.mean())**2)
        Sum_y_squared = sum((Y-Y.mean())**2)       
        corr = Sum_xy / np.sqrt(Sum_x_squared * Sum_y_squared)
    return corr
            
print(Pearson_correlation(x,y))    
print(Pearson_correlation(x,x))    

Output:

0.974894414261588
1.0

The above output shows that the relationship between x and y is 0.974894414261588 and x and x is 1.0

We can also find the correlation by using the numpy corrcoef function.

print(np.corrcoef(x, y))

Output:

[[ 1.         -0.97489441]
 [-0.97489441  1.        ]]

The above output shows the correlations between x&x, x&y, y&x, and y&y. 

Example

Import the necessary libraries 

import pandas as pd
from sklearn.datasets import load_diabetes
import seaborn as sns
import matplotlib.pyplot as plt

Loading load_diabetes Data from sklearn.dataset

# Load the dataset with frame
df = load_diabetes(as_frame=True)
# conver into pandas dataframe
df = df.frame
# Print first 5 rows
df.head()

Output: 

Find the Pearson correlations matrix by using the pandas command df.corr()

Syntax 

df.corr(method, min_periods,numeric_only )

method : In method we can choose any one from {'pearson', 'kendall', 'spearman'} pearson is the standard correlation coefficient matrix i.e default
min_periods : int This is optional. Defines th eminimum number of observations required per pair.
numeric_only : Default is False, Defines we want to compare only numeric or categorical object also

# Find the pearson correlations matrix
corr = df.corr(method = 'pearson')
corr

Output: 

The above table represents the correlations between each column of the data frame. The correlation between the self is 1.0, The negative correlation defined negative relationship means on increasing one column value second will decrease and vice-versa. The zeros correlation defines no relationship I.e neutral. and positive correlations define positive relationships meaning on increasing one column value second will also increase and vice-versa.

We can also find the correlations using numpy between two columns

# correaltions between age and sex columns
c = np.corrcoef(df['age'],df['sex'])
print('Correlations between age and sex\n',c)

Output:

Correlations between Age and Sex
 [[1.        0.1737371]
 [0.1737371 1.       ]]

We can match from the above correlation matrix table, it is almost the same result.

Plot the correlation matrix with the seaborn heatmap

plt.figure(figsize=(10,8), dpi =500)
sns.heatmap(corr,annot=True,fmt=".2f", linewidth=.5)
plt.show()

Output: