Machine Learning Notes
1. All the Import Modules Commands :
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
2. All the commands for Eda :
df.isna() / df.isna().sum()
df.info()
df.describe()
df.dropna( axis = 0,1 ) #0 for row and 1 for column
df.fillna()
To calculate mean :-
df['column_name'].mean()
To fill missing values by mean :-
x = df['column_name'].mean()
df['column_name'].fillna(x, inplace=True)
To read a csv file :-
df = pd.read_csv('cars.csv')
df["column_name"].unique()
df["column_name"].value_counts()
To replace a string by nan value :-
df['column_name'].replace("string",np.nan,inplace =True)
df['column_name'] = df['column_name'].astype("float")
To create a new df with specific data type :-
# df_cat / df_num = df with categorical / numerical data
df_cat = df.select_dtypes(object)
df_num = df.select_dtypes(['int64','float64'])
Steps to handle missing values :
#step1 - use replace
df['column_name'].replace("string",np.nan,inplace =True)
#step2 - change the datatype to float
df['column_name'] = df['column_name'].astype("float")
#step3 - calculate the mean for the cols
x = df['column_name'].mean()
#step4 - use fillna
df['column_name'].fillna(x, inplace=True)
Label Encoder :
from sklearn.preprocessing import LabelEncoder
for col in df_cat:
le=LabelEncoder()
df_cat[col] = le.fit_transform(df_cat[col])
To drop columns and rows :
df.drop('column_name', axis = 1) #for a single column
df.drop(['column_name','column_name'],axis=1) #multiple
df.drop(index_number) #to drop a Row
To handle outliers :
#Step1-: Make boxplot with two variable
Eg :- sns.boxplot(data=df,x='price',y='make')
#Step2-: Filter out the outliers
Eg :- df[(df['make']=='dodge') & (df['price']>10000)]
#Step3-: Drop the outliers
Eg :- df.drop(29,inplace=True)
Feature engineering : It is used to reduce the columns / features in the
data frame. Eg : if a data set has height and width
column ,we can create a new column = area ; a=l*b
and then remove height and width columns .
Skewness and handling Skewness :
from scipy.stats import skew
To find skewness of a column :
skew(df_num['column_name'])
Using for loop & plotting graph :
for col in df_num:
print(col)
print(skew(df_num[col]))
plt.figure()
sns.distplot(df_num[col])
plt.show()
#to find correlation
df_num.corr()
sns.heatmap(df_num.corr(), annot=True)
WE SHOULD NOT REMOVE THE SKEWNESS FOR THE COLUMN WHICH HAS
VERY HIGH CO-RELATION WITH TARGET, BECAUSE IF WE DO THAT THEN
THEIR CO-RELATION WITH THE TARGET WILL ALSO BE CHANGE.
ALSO NEVER REMOVE SKEWNESS OF A NEGATIVE COLUMNS , IT WILL GIVE
YOU A NAN VALUE.
To Handle Skewness either find the Square root or log of that
column :
df_num['column_name']= np.sqrt(df_num['column_name'])
Scaling :-
1. MinMax Scaler
from sklearn.preprocessing import MinMaxScaler
for col in df_new:
ms = MinMaxScaler()
df_new[col]=ms.fit_transform(df_new[[col]])
2. Standard Scaler
from sklearn.preprocessing import StandardScaler
for col in df_new:
sc = StandardScaler()
df_new[col]=sc.fit_transform(df_new[[col]])
Requirements for working with data in Sklearn :-
Feature and response should be seperated objects
Feature and response should be Numeric
Feature and response should be numpy array
Feature and response should have specific shape (2D)
x = df.iloc[:,:-1].values #Features -> independent Variable
y = df.iloc[:,-1].values # Response-> dependent variable
Taking care of missing data :-
from sklearn.impute import SimpleImputer
#step1: define the missing value & strategy
si = SimpleImputer(missing_values=np.nan, strategy='mean'
)
#step2: select the col that has missing values
si.fit(x[:,1:3])
#step3: fill the value using transform method to selected
cols and save it back
x[:,1:3] = si.transform(x[:,1:3])
Encoding categorical data ( One Hot Encoder ) : -
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
ct = ColumnTransformer(transformers= [('encoder',
OneHotEncoder(), [0])], remainder=' passthrough ')
#selecting and apply change at the same time
x = np.array(ct.fit_transform(x))
Splitting the dataset into the training set and test set :-
from sklearn.model_selection import train_test_split
xtrain, xtest, ytrain, ytest = train_test_split(x,y,
test_size=0.2, random_state = 1)
Feature Scaling :-
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
xtrain[:,3:] = sc.fit_transform(xtrain[:,3:])
xtest[:,3:] = sc.fit_transform(xtest[:,3:])
Linear regression model :-
#step 1-: Select a model from sklearn
from sklearn.linear_model import LinearRegression
#step 2 -: Create an object of your model
linreg = LinearRegression()
#step 3 -: Train your model
linreg.fit(xtrain, ytrain)
#step 4: Predict the value
ypred = linreg.predict(xtest)
