Experiment No.

1
Aim : To apply various data-preprocessing techniques on a data set to prepare it for machine learning algorithms. Theory What is
Data Preprocessing ? Data preprocessing is a data mining technique that involves transforming raw data into an understandable
format. Real-world data is often incomplete, inconsistent, and/or lacking in certain behaviors or trends, and is likely to contain many
errors. Data preprocessing is a proven method of resolving such issues.

Steps Check missing values Categorical variables (Label Encoding and One-Hot Enconding) Split data into training and testing sets
Feature scaling

Code and Output

In [1]:
import numpy as np
import pandas as pd

In [2]:
data = pd.read_csv('50_Startups.csv')

In [3]:
data.head(5)

Out[3]:

R&D Spend Administration Marketing Spend State Profit

0 165349.20 136897.80 471784.10 New York 192261.83

1 162597.70 151377.59 443898.53 California 191792.06

2 153441.51 101145.55 407934.54 Florida 191050.39

3 144372.41 118671.85 383199.62 New York 182901.99

4 142107.34 91391.77 366168.42 Florida 166187.94

In [4]:

data.shape

Out[4]:
(50, 5)

In [5]:

data.columns #features

Out[5]:
Index(['R&D Spend', 'Administration', 'Marketing Spend', 'State', 'Profit'], dtype='object')

Checking missing values

In [6]:
#check for missing values
data.isnull().any()
#It is observed that every column has missing values

Out[6]:
R&D Spend False
Administration False
Marketing Spend False
State False
Profit False
dtype: bool

Handling missing values

1. Drop rows having null values

2. Fill missing values with mean/median/mode or any relevant value

In [7]:
# Dropping null rows
data.dropna(inplace=True)
data.isnull().any()
#No null values now

Out[7]:
R&D Spend False
Administration False
Marketing Spend False
State False
Profit False
dtype: bool

In [8]:
print(data.shape)

(50, 5)

Handling categorical variables

In [17]:
data2 = pd.read_csv('50_Startups.csv')
data2.head()

Out[17]:

R&D Spend Administration Marketing Spend State Profit

0 165349.20 136897.80 471784.10 New York 192261.83

1 162597.70 151377.59 443898.53 California 191792.06

2 153441.51 101145.55 407934.54 Florida 191050.39

3 144372.41 118671.85 383199.62 New York 182901.99

4 142107.34 91391.77 366168.42 Florida 166187.94

In [18]:

data2['Profit'].unique()

Out[18]:
array([192261.83, 191792.06, 191050.39, 182901.99, 166187.94, 156991.12,
156122.51, 155752.6 , 152211.77, 149759.96, 146121.95, 144259.4 ,
141585.52, 134307.35, 132602.65, 129917.04, 126992.93, 125370.37,
124266.9 , 122776.86, 118474.03, 111313.02, 110352.25, 108733.99,
108552.04, 107404.34, 105733.54, 105008.31, 103282.38, 101004.64,
99937.59, 97483.56, 97427.84, 96778.92, 96712.8 , 96479.51,
 99937.59, 97483.56, 97427.84, 96778.92, 96712.8 , 96479.51,
90708.19, 89949.14, 81229.06, 81005.76, 78239.91, 77798.83,
71498.49, 69758.98, 65200.33, 64926.08, 49490.75, 42559.73,
35673.41, 14681.4 ])

In [160]:

from sklearn.preprocessing import LabelEncoder

label_encoder = LabelEncoder()

In [19]:

data_LE = data2.copy()
data_LE['State'] = label_encoder.fit_transform(data_LE['State'])

In [20]:

data_LE.head()

Out[20]:

R&D Spend Administration Marketing Spend State Profit

0 165349.20 136897.80 471784.10 2 192261.83

1 162597.70 151377.59 443898.53 0 191792.06

2 153441.51 101145.55 407934.54 1 191050.39

3 144372.41 118671.85 383199.62 2 182901.99

4 142107.34 91391.77 366168.42 1 166187.94

In [21]:
data_LE_df = pd.DataFrame(data_LE)

In [22]:
data_LE_df.dropna(inplace=True)

In [23]:
data_LE_df

Out[23]:

R&D Spend Administration Marketing Spend State Profit

0 165349.20 136897.80 471784.10 2 192261.83

1 162597.70 151377.59 443898.53 0 191792.06

2 153441.51 101145.55 407934.54 1 191050.39

3 144372.41 118671.85 383199.62 2 182901.99

4 142107.34 91391.77 366168.42 1 166187.94

5 131876.90 99814.71 362861.36 2 156991.12

6 134615.46 147198.87 127716.82 0 156122.51

7 130298.13 145530.06 323876.68 1 155752.60

8 120542.52 148718.95 311613.29 2 152211.77

9 123334.88 108679.17 304981.62 0 149759.96

10 101913.08 110594.11 229160.95 1 146121.95

11 100671.96 91790.61 249744.55 0 144259.40

12 93863.75 127320.38 249839.44 1 141585.52

13 R&D Spend Administration
91992.39 135495.07 Marketing
252664.93 Spend State
0 Profit
134307.35

14 119943.24 156547.42 256512.92 1 132602.65

15 114523.61 122616.84 261776.23 2 129917.04

16 78013.11 121597.55 264346.06 0 126992.93

17 94657.16 145077.58 282574.31 2 125370.37

18 91749.16 114175.79 294919.57 1 124266.90

19 86419.70 153514.11 0.00 2 122776.86

20 76253.86 113867.30 298664.47 0 118474.03

21 78389.47 153773.43 299737.29 2 111313.02

22 73994.56 122782.75 303319.26 1 110352.25

23 67532.53 105751.03 304768.73 1 108733.99

24 77044.01 99281.34 140574.81 2 108552.04

25 64664.71 139553.16 137962.62 0 107404.34

26 75328.87 144135.98 134050.07 1 105733.54

27 72107.60 127864.55 353183.81 2 105008.31

28 66051.52 182645.56 118148.20 1 103282.38

29 65605.48 153032.06 107138.38 2 101004.64

30 61994.48 115641.28 91131.24 1 99937.59

31 61136.38 152701.92 88218.23 2 97483.56

32 63408.86 129219.61 46085.25 0 97427.84

33 55493.95 103057.49 214634.81 1 96778.92

34 46426.07 157693.92 210797.67 0 96712.80

35 46014.02 85047.44 205517.64 2 96479.51

36 28663.76 127056.21 201126.82 1 90708.19

37 44069.95 51283.14 197029.42 0 89949.14

38 20229.59 65947.93 185265.10 2 81229.06

39 38558.51 82982.09 174999.30 0 81005.76

40 28754.33 118546.05 172795.67 0 78239.91

41 27892.92 84710.77 164470.71 1 77798.83

42 23640.93 96189.63 148001.11 0 71498.49

43 15505.73 127382.30 35534.17 2 69758.98

44 22177.74 154806.14 28334.72 0 65200.33

45 1000.23 124153.04 1903.93 2 64926.08

46 1315.46 115816.21 297114.46 1 49490.75

47 0.00 135426.92 0.00 0 42559.73

48 542.05 51743.15 0.00 2 35673.41

49 0.00 116983.80 45173.06 0 14681.40

Splitting into training and testing sets

In [26]:
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(data_LE_df,data_LE_df['Profit'],test_size=0.2)

/anaconda3/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This mo

dule was deprecated in version 0.18 in favor of the model_selection module into which all the refa
ctored classes and functions are moved. Also note that the interface of the new CV iterators are d
ctored classes and functions are moved. Also note that the interface of the new CV iterators are d
ifferent from that of this module. This module will be removed in 0.20.
"This module will be removed in 0.20.", DeprecationWarning)

In [27]:
X_train.head()

Out[27]:

R&D Spend Administration Marketing Spend State Profit

25 64664.71 139553.16 137962.62 0 107404.34

0 165349.20 136897.80 471784.10 2 192261.83

10 101913.08 110594.11 229160.95 1 146121.95

14 119943.24 156547.42 256512.92 1 132602.65

35 46014.02 85047.44 205517.64 2 96479.51

In [28]:
y_train.head()

Out[28]:
25 107404.34
0 192261.83
10 146121.95
14 132602.65
35 96479.51
Name: Profit, dtype: float64

Feature Scaling

In [29]:
from sklearn.preprocessing import StandardScaler
standard_X = StandardScaler()

In [30]:
X_train = standard_X.fit_transform(X_train)
X_test = standard_X.fit_transform(X_test)

In [31]:
pd.DataFrame(X_train) #SCALED

Out[31]:

0 1 2 3 4

0 -0.147778 0.768777 -0.732925 -1.248168 -0.078585

1 2.099133 0.672035 2.246595 1.187282 2.114855

2 0.683470 -0.286287 0.081064 -0.030443 0.922208

3 1.085838 1.387929 0.325194 -0.030443 0.572754

4 -0.563993 -1.217028 -0.129964 1.187282 -0.360975

5 -0.949166 0.003426 -0.422023 -1.248168 -0.832442

6 -1.590858 -0.053492 -1.561117 -1.248168 -2.475335

7 0.158509 1.286864 0.710993 1.187282 0.022449

8 -0.730373 -1.292275 -0.402355 -1.248168 -0.760949

9 0.5215450 0.9700481 0.5578052 1.1872823 0.3858104

10 1.316921 0.986533 0.926449 -0.030443 1.171146

11 -0.607378 -2.447162 -0.205725 -1.248168 -0.529776

12 -0.352435 -0.560869 -0.048589 -0.030443 -0.353236

13 0.964891 0.151737 0.372172 1.187282 0.503335

14 -1.244827 0.325357 -1.647149 1.187282 -1.051661

15 -1.578762 -2.430403 -1.964309 1.187282 -1.932723

16 -1.139408 -1.912880 -0.310728 1.187282 -0.755177

17 -1.561502 -0.096031 0.687583 -0.030443 -1.575565

18 -0.968390 -1.229294 -0.496328 -0.030443 -0.843843

19 1.631008 0.008009 1.455935 1.187282 1.872917

20 0.018321 0.342926 1.188029 1.187282 -0.140519

21 -1.095932 1.324489 -1.711408 -1.248168 -1.169496

22 -0.083778 -0.462735 0.755901 -0.030443 -0.044215

23 0.090208 0.935743 -0.767847 -0.030443 -0.121772

24 0.150110 0.114601 0.395109 -1.248168 0.427751

25 0.462077 0.620929 0.290849 -1.248168 0.616818

26 1.099212 1.102714 0.816992 1.187282 1.079621

27 1.413268 1.047333 -0.824374 -1.248168 1.180707

28 1.580460 -0.985886 1.303923 -0.030443 1.440884

29 1.352154 -0.679013 1.274406 1.187282 1.203160

30 -0.951188 0.313476 -0.169154 -0.030443 -0.510155

31 0.655773 -0.971355 0.264783 -1.248168 0.874064

32 -0.554798 1.429699 -0.082837 -1.248168 -0.354945

33 -1.063279 -0.811085 -0.643327 -1.248168 -1.006698

34 -1.568537 0.207705 -1.947316 1.187282 -1.176585

35 0.503839 0.323101 0.265630 -0.030443 0.804948

36 0.060431 0.157781 0.742964 -0.030443 -0.002386

37 0.110850 -0.167035 0.701417 -1.248168 0.207550

38 0.456649 -0.155796 0.667992 -0.030443 0.357287

39 0.337715 1.277416 -1.964309 1.187282 0.318772

Result
The data set was pre-processed by using label encoding, feature scaling (standardisation, normalization), checking for missing values
and splitting data into training and testing sets.

Conclusion
In Real world data are generally incomplete: lacking attribute values, lacking certain attributes of interest, or containing only
aggregate data. Noisy: containing errors or outliers. Inconsistent: containing discrepancies in codes or names. Hence, it is essential
to pre-process data so that algorithms can be applied without any hindrance.