Driver Distraction Detection (DDD) Project.

TABLE OF CONTENTS
1. PROJECT DESCRIBTION

• Introduction
• Problem Statment
• Tools
• Dataset
• Implementation
2. PREPARING

• Import libraries that will be used

• Exploration
3. PREPROCESSING

• Load training data

• Load testing data
• Visualize and preparing image dataset
• Convert data to batches of tensors

1. MODEL 1

• Define model 1
• Train model 1
• Visualize model 1 result
2. MODEL 2

• Define model 2
• Train model 2
• Visualize model 2 result
3. TESTING FINAL MODEL

PROJECT DESCRIBTION
linkcode

Introduction
According to the World Health Organization (WHO) survey, 1.3 million people worldwide die in
traffic accidents each year, making them the eighth leading cause of death and an additional 20-
50 millions are injured/ disabled. As per the report of National Crime Research Bureau (NCRB),
Govt. of India, Indian roads account for the highest fatalities in the world. There has been a
continuous increase in road crash deaths in India since 2006. The report also states that the total
number of deaths have risen to 1.46 lakhs in 2015 and driver error is the most common cause
behind these traffic accidents.The number of accidents because of distracted driver has been
increasing since few years.

National Highway Traffic Safety Administrator of United States (NHTSA) reports deaths of
3477 people and injuries to 391000 people in motor vehicle crashes because of distracted drivers
in 2015. In the United States, everyday approximately 9 people are killed and more than 1,000
are injured in road crashes that are reported to involve a distracted driver. NTHSA describes
distracted driving as “any activity that diverts attention of the driver from the task of driving”
which can be classified into Manual, Visual or Cognitive distraction. As per the definitions of
Center for Disease Control and Prevention (CDC), cognitive distraction is basically “driver’s
mind is off the driving”.

n other words, even though the driver is in safe driving posture, he is mentally distracted from
the task of driving. He might be lost in thoughts, daydreaming etc. Distraction because of
inattention, sleepiness, fatigue or drowsiness falls into visual distraction class where “drivers’s
eyes are off the road”. Manual distractions are concerned with various activities where “driver’s
hands are off the wheel”. Such distractions include talking or texting using mobile phones, eating
and drinking, talking to passengers in the vehicle, adjusting the radio, makeup etc.Nowadays,
Advanced Driver Assistance Systems (ADAS) are being developed to prevent accidents by
offering technologies that alert the driver to potential problems and to keep the car’s driver and
occupants safe if an accident does occur. But even today’s latest autonomous vehicles require the
driver to be attentive and ready to take the control of the wheel back in case of emergency.

Tesla autopilot’s crash with the white truck-trailor in Williston, Florida in May 2016 was the first
fatal crash in testing of autonomous vehicle. Recently in March 2018, Uber’s self driving car
with an emergency backup driver behind the wheel struck and killed a pedestrian in Arizona. In
both of these fatalities, the safety driver could have avoided the crashes but evidences reveal that
he was clearly distracted. This makes detection of distracted driver an essential part of the self
driving cars as well. We believe that distracted driver detection is utmost important for further
preventive measures. If the vehicle could detect such distractions and then warn the driver
against it, number of road crashes can be reduced. In this projects, we focus on detecting manual
distractions where driver is engaged in other activities than safe driving and also identify the
cause of distraction. We present a Convolutional Neural Network based approach for this
problem. We also attempt to reduce the computational complexity and memory requirement
while maintaining good accuracy which is desirable in real time applications.

Problem Statment
Given a dataset of 2D dashboard camera images, an algorithm needs to be developed to classify
each driver's behaviour and determine if they are driving attentively, wearing their seatbelt, or
taking a selfie with their friends in the backseat etc..? This can then be used to automatically
detect drivers engaging in distracted behaviours from dashboard cameras.
Following are needed tasks for the development of the algorithm:

1. Download and preprocess the driver images.

2. Build and train the model to classify the driver images.
3. Test the model and further improve the model using different techniques.

Tools
The project utilizes the following dependencies:

• Python 3.5: Tensorflow, Keras, Numpy, Scipy, seaborn, Matplotlib.

• NVIDIA Geforce GTX1060 GPU, CUDA, CuDNN.

Dataset
The provided data set has driver images, each taken in a car with a driver doing something in the
car (texting, eating, talking on the phone, makeup, reaching behind, etc). This dataset is obtained
from Kaggle.(State Farm Distracted Driver Detection competition).

Following are the file descriptions and URL’s from which the data can be obtained :

• imgs.zip - zipped folder of all (train/test) images

• sample_submission.csv - a sample submission file in the correct format
• driver_imgs_list.csv - a list of training images, their subject (driver) id, and class id

The 10 classes to predict

• c0: safe driving **
• c1: texting - right
• c2: talking on the phone - right
• c3: texting - left
• c4: talking on the phone - left
• c5: operating the radio
• c6: drinking
• c7: reaching behind
• c8: hair and makeup
• c9: talking to passenger

There are 22423 train images and 79725 test images.

Implementation
This project aims to develop a machine learning system that can detect and classify different
distracted states of car drivers. The main approach is to apply deep convolutional neural
networks (CNNs). We will explore and experiment various CNN architectures, leveraged pre-
trained networks (learning transfer).

Transfer Learning
Most of the time you won't want to train a whole convolutional network yourself. Modern
ConvNets training on huge datasets like ImageNet take weeks on multiple GPUs.

Instead, most people use a pretrained network either as a fixed feature extractor, or as an initial
network to fine tune.

In this notebook, you'll be using [VGGNet] trained on the [ImageNet dataset] as a feature
extractor. Below is a diagram of the VGGNet architecture, with a series of convolutional and
maxpooling layers, then three fully-connected layers at the end that classify the 1000 classes
found in the ImageNet database.

!img[VGG Architecture]
VGGNet is great because it's simple and has great performance, coming in second in the
ImageNet competition. The idea here is that we keep all the convolutional layers, but replace the
final fully-connected layer with our own classifier. This way we can use VGGNet as a fixed
feature extractor for our images then easily train a simple classifier on top of that.

PREPARING

Import libraries that will be used

In [58]:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

from keras.models import Sequential, Model

from keras.layers.core import Flatten, Dense, Dropout
from keras.layers.convolutional import Convolution2D, MaxPooling2D,
ZeroPadding2D
from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout,
BatchNormalization, GlobalAveragePooling2D
from keras.optimizers import SGD
from keras.applications.vgg16 import VGG16

from keras.utils import np_utils

from keras.preprocessing.image import ImageDataGenerator
from keras.preprocessing import image
from keras.callbacks import ModelCheckpoint, EarlyStopping

from IPython.display import FileLink,display, Image

from PIL import Image as I

from sklearn.model_selection import train_test_split

from sklearn.datasets import load_files
from sklearn.utils import shuffle
from sklearn.metrics import log_loss

from random import sample

import random

import matplotlib.pyplot as plt

import matplotlib.image as mpimg
import seaborn as sns

import numpy as np
import pandas as pd

from tqdm import tqdm

from glob import glob
import pickle
import zipfile
import os
import cv2
import timeit
import time
import h5py

In [59]:

# Create folder
models_dir = "saved_models"
if not os.path.exists(models_dir):
os.makedirs(models_dir)

In [60]:

# check if CUDA is available

if tf.test.is_gpu_available(cuda_only=True):
print('CUDA is available! Training on GPU ...')

IMG_SIZE = 244 if tf.test.is_gpu_available(cuda_only=True) else 160

COLOR_TYPE = 3
CLASSES = 10
EPOCHS = 30
BATCHES = 50
IMG_SIZE = 224
TEST_SIZE = 10

CUDA is available! Training on GPU ...

Exploration
Data exploration, preprocessing and analy will be conducted in great details to gain as much
information about the dataset as possible. All steps of a machine learning pipeline are included
and a summary is provided at the end of each section.

In [61]:

df =
pd.read_csv("../input/state-farm-distracted-driver-detection/driver_imgs_list.
csv")
df.head()

Out[61]:
subject classname img
0 p002 c0 img_44733.jpg
1 p002 c0 img_72999.jpg
2 p002 c0 img_25094.jpg
3 p002 c0 img_69092.jpg
4 p002 c0 img_92629.jpg
In [62]:

sns.set()

plt.figure(figsize = (10,5))
# Count the number of images per category
sns.countplot(x = 'classname', color = '#169DE3',data = df)

plt.title('Categories Distribution'.title(),size=22 , color = '#169DE3')

plt.xlabel('classname',size=17 , color = '#169DE3')
plt.ylabel('Count',size=17 , color = '#169DE3')

plt.show()
PREPROCESSING

Load Training data

In [63]:

# Load the dataset from Kaggle

def get_cv2_image(path, img_size, color_type):
# Loading as Grayscale image
if color_type == 1:
img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
# Loading as color image
elif color_type == 3:
img = cv2.imread(path, cv2.IMREAD_COLOR)
# Reduce size
img = cv2.resize(img[:500], (img_size, img_size))
return img

In [64]:
def load_trainning_data(img_size , color_type):
start_time = time.time()
training_images = []
training_labels = []

# Loop over the training folder

for class_ in tqdm(range(CLASSES)):

print('Loading directory c{}'.format(class_))

files = glob(os.path.join('../input/state-farm-distracted-driver-
detection/imgs/train', 'c' + str(class_), '*.jpg'))

for file in files:

img = get_cv2_image(file, img_size , color_type)
training_images.append(img)
training_labels.append(class_)

print("Data Loaded in {} Min".format((time.time() - start_time)/60))

return training_images, training_labels

In [65]:

X, y = load_trainning_data( IMG_SIZE , COLOR_TYPE)

0%| | 0/10 [00:00<?, ?it/s]

Loading directory c0

10%|█ | 1/10 [00:09<01:25, 9.50s/it]

Loading directory c1

20%|██ | 2/10 [00:18<01:14, 9.30s/it]

Loading directory c2

30%|███ | 3/10 [00:27<01:05, 9.32s/it]

Loading directory c3

40%|████ | 4/10 [00:36<00:55, 9.25s/it]

Loading directory c4

50%|█████ | 5/10 [00:45<00:45, 9.15s/it]

Loading directory c5

60%|██████ | 6/10 [00:54<00:36, 9.07s/it]

Loading directory c6

70%|███████ | 7/10 [01:04<00:27, 9.17s/it]

Loading directory c7

80%|████████ | 8/10 [01:11<00:17, 8.75s/it]

Loading directory c8

90%|█████████ | 9/10 [01:19<00:08, 8.39s/it]

Loading directory c9

100%|██████████| 10/10 [01:27<00:00, 8.28s/it]

Data Loaded in 1.45593421459198 Min

In [66]:

X[0]

Out[66]:

array([[[ 14, 20, 19],

[ 14, 20, 19],
[ 14, 20, 19],
...,
[227, 253, 229],
[232, 250, 235],
[197, 209, 202]],

[[ 14, 20, 19],

[ 14, 20, 19],
[ 14, 20, 19],
...,
[225, 254, 227],
[229, 251, 231],
[238, 254, 243]],

[[ 14, 20, 19],

[ 14, 20, 19],
[ 14, 20, 19],
...,
[205, 238, 203],
[224, 253, 226],
[230, 255, 237]],

...,

[[ 9, 11, 11],
[ 8, 10, 10],
 [ 8, 10, 10],
...,
[ 6, 10, 11],
[ 5, 9, 10],
[ 4, 8, 9]],

[[ 9, 11, 11],
[ 8, 10, 10],
[ 7, 9, 9],
...,
[ 6, 10, 11],
[ 5, 9, 10],
[ 4, 8, 9]],

[[ 8, 10, 10],
[ 7, 9, 9],
[ 6, 8, 8],
...,
[ 5, 10, 11],
[ 4, 9, 10],
[ 3, 8, 9]]], dtype=uint8)

In [67]:

# Convert Categorical data to numerical

y = np_utils.to_categorical(y, CLASSES)
y[0]

Out[67]:

array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)

In [68]:

# splitting train data to train and validation

X_train, X_valid, y_train, y_valid = train_test_split(X, y,
test_size=0.15,shuffle=True, random_state=2021)
print(X_train[0])

[[[ 36 44 37]
[ 34 43 33]
[ 32 42 30]
...
[255 254 230]
[255 254 227]
[255 253 224]]

[[ 31 39 32]
[ 30 39 29]
[ 29 38 28]
...
[255 255 232]
[255 255 227]
[252 254 222]]
 [[ 27 35 28]
[ 27 35 28]
[ 27 36 26]
...
[253 255 236]
[250 255 232]
[245 255 224]]

...

[[ 2 7 6]
[ 2 7 6]
[ 2 7 6]
...
[ 22 24 25]
[ 12 14 14]
[ 13 15 15]]

[[ 2 7 6]
[ 2 7 6]
[ 2 7 6]
...
[ 46 47 50]
[ 9 11 12]
[ 11 12 13]]

[[ 2 7 6]
[ 2 7 6]
[ 2 7 6]
...
[ 21 21 26]
[ 12 13 17]
[ 11 12 16]]]

In [69]:

# import gc
# del X
# gc.collect()

In [70]:

# del y
# gc.collect()

In [71]:

# convert data to numpy array

X_train = np.array(X_train, dtype=np.uint8).reshape(-
1,IMG_SIZE,IMG_SIZE,COLOR_TYPE)
X_valid = np.array(X_valid, dtype=np.uint8).reshape(-
1,IMG_SIZE,IMG_SIZE,COLOR_TYPE)

print('Train shape :', X_train.shape)

print('Number of train samples : ',X_train.shape[0])

print('Validation shape :', X_valid.shape)

print('Number of Validation samples : ',X_valid.shape[0])

Train shape : (19060, 224, 224, 3)

Number of train samples : 19060
Validation shape : (3364, 224, 224, 3)
Number of Validation samples : 3364

In [72]:

#shffle training data

random.shuffle(X_train)

In [73]:

#Normalization
#X_train = np.divide(X_train,255)

In [74]:

# X_valid = np.divide(X_valid,255)
# print(X_train[0],'/n/n')
# print(X_valid[0])

In [75]:

# pickle_out = open("X_train.pickle","wb")
# pickle.dump(X_train,pickle_out)
# pickle_out.close()

# pickle_out = open("y_train.pickle","wb")
# pickle.dump(y_train,pickle_out)
# pickle_out.close()

# pickle_out = open("X_valid.pickle","wb")
# pickle.dump(X_valid,pickle_out)
# pickle_out.close()

# pickle_out = open("y_valid.pickle","wb")
# pickle.dump(y_valid,pickle_out)
# pickle_out.close()

Load test data

In [76]:

def load_testing_data(test_size,img_size, color_type):

 files = sorted(glob(os.path.join('../input/state-farm-distracted-driver-
detection/imgs/test', '*.jpg')))
testing_image = []
testing_image_id = []

total = 0
files_size = len(files)

for file in tqdm(files):

if total == test_size:
break

file_base = os.path.basename(file)
img = get_cv2_image(file, img_size, color_type)
testing_image.append(img)
testing_image_id.append(file_base)

total += 1
return testing_image, testing_image_id

In [77]:

test_data, test_ids = load_testing_data(TEST_SIZE, IMG_SIZE, COLOR_TYPE)

test_data = np.array(test_data, dtype=np.uint8)
test_data = test_data.reshape(-1,IMG_SIZE,IMG_SIZE,COLOR_TYPE)

0%| | 0/79726 [00:00<?, ?it/s]

In [78]:

print('Test shape:', test_data.shape)

print(test_data.shape[0], 'Test samples')

Test shape: (10, 224, 224, 3)

10 Test samples

Visualize and preparing image dataset

In [79]:

# mapping categotical
CAT_MAP = {'c0': 'Safe driving',
'c1': 'Texting - right',
'c2': 'Talking on the phone - right',
'c3': 'Texting - left',
'c4': 'Talking on the phone - left',
'c5': 'Operating the radio',
'c6': 'Drinking',
 'c7': 'Reaching behind',
'c8': 'Hair and makeup',
'c9': 'Talking to passenger'}

In [80]:

plt.figure(figsize = (12, 20))

#image_count = 1
DIR = '../input/state-farm-distracted-driver-detection/imgs/train/'

for directory in os.listdir(DIR):

if directory[0] != '.':
for i, file in enumerate(os.listdir(DIR + directory)):
if i == 2:
break
else:
#fig = plt.subplot(2, 2, image_count)
#image_count += 1
image = mpimg.imread(DIR + directory + '/' + file)
plt.imshow(image)
plt.title(CAT_MAP[directory])
Convert data to batches of tensors
In [81]:

#Preparing data augmentation

'''Generate batches of tensor image data with real-time data augmentation.
The data will be looped over (in batches).'''

train_gen = ImageDataGenerator(rescale = 1.0/255,

height_shift_range=0.5,
width_shift_range = 0.5,
rotation_range=30,
validation_split = 0.2)

valid_gen = ImageDataGenerator(rescale=1.0/ 255, validation_split = 0.2)

In [82]:

'''Takes the dataframe and the path to a directory + generates batches.

The generated batches contain => augmented/normalized data.'''

training_generator = train_gen.flow_from_directory('../input/state-farm-
distracted-driver-detection/imgs/train',
target_size = (IMG_SIZE,
IMG_SIZE),
batch_size = BATCHES,
shuffle=True,
class_mode='categorical',
subset="training")

validation_generator = valid_gen.flow_from_directory('../input/state-farm-
distracted-driver-detection/imgs/train',
target_size = (IMG_SIZE,
IMG_SIZE),
batch_size = BATCHES,
shuffle=False,
class_mode='categorical',
subset="validation")

Found 17943 images belonging to 10 classes.

Found 4481 images belonging to 10 classes.

MODEL1
To define a model for training we'll follow these steps:

1. Load in a pre-trained VGG16 model.

2. "Freeze" all the parameters, so the net acts as a fixed feature extractor.
3. Replace the last layer with a linear classifier of our own .

Freezing simply means that the parameters in the pre-trained model will
not change during training.
Failed Model

In [83]:

# def VGG_16(weights_path=None):

# model = Sequential()
# # Layer 1 (CONV)
#
model.add(ZeroPadding2D((1,1),input_shape=(COLOR_TYPE,IMG_SIZE,IMG_SIZE)))
# model.add(Convolution2D(64, 3, 3, activation='relu'))
# model.add(ZeroPadding2D((1,1)))
# model.add(Convolution2D(64, 3, 3, activation='relu'))
# model.add(MaxPooling2D((2,2), strides=(2,2)))
# # Layer 2 (CONV)
# model.add(ZeroPadding2D((1,1)))
# model.add(Convolution2D(128, 3, 3, activation='relu'))
# model.add(ZeroPadding2D((1,1)))
# model.add(Convolution2D(128, 3, 3, activation='relu'))
# model.add(MaxPooling2D((2,2), strides=(2,2)))
# # Layer 3 (CONV)
# model.add(ZeroPadding2D((1,1)))
# model.add(Convolution2D(256, 3, 3, activation='relu'))
# model.add(ZeroPadding2D((1,1)))
# model.add(Convolution2D(256, 3, 3, activation='relu'))
# model.add(ZeroPadding2D((1,1)))
# model.add(Convolution2D(256, 3, 3, activation='relu'))
# model.add(MaxPooling2D((2,2), strides=(2,2)))
# # Layer 4 (CONV)
# model.add(ZeroPadding2D((1,1)))
# model.add(Convolution2D(512, 3, 3, activation='relu'))
# model.add(ZeroPadding2D((1,1)))
# model.add(Convolution2D(512, 3, 3, activation='relu'))
# model.add(ZeroPadding2D((1,1)))
# model.add(Convolution2D(512, 3, 3, activation='relu'))
# model.add(MaxPooling2D((2,2), strides=(2,2)))
# # Layer 5 (CONV)
# model.add(ZeroPadding2D((1,1)))
# model.add(Convolution2D(512, 3, 3, activation='relu'))
# model.add(ZeroPadding2D((1,1)))
# model.add(Convolution2D(512, 3, 3, activation='relu'))
# model.add(ZeroPadding2D((1,1)))
# model.add(Convolution2D(512, 3, 3, activation='relu'))
# model.add(MaxPooling2D((2,2), strides=(2,2)))
# # Layer 6 (MPL)
# model.add(Flatten())
# model.add(Dense(4096, activation='relu'))
# model.add(Dropout(0.5))
# model.add(Dense(4096, activation='relu'))
# model.add(Dropout(0.5))
# model.add(Dense(10, activation='softmax'))

# if weights_path:
# model.load_weights(weights_path)
# return model

In [84]:

''' This callback will stop the training when there is no improvement in
the validation loss for three consecutive epochs.'''

# Model weights are saved at the end of every epoch

early_stopping = EarlyStopping(monitor='val_loss', mode='min', verbose=1)

In [85]:

'''ModelCheckpoint callback is used in conjunction with training using

model.fit()
to save a model or weights (in a checkpoint file) at some interval, so the
model
or weights can be loaded later to continue the training from the state
saved.'''

checkpoint =
ModelCheckpoint(filepath='saved_models/weights_best_vgg16_model.hdf5',
monitor='val_loss', mode='max',
verbose=1, save_best_only=True)

In [86]:

train_samples = 17943
valid_samples = 4481

Define model 1
In [87]:

!rm -f input/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels.h5

In [88]:

def VGG16_MODEL(img_rows=IMG_SIZE, img_cols=IMG_SIZE, color_type=3):

# Remove fully connected layer and replace
# with softmax for classifying 10 classes
model_vgg16_1 = VGG16(weights="imagenet", include_top=False)

# Freeze all layers of the pre-trained model

for layer in model_vgg16_1.layers:
layer.trainable = False

x = model_vgg16_1.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
predictions = Dense(CLASSES, activation = 'softmax')(x)
 model = Model(input = model_vgg16_1.input, output = predictions)

return model

In [89]:

print("Model 1 network...")
model_vgg16_1 = VGG16_MODEL(img_rows=IMG_SIZE, img_cols=IMG_SIZE)

model_vgg16_1.summary()

model_vgg16_1.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])

Model 1 network...
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_3 (InputLayer) (None, None, None, 3) 0
_________________________________________________________________
block1_conv1 (Conv2D) (None, None, None, 64) 1792
_________________________________________________________________
block1_conv2 (Conv2D) (None, None, None, 64) 36928
_________________________________________________________________
block1_pool (MaxPooling2D) (None, None, None, 64) 0
_________________________________________________________________
block2_conv1 (Conv2D) (None, None, None, 128) 73856
_________________________________________________________________
block2_conv2 (Conv2D) (None, None, None, 128) 147584
_________________________________________________________________
block2_pool (MaxPooling2D) (None, None, None, 128) 0
_________________________________________________________________
block3_conv1 (Conv2D) (None, None, None, 256) 295168
_________________________________________________________________
block3_conv2 (Conv2D) (None, None, None, 256) 590080
_________________________________________________________________
block3_conv3 (Conv2D) (None, None, None, 256) 590080
_________________________________________________________________
block3_pool (MaxPooling2D) (None, None, None, 256) 0
_________________________________________________________________
block4_conv1 (Conv2D) (None, None, None, 512) 1180160
_________________________________________________________________
block4_conv2 (Conv2D) (None, None, None, 512) 2359808
_________________________________________________________________
block4_conv3 (Conv2D) (None, None, None, 512) 2359808
_________________________________________________________________
block4_pool (MaxPooling2D) (None, None, None, 512) 0
_________________________________________________________________
block5_conv1 (Conv2D) (None, None, None, 512) 2359808
_________________________________________________________________
block5_conv2 (Conv2D) (None, None, None, 512) 2359808
_________________________________________________________________
block5_conv3 (Conv2D) (None, None, None, 512) 2359808
_________________________________________________________________
block5_pool (MaxPooling2D) (None, None, None, 512) 0
_________________________________________________________________
global_average_pooling2d_3 ( (None, 512) 0
_________________________________________________________________
dense_5 (Dense) (None, 1024) 525312
_________________________________________________________________
dense_6 (Dense) (None, 10) 10250
=================================================================
Total params: 15,250,250
Trainable params: 535,562
Non-trainable params: 14,714,688
_________________________________________________________________

/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:16: UserWarning:
Update your `Model` call to the Keras 2 API: `Model(inputs=Tensor("in...,
outputs=Tensor("de...)`
app.launch_new_instance()

Train model 1
In [90]:

history = model_vgg16_1.fit_generator(training_generator,
steps_per_epoch = train_samples // BATCHES,
epochs = EPOCHS,
callbacks=[early_stopping, checkpoint],
verbose = 1,
class_weight='auto',
validation_data = validation_generator,
validation_steps = valid_samples // BATCHES)

Epoch 1/30
358/358 [==============================] - 313s 875ms/step - loss: 2.1577 -
acc: 0.2118 - val_loss: 1.8197 - val_acc: 0.3360

Epoch 00001: val_loss improved from -inf to 1.81966, saving model to

saved_models/weights_best_vgg16_model.hdf5
Epoch 2/30
358/358 [==============================] - 301s 842ms/step - loss: 1.8509 -
acc: 0.3527 - val_loss: 1.5312 - val_acc: 0.4489

Epoch 00002: val_loss did not improve from 1.81966

Epoch 3/30
358/358 [==============================] - 300s 838ms/step - loss: 1.6851 -
acc: 0.4112 - val_loss: 1.4860 - val_acc: 0.5015

Epoch 00003: val_loss did not improve from 1.81966

Epoch 4/30
358/358 [==============================] - 300s 838ms/step - loss: 1.5848 -
acc: 0.4504 - val_loss: 1.1745 - val_acc: 0.6125

Epoch 00004: val_loss did not improve from 1.81966

Epoch 5/30
358/358 [==============================] - 300s 838ms/step - loss: 1.4937 -
acc: 0.4851 - val_loss: 0.9911 - val_acc: 0.6570

Epoch 00005: val_loss did not improve from 1.81966

Epoch 6/30
358/358 [==============================] - 300s 837ms/step - loss: 1.4455 -
acc: 0.4984 - val_loss: 1.1457 - val_acc: 0.5922

Epoch 00006: val_loss did not improve from 1.81966

Epoch 00006: early stopping

In [91]:

model_vgg16_1.load_weights('saved_models/weights_best_vgg16_model.hdf5')

Visualize model 1 results

from backcalls

Plotting the accuracy and loss of the train and validation data to improve our
model

In [92]:

def plot_train_history(history):
# Summarize history for accuracy
plt.figure(figsize = (8, 5))
#plt.xticks(np.arange(0, 10))
#plt.yticks(np.arange(0, 100))
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

# Summarize history for loss

plt.figure(figsize = (8, 5))
#plt.xticks(np.arange(0, 10))
#plt.yticks(np.arange(0, 100))
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='lower left')
plt.show()

In [93]:
plot_train_history(history)
In [94]:

# Evaluate the performance of the new model

score = model_vgg16_1.evaluate_generator(validation_generator,
valid_samples // BATCHES, verbose = 1)
print("validation Score:", score[0])
print("validation Accuracy:", score[1])

89/89 [==============================] - 27s 308ms/step

validation Score: 1.8196598204334131
validation Accuracy: 0.335955056288604

MODEL 2

Define model 2
In [95]:

!rm -f input/vggweights/vgg16_weights.h5
In [96]:

def VGG16_MODEL(img_rows=IMG_SIZE, img_cols=IMG_SIZE, color_type=3):

# Remove fully connected layer and replace
# with softmax for classifying 10 classes
vgg16_model_2 = VGG16(weights="imagenet", include_top=False)

# Freeze all layers of the pre-trained model

for layer in vgg16_model_2.layers:
layer.trainable = False

x = vgg16_model_2.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
predictions = Dense(CLASSES, activation = 'softmax')(x)

model = Model(input = vgg16_model_2.input, output = predictions)

return model

In [97]:

print("Loading network...")
model_vgg16_2 = VGG16_MODEL(img_rows=IMG_SIZE, img_cols=IMG_SIZE)

model_vgg16_2.summary()

model_vgg16_2.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])

Loading network...
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_4 (InputLayer) (None, None, None, 3) 0
_________________________________________________________________
block1_conv1 (Conv2D) (None, None, None, 64) 1792
_________________________________________________________________
block1_conv2 (Conv2D) (None, None, None, 64) 36928
_________________________________________________________________
block1_pool (MaxPooling2D) (None, None, None, 64) 0
_________________________________________________________________
block2_conv1 (Conv2D) (None, None, None, 128) 73856
_________________________________________________________________
block2_conv2 (Conv2D) (None, None, None, 128) 147584
_________________________________________________________________
block2_pool (MaxPooling2D) (None, None, None, 128) 0
_________________________________________________________________
block3_conv1 (Conv2D) (None, None, None, 256) 295168
_________________________________________________________________
block3_conv2 (Conv2D) (None, None, None, 256) 590080
_________________________________________________________________
block3_conv3 (Conv2D) (None, None, None, 256) 590080
_________________________________________________________________
block3_pool (MaxPooling2D) (None, None, None, 256) 0
_________________________________________________________________
block4_conv1 (Conv2D) (None, None, None, 512) 1180160
_________________________________________________________________
block4_conv2 (Conv2D) (None, None, None, 512) 2359808
_________________________________________________________________
block4_conv3 (Conv2D) (None, None, None, 512) 2359808
_________________________________________________________________
block4_pool (MaxPooling2D) (None, None, None, 512) 0
_________________________________________________________________
block5_conv1 (Conv2D) (None, None, None, 512) 2359808
_________________________________________________________________
block5_conv2 (Conv2D) (None, None, None, 512) 2359808
_________________________________________________________________
block5_conv3 (Conv2D) (None, None, None, 512) 2359808
_________________________________________________________________
block5_pool (MaxPooling2D) (None, None, None, 512) 0
_________________________________________________________________
global_average_pooling2d_4 ( (None, 512) 0
_________________________________________________________________
dense_7 (Dense) (None, 1024) 525312
_________________________________________________________________
dense_8 (Dense) (None, 10) 10250
=================================================================
Total params: 15,250,250
Trainable params: 535,562
Non-trainable params: 14,714,688
_________________________________________________________________

/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:16: UserWarning:
Update your `Model` call to the Keras 2 API: `Model(inputs=Tensor("in...,
outputs=Tensor("de...)`
app.launch_new_instance()

Train model 2
In [98]:

history = model_vgg16_2.fit_generator(training_generator,
steps_per_epoch = train_samples // BATCHES,
epochs = EPOCHS,
callbacks=[early_stopping, checkpoint],
verbose = 1,
class_weight='auto',
validation_data = validation_generator,
validation_steps = valid_samples // BATCHES)

Epoch 1/30
358/358 [==============================] - 300s 839ms/step - loss: 2.1503 -
acc: 0.2163 - val_loss: 1.8277 - val_acc: 0.2980

Epoch 00001: val_loss improved from 1.81966 to 1.82766, saving model to

saved_models/weights_best_vgg16_model.hdf5
Epoch 2/30
358/358 [==============================] - 301s 842ms/step - loss: 1.8442 -
acc: 0.3558 - val_loss: 1.6410 - val_acc: 0.4182

Epoch 00002: val_loss did not improve from 1.82766

Epoch 3/30
358/358 [==============================] - 300s 838ms/step - loss: 1.6813 -
acc: 0.4182 - val_loss: 1.4178 - val_acc: 0.4863

Epoch 00003: val_loss did not improve from 1.82766

Epoch 4/30
358/358 [==============================] - 300s 837ms/step - loss: 1.5661 -
acc: 0.4608 - val_loss: 1.1625 - val_acc: 0.5814

Epoch 00004: val_loss did not improve from 1.82766

Epoch 5/30
358/358 [==============================] - 300s 839ms/step - loss: 1.4935 -
acc: 0.4850 - val_loss: 1.0131 - val_acc: 0.6540

Epoch 00005: val_loss did not improve from 1.82766

Epoch 6/30
358/358 [==============================] - 300s 837ms/step - loss: 1.4302 -
acc: 0.5061 - val_loss: 1.2649 - val_acc: 0.5459

Epoch 00006: val_loss did not improve from 1.82766

Epoch 00006: early stopping

Visualize model 2 results

from backcalls

In [99]:

def plot_train_history(history):
# Summarize history for accuracy
plt.figure(figsize = (8, 5))
#plt.xticks(np.arange(0, 10))
#plt.yticks(np.arange(0, 100))
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

# Summarize history for loss

plt.figure(figsize = (8, 5))
#plt.xticks(np.arange(0, 10))
#plt.yticks(np.arange(0, 100))
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
 plt.title('Model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='lower left')
plt.show()

In [100]:

plot_train_history(history)
In [101]:

# Evaluate the performance of the new model

score = model_vgg16_2.evaluate_generator(validation_generator,
valid_samples // BATCHES, verbose = 1)
print("Validation Score:", score[0])
print("Validation Accuracy:", score[1])

89/89 [==============================] - 27s 305ms/step

Validation Score: 1.2709403820754437
Validation Accuracy: 0.5435955063178298

TESTING FINAL MODEL

In [102]:

def prediction():
for i in np.arange(10):
img_brute = test_data[i]

im = cv2.resize(cv2.cvtColor(img_brute, cv2.COLOR_BGR2RGB),
(IMG_SIZE,IMG_SIZE)).astype(np.float32) / 255.0
 im = np.expand_dims(im, axis =0)

img_display = cv2.resize(img_brute,(IMG_SIZE,IMG_SIZE))
plt.imshow(img_display, cmap='gray')

y_preds = model_vgg16_1.predict(im, batch_size=BATCHES, verbose=1)

print(y_preds)
y_prediction = np.argmax(y_preds)
print('Y Prediction: {}'.format(y_prediction))
print('Predicted as:
{}'.format(CAT_MAP.get('c{}'.format(y_prediction))))

plt.show()

In [103]:

prediction()

1/1 [==============================] - 0s 109ms/step

[[0.02529937 0.17061111 0.10869354 0.21523008 0.09752461 0.03731625
0.07626924 0.0718556 0.11949571 0.07770445]]
Y Prediction: 3
Predicted as: Texting - left

1/1 [==============================] - 0s 6ms/step

[[0.14406215 0.07679002 0.06996788 0.11225218 0.0744263 0.16400713
0.03309441 0.11207563 0.13081154 0.08251275]]
Y Prediction: 5
Predicted as: Operating the radio
1/1 [==============================] - 0s 7ms/step
[[0.11309579 0.316222 0.07044181 0.06531883 0.01432761 0.01459904
0.0303529 0.2020727 0.08494909 0.08862015]]
Y Prediction: 1
Predicted as: Texting - right

1/1 [==============================] - 0s 7ms/step

[[0.02271187 0.03968159 0.2648685 0.02466394 0.02365336 0.02933127
0.03403525 0.2215907 0.29069212 0.04877144]]
Y Prediction: 8
Predicted as: Hair and makeup

1/1 [==============================] - 0s 7ms/step

[[0.06332105 0.10492047 0.05344269 0.35476425 0.10214324 0.01553122
0.07478541 0.10169374 0.09167489 0.03772296]]
Y Prediction: 3
Predicted as: Texting - left
1/1 [==============================] - 0s 7ms/step
[[0.03823161 0.18702559 0.06736635 0.16965327 0.07644825 0.06013601
0.11681363 0.11435039 0.11560682 0.05436817]]
Y Prediction: 1
Predicted as: Texting - right

1/1 [==============================] - 0s 7ms/step

[[0.01816014 0.17984138 0.17333056 0.1425233 0.05690064 0.00791651
0.06918186 0.12251849 0.19035022 0.03927679]]
Y Prediction: 8
Predicted as: Hair and makeup
1/1 [==============================] - 0s 7ms/step
[[0.0366651 0.12666202 0.13342209 0.03730501 0.03417845 0.04574389
0.04998513 0.29226276 0.158625 0.08515056]]
Y Prediction: 7
Predicted as: Reaching behind

1/1 [==============================] - 0s 6ms/step

[[0.04912293 0.07638273 0.15630975 0.03469994 0.03973481 0.10763451
0.05124296 0.19937329 0.22215132 0.06334769]]
Y Prediction: 8
Predicted as: Hair and makeup

1/1 [==============================] - 0s 6ms/step

[[0.04716836 0.13811314 0.12138977 0.04751166 0.01918852 0.02311692
0.03819726 0.44342202 0.07954597 0.04234634]]
Y Prediction: 7
Predicted as: Reaching behind