02_asl about:srcdoc

Image Classification of an American Sign Language

Dataset

In this section, we will perform the data preparation, model creation, and model training steps we observed in
the last section using a different dataset: images of hands making letters in American Sign Language
(http://www.asl.gs/).

Objectives

Prepare image data for training

Create and compile a simple model for image classification
Train an image classification model and observe the results

American Sign Language Dataset

The American Sign Language alphabet (http://www.asl.gs/) contains 26 letters. Two of those letters (j and z)
require movement, so they are not included in the training dataset.

Kaggle

This dataset is available from the website Kaggle (http://www.kaggle.com), which is a fantastic place to find
datasets and other deep learning resources. In addition to providing resources like datasets and "kernels" that
are like these notebooks, Kaggle hosts competitions that you can take part in, competing with others in
training highly accurate models.

If you're looking to practice or see examples of many deep learning projects, Kaggle is a great site to visit.

Loading the Data

1 of 11 12/07/2021, 04:59 pm
02_asl about:srcdoc

This dataset is not available via Keras in the same way that MNIST is, so let's learn how to load custom data.
By the end of this section we will have x_train , y_train , x_valid , and y_valid variables as
before.

Reading in the Data

The sign language dataset is in CSV (https://en.wikipedia.org/wiki/Comma-separated_values) (Comma

Separated Values) format, the same data structure behind Microsoft Excel and Google Sheets. It is a grid of
rows and columns with labels at the top, as seen in the train (data/asl_data/sign_mnist_train.csv) and valid
(data/asl_data/sign_mnist_valid.csv) datasets (they may take a moment to load).

To load and work with the data, we'll be using a library called Pandas (https://pandas.pydata.org/), which is a
highly performant tool for loading and manipulating data. We'll read the CSV files into a format called a
DataFrame (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html).

In [1]: import pandas as pd

Pandas has a read_csv (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)

method that expects a csv file, and returns a DataFrame:

In [2]: train_df = pd.read_csv("data/asl_data/sign_mnist_train.csv")

valid_df = pd.read_csv("data/asl_data/sign_mnist_valid.csv")

Exploring the Data

Let's take a look at our data. We can use the head (https://pandas.pydata.org/pandas-docs/stable/reference
/api/pandas.DataFrame.head.html) method to print the first few rows of the DataFrame. Each row is an image
which has a label column, and also, 784 values representing each pixel value in the image, just like with
the MNIST dataset. Note that the labels currently are numerical values, not letters of the alphabet:

2 of 11 12/07/2021, 04:59 pm
02_asl about:srcdoc

In [3]: train_df.head()

Out[3]:
label pixel1 pixel2 pixel3 pixel4 pixel5 pixel6 pixel7 pixel8 pixel9 ... pixel775 pixel776

0 3 107 118 127 134 139 143 146 150 153 ... 207

1 6 155 157 156 156 156 157 156 158 158 ... 69

2 2 187 188 188 187 187 186 187 188 187 ... 202

3 2 211 211 212 212 211 210 211 210 210 ... 235

4 12 164 167 170 172 176 179 180 184 185 ... 92

5 rows × 785 columns

Extracting the Labels

As with MNIST, we would like to store our training and validation labels in y_train and y_valid
variables. Here we create those variables and then delete the labels from our original dataframes, where they
are no longer needed:

In [4]: y_train = train_df['label']

y_valid = valid_df['label']
del train_df['label']
del valid_df['label']

Extracting the Images

As with MNIST, we would like to store our training and validation images in x_train and x_valid
variables. Here we create those variables:

In [5]: x_train = train_df.values

x_valid = valid_df.values

Summarizing the Training and Validation Data

We now have 27,455 images with 784 pixels each for training...

In [6]: x_train.shape

Out[6]: (27455, 784)

3 of 11 12/07/2021, 04:59 pm
02_asl about:srcdoc

...as well as their corresponding labels:

In [7]: y_train.shape

Out[7]: (27455,)

For validation, we have 7,172 images...

In [8]: x_valid.shape

Out[8]: (7172, 784)

...and their corresponding labels:

In [9]: y_valid.shape

Out[9]: (7172,)

Visualizing the Data

To visualize the images, we will again use the matplotlib library. We don't need to worry about the details of
this visualization, but if interested, you can learn more about matplotlib (https://matplotlib.org/) at a later time.

Note that we'll have to reshape the data from its current 1D shape of 784 pixels, to a 2D shape of 28x28
pixels to make sense of the image:

In [10]: import matplotlib.pyplot as plt

plt.figure(figsize=(40,40))

num_images = 20
for i in range(num_images):
row = x_train[i]
label = y_train[i]

image = row.reshape(28,28)
plt.subplot(1, num_images, i+1)
plt.title(label, fontdict={'fontsize': 30})
plt.axis('off')
plt.imshow(image, cmap='gray')

Exercise: Normalize the Image Data

4 of 11 12/07/2021, 04:59 pm
02_asl about:srcdoc

As we did with the MNIST dataset, we are going to normalize the image data, meaning that their pixel values,
instead of being between 0 and 255 as they are currently:

In [11]: x_train.min()

Out[11]: 0

In [12]: x_train.max()

Out[12]: 255

...should be floating point values between 0 and 1. Use the following cell to work. If you get stuck, look at the
solution below.

In [13]: # TODO: Normalize x_train and x_valid.

Solution

Click on the '...' below to show the solution.

In [14]: x_train = x_train / 255

x_valid = x_valid / 255

Exercise: Categorize the Labels

As we did with the MNIST dataset, we are going to categorically encode the labels. Recall that we can use
the keras.utils.to_categorical (https://www.tensorflow.org/api_docs/python/tf/keras/utils/to_categorical) method
to accomplish this by passing it the values to encode, and, the number of categories to encode it into. Do your
work in the cell below. We have imported keras and set the number of categories (24) for you.

In [15]: import tensorflow.keras as keras

num_classes = 24

In [16]: # TODO: Categorically encode y_train and y_valid.

Solution

Click on the '...' below to show the solution.

5 of 11 12/07/2021, 04:59 pm
02_asl about:srcdoc

In [17]: y_train = keras.utils.to_categorical(y_train, num_classes)

y_valid = keras.utils.to_categorical(y_valid, num_classes)

Exercise: Build the Model

The data is all prepared, we have normalized images for training and validation, as well as categorically
encoded labels for training and validation.

For this exercise we are going to build a sequential model. Just like last time, build a model that:

Has a dense input layer. This layer should contain 512 neurons, use the relu activation function, and
expect input images with a shape of (784,)
Has a second dense layer with 512 neurons which uses the relu activation function
Has a dense output layer with neurons equal to the number of classes, using the softmax activation
function

Do your work in the cell below, creating a model variable to store the model. We've imported the Keras
Sequental (https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) model class and Dense
(https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) layer class to get you started. Reveal the
solution below for a hint:

In [18]: from tensorflow.keras.models import Sequential

from tensorflow.keras.layers import Dense

In [19]: # TODO: build a model following the guidelines above.

Solution

Click on the '...' below to show the solution.

In [20]: model = Sequential()

model.add(Dense(units = 512, activation='relu', input_shape=(784,)))
model.add(Dense(units = 512, activation='relu'))
model.add(Dense(units = num_classes, activation='softmax'))

Summarizing the Model

Run the cell below to summarize the model you just created:

6 of 11 12/07/2021, 04:59 pm
02_asl about:srcdoc

In [21]: model.summary()

Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 512) 401920
_________________________________________________________________
dense_1 (Dense) (None, 512) 262656
_________________________________________________________________
dense_2 (Dense) (None, 24) 12312
=================================================================
Total params: 676,888
Trainable params: 676,888
Non-trainable params: 0
_________________________________________________________________

Compiling the Model

We'll compile (https://www.tensorflow.org/api_docs/python/tf/keras/Sequential#compile) our model with the

same options as before, using categorical crossentropy (https://www.tensorflow.org/api_docs/python/tf/keras
/losses/CategoricalCrossentropy) to reflect the fact that we want to fit into one of many categories, and
measuring the accuracy of our model:

In [22]: model.compile(loss='categorical_crossentropy', metrics=['accuracy'])

Exercise: Train the Model

Use the model's fit method to train it for 20 epochs using the training and validation images and labels
created above:

In [23]: # TODO: Train the model for 20 epochs.

Solution

Click on the '...' below to show the solution.

7 of 11 12/07/2021, 04:59 pm
02_asl about:srcdoc

In [24]: model.fit(x_train, y_train, epochs=20, verbose=1, validation_data=(x_va

lid, y_valid))

8 of 11 12/07/2021, 04:59 pm
02_asl about:srcdoc

Epoch 1/20
858/858 [==============================] - 2s 3ms/step - loss: 1.9568
- accuracy: 0.3751 - val_loss: 1.7991 - val_accuracy: 0.4265
Epoch 2/20
858/858 [==============================] - 2s 3ms/step - loss: 0.9601
- accuracy: 0.6735 - val_loss: 0.8740 - val_accuracy: 0.6828
Epoch 3/20
858/858 [==============================] - 2s 3ms/step - loss: 0.5910
- accuracy: 0.8003 - val_loss: 0.9801 - val_accuracy: 0.7057
Epoch 4/20
858/858 [==============================] - 2s 3ms/step - loss: 0.4023
- accuracy: 0.8731 - val_loss: 1.1534 - val_accuracy: 0.6959
Epoch 5/20
858/858 [==============================] - 2s 3ms/step - loss: 0.2888
- accuracy: 0.9100 - val_loss: 0.6707 - val_accuracy: 0.8422
Epoch 6/20
858/858 [==============================] - 2s 3ms/step - loss: 0.2470
- accuracy: 0.9320 - val_loss: 0.7619 - val_accuracy: 0.8349
Epoch 7/20
858/858 [==============================] - 2s 3ms/step - loss: 0.2291
- accuracy: 0.9432 - val_loss: 1.0552 - val_accuracy: 0.7916
Epoch 8/20
858/858 [==============================] - 2s 3ms/step - loss: 0.2081
- accuracy: 0.9509 - val_loss: 0.8447 - val_accuracy: 0.8444
Epoch 9/20
858/858 [==============================] - 2s 3ms/step - loss: 0.1903
- accuracy: 0.9582 - val_loss: 1.1173 - val_accuracy: 0.8249
Epoch 10/20
858/858 [==============================] - 2s 3ms/step - loss: 0.1854
- accuracy: 0.9616 - val_loss: 1.0967 - val_accuracy: 0.8015
Epoch 11/20
858/858 [==============================] - 2s 3ms/step - loss: 0.1535
- accuracy: 0.9659 - val_loss: 1.1120 - val_accuracy: 0.8533
Epoch 12/20
858/858 [==============================] - 2s 3ms/step - loss: 0.1618
- accuracy: 0.9672 - val_loss: 1.1863 - val_accuracy: 0.8165
Epoch 13/20
858/858 [==============================] - 2s 3ms/step - loss: 0.1452
- accuracy: 0.9717 - val_loss: 1.3223 - val_accuracy: 0.8228
Epoch 14/20
858/858 [==============================] - 2s 3ms/step - loss: 0.1378
- accuracy: 0.9733 - val_loss: 1.2315 - val_accuracy: 0.8433
Epoch 15/20
858/858 [==============================] - 2s 3ms/step - loss: 0.1503
- accuracy: 0.9738 - val_loss: 1.9701 - val_accuracy: 0.7850
Epoch 16/20
858/858 [==============================] - 2s 3ms/step - loss: 0.1282
- accuracy: 0.9772 - val_loss: 1.2698 - val_accuracy: 0.8397
Epoch 17/20
858/858 [==============================] - 2s 3ms/step - loss: 0.1218
- accuracy: 0.9794 - val_loss: 3.9590 - val_accuracy: 0.6695
Epoch 18/20
858/858 [==============================] - 2s 3ms/step - loss: 0.1364
- accuracy: 0.9775 - val_loss: 1.4328 - val_accuracy: 0.8730
Epoch 19/20
858/858 [==============================] - 2s 3ms/step - loss: 0.1338

9 of 11 12/07/2021, 04:59 pm
02_asl about:srcdoc

- accuracy: 0.9790 - val_loss: 1.5981 - val_accuracy: 0.8608

Epoch 20/20
858/858 [==============================] - 2s 3ms/step - loss: 0.1341
- accuracy: 0.9794 - val_loss: 1.7698 - val_accuracy: 0.8480
Out[24]: <tensorflow.python.keras.callbacks.History at 0x7fae4827ccc0>

Discussion: What happened?

We can see that the training accuracy got to a fairly high level, but the validation accuracy was not as high.
What happened here?

Think about it for a bit before clicking on the '...' below to reveal the answer.

This is an example of the model learning to categorize the training data, but performing poorly against new
data that it has not been trained on. Essentially, it is memorizing the dataset, but not gaining a robust and
general understanding of the problem. This is a common issue called overfitting. We will discuss overfitting in
the next two lectures, as well as some ways to address it.

Summary

In this section you built your own neural network to perform image classification that is quite accurate.
Congrats!

At this point we should be getting somewhat familiar with the process of loading data (incuding labels),
preparing it, creating a model, and then training the model with prepared data.

Clear the Memory

Before moving on, please execute the following cell to clear up the GPU memory. This is required to move on
to the next notebook.

In [25]: import IPython

app = IPython.Application.instance()
app.kernel.do_shutdown(True)

Out[25]: {'status': 'ok', 'restart': True}

Next

10 of 11 12/07/2021, 04:59 pm
02_asl about:srcdoc

Now that you have built some very basic, somewhat effective models, we will begin to learn about more
sophisticated models, including Convolutional Neural Networks.

11 of 11 12/07/2021, 04:59 pm