Training of Recurrent Neural Networks (RNN) in TensorFlow

Recurrent Neural Networks (RNNs) are a type of neural network designed to handle sequential data. They maintain hidden states that capture information from previous steps. In this article we will be learning to implement RNN model using TenserFlow.

Here we will be using a clothing brands reviews as dataset and will be using RNN to analyze there reviews.

1. Importing Libraries

We will be importing Pandas, NumPy, Matplotlib, Seaborn, TensorFlow, Keras, NLTK and Scikit-learn for implemntation.

import warnings
from tensorflow.keras.utils import pad_sequences
from tensorflow.keras.preprocessing.text import Tokenizer
from sklearn.model_selection import train_test_split
import tensorflow as tf
from tensorflow import keras
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px
import numpy as np

import re
import nltk
nltk.download('all')
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
lemm = WordNetLemmatizer()

warnings.filterwarnings("ignore")

2. Loading the Dataset

You can download the dataset used in this article from here. We load the dataset using pd.read_csv() and display the first 7 rows with data.head(7). We remove rows where the Class Name column is null to clean the data.

data = pd.read_csv("Clothing Review.csv")
data.head(7)

data = data[data['Class Name'].isnull() == False]

Output:

We can use data.shape to give us dimensions of the dataset.

print(data.shape)

(23472, 10)

3. Performing Exploratory Data Analysis

EDA helps one understand how the data is distributed. To perform EDA one must perform various visualizing techniques so that one can understand the data before building a model.

We use sns.countplot() to visualize the count of each category in the ‘Class Name’ column. The plt.xticks(rotation=90) rotates the x-axis labels for better readability.

sns.countplot(data=data, x='Class Name', palette='rainbow')
plt.xticks(rotation=90)
plt.show()

Output:

We create a figure with size 12x5 inches using plt.subplots():

• Using plt.subplot(1, 2, 1) we plot a countplot of the Rating column
• Using plt.subplot(1, 2, 2) we plot a countplot of the Recommended IND column.

plt.subplots(figsize=(12, 5))
plt.subplot(1, 2, 1)
sns.countplot(data=data, x='Rating',palette="deep")

plt.subplot(1, 2, 2)
sns.countplot(data=data, x="Recommended IND", palette="deep")
plt.show()

Output:

We create a histogram using px.histogram() to show the frequency distribution of Age. The histogram includes a box plot as a marginal plot. Data is colored based on Recommended IND values using green and red colors. Number of bins is set to the range from 18 to 65. fig.update_layout() adjusts the gap between bars.

fig = px.histogram(data, marginal='box',
                   x="Age", title="Age Group",
                   color="Recommended IND",
                   nbins=65-18,
                   color_discrete_sequence=['green', 'red'])
fig.update_layout(bargap=0.2)

Output:

The histogram on the bottom shows age distribution with green bars for recommended individuals and red bars for non-recommended ones. The box plots at the top display the spread and outliers of ages for each recommendation group helping to visualize differences in age distribution between the two groups.

We can visualize the distribution of the age columns data along with the Rating.

fig = px.histogram(data,
                   x="Age",
                   marginal='box',
                   title="Age Group",
                   color="Rating",
                   nbins=65-18,
                   color_discrete_sequence
                   =['black', 'green', 'blue', 'red', 'yellow'])
fig.update_layout(bargap=0.2)

Output:

The histogram at the bottom represents the count of individuals in each age group with bars color coded by rating from 1 to 5. The boxplots at the top provide a summary of age distribution for each rating showing the median, interquartile range and outliers. It helps to analyze how ratings vary with age groups.

4. Prepare the Data to build Model

Since we are working on the NLP-based dataset it could be valid to use Text columns as the feature. So we select the features that are text and the Rating column is used for Sentiment Analysis. By the above Rating counterplot we can observe that there is too much of an imbalance between the rating. So all the rating above 3 is made as 1 and below 3 as 0. 

def filter_score(rating):
    return int(rating > 3)

features = ['Class Name', 'Title', 'Review Text']

X = data[features]
y = data['Rating']
y = y.apply(filter_score)

5. Text Preprocessing

The text data we have comes with too much noise. This noise can be in form of repeated words or commonly used sentences. In text preprocessing we need the text in the same format so we first convert the entire text into lowercase and then perform Lemmatization to remove the superposition of the words. Since we need clean text we also remove common words such as Stopwords and punctuation. 

Note: Lemmatization and stop words are some concepts used in NLP and they are apart from RNN.

def toLower(data):
    if isinstance(data, float):
        return '<UNK>'
    else:
        return data.lower()

stop_words = stopwords.words("english")

def remove_stopwords(text):
    no_stop = []
    for word in text.split(' '):
        if word not in stop_words:
            no_stop.append(word)
    return " ".join(no_stop)

def remove_punctuation_func(text):
    return re.sub(r'[^a-zA-Z0-9]', ' ', text)

X['Title'] = X['Title'].apply(toLower)
X['Review Text'] = X['Review Text'].apply(toLower)

X['Title'] = X['Title'].apply(remove_stopwords)
X['Review Text'] = X['Review Text'].apply(remove_stopwords)

X['Title'] = X['Title'].apply(lambda x: lemm.lemmatize(x))
X['Review Text'] = X['Review Text'].apply(lambda x: lemm.lemmatize(x))

X['Title'] = X['Title'].apply(remove_punctuation_func)
X['Review Text'] = X['Review Text'].apply(remove_punctuation_func)

X['Text'] = list(X['Title']+X['Review Text']+X['Class Name'])


X_train, X_test, y_train, y_test = train_test_split(
    X['Text'], y, test_size=0.25, random_state=42)

If you notice at the end of the code, we have created a new column "Text" which is of type list. The reason we did this is that we need to perform Tokenization on the entire feature taken to train the model. 

6. Tokenization

In Tokenization we convert the text into Vectors. Keras API supports text pre-processing. This API consists of Tokenizer that takes in the total num_words to create the Word index. OOV stands for out of vocabulary this is triggered when new text is encountered.  Also remember that we fit_on_texts only on training data and not testing. 

Vectors are numeric representation of data so that machine can easily be understand and use it for training.

tokenizer = Tokenizer(num_words=10000, oov_token='<OOV>')
tokenizer.fit_on_texts(X_train)

7. Padding the Text Data

Keras preprocessing helps in organizing the text. Padding helps in building models of the same size that further becomes easy to train neural network models. The padding adds extra zeros to satisfy the maximum length to feed a neural network. If the text length exceeds then it can be truncated from either the beginning or end. By default it is pre, we can set it to post or leave it as it is.

Both padding and tokenization are a part of NLP processing not specific for RNN task.

train_seq = tokenizer.texts_to_sequences(X_train)
test_seq = tokenizer.texts_to_sequences(X_test)

train_pad = pad_sequences(train_seq,
                          maxlen=40,
                          truncating="post",
                          padding="post")
test_pad = pad_sequences(test_seq,
                         maxlen=40,
                         truncating="post",
                         padding="post")

8. Building a Recurrent Neural Network (RNN) in TensorFlow

Now that the data is ready, the next step is building a Simple Recurrent Neural network. Before training with SImpleRNN, the data is passed through the Embedding layer to perform the equal size of Word Vectors. 

Note: We use return_sequences = True only when we need another layer to stack.

from tensorflow import keras

model = keras.models.Sequential()
model.add(keras.layers.Embedding(input_dim=10000, output_dim=128, input_length=40))
model.add(keras.layers.SimpleRNN(64, return_sequences=True))
model.add(keras.layers.SimpleRNN(64))
model.add(keras.layers.Dense(128, activation="relu"))
model.add(keras.layers.Dropout(0.4))
model.add(keras.layers.Dense(1, activation="sigmoid"))

model.build(input_shape=(None, 40))

model.summary()

Output:

9. Training the Model

In Tensorflow after developing a model, it needs to be compiled using the three important parameters i.e Optimizer, Loss Function and Evaluation metrics. We will be training the model on the train_pad which we preprocessed and use 5 epochs to calculate the accuracy on y_train dataset.

model.compile(loss="binary_crossentropy",
              optimizer="adam",
              metrics=["accuracy"])
              

history = model.fit(train_pad,
                    y_train,
                    epochs=5)

Output:

Here we can see our model got a accuracy of 92.86% which is pretty good for a RNN model.

You can also make RNN model using PyTorch and for that you can refer to this article: Implementing Recurrent Neural Networks in PyTorch