How to use PyTorch for sentiment analysis on textual data?

Sentiment Analysis is a natural language processing (NLP) task that involves identifying the emotion present in a given text. The amount of textual data on social media, consumer reviews, and other platforms is increasing, making sentiment analysis more and more crucial.

In this article, we present a step-by-step guide to performing sentiment analysis using PyTorch

Sentiment analysis on Textual Data using PyTorch

Sentiment analysis is a natural language processing (NLP) task that involves determining the sentiment expressed in a piece of text. PyTorch provides a flexible and efficient platform for building neural networks, making it well-suited for sentiment analysis tasks.

PyTorch for sentiment analysis on textual data is implemented with following steps:

Step 1: Importing Necessary Libraries

To begin, we import essential libraries, laying the foundation for our sentiment analysis implementation with PyTorch.

import torch
from torchtext import data
from torch.nn.utils.rnn import pad_sequence
from collections import Counter
import spacy
import torch.optim as optim
from torchtext.data import BucketIterator
from pprint import pprint  # Importing pprint for pretty-printing

Step 2: Loading the Dataset

Spacy tokenizer is used alongside TorchText for tokenization purposes due to its effectiveness in splitting text into meaningful tokens.

The TEXT and LABEL are Field objects, representing the text and label fields of the dataset.

• TEXT is responsible for tokenizing and preprocessing the textual data.
• LABEL is responsible for handling the labels associated with the data

# Download and load the spaCy language model
!python -m spacy download en_core_web_sm
nlp = spacy.load('en_core_web_sm')

# Define data fields
TEXT = data.Field(include_lengths=True)
LABEL = data.LabelField(dtype=torch.float)

# Load the IMDB dataset using torchtext.datasets
from torchtext.datasets import IMDB
train_data, test_data = IMDB.splits(TEXT, LABEL)

Step 3: Tokenizing and Removing Punctuation

• We tokenize each review using the Spacy tokenizer with torchtext for loading the data. The tokenized reviews are stored in the 'text' field of the dataset.
• All individual words are extracted from the tokenized reviews and with count of the occurrences of each word.
• Each tokenized review is encoded into a list of integers using the created dictionary that maps each unique word to an integer.
• We use 'pad_sequence' to ensure that all encoded reviews have the same length by padding with zeros.

Step 1: Tokenizing and removing punctuation

# Tokenization function using spaCy
def tokenize(text):
    return [token.text for token in nlp(text)]

TEXT.tokenize = tokenize # tokenization and remove punctuation
TEXT.build_vocab(train_data, max_size=10000) # Building vocabulary
LABEL.build_vocab(train_data)
# Tokenize and preprocess reviews
tokenized_reviews = [data.Example.fromlist([review.text, None], [('text', TEXT), ('label', None)]) for review in train_data.examples]
all_words = [word for example in tokenized_reviews for word in example.text]

Step 2: Count and sort all words based on count. Also, a dictionary to convert words to Integers based on the number of occurrence of the word.

word_counts = Counter(all_words) # Sorting based on occurrences
sorted_words = sorted(word_counts, key=word_counts.get, reverse=True)
# dictionary for integers
word_to_int = {word: idx + 1 for idx, word in enumerate(sorted_words)} 

Step 3: Encode review in list of Integer with dictionary

encoded_reviews = [[word_to_int[word] for word in example.text] for example in tokenized_reviews] # Encoding reviews into lists of integers

# Making encoded reviews of the same length
padded_reviews = pad_sequence([torch.tensor(review) for review in encoded_reviews])

# Pretty-print a subset of the outputs
pprint({
    "Tokenized Reviews": [example.text[:10] for example in tokenized_reviews[:5]],  # Show first 5 reviews, first 10 tokens
    "All Words": all_words[:50],  # Show first 50 words
    "Sorted Words": sorted_words[:50],  # Show first 50 sorted words
    "Word to Integer Dictionary": {k: word_to_int[k] for k in list(word_to_int)[:10]},  # Show first 10 entries
    "Encoded Reviews": encoded_reviews[:5],  # Show first 5 encoded reviews
    "Padded Reviews Shape": padded_reviews.shape  # Show shape of padded_reviews
})

Output:

# Sample Output 
{'All Words': ['Michelle',
               'Rodriguez',
               'is',
               'the',
               'defining',
               'actress',]
'Encoded Reviews': [[3450,
                      11802,
                      6,
                      1,
                      10912,
                      684,
                      34,
                      85,
                      26,
                      1,
                      18108,
                      1601,
                      16]]
'Padded Reviews Shape': torch.Size([2470, 25000]),
 'Sorted Words': ['the',
                  'a',
                  'and',
                  'of',
                  'to',
                  'is',
                  'in',
                  'I',
                  'that',
                  'this',
                  'it',
                  '/><br',
                  'was',
                  'as',
                  'with',
                  'for']
 'Tokenized Reviews': [['Michelle',
                        'Rodriguez',
                        'is',
                        'the',
                        'defining',
                        'actress',
                        'who',
                        'could',
                        'be',
                        'the']]
 'Word to Integer Dictionary': {'I': 8,
                                'a': 2,
                                'and': 3,
                                'in': 7,
                                'is': 6,
                                'of': 4,
                                'that': 9,
                                'the': 1,
                                'this': 10,
                                'to': 5}}

This code essentially demonstrates the process of creating a simple dataset from raw reviews, tokenizing the text, and preparing the data for further processing, such as training a machine learning model for sentiment analysis.

Transforming Sentiment Labels

Converting sentiment labels to numeric form (e.g., assigning 0 for negative and 1 for positive) - streamlines model training. This approach equips the model with a numerical target it can predict; thus enhancing its performance.

# Get the labels from the dataset
labels = [example.label for example in train_data.examples]

# Convert labels to numeric format
numeric_labels = [1 if label == 'positive' else 0 for label in labels]

# Print the numeric labels
print("Numeric Labels:", numeric_labels)

Output:

# Sample Output
Numeric Labels: [1, 0, 1, 0, 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]

Splitting the dataset

We use the training set for rigorous model training; meanwhile, the validation set for functioning as an independent evaluator.

• The first two lines convert the padded_reviews and numeric_labels lists into PyTorch tensors. Convert the features (padded_reviews) and labels (numeric_labels) to PyTorch tensors as it is important to convert it into a tensor to enable efficient processing within PyTorch.
• Check and Correct Size Mismatch: This conditional statement checks if the number of samples in padded_reviews_tensor matches the number of samples in numeric_labels_tensor. If there is a mismatch, it ensures that both tensors have the same number of samples by trimming the longer tensor to match the length of the shorter one. This step ensures that each text review is paired with its corresponding sentiment label correctly.

The subsequent lines of code split the dataset into training and validation sets, extract features and labels for training and validation, and convert them into tensors for further processing.

padded_reviews_tensor = torch.tensor(padded_reviews, dtype=torch.long)
numeric_labels_tensor = torch.tensor(numeric_labels, dtype=torch.float)

# Check if sizes match and correct if needed
if padded_reviews_tensor.size(0) != numeric_labels_tensor.size(0):
    min_size = min(padded_reviews_tensor.size(0), numeric_labels_tensor.size(0))
    padded_reviews_tensor = padded_reviews_tensor[:min_size]
    numeric_labels_tensor = numeric_labels_tensor[:min_size]

# Combine features and labels into a TensorDataset
dataset = torch.utils.data.TensorDataset(padded_reviews_tensor, numeric_labels_tensor)

# Calculate the size of the training set
train_size = int(0.8 * len(dataset))
valid_size = len(dataset) - train_size

# Split the dataset into training and validation sets
train_data, valid_data = torch.utils.data.random_split(dataset, [train_size, valid_size])

# Extract features and labels for training and validation
X_train, y_train = zip(*train_data)
X_valid, y_valid = zip(*valid_data)

# Convert X_train, y_train, X_valid, y_valid to tensors (if needed)
X_train_tensor = torch.stack(X_train)
y_train_tensor = torch.stack(y_train)
X_valid_tensor = torch.stack(X_valid)
y_valid_tensor = torch.stack(y_valid)

# Print the shapes of the tensors (for verification)
print("X_train_tensor shape:", X_train_tensor.shape)
print("y_train_tensor shape:", y_train_tensor.shape)
print("X_valid_tensor shape:", X_valid_tensor.shape)
print("y_valid_tensor shape:", y_valid_tensor.shape)

# Print shapes of training and validation sets
print(f"Training Set - Features: {len(X_train)}, Labels: {len(y_train)}")
print(f"Validation Set - Features: {len(X_valid)}, Labels: {len(y_valid)}")

Output:

X_train_tensor shape: torch.Size([1976, 25000])
y_train_tensor shape: torch.Size([1976])
X_valid_tensor shape: torch.Size([494, 25000])
y_valid_tensor shape: torch.Size([494])

Training Set - Features: 1976, Labels: 1976
Validation Set - Features: 494, Labels: 494

Model Building

In this instance, we employ a Long Short-Term Memory (LSTM) model renowned for its ability to capture sequential dependencies within data.

Incorporate an embedding layer, LSTM layer, and a linear layer into the model architecture using PyTorch's 'nn.Module' class.

class SentimentModel(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.rnn = nn.LSTM(embedding_dim, hidden_dim)
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, text, text_lengths):
        embedded = self.embedding(text)
        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths.to('cpu'), enforce_sorted=False)
        packed_output, (hidden, cell) = self.rnn(packed_embedded)
        output = self.fc(hidden[-1])
        return output

Initialization and Training

The code snippet sets up the necessary components for training a sentiment analysis model using PyTorch, including data iterators, model architecture, optimizer, and loss criterion, and then proceeds to train the model using the training data.

BucketIterator provided by the torchtext.data module automatically iterates batches sequences of similar lengths together to minimize padding within batches, which improves efficiency during training. By specifying a batch size of 64 and enabling shuffling, we ensure that the data is shuffled and processed in batches of 64 samples.

# Create iterators
train_iterator, valid_iterator = BucketIterator.splits(
    (train_data, test_data), batch_size=64, sort_key=lambda x: len(x.text), shuffle=True
)
# Define hyperparameters
vocab_size = len(TEXT.vocab)
embedding_dim = 100
hidden_dim = 256
output_dim = 1  # Binary classification
# Create model, optimizer, and criterion objects
model = SentimentModel(vocab_size, embedding_dim, hidden_dim, output_dim)
optimizer = optim.Adam(model.parameters())
criterion = nn.BCEWithLogitsLoss()

# Training the model
def train_model(model, iterator, optimizer, criterion, epochs=3):
    model.train()
    for epoch in range(epochs):
        epoch_loss = 0
        for batch in iterator:
            optimizer.zero_grad()
            text, text_lengths = batch.text
            predictions = model(text, text_lengths).squeeze(1)
            loss = criterion(predictions, batch.label)
            loss.backward()
            optimizer.step()
            epoch_loss += loss.item()
        print(f'Epoch {epoch+1}/{epochs}, Loss: {epoch_loss / len(iterator):.4f}')
train_model(model, train_iterator, optimizer, criterion)

Output:

#Sample output
Epoch 1/3, Loss: 0.6802
Epoch 2/3, Loss: 0.6548
Epoch 3/3, Loss: 0.6301

The section delves into three key aspects: model initialization, optimizer setup, and the training process. It specifically addresses these components for a predetermined number of epochs.

Evaluation

# Define evaluate_model function
def evaluate_model(model, iterator, criterion):
    model.eval()
    total_correct = 0
    total_examples = 0
    with torch.no_grad():
        for batch in iterator:
            text, text_lengths = batch.text
            predictions = model(text, text_lengths).squeeze(1)
            rounded_preds = torch.round(torch.sigmoid(predictions))
            total_correct += (rounded_preds == batch.label).sum().item()
            total_examples += batch.label.size(0)
    accuracy = total_correct / total_examples
    return accuracy

accuracy = evaluate_model(model, valid_iterator, criterion)
print(f'Validation Accuracy: {accuracy * 100:.2f}%')

Output:

# Sample Output
Validation Accuracy: 86.45%

Understanding and implementing each of these steps enables you to construct a comprehensive pipeline for sentiment analysis using PyTorch; this ensures not only effective handling of textual data but also robust model training.

Conclusion

The comprehensive guide demonstrates the intricacies of sentiment analysis using PyTorch, from data preprocessing to model training and evaluation.