Cheat sheet

Deep Learning with PyTorch Datasets and dataloaders The training loop
# Create a dataset from a pandas DataFrame with TensorDataset()
# Set model to training mode

X = df[feature_columns].values
model.train()

Learn PyTorch online at www.DataCamp.com y = df[target_column].values

# Set a loss criterion and an optimizer

loss_criterion = nn.MSELoss()

dataset = TensorDataset(torch.tensor(X).float(), torch.tensor(y).float())

optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.95)

# Loop over chunks of data in the training set

# Load the data in batches with DataLoader()

for data in dataloader:

dataloader = DataLoader(dataset, batch_size=n, shuffle=True) # Set the gradients to zero with .zero_grad()

optimizer.zero_grad()

# Get features and targets for current chunk of data

Definitions Preprocessing features, targets = data

# Run a "forward pass" to fit the model to the data

predictions = model(data)

# Calculate loss

# One-hot encode categorical variables with one_hot()

loss = loss_criterion(predictions, targets)

PyTorch is one of the most popular deep learning frameworks, with a syntax similar to NumPy F.one_hot(torch.tensor([0, 1, 2]), num_classes=3) # Returns tensor of 0s and 1s # Calculate gradients using backprogagation

In the context of PyTorch, you can think of a Tensor as a NumPy array that can be run on a CPU or a GPU, and has a loss.backward()

method for automatic differentiation (needed for backpropagation) # Update the model parameters

TorchText, TorchVision, and TorchAudio are Python packages that provide PyTorch with functionality for text, image,
Sequential model architecture optimizer.step()

and audio data respectively

A neural network consists of neurons that are arranged into layers. Input values are passed to the first layer of neural # Create a linear layer with m inputs, n outputs with Linear()
The evaluation loop
networks. Each neuron has two properties: a weight and a bias. The output of a neuron in a neural network is a lnr = nn.Linear(m, n)

weighted sum of its inputs, plus the bias. The output is passed on to any connected neurons in the next layer, and this
continues until the final layer of the network is reached # Get weight of layer with .weight
# Set model to evaluation mode

lnr.weight

model.eval()

An activation function is a transformation of the output from a neuron, and is used to introduce non-linearity into the # Create accuracy metric with Accuracy()

calculations metric = torchmetrics.Accuracy(task="multiclass", num_classes=3)

# Get bias of layer with .bias

# Loop of chunks of data in the validation set

Backpropagation is an algorithm used to train neural networks by iteratively adjusting the weights and biases of each lnr.bias

for i, data in enumerate(dataloader, 0):

neuron # Get features and targets for current chunk of data

# Create a sigmoid activation layer for binary classification with Sigmoid()

features, targets = data

Saturation is when the output from a neuron reaches a maximum or minimum value beyond which it cannot change.
This can reduce learning performance, and an activation function such as ReLU may be needed to avoid the nn.Sigmoid()

# Run a "forward pass" to fit the model to the data

phenomenon predictions = model(data)

# Create a softmax activation layer for multi-class classification with Softmax()

# Calculate accuracy over the batch

The loss function quantifies the difference between the predicted output of a model and the actual target output nn.Softmax(dim=-1)

accuracy = metric(output, predictions.argmax(dim=-1))

# Calculate accuracy over all the validation data

The optimizer is an algorithm to adjust the parameters (neuron weights and biases) of a neural network during the accuracy = metric.compute()

# Create a rectified linear unit activation layer to avoid saturation with ReLU()
print(f"Accuracy on all data: {accuracy}")

training process in order to minimize the loss function

nn.ReLU()

# Reset the metric for the next dataset (training or validation)

The learning rate controls the step size of the optimizer. If the learning rate is too low the optimization will take too metric.reset()
long. If it is too high, the optimizer will not effectively minimize the loss function leading to poor predictions # Create a leaky rectified linear unit activation layer to avoid saturation with LeakyReLU()

nn.LeakyReLU(negative_slope=0.05)

Momentum controls the inertia of the optimizer. If momentum is too low, the optimizer can get stuck at a local
minimum and give the wrong answer. If it is too high, the optimizer can fail to converge and not give an answer
# Create a dropout layer to regularize and prevent overfitting with Dropout()

Transfer learning and fine-tuning

Transfer learning is reusing a model trained on one task for a second similar task to accelerate the training process nn.Dropout(p=0.5)

# Save a layer of a model to a file with save()

Fine-tuning is a type of transfer learning where early layers are frozen, and only the layers close to the output are # Create a sequential model from layers
torch.save(layer, 'layer.pth')

trained model = nn.Sequential(

nn.Linear(n_features, i),
# Load a layer of a model from a file with load()

Accuracy is a metric to determine how well a model fits a dataset. It quantifies the proportion of correctly predicted new_layer = torch.load('layer.pth')

outcomes (either classifications or predictions) compared to the total number of data points in the dataset. nn.Linear(i, j), # Input size must match output from previous layer

nn.Linear(j, n_classes),

# Freeze the weight for layer 0 with .requires_grad

nn.Softmax(dim=-1) # Activation layer comes last

for name, param in model.named_parameters():

Importing PyTorch ) if name == "0.weight":

param.requires_grad = False

# Import the top-level package for core functionality

Fitting a model and calculating loss
import torch

# Import neural network functionality

# Fit a model to input data with model where model is a variable created by, e.g., Sequential()

from torch import nn

prediction = model(input_data).double()

# Import functional programming tools

import torch.nn.functional as F

# Get target values

actual = torch.tensor(target_values).double()

# Import optimization functionality

import torch.optim as optim

# Calculate the mean-squared error loss for regression with MSELoss()

mse_loss = nn.MSELoss()(prediction, actual) # Returns tensor(x) 

# Import dataset functions

from torch.utils.data import TensorDataset, DataLoader

# Calculate the L1 loss for robust regression with SmoothL1Loss()

Learn Python Online at
# Import evaluation metrics

import torchmetrics
l1_loss = nn.SmoothL1Loss()(prediction, actual) # Returns tensor(x) 

www.DataCamp.com
# Calculate binary cross-entropy loss for binary classification with BCELoss()

bce_loss = nn.BCELoss()(prediction, actual) # Returns tensor(x) 

Working with tensors # Calculate cross-entropy loss for multi-class classification with CrossEntropyLoss()

ce_loss = nn.CrossEntropyLoss()(prediction, actual) # Returns tensor(x) 

# Create tensor from list with tensor()

tnsr = torch.tensor([1, 3, 6, 10])

# Calculate the gradients via backprogagation with .backward()

# Get data type of tensor elements with .dtype

loss.backward()
tnsr.dtype # Returns torch.int64

# Get dimensions of tensor with .Size()

tnsr.shape # Returns torch.Size([4])

Working with optimizers

# Get memory location of tensor with .device

tnsr.device # Returns cpu or gpu

# Create a tensor of zeros with zeros()

# Create a stochastic gradient descent optimizer with SGD(), setting learning rate and momentum

tnsr_zrs = torch.zeros(2, 3)

optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.95)

# Create a random tensor with rand()

tnsr_rndm = torch.rand(size=(3, 4)) # Tensor has 3 rows, 4 columns # Update neuron parameters with .step()

optimizer.step()