Torch.nn Dropout Method in Python PyTorch

Making some of the random elements of an input tensor zero has been proven to be an effective technique for regularization during the training of a neural network. To achieve this task, we can apply torch.nn.Dropout(). It zeroes some of the elements of the input tensor.

An element will be zeroed with the given probability p. It uses a Bernoulli distribution to take samples of the element being zeroed. It does not support complex-valued inputs.

Syntax

torch.nn.Dropout(p=0.5)

The default probability of an element to zeroed is set to 0.5

Steps

We could use the following steps to randomly zero some of the elements of an input tensor with a given probability p −

• Import the required library. In all the following examples, the required Python library is torch. Make sure you have already installed it.

import torch

• Define an input tensor input.

input = torch.randn(5,2)

• Define the Dropout layer dropout passing the probability p as an optional parameter.

dropout = torch.nn.Dropout(p= 0.5)

• Apply the above defined dropout layer dropout on the input tensor input.

output = dropout(input)

• Print the tensor containing softmax values.

print(output)

Example 1

In the following Python program, we use p = 0.5. It defines that the probability of an element being zeroed is 50%.

# Import the required library
import torch

# define a tensor
tensor = torch.randn(4)

# print the tensor
print("Input tensor:",tensor)

# define a dropout layer to randomly zero some elements
# with probability p=0.5
dropout = torch.nn.Dropout(.5)

# apply above defined dropout layer
output = dropout(tensor)

# print the output after dropout
print("Output Tensor:",output)

Output

Input tensor: tensor([ 0.3667, -0.9647, 0.8537, 1.3171])
Output Tensor: tensor([ 0.7334, -1.9294, 0.0000, 0.0000])

Example 2

In the python3 program below we use p = 0.25. It means there is a 25% chance of an element of input tensor to be zeroed.

# Import the required library
import torch

# define a tensor
tensor = torch.randn(4, 3)

# print the tensor
print("Input tensor:
",tensor)

# define a dropout layer to randomly zero some elements
# with probability p=0.25
dropout = torch.nn.Dropout(.25)

# apply above defined dropout layer
output = dropout(tensor)

# print the output after dropout
print("Output Tensor:
",output)

Output

Input tensor:
   tensor([[-0.6748, -1.1602, 0.2891],
      [-0.7531, -1.3444, 0.2013],
      [-1.3212, 1.2103, -0.6457],
      [ 0.9957, -0.2670, 1.6593]])
Output Tensor:
   tensor([[-0.0000, -0.0000, 0.0000],
      [-1.0041, -0.0000, 0.2683],
      [-1.7616, 0.0000, -0.0000],
      [ 1.3276, -0.3560, 2.2124]])