Multi Dimensional Inputs in Pytorch Linear Method in Python

In PyTorch, the torch.nn.Linear class is a linear layer that applies a linear transformation to the input data. It is called linear transformation because it applies the linear equation. i.e

y = xA^{T}+b

Here

• x : input data of one or more dimensions
• A : weight
• b : bias

syntax: torch.nn.Linear(in_features, out_features, bias=True, device=None, dtype=None)

 Here: 

• in_features : Input features will be an integer value. It denotes the number of variables or columns for the rectangular data frame.
•  out_features : Output features will also be an integer value. It denotes the number of output value. We can get1 or more output values.
• bias : It take the boolean value True or False, For default it is True.
• device (Optional) :  For default it is None. Here we can change of GPU.
• dtype (Optional) :  For default it is None. We can change to any float value.

The torch.nn.Linear class expects the input data to be in the form of a 1-D,2-D, or 3-D tensor, with the last dimension being the input size and the other dimensions representing the batch size and other dimensions.

When working with multi-dimensional input data, the torch.nn.Linear class will apply the linear transformation to the last dimension of the input tensor. It means that the last dimension of the input tensor is considered as input size 

In this way, we can use the torch.nn.Linear class to apply a linear transformation to multi-dimensional input data like images, videos, etc. This can be useful in various applications like image classification, object detection, etc.

For example, if you're working with a 3-D tensor representing a set of images, each image having a number of channels, the last dimension of the tensor will be the number of pixels, and the other dimensions will represent the batch size, and the number of channels. The torch.nn.Linear layer will apply a linear transformation to the number of pixels, and the other dimensions will be preserved.

Example 1:

Step 1: Import the necessary packages

# Import the library
import torch
import torch.nn as nn

Step 2: Define the Input Tensor

# Define the input data as a 2-D tensor
x = torch.tensor([[0.,1.],[2.,3.],[4.,5.]])
#Shape
print('input_data:', x.shape)

Output:

input_data: torch.Size([3, 2])

Step 3:  Define the output size and Create the Linear transformation layer

# Create an instance of the nn.Linear class
output_size = 1
linear_layer = nn.Linear(in_features=2,
                         out_features=output_size,
                         bias = True,
                         dtype= torch.float)

Step 4: Apply the linear transformation to the input data. and print the output.

# Apply the linear transformation to the input data
y = linear_layer(x)

#print the outputs
print('output:',y)

#print the outputs shape
print('output Shape:', y.shape)

Output:

output:
 tensor([[-0.5798],
        [-2.1357],
        [-3.6916]], grad_fn=<AddmmBackward0>)
output Shape: torch.Size([3, 1])

Check the Result:

# Linear layer weight
A = linear_layer.weight
#Bias
b = linear_layer.bias

#Output
y = x @ A.transpose(0,1) + b

#print the outputs
print(y)

Output:

tensor([[-0.5798],
        [-2.1357],
        [-3.6916]], grad_fn=<AddBackward0>)

Example 2

# Import the library
import torch
import torch.nn as nn

# Define the input data as a 3-D tensor
x = torch.tensor([[[0.,1.,2.],
                   [3.,4.,5.],
                   [6.,7.,8.]]])

print('input_data:', x.shape)

# Create an instance of the nn.Linear class
output_size = 5
linear_layer = nn.Linear(in_features=x.size()[-1],
                         out_features=output_size,
                         bias = True,
                         dtype= torch.float)

# Apply the linear transformation to the input data
y = linear_layer(x)
#print the outputs
print('\noutput:\n',y)

#print the outputs shape
print('\noutput Shape:', y.shape)

Output:

input_data: torch.Size([1, 3, 3])

output:
 tensor([[[ 0.8020,  0.7344, -0.1366,  0.7514, -0.0804],
         [ 3.7831,  1.5299, -0.1147, -0.1602,  0.8117],
         [ 6.7643,  2.3254, -0.0927, -1.0718,  1.7038]]],
       grad_fn=<ViewBackward0>)

output Shape: torch.Size([1, 3, 5])

Check the Results:

# Linear layer weight
A = linear_layer.weight
#Bias
b = linear_layer.bias

#Output y= xA^T + b
##Matrix product of two tensors
y = x.matmul(A.transpose(0,1)) + b

#print the outputs
print(y)

Output:

tensor([[[ 0.8020,  0.7344, -0.1366,  0.7514, -0.0804],
         [ 3.7831,  1.5299, -0.1147, -0.1602,  0.8117],
         [ 6.7643,  2.3254, -0.0927, -1.0718,  1.7038]]],
       grad_fn=<AddBackward0>)

Example 3:

# Import the library
import torch
import torch.nn as nn

# Define the input data as a 3-D tensor
batch_size = 2
num_channels = 3
input_size = 4

input_data = torch.randn(size =(batch_size, num_channels, input_size))
print('input Shape :', input_data.shape)
print('Input\n',input_data)

# Create an instance of the nn.Linear class
output_size = 2
linear_layer = nn.Linear(in_features=input_data.size()[-1],
                         out_features=output_size,
                         bias = True)

# Apply the linear transformation to the input data
output = linear_layer(input_data)
print('\noutput shape:', output.shape)
print('output:\n',output)

Output

input Shape : torch.Size([2, 3, 4])
Input
 tensor([[[ 0.2777, -0.1791, -0.0452,  1.4240],
         [-0.3565, -1.0471,  0.2848,  0.4885],
         [ 0.4119,  0.4591, -0.9893,  0.7301]],

        [[-0.8063, -0.7165,  1.6583,  1.2563],
         [ 0.0972, -0.3562,  1.1938,  0.4868],
         [ 2.1877,  0.3338,  0.7358, -1.3842]]])

output shape: torch.Size([2, 3, 2])
output:
 tensor([[[ 0.5222,  0.1193],
         [ 0.0817, -0.4504],
         [ 0.2597,  0.4163]],

        [[ 0.7127, -0.3072],
         [ 0.4723, -0.3006],
         [-0.0074, -0.4597]]], grad_fn=<ViewBackward0>)
         

Check the Results:

# Linear layer weight
A = linear_layer.weight
#Bias
b = linear_layer.bias

#Output y= xA^T + b
##Matrix product of two tensors
y = input_data.matmul(A.transpose(0,1)) + b

#print the outputs
print(y)

Output:

tensor([[[ 0.5222,  0.1193],
         [ 0.0817, -0.4504],
         [ 0.2597,  0.4163]],

        [[ 0.7127, -0.3072],
         [ 0.4723, -0.3006],
         [-0.0074, -0.4597]]], grad_fn=<AddBackward0>)

Explanation:

Here, input_data is 3D tensor having batch_size, num_channels and input_size as dimensions. torch.nn.Linear(in_features=input_data.size()[-1], out_features=output_size) creates a Linear layer with input size being the last dimension of input_data (i.e. input_size) and output size being output_size.

The output of this operation will be a 3D tensor having dimensions batch_size, num_channels and output_size. The output size is defined by the out_features parameter when creating an instance of the nn.Linear class.

It's also important to note that this script is only for linear layers and for model building, we need to perform forward and backward pass, And there a non-linear activation function like ReLU, tanh, etc. will be added based on the use case, which to decide the weights and biases of the model.

You can also change the shape of the input_data as per your requirement, for example, you can use a 4D tensor for 3D convolution, and so on.