Pytorch - Index-based Operation

PyTorch is a python library developed by Facebook to run and train deep learning and machine learning algorithms. Tensor is the fundamental data structure of the machine or deep learning algorithms and to deal with them, we perform several operations, for which PyTorch library offers many functionalities.

Tensor operations that handle indexing on some particular row or column for copying, adding, filling values/tensors are said to be index-based developed operation. There are two types of index-based operations in PyTorch, one is in-place operations and the other is out-of-place operations.  The basic difference between the two is in-place operation directly changes the values of the tensors without making any copy of that whereas out of place operations don't. Following are the operations:-

1. index_add_
2. index_add
3. index_copy_
4. index_copy
5. index_fill_
6. index_fill
7. index_put_
8. index_put
9. index_select

Now we discuss each of the functions with proper examples.

1. index_add_:  Adds the given tensor elements to the self tensor along the order given in the matrix.

Syntax:

 index_add_(dim,index,ensor)---> Tensor

 Parameters:

• dim: dimension along which index to add. '0' stands for column and '1' stands for row.
• index: indices of the tensor to select from. It can be LongTensor or IntTensor.
• tensor: tensor containing the values to add.

Example 1: We take a zero vector 'x', te tensor of size(3,5) and index tensor. Accumulating the resultant vector along rows we get the output.

#importing libraries
import torch

x=torch.zeros(5,5)
te=torch.tensor([[1,3,5,7,9],[1,3,5,7,9],[1,3,5,7,9]],dtype=torch.float32)
index0=torch.tensor([0,2,4])
#adding tensor te to x along row of the given order
x.index_add_(0,index0,te)

Output:

tensor([[1., 3., 5., 7., 9.],
        [0., 0., 0., 0., 0.],
        [1., 3., 5., 7., 9.],
        [0., 0., 0., 0., 0.],
        [1., 3., 5., 7., 9.]])

 Example 2:

#importing libraries
import torch

y=torch.ones(5,5)#unitvector
index2=torch.tensor([0,1,1,1,2])
ten=torch.randn(1,5)
#adding values to y along the column with given order
y.index_add_(1,index2,ten)

Output:

tensor([[0.9460, 0.4762, 1.2219, 1.0000, 1.0000],
        [0.9460, 0.4762, 1.2219, 1.0000, 1.0000],
        [0.9460, 0.4762, 1.2219, 1.0000, 1.0000],
        [0.9460, 0.4762, 1.2219, 1.0000, 1.0000],
        [0.9460, 0.4762, 1.2219, 1.0000, 1.0000]])

2.index_add: It is the out-of place version of the above function. This adds a given tensor to the self tensor temporarily. Parameters and syntax are the out-of-placesame as above.

Example 3:

import torch

y=torch.ones(5,5)

index2=torch.tensor([0,1,1,1,2])
ten=torch.randn(1,5)
print("Indexed Matrix:\n",y.index_add(1,index2,ten))
print ("Printing Indexed Matrix again:\n",y)

Output:

Indexed Matrix:
 tensor([[-0.2811, -1.0776,  2.2697,  1.0000,  1.0000],
        [-0.2811, -1.0776,  2.2697,  1.0000,  1.0000],
        [-0.2811, -1.0776,  2.2697,  1.0000,  1.0000],
        [-0.2811, -1.0776,  2.2697,  1.0000,  1.0000],
        [-0.2811, -1.0776,  2.2697,  1.0000,  1.0000]])
Printing Indexed Matrix again:
 tensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]])

3.index_copy_:  Copies the elements of a given tensor to the input tensor by selecting the indices in the order given in 'index'.

Syntax:

index_copy_(dim,index,tensor)---> Tensor

Parameters:

• dim: dimension along which index to copy. It is of 'int' format.
• index: indices of the tensor to select from. It can be either IntTensor or LongTensor.
• tensor: tensor containing the values to be copied.

Example 4:  Here elements of 'a' is being replaced by 't1' tensor in an order given in the 'index3'.

#importing libraries
import torch

a=torch.ones(4,4)#unit vector
t1=torch.randn(2,4)
index3=torch.tensor([1,3])

#copying elements of t1 ensor to 'a' in given order of index
a.index_copy_(0,index3,t1)

Output:

tensor([[ 1.0000,  1.0000,  1.0000,  1.0000],
        [-0.1918, -1.2089,  0.3229, -0.1831],
        [ 1.0000,  1.0000,  1.0000,  1.0000],
        [ 0.7521,  0.8424, -0.8698, -0.3908]])

Example 5: In the below example we will receive an error. This is because the dim^th dimension of the tensor is not equal to the length of the index. So we have to keep in mind that dim^th dimension of the tensor must have the same size as the length of the index.

#importing libraries
import torch

y=torch.ones(5,5)
index1=torch.tensor([0,1,2,3,4])
te=torch.tensor([[1,3,5,7,9],[1,3,5,7,9],[1,3,5,7,9]],dtype=torch.float32)
y.index_copy_(1,index1,te)

Output:

RuntimeError                              Traceback (most recent call last)
<ipython-input-8-25e4150d5bd7> in <module>
      1 y=torch.ones(5,5)
      2 index1=torch.tensor([0,1,2,3,4])
----> 3 y.index_copy_(1,index1,te)

RuntimeError: index_copy_(): Source/destination tensor must have same slice shapes.
Destination slice shape: 5 at dimension 1 and source slice shape: 3 at dimension 0.

Example 6: In this, we will copy the given tensor to the self tensor keeping in mind that dim^th dimension of the tensor must have the same size as the length of the index.

import torch

b=torch.ones(4,4)
t2=torch.randn(4,2)

index4=torch.tensor([0,1])
b.index_copy_(1,index4,t2)

Output:

tensor([[-0.3964, -0.3859,  1.0000,  1.0000],
        [ 2.6910, -0.9394,  1.0000,  1.0000],
        [ 0.3591, -0.2262,  1.0000,  1.0000],
        [ 1.2102, -0.8340,  1.0000,  1.0000]])

 4.index_copy: This is the out-of-place index-based operation for replacing elements of input tensor with a given tensor. Syntax, parameters are the same as the above.

5.index_fill_:  'Val' value is filled with the elements of 'x' along with the order of indices given in the vector 'index'.

Syntax:

index_fill_(dim, index, val) → Tensor

Parameters:

• dim: dimension along which to fill. It is of 'int' format.
• index:  filling in order of the indices given in this index vector. It can be either IntTensor or LongTensor.
• val(float): value with which to fill in.

Example 7: In this example, we will declare a tensor with random elements and then fill it with '4' along with the given indices.

#importing libraries
import torch
c=torch.randn(4,4)

index5=torch.tensor([0,2])

#filling 4 within the elements of the tensor 'c' along the indices 0,2
c.index_fill_(0,index5,4)
print(c)

Output:

 tensor([[ 4.0000,  4.0000,  4.0000,  4.0000],
        [ 0.4762,  0.0056,  0.3258,  1.1345],
        [ 4.0000,  4.0000,  4.0000,  4.0000],
        [-0.1490, -0.6543,  0.9755,  1.8087]])

 Example 8: Similarly we perform a filling operation along with the columns.

d=torch.randn(5,5)

d.index_fill(1,index5,2)
print(d)

Output:

 tensor([[ 0.5978, -1.2461, -0.8794, -1.0175,  0.8938],
        [-0.6374,  1.0848,  0.1291,  0.6658,  0.3081],
        [-0.9686, -0.8212, -0.5223, -0.3208, -1.7718],
        [-0.1153, -1.2552, -0.4119, -1.1293,  0.2266],
        [ 1.2610,  0.2618, -1.5528,  0.7805,  1.3730]])

 6. index_fill: This is the out of place index-based operation for filling the elements with tensor with 'val'. Syntax, parameters are the same as the above.

7.index_put_: This operation puts the value of 'val' into the self tensor using the indices of the given 'index'.

Syntax:

index_put_(indices, values, accumulate=False) → Tensor

Parameters:

• indices: It is the tuple of LongTensor which is used to index to self.
• values: Tensor with values that need to be put into the target.
• accumulate: whether to accumulate or not.

Example 9: Here we take the target vector and replaces the values of  value tensor as mentioned in the 

#importing libraries
import torch
 
target=torch.zeros([4,4])
indices = torch.LongTensor([[0,1],[1,2],[3,1],[1,0]])#indices to which values to be put
value = torch.ones(indices.shape[0])
#tuple of the index tensor is passed along with the value
target.index_put_(tuple(indices.t()), value)

Output:

tensor([[0., 1., 0., 0.],
       [1., 0., 1., 0.],
       [0., 0., 0., 0.],
       [0., 1., 0., 0.]])

 Note: We have to take the transpose of the indices tensor or else an error will occur.

Example 10: In this example, we keep the accumulate function as true which means the elements in value are added to the target.

e=torch.ones([4,4])
indices2=torch.LongTensor([[0,1],[0,1],[2,1]])
value2=torch.zeros(indices2.shape[0])

e.index_put_(tuple(indices2.t()),value2,accumulate=True)

Output:
tensor([[1., 1., 1., 1.],
       [1., 1., 1., 1.],
       [1., 1., 1., 1.],
       [1., 1., 1., 1.]])

8. index_fill: This is the out of place version of index_fill_. Syntax, parameters are the same as above.

9. index_select: A tensor is returned with indices as mentioned, by selecting from the target tensor.

Syntax:

torch.index_select(input, dim, index, out=None) 

Parameters:

• input(Tensor): Tensor from which the indices will be selected.
• dim(int): dimension along which to select.
• index: It contains indices to the index.

Example 11: We pick up 0,1 indices of the tensor 'm' along dim=0 i.e. rows and print the output.

#importing libraries
import torch

m=torch.randn(3,4)
print('Original matrix:\n',m)
indices=torch.tensor([0,1])
print("Indexed Matrix:\n",torch.index_select(m, 0, indices))

Output:

Original matrix:
 tensor([[ 0.2008, -0.2637,  2.1216, -0.2892],
        [-0.4059, -1.6054, -2.5022, -0.2912],
        [-0.3246,  0.4751, -0.1018, -0.6707]])
Indexed Matrix:
 tensor([[ 0.2008, -0.2637,  2.1216, -0.2892],
        [-0.4059, -1.6054, -2.5022, -0.2912]])

  Reference: https://docs.pytorch.org/docs/stable/tensors.html