● Need for Upsampling in GANs

● How to Use the Upsampling Layer

● How to Use the Transpose Convolutional Layer

Need for Upsampling in Generative Adversarial

Networks
Generative Adversarial Networks are an architecture for neural networks for training a
generative model.

The architecture is comprised of a generator and a discriminator model, both of which are
implemented as a deep convolutional neural network. The discriminator is responsible for
classifying images as either real (from the domain) or fake (generated). The generator is
responsible for generating new plausible examples from the problem domain.

The generator works by taking a random point from the latent space as input and outputting
a complete image, in a one-shot manner.

A traditional convolutional neural network for image classification, and related tasks, will use
pooling layers to downsample input images. For example, an average pooling or max
pooling layer will reduce the feature maps from a convolutional by half on each dimension,
resulting in an output that is one quarter the area of the input.

Convolutional layers themselves also perform a form of downsampling by applying each

filter across the input images or feature maps; the resulting activations are an output feature
map that is smaller because of the border effects. Often padding is used to counter this
effect.

The generator model in a GAN requires an inverse operation of a pooling layer in a

traditional convolutional layer. It needs a layer to translate from coarse salient features to a
more dense and detailed output.

A simple version of an unpooling or opposite pooling layer is called an upsampling layer. It

works by repeating the rows and columns of the input.
A more elaborate approach is to perform a backwards convolutional operation, originally
referred to as a deconvolution, which is incorrect, but is more commonly referred to as a
fractional convolutional layer or a transposed convolutional layer.

Both of these layers can be used on a GAN to perform the required upsampling operation to
transform a small input into a large image

output.

In the following sections, we will take a closer look at each and develop an intuition for how
they work so that we can use them effectively in our GAN models.

How to Use the UpSampling2D Layer

Perhaps the simplest way to upsample an input is to double each row and column.

For example, an input image with the shape 2×2 would be output as 4×4.

1 1, 2

2 Input = (3, 4)

4 1, 1, 2, 2

5 Output = (1, 1, 2, 2)

6 3, 3, 4, 4

7 3, 3, 4, 4

Worked Example Using the UpSampling2D Layer

The Keras deep learning library provides this capability in a layer called UpSampling2D.

It can be added to a convolutional neural network and repeats the rows and columns
provided as input in the output. For example:

1 ...

2 # define model
 3 model = Sequential()

4 model.add(UpSampling2D())

We can demonstrate the behavior of this layer with a simple contrived example.

First, we can define a contrived input image that is 2×2 pixels. We can use specific values
for each pixel so that after upsampling, we can see exactly what effect the operation had on
the input.

1 ...

2 # define input data

3 X = asarray([[1, 2],

4 [3, 4]])

5 # show input data for context

6 print(X)

Once the image is defined, we must add a channel dimension (e.g. grayscale) and also a
sample dimension (e.g. we have 1 sample) so that we can pass it as input to the model.

1 ...

2 # reshape input data into one sample a sample with a channel

3 X = X.reshape((1, 2, 2, 1))

We can now define our model.

The model has only the UpSampling2D layer which takes 2×2 grayscale images as input
directly and outputs the result of the upsampling operation.

1 ...

2 # define model

3 model = Sequential()

4 model.add(UpSampling2D(input_shape=(2, 2, 1)))

5 # summarize the model

6 model.summary()

We can then use the model to make a prediction, that is upsample a provided input image.
 1 ...

2 # make a prediction with the model

3 yhat = model.predict(X)

The output will have four dimensions, like the input, therefore, we can convert it back to a
2×2 array to make it easier to review the result.

1 ...

2 # reshape output to remove channel to make printing easier

3 yhat = yhat.reshape((4, 4))

4 # summarize output

5 print(yhat)

Tying all of this together, the complete example of using the UpSampling2D layer in Keras
is provided below.

1 # example of using the upsampling layer

2 from numpy import asarray

3 from keras.models import Sequential

4 from keras.layers import UpSampling2D

5 # define input data

6 X = asarray([[1, 2],

7 [3, 4]])

8 # show input data for context

9 print(X)

10 # reshape input data into one sample a sample with a channel

11 X = X.reshape((1, 2, 2, 1))

12 # define model

13 model = Sequential()

14 model.add(UpSampling2D(input_shape=(2, 2, 1)))

15 # summarize the model

16 model.summary()

17 # make a prediction with the model

18 yhat = model.predict(X)
 19 # reshape output to remove channel to make printing easier

20 yhat = yhat.reshape((4, 4))

21 # summarize output

22 print(yhat)

Running the example first creates and summarizes our 2×2 input data.

Next, the model is summarized. We can see that it will output a 4×4 result as we expect,
and importantly, the layer has no parameters or model weights. This is because it is not
learning anything; it is just doubling the input.

Finally, the model is used to upsample our input, resulting in a doubling of each row and
column for our input data, as we expected.

1 [[1 2]

2 [3 4]]

4 _________________________________________________________________

5 Layer (type) Output Shape Param #

6 =================================================================

7 up_sampling2d_1 (UpSampling2 (None, 4, 4, 1) 0

8 =================================================================

9 Total params: 0

10 Trainable params: 0

11 Non-trainable params: 0

12 _________________________________________________________________

13

14

15 [[1. 1. 2. 2.]

16 [1. 1. 2. 2.]

17 [3. 3. 4. 4.]

18 [3. 3. 4. 4.]]

By default, the UpSampling2D will double each input dimension. This is defined by the ‘size‘
argument that is set to the tuple (2,2).
You may want to use different factors on each dimension, such as double the width and
triple the height. This could be achieved by setting the ‘size‘ argument to (2, 3). The result of
applying this operation to a 2×2 image would be a 4×6 output image (e.g. 2×2 and 2×3). For
example:

1 ...

2 # example of using different scale factors for each dimension

3 model.add(UpSampling2D(size=(2, 3)))

Additionally, by default, the UpSampling2D layer will use a nearest neighbor algorithm to fill
in the new rows and columns. This has the effect of simply doubling rows and columns, as
described and is specified by the ‘interpolation‘ argument set to ‘nearest‘.

Alternately, a bilinear interpolation method can be used which draws upon multiple
surrounding points. This can be specified via setting the ‘interpolation‘ argument to ‘bilinear‘.
For example:

1 ...

2 # example of using bilinear interpolation when upsampling

3 model.add(UpSampling2D(interpolation='bilinear'))