Deep Convolutional AutoEncoder-based Lossy

Image Compression
Zhengxue Cheng∗ , Heming Sun, Masaru Takeuchi∗ , and Jiro Katto∗
∗ Graduate
School of Fundamental Science and Engineering, Waseda University, Tokyo, Japan
Email: [email protected], [email protected], [email protected], [email protected]

Abstract—Image compression has been investigated as a funda- a quantizer cannot be directly incorporated into autoencoder
mental research topic for many decades. Recently, deep learning optimization. Thus, the works [4] and [5] proposed a differen-
has achieved great success in many computer vision tasks, and tiable approximation for quantization and entropy rate estima-
arXiv:1804.09535v1 [cs.CV] 25 Apr 2018

is gradually being used in image compression. In this paper, we

present a lossy image compression architecture, which utilizes the tion for an end-to-end training with gradient backpropagation.
advantages of convolutional autoencoder (CAE) to achieve a high Unlike those works, the work [6] used an LSTM recurrent
coding efficiency. First, we design a novel CAE architecture to network for compressing small thumbnail images (32 × 32),
replace the conventional transforms and train this CAE using a and used a binarization layer to replace the quantization and
rate-distortion loss function. Second, to generate a more energy- entropy coder. This approach was further extended in [7]
compact representation, we utilize the principal components
analysis (PCA) to rotate the feature maps produced by the for compressing full-resolution images. These works achieved
CAE, and then apply the quantization and entropy coder to promising coding performance; however, there is still room
generate the codes. Experimental results demonstrate that our for improvement, because they did not analyze the energy
method outperforms traditional image coding algorithms, by compaction property of the generated feature maps and did
achieving a 13.7% BD-rate decrement on the Kodak database not use a real entropy coder to generate the final codes.
images compared to JPEG2000. Besides, our method maintains
a moderate complexity similar to JPEG2000. In this paper, we propose a convolutional autoencoder
I. I NTRODUCTION (CAE) based lossy image compression architecture. Our main
Image compression has been a fundamental and significant contributions are twofold.
research topic in the field of image processing for several 1) To replace the transform and inverse transform in tradi-
decades. Traditional image compression algorithms, such as tional codecs, we design a symmetric CAE structure with
JPEG [1] and JPEG2000 [2], rely on the hand-crafted en- multiple downsampling and upsampling units to generate
coder/decoder (codec) block diagram. They use the fixed feature maps with low dimensions. We optimize this CAE
transform matrixes, i.e. Discrete cosine transform (DCT) and using an approximated rate-distortion loss function.
wavelet transform, together with quantization and entropy 2) To generate a more energy-compact representation, we
coder to compress the image. However, they are not expected propose a principal components analysis (PCA)-based
to be an optimal and flexible image coding solution for all rotation to generate more zeros in the feature maps.
types of image content and image formats. Then, the quantization and entropy coder are utilized to
Deep learning has been successfully applied in various compress the data further.
computer vision tasks and has the potential to enhance the Experimental results demonstrate that our method outperforms
performance of image compression. Especially, the autoen- JPEG and JPEG2000 in terms of PSNR, and achieves a 13.7%
coder has been applied in dimensionality reduction, compact BD-rate decrement compared to JPEG2000 with the popular
representations of images, and generative models learning [3]. Kodak database images. In addition, our method is computa-
Thus, autoencoders are able to extract more compressed codes tionally more appealing compared to other autoencoder based
from images with a minimized loss function, and are expected image compression methods.
to achieve better compression performance than existing im- The rest of this paper is organized as follows. Section II
age compression standards including JPEG and JPEG2000. presents the proposed CAE based image compression architec-
Another advantage of deep learning is that although the ture, which includes the design of the CAE network architec-
development and standardization of a conventional codec ture, quantization, and entropy coder. Section III summarizes
has historically taken years, a deep learning based image the experimental results and compares the rate-distortion (RD)
compression approach can be much quicker with new media curves of the proposed CAE with those of existing codecs.
contents and new media formats, such as 360-degree image Conclusion and future work are given in Section IV.
and virtual reality (VR) [4]. Therefore, deep learning based
image compression is expected to be more general and more II. P ROPOSED C ONVOLUTIONAL AUTOENCODER BASED
efficient. I MAGE C OMPRESSION
Recently, some approaches have been proposed to take The block diagram of the proposed image compression
advantage of the autoencoder for image compression. Due to based on CAE is illustrated in Fig.1. The encoder part includes
the inherent non-differentiability of round-based quantization, the pre-processing steps, CAE computation, PCA rotation,
Fig. 1: Block diagram of the proposed CAE based image compression. (The detailed block for downsampling/upsampling is
shown in Fig. 2)

quantization, and entropy coder. The decoder part mirrors the

architecture of the encoder.
To build an effective codec for image compression, we train
this approach in two stages. First, a symmetric CAE network
is designed using convolution and deconvolution filters. Then,
we train this CAE greedily using an RD loss function with an
added uniform noise, which is used to imitate the quantization
noises during the optimizing process. Second, by analyzing
the produced feature maps from the pre-trained CAE, we Fig. 2: Downsampling/Upsampling Units with two
utilize the PCA rotation to produce more zeros for improving (De)Convolution Filters.
the coding efficiency further. Subsequently, quantization and
entropy coder are used to compress the rotated feature maps
and the side information for PCA (matrix U) to generate setting the stride as 2, we can get downsampled feature maps.
the compressed bitstream. Each of these components will be The padding size is set as one to maintain the same size as the
discussed in detail in the following. input. Unlike the work [4], we do not use residual networks
A. CAE Network and sub-pixel convolutions, instead, we apply deconvolution
As the pre-processing steps before the CAE design, the raw filters to achieve a symmetric and simple CAE network.
RGB image is mapped to YCbCr images and normalized to In traditional codecs, the quantization is usually imple-
[0,1]. For general purposes, we design the CAE for each luma mented using the round function (denoted as [·]), and the
or chroma component; therefore, the CAE network handles derivative of the round function is almost zero except at the
inputs of size H × W × 1. When the size of raw image is integers. Due to the non-differentiable property of rounding
larger than H×W , the image will be split into non-overlapping function, the quantizer cannot be directly incorporated into
H × W patches, which can be compressed independently. the gradient-based optimization process of CAE. Thus, some
The CAE network can be regarded as an analysis transform smooth approximations are proposed in related works. Theis et
with the encoder function, y = fθ (x), and a synthesis al. [4] proposed to replace the derivative in the backward pass
d
transform with the decoder function, x̂ = gφ (y), where x, of back propagation as dy ([y]) ≈ 1. Balle et al. [5] replaced
x̂, and y are the original images, reconstructed images, and the quantization by an additive uniform noise as [y] ≈ y + µ.
the compressed data, respectively. θ and φ are the optimized Toderici et al. [6] used a stochastic binarization function as
parameters in the encoder and decoder, respectively. b(y) = −1 when y < 0, and b(y) = 1 otherwise. In our
To obtain the compressed representation of the input im- method, we use the simple uniform noises intuitively to imitate
ages, downsampling/upsampling operations are required in the quantization noises during the CAE training. After CAE
the encoding/decoding process of CAE. However, consecutive training, we apply the real round-based quantization in the
downsampling operations will reduce the quality of the recon- final image compression. The network architecture of CAE
structed images. In the work [4], it points out that the super is shown in Fig. 1, in which Ni denotes the number of
resolution is achieved more efficiently by first convolving filters in each convolution layer and determines the number
images and then upsampling them. Therefore, we propose a of generated feature maps.
pair of convolution/deconvolution filters for upsampling or As for the activation function in each convolution layer,
downsampling, as shown in Fig. 2, where Ni denotes the we utilize the Parametric Rectified Linear Unit (PReLU)
number of filters in the convolution or deconvolution block. By function [8], instead of the ReLU which is commonly used in
 (a) (b) (c)

Fig. 3: The effect of activation function in CAE. (d) (e) (f)

the related works. The performance with ReLU and PReLU

functions are shown in Fig. 3. Compared to ReLU, PReLU can
improve the quality of the reconstructed images, especially for
high bit rate. Inspired by the rate-distortion cost function in (g) (h) (i)
the traditional codecs, the loss function of CAE is defined as
Fig. 4: Examples of three images and their corresponding
J(θ, φ; x) = ||x − x̂||2 + λ · ||y||2
(1) feature maps arranged in raster-scan order (N6 = 32): (a)(d)(g)
= ||x − gφ (fθ (x) + µ)||2 + λ · ||fθ (x)||2 Raw images, (b)(e)(h) Generated 32 feature maps for Y-
component by CAE, and the size of each feature map is
where ||x − x̂||2 denotes the mean square error (MSE) distor- H W
8 × 8 , (c)(f)(i) Rotated Y feature maps by PCA, arranged
tion between the original images x and reconstructed images in vertical scan order.
x̂. µ is the uniform noise. λ controls the tradeoff between
the rate and distortion. ||fθ (x)||2 denotes the amplitude of the
compressed data y, which reflects the number of bits used to in columns to form the matrix U . Here, the first column is the
encode the compressed data. In this work, the CAE model was principal eigenvector corresponding to the largest eigenvalue,
optimized using Adam [9], and was applied to images with the second column is the second eigenvector, and so on. The
the size of H × W . We used a batch size of 16 and trained third step is to rotate the N6 -dimensional data y by computing
the model up to 8 × 105 iterations, but the model reached
convergence much earlier. The learning rate was kept at a fixed yrot = U T y (3)
value of 0.0001, and the momentum was set as 0.9 during the
training process. By computing yrot , we can ensure that the first feature maps
have the largest value, and the features maps are sorted in
B. PCA Rotation, Quantization, and Entropy Coder descending order. Experimental results demonstrate that the
After the CAE computation, an image representation with vertical-scan order for the feature maps works a little better
a size of H8 × W8 × N6 is obtained for each H × W × 1 input, than diagonal scan and horizontal scan; therefore, we arrange
where N6 denotes the number of filters in the sixth convolution the feature maps in vertical scan as shown in the third column
layer of the encoder part. Three examples of the feature maps of Fig. 4. It can be observed that more zeros are generated in
for the 512 × 512 images cropped from Kodak databases [11] the bottom-right corner and large values are centered in the
are demonstrated in the second column of Fig. 4. It can be top-left corner in the rotated feature maps, which can benefit
observed that each feature map can be regarded as one high- the entropy coder to achieve large compression ratio.
level representation of the raw images. After the PCA rotation, the quantization is performed as
To obtain a more energy-compact representation, we decor-
relate each feature map by utilizing the principle component y 0 = [2B−1 · yrot ] (4)
analysis (PCA), because PCA is an unsupervised dimensional-
where B denotes the number of bits for the desired precision,
ity reduction algorithm and is suitable for learning the reduced
which is set as 12 in our model.
features as a supplementary of CAE. The generated feature
As for the entropy coder, we use the JPEG2000 entropy
maps are denoted as y = H8 × W 8 × N6 , and y is reshaped as coder to decompose y 0 into bitplanes and apply the adaptive
N6 -dimensional data. PCA is performed using the following
binary arithmetic coder. It is noted that JPEG2000 entropy
steps. The first step is to compute the covariance matrix of z
coder applies EBCOT (Embedded block coding with opti-
as follows: m
1 X mized truncation) algorithm to achieve a desired rate R, which
Σ= (y)(y)T (2) is also referred to as post-compression RD optimization. In our
m 1
method, the feature maps rotated by PCA have many zeros;
where m is the number of samples for y. The second step is therefore, assigning the target bits R can further improve the
to compute the eigenvectors of Σ and stack the eigenvectors coding efficiency.
 In the decoder part, de-quantization is performed as
y0
ỹ = (5)
2B−1
After obtaining the float-point number ỹ from the bitstream,
we recover the feature maps from the rotated data by using
ŷ = U ỹ (6)
Then, the CAE decoder network will reconstruct the images
using x̂ = gφ (ŷ). The side information of PCA rotation is the
matrix U with a dimension of N6 × N6 for each image. We
also quantize U and encode it. The bits for U is added to the
final rate as the side information in the experimental results.
III. E XPERIMENTAL R ESULTS
A. Experimental Setup
We use a subset of the ImageNet database [10] consisting
of 5500 images to train the CAE network. In our experiments,
H and W are set as 128; therefore, the images that are input to Fig. 5: RD curves of color images for the proposed CAE,
the CAE are split to a size of 128 × 128 patches. The numbers JPEG, and JPEG2000
of filters, i.e. Ni , i ∈ [1, 6] in convolutional layers are set as
{32, 32, 64, 64, 64, 32}, respectively. The decoder part mirrors
the encoder part. The luma component is used to train the CAE
network. Mean square error is used in the loss function during
the training process in order to measure the distortion between
the reconstructed images and original images. For testing, we
use the commonly used Kodak lossless image database [11]
24bpp 0.290bpp 0.297bpp 0.293bpp
with 24 uncompressed 768 × 512 or 512 × 768 images. In our
CAE training process, λ is set as one and the uniform noise
µ is set as [− 2110 , 2110 ].
In order to measure the coding efficiency of the proposed
CAE-based image compression method, the rate is measured
in terms of bit per pixel (bpp). The quality of the reconstructed
images is measured using the quality metrics PSNR and MS- 24bpp 0.283bpp 0.300bpp 0.294bpp
SSIM [12], which measure the objective quality and perceived
quality, respectively.
B. Coding Efficiency Performance
We compare our CAE-based image compression with JPEG
and JPEG2000. The color space in this experiment is YUV444. 24bpp 0.318bpp 0.299bpp 0.295bpp
Since the human visual system is more sensitive to the luma (a) Raw (b) JPEG (c) JPEG2000 (d) CAE
component than chroma components, it is common to assign Fig. 6: Examples of raw image (a) and reconstructed images
the weights 68 , 81 , and 18 to the Y, Cb, and Cr components, (300 × 300) cropped from Kodak images using (b)JPEG,
respectively. The RD curves for the images red door and (c)JPEG2000 and (d)CAE.
a girl are shown in Fig. 5. The coding efficiency of CAE
is better than those of both JPEG2000 and JPEG in terms
of PSNR. In terms of MS-SSIM, CAE is better than JPEG The rate-distortion performance can be evaluated quantita-
and comparable with JPEG2000, because optimizing MSE in tively in terms of the average coding efficiency differences,
CAE training leads to better PSNR characteristic, but not MS- BD-rate (%) [13]. While calculating the BD-rate, the rate is
SSIM. Besides, CAE handles a fixed input size of 128 × 128; varied from 0.12bpp to 2.4bpp and the quality is evaluated
therefore, block boundary artifacts appear in some images. It is by using PSNR. With JPEG2000 as the benchmark, the BD-
expected that adding perceptual quality matrices into the loss rate results for 24 images in the Kodak database are listed in
function will improve the MS-SSIM performance, which will Fig. 7. On average, for the 24 images in the Kodak database,
be carried out in our future work. Examples of reconstructed our method achieves 13.7% BD-rate saving compared to
patches are shown in Fig. 6. We can observe that the subjective JPEG2000.
quality of the reconstructed images for CAE is better than We also compare our proposed CAE-based method with
JPEG and comparable with that of JPEG2000. Balle’s work, which released the source code for gray im-
 method could achieve comparable complexity with those of
JPEG and JPEG2000, which are implemented by C language.
Thus, it proves that our method has relatively low complexity.

TABLE I: Average running time comparison.

Codec Time (s)
JPEG 0.39
JPEG2000 0.59
Balle’s work[5] with CPU 7.39
Propose CAE with CPU 2.29
Propose CAE with GPU 0.67
Fig. 7: BD-rate of the proposed CAE with JPEG2000 as the
benchmark.
IV. C ONCLUSION AND F UTURE W ORK
In this paper, we proposed a convolutional autoencoder
based image compression architecture. First, a symmetric CAE
architecture with multiple downsampling and upsampling units
was designed to replace the conventional transforms. Then
this CAE was trained by using an approximated rate-distortion
function to achieve high coding efficiency. Second, we applied
the PCA to the feature maps for a more energy-compact
representation, which can benefit the quantization and entropy
coder to improve the coding efficiency further. Experimental
(a) (b) results demonstrate that our method outperforms conventional
traditional image coding algorithms and achieves a 13.7%
Fig. 8: RD curves of gray images for our proposed CAE and
Balle’s work. BD-rate decrement compared to JPEG2000 on the Kodak
database images. In our future work, we will add perceptual
quality matrices, such as MS-SSIM or the quality predicted
ages [5]. For a fair comparison, we give the comparison results by neural networks in [16], into the loss function to improve
for gray images. For Balle’s work, the rate is estimated by the the MS-SSIM performance. Besides, the generative adversarial
entropy of the discrete probability distribution of the quantized network (GAN) shows more promising performance than
vector, which is the lower bound of the rate. In our work, using autoencoder only; therefore, we will utilize GAN to
the rate is calculated by the real file size (kb) divided by the improve the coding efficiency further.
resolution of the tested images. Two examples of RD curves R EFERENCES
are shown in Fig. 8. Our method exhibits better RD curves [1] G. K Wallace, “The JPEG still picture compression standard”, IEEE
than Balle’s work for some test images, such as Fig. 8(a), Trans. on Consumer Electronics, vol. 38, no. 1, pp. 43-59, Feb. 1991.
[2] Majid Rabbani, Rajan Joshi, “An overview of the JPEG2000 still image
but exhibits slightly worse RD performance for some images, compression standard”, ELSEVIER Signal Processing: Image Commu-
such as Fig. 8(b). On average, the performance of our proposed nication, vol. 17, no, 1, pp. 3-48, Jan. 2002.
[3] P. Vincent, H. Larochelle, Y. Bengio and P.-A. Manzagol, “Extracting
method CAE is comparable with Balle’s work, even though and composing robust features with denoising autoencoders”, Intl. conf.
the CAE used an actual entropy coder against the ideal entropy on Machine Learning (ICML), pp. 1096-1103, July 5-9. 2008.
[4] Lucas Theis, Wenzhe Shi, Andrew Cunninghan and Ferenc Huszar,
of Balle’s work. “Lossy Image Compression with Compressive Autoencoders”, Intl. Conf.
on Learning Representations (ICLR), pp. 1-19, April 24-26, 2017.
C. Complexity Performance [5] Johannes Balle, Valero Laparra, Eero P. Simoncelli, “End-
to-End Optimized Image Compression”, Intl. Conf. on
Our experiments are performed on a PC with 4.20 GHz Intel Learning Representations (ICLR), pp. 1-27, April 24-26, 2017.
http://www.cns.nyu.edu/∼lcv/iclr2017/
Core i7-7700K CPU, 16GB RAM and GeForce GTX 1080 [6] G. Toderici, S. M.O’Malley, S. J. Hwang, et al., “Variable rate image
GPU. The pre-processing steps for the images and Balle’s compression with recurrent neural networks”, arXiv: 1511.06085, 2015.
[7] G, Toderici, D. Vincent, N. Johnson, et al., “Full Resolution Image
codec [5] are implemented using Matlab script in Matlab Compression with Recurrent Neural Networks”, IEEE Conf. on Computer
R2016b environment. The codecs of JPEG and JPEG2000 Vision and Pattern Recognition (CVPR), pp. 1-9, July 21-26, 2017.
[8] K. He, X. Zhang, S. Ren and J. Sun, “Delving Deep into Rectifiers: Sur-
can be found from [14] and [15], implemented with CPU. passing Human-Level Performance on ImageNet Classification”, IEEE
Balle released only their CPU implementation. Running time Intl. Conf. on Computer Vision (ICCV), pp. 1026-1034, Santiago, 2015.
[9] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization”,
refers to one complete encoder and decoder process for one arXiv:1412.6980, pp.1-15, Dec. 2014.
color image with a resolution of 768 × 512, while Balle’s time [10] J. Deng, W. Dong, R. Socher, L. Li, K. Li and L. Fei-Fei, “ImageNet:
A Large-Scale Hierarchical Image Database”, IEEE Conf. on Computer
refers to the gray image. The running time comparison for Vision and Pattern Recognition, pp. 1-8, June 20-25, 2009.
each image for different image compression methods is listed [11] Kodak Lossless True Color Image Suite, Download from
http://r0k.us/graphics/kodak/
in Table I. It can be observed that our CAE-based method [12] Z. Wang, E. P. Simoncelli and A. C. Bovik, “Multiscale structural
achieves lower complexity than Balle’s method [5] when it is similarity for image quality assessment”, The 36-th Asilomar Conference
on Signals, Systems and Computers, Vol.2, pp. 1398-1402, Nov. 2013.
run by the CPU, because we have designed a relatively simple [13] G. Bjontegaard, “Calculation of Average PSNR Differences between
CAE architecture. Besides, with GPU implementation, our RDcurves”, ITU-T VCEG, Document VCEG-M33, Apr. 2001.
[14] JPEG official software libjpeg, https://jpeg.org/jpeg/software.html
[15] JPEG2000 official software OpenJPEG,
https://jpeg.org/jpeg2000/software.html
[16] Z. Cheng, M. Takeuchi, J. Katto, “A Pre-Saliency Map Based Blind
Image Quality Assessment via Convolutional Neural Networks”, IEEE
Intl. Symposium on Multimedia, pp. 1-6, Dec. 11-13, 2017.