Edge-Aware Color Image Manipulation

by Combination of Low-Pass Linear Filter

and Morphological Processing of Its
Residuals

Marcin Iwanowski(B)

Institute of Control and Industrial Electronics, Warsaw University of Technology,

ul. Koszykowa 75, 00-662 Warsaw, Poland
[email protected]

Abstract. In the paper, a method of edge-preserving processing of

color images, called LPMPR (Low-Pass filter with Morphologically Pro-
cessed Residuals), is proposed. It combines linear low-pass filtering with
non-linear techniques, that allow for selecting meaningful regions of the
image, where edges should be preserved. The selection of those regions
is based on morphological processing of the linear filter residuals and
aims to find meaningful regions characterized by edges of high ampli-
tude and appropriate size. To find them, two methods of morphological
image processing are used: reconstruction operator and area opening.
The meaningful reconstructed regions are finally combined with the low-
pass filtering result to recover the edges’ original shape. Besides, the
method allows for controlling the contrast of the output image. The pro-
cessing result depends on four parameters, the choice of which allows
for adjusting the processed image to particular requirements. Results of
experiments, showing example filtering results, are also presented in the
paper.

Keywords: Image processing · Image filtering · Mathematical

morphology · Edge-preserving smoothing · Contrast enhancement

1 Introduction

The image low-pass filtering using linear (convolution) filters is one of the sim-
plest, oldest, and most often applied image processing techniques. It allows for
image blurring that makes image regions smooth by reducing the high-frequency
image components. There are two kinds of those components: irrelevant noise
and meaningful edges. To the first kind belong all image content, the blurring of
which is desirable: real noise and small objects referred to, e.g. details without
any importance, skin defects on portrait images, the texture of land cover ele-
ments, etc. The second kind of high-frequency component refers to rapid changes
of image intensity value related to boundaries of objects that usually should be
c Springer Nature Switzerland AG 2020
L. J. Chmielewski et al. (Eds.): ICCVG 2020, LNCS 12334, pp. 59–71, 2020.
https://doi.org/10.1007/978-3-030-59006-2_6
60 M. Iwanowski

visible without any blurring influencing contrast at edges. The classic linear
low-pass filters modify both high-frequency components.
The proposed LPMPR (Low-Pass filter with Morphologically Processed
Residuals) method combines the linear low-pass filtering with non-linear pro-
cessing of its residual in order to find and restore the meaningful edges on the
low-pass filtered image1 and is based on the concept described in [6]. The work-
flow of the original approach is as follows. At first, the image is filtered using
the Gaussian filter. Next, its residual is computed and split into two fractions
referred to subsets of pixels of the positive and the absolute value of negative
fraction. Both fractions are subject to thresholding to find strong edges (high-
contrast ones). The thresholding result is next used as markers for the morpho-
logical reconstruction of residuals. Reconstructed residuals that do not contain
irrelevant image components are added to the low-pass filtered image to recover
meaningful edges.
In this paper, the above method is extended by adding three essential func-
tionalities, that overcomes three imperfections of the original. The first one refers
to selecting the relevant regions that have originally been based on the resid-
ual only amplitude. In the proposed approach, it is extended by adding the
second, size-criterion. Secondly, the original method does not allow to control
the contrast of the output image. To solve this problem, a proposed solution
introduces an additional parameter – a contrast coefficient that provides for
increasing/reducing the contrast of the output image. Finally, last but not least,
the original method was restricted to gray-tone images only. In this paper, a way
of extending it to color images is proposed.
The paper is organized as follows. Section 2 contains the discussion on related
papers. In Sect. 3, the graytone case is described. Section 4 is dedicated to color
image processing using the proposed approach. In Sect. 5, the examples are
shown. Finally, Sect. 6 concludes the paper.

2 Related Work

The problem of edge-preservation in smoothing filters has already been addressed

in the literature in numerous papers. The Kuwahara-Nagao filters [7,8], are based
on the division of the pixel neighborhood into four overlapping regions. Based on
the standard deviation, computed within these regions, the appropriate region-
mean is chosen as the filter response.
The anisotropic diffusion approach [9] is based on the gradients of the fil-
tering image to restrict the diffusion process that prevents edge blurring. This
approach also exploits an idea typical for many other edge-preserving methods –
it introduces a guidance image, according to which there exists a supplementary
image that is used to pass the information on the way the output pixel value is
computed.

1
The MATLAB code of the proposed filtering method is available at https://www.
mathworks.com/matlabcentral/fileexchange/77581-lpmpr-image-filter.
 Image Manipulation by Low-Pass Linear and Morphological Processing 61

One of the most popular filters, based on a similar concept, is a bilateral

filter [2,12]. In this filter, typical for the linear filters, the weighting has been
modified. The special factor that depends on the pixel values differences has
been added. It allowed for reducing the blurring in the edge regions. Among
other filters of similar type are [5,10]. Relations between most popular edge-
preserving filters has been studied in [1].
In [13], the Gaussian filter residuals are linearly processed in an iterative way
to get better preservation of edges in the final image. An application of morpho-
logical processing to edge-aware filtering was described in [3]. The method is
based on adaptive geodesic time propagation.
Among recent developments in this field, one can mention [4], where a flexible
filter based on a linear model was proposed. It that may be used not only to
image filtering, but to perform also some other image processing tasks like e.g.
transfer structures of guidance image to the output one etc.

3 Graylevel Images

3.1 The Original Approach

The original approach [6] has been proposed for gray-level images. It is based on
the residual of the Gaussian filter, denoted as:

Ilf = Iin ∗ L, (1)

where Iin stands for the original gray-level image, ∗ is the convolution operator
and L stands for the mask of the Gaussian (or any other linear low-pass) filter.
The linear filter residual is defined as:

Ires = Iin − Ilf (2)

Further processing of the residual is based on operators that are defined on

positive-valued images. Due to this fact, the Ires , which consists of both positive
and negative values, is split into two fractions – positive and negative:

Ires (p) if Ires (p) > 0
∀p Ires+ (p) = (3)
0 if Ires (p) ≤ 0

−Ires (p) if Ires (p) < 0
∀p Ires− (p) = (4)
0 if Ires (p) ≥ 0
Both fractions of the residual (Ires+ , Ires− ) are further processed in order to
cut-off the irrelevant variations of the residual while preserving significant ones
based on the amplitude of the residual. The processing rests on the morpholog-
ical operator M, based on the reconstruction that selects relevant parts of the
residuum and preserves their original structure. Finally, relevant parts of the
62 M. Iwanowski

residuals are added to the filtered image so that meaningful edges are recovered,
while the image remains blurred within the irrelevant image regions:

Iout = Ilf + M(Ires+ ) − M(Ires− ). (5)

The operator M, which is the same for both residuals is defined as:
  
M(I) = RI min I, St (I)|{min{I},max{I}} , (6)

where RI (A) stands for the morphological reconstruction of the gray-level mask
image I from (also gray-level) markers A, ‘|’ is a mapping operator that converts
a binary image into gray-level one by replacing original values of 0’s and 1’s by
two given ones. Finally, ‘min’ is a point-wise minimum operator of two images.
The most important element of the definition of M (Eq. 6) is the binary marker
S that contains meaningful, relevant regions where contrast should be preserved.
The choice of these regions is based on the amplitude of the residuum:

St (I) = (I ≥ t). (7)

S is thus the selection operator that extracts (in a form of a binary image)
regions of I of amplitude higher than given threshold t.

Fig. 1. Simple graylevel image filtering: original image (a), result of the Gaussian
filtering with σ = 15 (b), filtering with t = 0.1 positive/negative mask (c) and filtering
result (d)

An example of image filtering using the above approach is shown in Fig. 1.

The figure exhibits results of filtering of a test image, with various t values.
In addition, the binary mask (markers of the meaningful regions) are drawn
as white (positive binary mask: St (Ires+ )) and black (negative one: St (Ires− )),
gray is used to indicate regions where both masks are equal to 0. The increase
of the threshold results in lower number of detected regions that are further
reconstructed and used to recover the original content of the image. Finally,
the number of regions where the original sharpness of regions is recovered, is
decreasing.
 Image Manipulation by Low-Pass Linear and Morphological Processing 63

3.2 Meaningful Region Selection Using Size Criterion

In the original workflow of the method, binary masks, obtained by thresholding,

contain boundaries of meaningful regions. The ‘meaningfulness’ is estimated by
means of the amplitude of residuals. However, in fact, the amplitude in not the
only factor that determines the importance of image region. One may easily
imagine image elements characterized by high amplitude of residual that are not
important for image understanding. For example, addition of a salt-and-pepper
noise to the original image introduces a lot of tiny (one-pixel-size) image elements
of high amplitude on the input image and consequently modifies the residual
image by adding high-amplitude elements. As such, they would be detected as
meaningful regions, which is not desirable.
In order to tackle with this problem, an additional step is added to the
original method. The binary mask is filtered using the area opening filter [11].
This filter removes from the binary image, all connected components of size lower
that a given size threshold s (size-coefficient). The equation 7 is thus extended to:

St,s (I) = (I ≥ t) ◦ (s), (8)

where ◦(s) is the area opening, removing components smaller than given size
defined by s (number of pixels in the connected component of the thresholded
fraction of the residual).

Fig. 2. Results of filtering for various s values – Gaussian with σ = 15, t = 0.05:
positive/negative mask and filtering result for s = 20 (a), (b); s = 110 (c), (d)

The result of applying the size coefficient s is shown in Fig. 2. Comparing

to Fig. 1 one may observe that in both cases, when the coefficient (t in case of
Fig. 1 and s in case of Fig. 2) increases, the number of selected regions decreases.
Contrary, however, to the original case, when coefficient s is used, the regions are
removed based on their size. It allows thus to reject small – in terms of a number
of pixels – objects from the residual even if their amplitude is high. Finally, it
allows for keeping these regions blurred on the final image.
64 M. Iwanowski

3.3 Contrast Control

One of the classic approaches to image contrast enhancement is the based on

the subtraction (or addition – depending on the filter mask coefficients) of the
high-pass filter from the image itself. It is based on the high-pass filter property
that allows for detecting the local variations of image pixel values. The differ-
ence between low-pass filter and the image itself is another way of getting the
high-pass filtering result. In the proposed method, the morphologically-processed
residuals refer to image high-frequency components containing regions of the
amplitude above the threshold t. In order to get control of the output image
contrast, a contrast control coefficient c is introduced in the Eq. 5, so that the
modified formula is defined now as follows:

Iout = Ilf + (M(Ires+ ) − M(Ires− ))) · c. (9)

Depending on c, the contrast is either enhanced (c > 1), preserved (c = 1)

or reduced (0 < c < 1). An example showing how the contrast control coeffi-
cient affects the result of processing is shown in Fig. 3. In addition, the contrast
enhancement result without morphological processing of residuals is shown. Com-
paring picture Fig. 3(c) and (d) one may see how the morphological processing
affects on the number of small image details visible on the final image. Pro-
posed processing allows for rejecting details of low importance so that finally
only contrast of meaningful regions is enhanced.

Fig. 3. The influence of the contrast factor c to the final result. The original ‘Lena’
image (a), LPMPR filtering without contrast modification, c = 1, t = 0.2 (b), contrast
enhancement c = 1.5, t = 0.2 (c), contrast enhancement c = 1.5 without residual
filtering (t = 0) (d)

4 Color Image Processing

The proposed method is based on both linear and non-linear techniques. The
extension of linear methods from gray-level case to color one is usually achieved
by independent processing of color components. Contrary to the linear case, the
ways of extending non-linear techniques to color case are more differentiated.
 Image Manipulation by Low-Pass Linear and Morphological Processing 65

The main reason are problems caused by non-linear independent processing of

separated channels. They usually appear as modification of colors within the
final color image, that in worst case have a form of so-called false colors i.e.
colors that are not present on the original image.
To solve this problem, an application of a single binary mask, the same for all
channel is proposed. Thanks to that, for each color component, the same regions
are marked as meaningful, and consequently the same regions are reconstructed
on all color channels. The proposed way of producing a single mask is based on
the assumption that the primary role for interpreting the presence of objects on
the image plays image luminance. For the input RGB color space, it is computed
as the weighted average of components. Thanks to the linear properties of the
first part of processing, instead of starting from the luminance of the input image,
the luminance of residuals is computed giving the same result:

Lres+ = lum(Ires+ ); Lres− = lum(Ires− ), (10)

where upper indexes R, G, B determines the color component and lum(I) =

0.3I R + 0.6I G + 0.1I B is the luminance.

Fig. 4. Color processing (Gaussian with σ = 11, t = 0.12, c = 1.8, s = 0). Original
image (a), blurred image (b), independent processing of color channels (c), luminance-
driven processing (d) (Color figure online)

An example of color processing is shown in Fig. 4 – the quality of image

obtained by independent channel processing is visibly lower than the result of
applying the proposed approach.
Considering all described extensions, the complete processing scheme of color
images may be summarized by following algorithm:

– Input color image Iin , output color image Iout

– Parameters: linear filter mask L, amplitude threshold t, contrast coefficient
c, size threshold s
66 M. Iwanowski

1. Filter Iin using linear filter with mask L to get Ilf (Eq. 1)
2. Compute Ires , Ires+ , Ires− (Eq. 2, 3, 4)
3. Compute residuals of the luminance Lres+ and Lres− (Eq. 2)
4. For each residual (Lres+ , Lres− ) create binary mask (St,s (Lres+ ), St,s (Lres− ))
applying given t and s (Eq. 8)
cc
5. For each cc ∈ {R, G, B} perform reconstruction of Ires− with mask
cc
St,s (Lres− ) and get M(Ires− ) (Eq. 6)
cc cc
6. Do the same using Ires+ and St,s (Lres+ ) to get M(Ires+ ) (Eq. 6)
7. Combine residuals with If ilt and get final image Iout (Eq. 9) for given c

The control of the proposed method is performed by means of three param-

eters t, s and c. Depending on their choice one controls level of details recovered
based on the amplitude of residual (t), as well as the size of particles residual
(s). Moreover, thanks to parameter c one controls also the contrast of the output
image. An example showing how the choice of parameters t and s influences the
final result is shown in Fig. 5.

Fig. 5. Results of processing with various parameter sets (c = 1.5 and σ = 5 are
constant): t = 0.05, s = 0 (a); t = 0.15, s = 0 (b); t = 0.15, s = 2 (c); t = 0.15,
s = 8 (d)

5 Results

The comparison of the proposed method with guided and bilateral filters (with
default parameters in their MATLAB implementations) is shown in Fig. 6. The
proposed approach allows for better extracting the relevant object (not blurred)
from the blurred background. In the case of its competitors the increase of the
expected blurring would affect also to quality and sharpness of relevant object.
 Image Manipulation by Low-Pass Linear and Morphological Processing 67

Fig. 6. Original image (a), filtered using the proposed method (b), using guided filter
(c) and bilateral filter (d) (Color figure online)

Due to its four parameters, the LPMPR filters are flexible. They may be
adjusted to get filtering results focused on preserving edges of a particular object
or content classes of the specific image. The image shown in Fig. 7 is a part of
a high-resolution satellite image showing a mountain house located close to the
lake with different types of landcover classes around: trees, bushes, water, stones,
underwater stones, grass, road, etc. Depending on the choice of parameters,
various classes might remain sharp, while the remainder of the image is blurred.
The original image shown at the position (a) has been filtered using multiple
sets of parameters, that are printed below figures at particular positions. All
the operations were performed on the original image (a) with Gaussian filtering
with σ = 15 (b). The results of LPMPR are the following. Having a look at
the position (c) one may see that only the smallest stones at the lakeshore are
blurred (low value of t), in case of (d), for t = 0.1 similar results are obtained
for lower s but now, also the underwater stones disappear. On position (e) it
is visible that, when increasing s, the area covered by bushes and grass became
blurred, on (f) further increase of t results in more robust filtering results. The
remainder of cases illustrates the contrast issues, on (g) high values of t and
s produces an image where most of the area is blurred apart from the house
and some other high-contrasted details. The next one, (h), shows what happens
when going down with s and, in the same, increasing c. It results in a highly
contrasted final image but with small details of texture blurred. The image at
the last position (i) is presented to compare with (h) – it depicts the same level
of contrast enhancement but without morphological processing of residuals is
the result of classic contrast enhancement method.
The method may be also successively applied, to filter portrait images of
human faces, where skin smoothing is one of the most popular tasks. Its appli-
cation allows for reducing considerably the noise level inside the skin regions
(what makes the skin smooth) and preserving the contrast on image edges (face
and its details outlines). The example of such an application is shown in Fig. 8.
68 M. Iwanowski

original Gaussian blur σ = 15 t = 0.01, s = 100, c = 1.3

(a) (b) (c)

t = 0.1, s = 20, c = 1.3 t = 0.1, s = 90, c = 1.3 t = 0.2, s = 90, c = 1.3

(d) (e) (f)

t = 0.3, s = 160, c = 1.3 t = 0.3, s = 10, c = 2.4 t = 0, s = 0, c = 2.4

(g) (h) (i)

Fig. 7. LPMPR filtering results of the satellite image with various land-cover classes,
for filtering parameters of see captions below images for description see the text (Color
figure online)
 Image Manipulation by Low-Pass Linear and Morphological Processing 69

Fig. 8. Photo enhancement. The original picture (a), filtered one (b), enlarged part of
the original (c) and of the filtered one (d)

In most of the previous examples, the contrast coefficient was set to values
slightly above 0 (1 < c < 1.8). This results in increasing pixel values difference
on edges and improve the contrast of meaningful image objects. This range
of possible changes of c gives, as a result, natural-looking images with visibly
increased contrast of details. Further increase of c (c > 2) causes a somehow
unnatural outlook of images. The boundaries between image regions become so
sharp that its color becomes either black or white. The extremely high contrast
makes image object looking comics-like – flat regions of relatively uniform color
value separated by black or white lines. An effect of transferring an image into
comics is shown in Fig. 9.

Fig. 9. Two examples of comics effect. Filtering with (in both cases) with t = 0.03,
c = 3, s = 1000.
70 M. Iwanowski

6 Conclusions
In the paper, a novel method for edge-aware processing of color images, called
LPMPR, has been proposed. It is based on the principle of extracting the resid-
uals of the low-pass linear filter that is further processed using a non-linear
approach based on morphological operators of area opening and reconstruction.
The result of the latter processing is added back to the low-pass filtering result.
Non-linear filtering aims to localize and extract meaningful edges reconstructed
on the final image so that blurring is not present within their regions. Four
parameters control the method’s behavior: the mask of the Gaussian (on any
other low-pass linear) filter, the threshold t, size coefficient s, and contrast coef-
ficient c. Depending on the choice of these parameters, the final result may be
adapted to particular requirements. The method may be applied to perform
a controllable edge-aware image blur with contrast improvement, to improve
the quality of images with a controlled level of details preserved. The proposed
LPMPR method has been illustrated in the paper by examples demonstrating
its usefulness.

References
1. Barash, D.: Fundamental relationship between bilateral filtering, adaptive smooth-
ing, and the nonlinear diffusion equation. IEEE Trans. Pattern Anal. Mach. Intell.
24(6), 844–847 (2002)
2. Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range
images. ACM Trans. Graph. 21(3), 257–266 (2002)
3. Grazzini, J., Soille, P.: Edge-preserving smoothing using a similarity measure in
adaptive geodesic neighbourhoods. Pattern Recogn. 42(10), 2306–2316 (2009).
Selected papers from the 14th IAPR International Conference on Discrete Geom-
etry for Computer Imagery 2008
4. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach.
Intell. 35(6), 1397–1409 (2013)
5. Hong, V., Palus, H., Paulus, D.: Edge preserving filters on color images. In: Bubak,
M., van Albada, G.D., Sloot, P.M.A., Dongarra, J. (eds.) ICCS 2004. LNCS, vol.
3039, pp. 34–40. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-
25944-2 5
6. Iwanowski, M.: Morphological processing of Gaussian residuals for edge-preserving
smoothing. In: Angulo, J., Velasco-Forero, S., Meyer, F. (eds.) ISMM 2017. LNCS,
vol. 10225, pp. 331–341. Springer, Cham (2017). https://doi.org/10.1007/978-3-
319-57240-6 27
7. Kuwahara, M., Hachimura, K., Eiho, S., Kinoshita, M.: Processing of RI-
angiocardiographic images. In: Preston, K., Onoe, M. (eds.) Digital Processing
of Biomedical Images, pp. 187–202. Springer, Boston (1976). https://doi.org/10.
1007/978-1-4684-0769-3 13
8. Nagao, M., Matsuyama, T.: Edge preserving smoothing. Comput. Graph. Image
Process. 9(4), 394–407 (1979)
9. Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion.
IEEE Trans. Pattern Anal. Mach. Intell. 12(7), 629–639 (1990)
 Image Manipulation by Low-Pass Linear and Morphological Processing 71

10. Saint-Marc, P., Chen, J., Medioni, G.: Adaptive smoothing: a general tool for early
vision. IEEE Trans. Pattern Anal. Mach. Intell. 13(6), 514–529 (1991)
11. Soille, P.: Morphological Image Analysis: Principles and Applications. Springer,
Berlin (2004). https://doi.org/10.1007/978-3-662-05088-0
12. Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Sixth
International Conference on Computer Vision (IEEE Cat. No.98CH36271), pp.
839–846, January 1998
13. Wheeler, M.D., Ikeuchi, K.: Iterative smoothed residuals: a low-pass filter for
smoothing with controlled shrinkage. IEEE Trans. Pattern Anal. Mach. Intell.
18(3), 334–337 (1996)