information

Article
Enhancement of Low Contrast Images Based on
Effective Space Combined with Pixel Learning
Gengfei Li 1,2, *, Guiju Li 1 and Guangliang Han 1
1 Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences,
#3888 Dongnanhu Road, Changchun 130033, China; [email protected] (G.L.); [email protected] (G.H.)
2 University of Chinese Academy of Science, #19 Yuquan Road, Beijing 100049, China
* Correspondence: [email protected]; Tel.: +86-177-1023-7905

Received: 19 September 2017; Accepted: 27 October 2017; Published: 1 November 2017

Abstract: Images captured in bad conditions often suffer from low contrast. In this paper, we
proposed a simple, but efficient linear restoration model to enhance the low contrast images.
The model’s design is based on the effective space of the 3D surface graph of the image. Effective space
is defined as the minimum space containing the 3D surface graph of the image, and the proportion
of the pixel value in the effective space is considered to reflect the details of images. The bright
channel prior and the dark channel prior are used to estimate the effective space, however, they may
cause block artifacts. We designed the pixel learning to solve this problem. Pixel learning takes the
input image as the training example and the low frequency component of input as the label to learn
(pixel by pixel) based on the look-up table model. The proposed method is very fast and can restore
a high-quality image with fine details. The experimental results on a variety of images captured in
bad conditions, such as nonuniform light, night, hazy and underwater, demonstrate the effectiveness
and efficiency of the proposed method.

Keywords: image enhancement; restoration model; nonilluminated; nighttime; dehaze; nighttime;

underwater

1. Introduction
In the image acquisition process, the low illumination in nonuniformly illuminated environment
or light scattering caused by turbid medium in the foggy/underwater environment will lead to low
image contrast. Based on the variety of bad conditions, however, it is difficult to enhance the image
of these conditions through a unified approach. Though traditional methods such as the histogram
equalization will deal with all these low contrast images, most results are show to be uncomfortable
for the human visual system. Therefore, most of them establish a specific recovery model based on the
distinctive physical environment to enhance the images.
Dealing with light compensation is usually done by using the Retinex (retina-cortex) model,
which is built on the human visual system [1–3]. The early single-scale Retinex algorithm proposed
by Jobson [4] can either provide dynamic range compression on a small scale or tonal rendition on a
large scale. Therefore, Jobson continued his research and proposed the MSR (multiscale Retinex)
algorithm [5], which has been the most widely used in recent years. Most improved Retinex
algorithms [6–15] are based on MSR. However, the Gaussian filtering used by the MSR algorithm
calculates a large number of floating data, which makes the algorithm take too much time.
Therefore, for practical use, Jiang et al. [15] used hardware acceleration to implement the MSR
algorithm. In addition, some research [16] used the dehaze model instead of the Retinex model
to deal with the negative input image for light compensation or using the bright channel prior with
the guided filter [17] for a quick lighting compensation.

Information 2017, 8, 135; doi:10.3390/info8040135 www.mdpi.com/journal/information

Information 2017, 8, 135 2 of 20

Early haze removal algorithms require multiple frames or additional depth map information [18–20].
Since Fattal et al. [21] and Tan et al. [22] proposed a single-frame dehaze algorithm relying on stronger
priors or assumptions, the single-frame dehaze algorithms have become a research focus. Subsequently,
He et al. [23] proposed the DCP (dark channel prior) for haze removal, which laid the foundation
for the dehazing algorithm in recent years. Combining the DCP with the guided filter [24] is also
the most efficient method for the dehazing algorithm. Since then, the principal study of the fog
algorithm has focused on the matting technique of transmittance [24–30]. Meng et al. [25] applied a
weighted L1-norm-based contextual regularization to optimize the estimation of the unknown scene
transmission. Sung et al. [26] used a fast guided filter that was combined with the up/down samples to
optimize the performance time. Zhang et al. [28] used the five-dimensional feature vectors to recover
the transmission values by finding their nearest neighbors from the fixed points. Li et al. [30] computed
a spatially varying atmospheric light map to predict the transmission and refined it by the guided
filter [24]. The guided filter [24] is an O(n) time edge-preserving filter with the similar result of bilateral
filter. However, in the application, the well-refined methods are hard to perform on a video system due
to the time cost, and the fast local filters like guided/bilateral filters would concentrate the blurring
near the strong edges, then introducing halos.
There have been several attempts to restore and enhance the underwater image [31–33].
However, there is no general restoration model for such degraded images. Research mainly applies
the white balance [32–34] to correct the color bias at first and then uses a series of contrast stretch
processing to enhance the visibility of underwater images.
Although these studies have applied different models and methods for image enhancement,
the essential goal of all of these is to stretch the contrast. In this paper, we proposed a unified
restoration model for these low-contrast degraded images to reduce the human operations or even in
some multidegradation environments. Moreover, due to the artifacts produced by the patch-based
methods we applied in our approach, the pixel learning refinement is proposed.

2. Background

2.1. Retinex Model

The Retinex theory was proposed by Land et al. based on the property of the human visual system,
which is commonly referred to as the color constancy. The main goal of this model is to decompose
the given image I into the illuminated image L and the reflectance image R. Then, the output of the
reflectance image R is its result. The basic Retinex model is given as:

I ( x ) = L ( x ).∗ R ( x ), (1)

where x is the image coordinates and operator .* denotes the matrix point multiplication. The main
parameter of the Retinex algorithm is the Gaussian filter’s radius. A large radius can get obtain color
recovery, and a small radius can retain more details. Therefore, the most commonly used Retinex
algorithm is multiscale. Early research on Retinex was mainly for light compensation. In recent years,
a large number of studies has been undertaken to enhance the image with hazy [10] and underwater
images [8,31] using the Retinex algorithm. However, these methods are developed with their own
framework for a specific environment.

2.2. Dehaze Model

The most widely-used dehaze model in computer vision is:

I( x ) = J( x ).∗ T( x ) + (1 − T( x )) × A, (2)
Information 2017, 8, 135 3 of 20

where I is the hazy image, J is the clear image, T is the medium transmittance image and A is the
global atmospheric light, usually constant. In the methods of estimating the transmittance image T,
the DCP is the simplest and most widely-used method, which is given by:

dark p (I( x )) = min (Ic ( x ))

c∈(r,g,b)
, (3)
dark b (I( x )) = min (dark p (I(y)))
y∈Ω( x )

Information 2017, 8, 135 3 of 20

where darkp represents the pixel dark channel operator and darkb represents the block dark channel
operator. I is the input image; x is the image coordinates; c is the color channel index; and y ∈ Ω( x )
where darkp represents the pixel dark channel operator and darkb represents the block dark channel
indicates that y is a local patch centered at x. The main property of the block dark channel image is that
operator. I is the input image; x is the image coordinates; c is the color channel index; and y ∈Ω( x)
in the haze-free image, the intensity of its block dark channel image tends to be zero. Assuming that
indicates that y is a local patch centered at x. The main property of the block dark channel image is
the atmospheric light A is given, the estimated expression of the transmittance T can be obtained by
that in the haze-free image, the intensity of its block dark channel image tends to be zero. Assuming
putting the block dark channel operator on both sides of (2):
that the atmospheric light A is given, the estimated expression of the transmittance T can be
obtained by putting the block dark channel operator on both sides of (2):
I( x )
T( x ) = 1 − dark b ( ), (4)
I (A
x)
T( x ) = 1 − darkb ( ), (4)
A
With transmittance T and the given atmospheric light A, we can enhance the hazy image according
With (2).
to Equation transmittance T and the
However, Equation (4)given atmospheric
can only be used aslight A, we canestimation
a preliminary enhance the hazy the
because image
block
according to Equation (2). However, Equation (4) can only be used as a preliminary
dark channel image can produce the block artifacts due to the minimum filtering. Therefore, a refine estimation
because
process the blockbefore
is required dark channel image In
the recovery. canaddition,
produce the the block
dehaze artifacts
modelduehas to the minimum
some filtering.
physical defects that
Therefore, a refine process is required before the recovery. In addition, the dehaze model has some
can result in the image looking dim after haze removal.
physical defects that can result in the image looking dim after haze removal.
In the next section, we present a simple linear model for enhancing all the mentioned low contrast
In the next section, we present a simple linear model for enhancing all the mentioned low
images based on an observation of the images’ 3D surface graph.
contrast images based on an observation of the images’ 3D surface graph.
3. Proposed Model
3. Proposed Model
3.1. Enhance Model Based on Effective Space
3.1. Enhance Model Based on Effective Space
The effective space comes from the observation of the 3D surface graph of the images.
The effective space comes from the observation of the 3D surface graph of the images. The three
The three coordinate axes of the 3D surface graph are the image width (x-axis), the image height
coordinate axes of the 3D surface graph are the image width (x-axis), the image height (y-axis) and
(y-axis) and the pixel value (z-axis), and all the pixel points of an image can be connected to form a
the pixel value (z-axis), and all the pixel points of an image can be connected to form a surface, which
surface, which is the 3D surface graph. In this paper, we call the minimum space of the 3D surface
is the 3D surface graph. In this paper, we call the minimum space of the 3D surface graph an
graph an effective
effective space. Inspace.
order In
to order to show
show the law ofthe law ofspace,
effective effective
we space, we the
transform transform the color
color images into images
gray
into gray scale and project their 3D surface graph on the x-z plane, as shown in Figure 1 (in the 2D
scale and project their 3D surface graph on the x-z plane, as shown in Figure 1 (in the 2D projection,
projection,
the x-axisthe x-axis
is the is the
image image
width, andwidth, they-axis
and isthe
the y-axis pixel is the pixel value).
value).

(a)

Figure 1. Cont.
Information 2017, 8, 135 4 of 20
Information 2017, 8, 135 4 of 20

(b)

(c)

(d)

Figure 1. Images and the projection of their 3D surface graphs on the x-z plane. In the 2D projection,
Figure 1. Images and the projection of their 3D surface graphs on the x-z plane. In the 2D projection,
the x-axis is the image width, and the y-axis is the pixel value: (a) the clear images; (b) the low
the x-axis is the image width, and the y-axis is the pixel value: (a) the clear images; (b) the low
illumination images; (c) the hazy images; (d) the underwater images.
illumination images; (c) the hazy images; (d) the underwater images.
Information 2017, 8, 135 5 of 20

As can be seen, the projection of clear images is almost filled by the domain of the pixel value.
On the contrary, the degraded images’ are compressed. Therefore, we assume that the proportion
of the pixel value in the effective space denotes the detail information of the image. Since the image
size is fixed in processing, we estimate the two smooth surfaces of the effective space that are the
upper surface U (up) and the lower surface D (down) to obtain the proportion of the pixel value in the
effective space. It should be noted that in the ideal clear image, U is a constant plane at 255, while D is
zero. According to the relationship of the proportion, we can establish a linear model by:

I − DI J − DJ J−0
= = , (5)
UI − DI UJ − DJ 255 − 0

where I represents the input degraded image, J represents the ideal clear image, UI and UJ represent
the upper surface of the effective space of I and J, respectively, and similarly, DI and DJ represent
the lower surface. Once we obtain U and D, we can enhance the low contrast images according to
Equation (5). However, the denominator in Equation (5) can be zero when the pixel value of D is equal
to U. Therefore, we introduce a small parameter in the denominator to avoid the division by zero.
Our final enhanced model is given as:

I − DI
J = U J .∗ , (6)
UI − DI + λ

where D is estimated by the block dark channel operation, according to Equation (3), U is estimated
by the block bright channel operation, which can be calculated by Equation (3) with the min function
replaced by the max function and λ is a small factor to prevent the division by zero, usually set to
0.01. UJ denotes the light intensity of the enhanced images, ideally set to the maximum gray level
(the eight-bit image is 255). The images always looks dim after haze removal based on Equation (2).
Our model can improve this phenomenon while combining the Retinex model and the dehaze model
so that our model can be adapted according to the scene requirements.

3.2. Relationship with Retinex Model and Dehaze Model

If an image is haze free, the intensity of its block dark channel image is always very low and
tends to zero. Similarly, if an image has abundant light supply, the intensity of its block bright channel
image usually approaches the maximum value of the image dynamic range. The main cause of the
dim restoration of the dehaze algorithm is the overestimated atmospheric light A. Considering a hazy
image without color bias, the atmospheric light A is approximately equal in each channel of the color
image; in other words, AR = AG = AB . Due to this, Equation (4) can be rewritten as:

dark b (I)
T = 1− , (7)
A
where A represents the light intensity of all three channels. Putting Equation (7) into Equation (2),
we can remove the haze with the style of our model as Equation (6):

I−A I−A I − DI
J= +A = DI
+ A = A J .∗ , (8)
T 1− A A − DI

where, in our model, AJ denotes the illumination of the clear image and A is the input
image’s illumination; whereas, in the fog model, AJ = A, which both represent the input
image’s illumination. Due to the impact of the clouds, the light intensity is usually low when the
weather is rainy or foggy. This can lead to the result of haze removal looking dim. On the other hand,
most haze removal research assumes that the atmospheric light intensity is uniform, and A is estimated
as a constant. According to Equation (9), if the input image has sufficient illumination, which means
A = 255 = UJ = AJ , the proposed model is equivalent to the dehaze model. Nevertheless, the light
Information 2017, 8, 135 6 of 20

intensity of a real scene is always nonuniform. As can be seen in Equation (9), when DI remains the
same, the larger the A we give, the smaller the J we have. On the whole, a large estimated value
of the constant A will be great in the thick foggy region, but for the foreground region with only a
little mist, it is too large. This is the main cause of haze removal always becoming a dim restoration.
Moreover, if the input image I is a haze-free image, it will be the typical low contrast problem of
Retinex. According to the constant zero tendency of the block dark channel image and Equation (1),
we have a Retinex model with the style of Equation (6):

I I−0 I − DI
J = 255 × R = 255 × = 255 × = 255 × , (9)
L L−0 L − DI

where R ∈ [0, 1] represents the reflectivity in Equation (1). In order to unify the dynamic range of the
output image J, we multiply R by 255 of the eight-bit image. According to Equation (9), if we estimate
the illuminated image L by a Gaussian filter, the proposed model is equivalent to the Retinex model.
It is notable that the bright channel prior is also a good method to estimate L, and the bright channel
prior will be faster than the Gaussian filter due to the integer operation. The study of Wang et al. [17]
shows that the bright channel prior has a significant effect in terms of light compensation.
Due to the constant zero tendency of the block dark channel image, the proposed model will
automatically turn into the Retinex model to compensate for the light intensity of the image in the
haze-free region. Besides, the model mainly modified the two assumptions of the dehaze model that
AJ is equal to A, and A is a constant; so that the proposed algorithm can increase the exposure of
J when the input I is a hazy image. However, as a result of the use of the bright/dark channel prior,
a refinement process is necessary to improve the block artifacts produced by the min/max filter used
in the priors.

4. Pixel Learning for Refinement

4.1. Pixel Learning

Pixel learning (PL) is a kind of edge-preserving smoothing, which is similar to online learning.
Generally, for the edge-preserving smoothing, the cost function is firstly established:

S( x ) = arg min(kI( x ) − S( x )k + k∇ S( x )k), (10)

where S represents the process of smoothing and I represents the input image. In Equation (10),
the first penalty term kI( x ) − S( x )k forces proximity between I and S, which can preserve the sharp
gradient edges of I for S, and this will help reduce the artifacts, such as the haloes. The second penalty
term λ × k∇ S( x )k is the degree of similarity between the estimated component and the input image,
and it forces the spatial smoothness on the processed image S. In the previous research results, there
were a number of methods to solve the optimization problem of Equation (10), in which the most
practical of the methods was the guided filter [24]. However, [24] still retained the halo artifacts due to
the frequent use of the mean filter. Although, there are some improvements such as applying iteration
optimization [27] or solving the equation of higher-order [24] to optimize the model, these algorithms
are usually too time consuming for a practical use due to the large number of pixels in an input image
as training examples for an optimization problem. Therefore, we introduce the idea of online learning
to overcome the time-consuming process of optimization.
Online learning is different from batch learning. Its samples come in sequence; in other words,
a one-by-one processing. Then, the classifier is updated according to each new sample. If the input
samples are large enough, the online learning can converge in only one scanning. As an online
learning technique, the pixel learning outputs the results pixel-by-pixel while learning from the input
image to perform the iteration optimization in just one instance of scanning of the whole input image.
The input image Ip (pixel level bright/dark channel or grayscale image) is taken as the training
Information 2017, 8, 135 7 of 20

example, and the mean filter result of its block bright/dark channel image Ibm is used as the label for
learning. As for most machine learning algorithms, we use the square of the difference between the
predictor and the label as the cost function, i.e., E(Ip( x )) = kY( x ) − Ibm( x )k2 , where Y(x) represents
the estimation of the current input pixel value Ip(x). The pixel learning should obtain a convergent
result by one-time scan like the online learning. However, the convergence of learning usually starts
with a large error and converge slowly, which could produce noises during the pixel by pixel learning.
Considering Equation (10), the fusion of the low-frequency image and the original image which
contains the high-frequency information can make the initial output error smaller, so7 ofthat
Information 2017, 8, 135 20
the fast
convergence and the noise suppression can be achieved. Here α fusion is applied as the learning
represents which
iterative equation, the estimation
is the of the current
simplest inputmethod.
fusion pixel valueItIp(x). The pixel
is given as: learning should obtain a
convergent result by one-time scan like the online learning. However, the convergence of learning
usually starts with a large error and converge slowly, which∗ could produce noises during the pixel
P(Equation
by pixel learning. Considering x ).∗ the
x ) = Y((10), (1 −ofα)the
α +fusion . Ibm ( x ),
low-frequency image and the original (11)
image which contains the high-frequency information can make the initial output error smaller, so
where x isthat
thetheimage coordinates,
fast convergence noisedenotes
and theY(x) suppression the
canestimation
be achieved. result of the
Here α fusion input as
is applied pixel
the value at
learning iterative
x as the forward equation,
propagation which is Ibm(x)
process, the simplest fusion method.
denotes the pixelIt isvalue
given as:
of a low-frequency image at
x as the label, α is the fusion weight as P( xthe ( x).*α + (1 − αof
) = Ygradient ).*the
Ibm(iterative
x) , step and P(x) is (11) the learning
results as the
wherepredictor. Note
x is the image that when
coordinates, Y(x)E(Ip(x))
denotes theis large,
estimation thisresult
means there
of the inputispixel
an edge that
value at x asshould be
preserved.theSimilarly, when E(Ip(x))
forward propagation process,isIbm(x)
small,denotes
this means
the pixel there
valueisofaa texture that should
low-frequency image at be
x assmoothed.
the label, αwe
As a consequence, is the fusion the
design weight as theweight
fusion gradientαofbased
the iterative step and P(x) is the learning results as
on E(Ip(x)):
the predictor. Note that when E(Ip(x)) is large, this means there is an edge that should be preserved.
Similarly, when E(Ip(x)) is small, this means thereE(Ipis( xa))texture that should be smoothed. As a
α(weight
consequence, we design the fusion x ) = α based on E(Ip(x)): 2 , (12)
E(Ip( x )) + thred1
E (Ip( x))
α ( x) = 2
, (12)
where thred1 is used to judge whether the detailE (needs Ip( x)) + thred
to be 1 preserved, and a small thred1 preserves

more details and

where reduces
thred1 is usedthe halowhether
to judge artifacts
the obviously.
detail needs toItbe ispreserved,
clear that andif aE(Ip(x)) is much
small thred1 larger than
preserves
α ≈details
thred1, thenmore 1; andandif reduces
E(Ip(x))theishalo artifacts
small enough,obviously.
then α ≈clear
It is that if E(Ip(x))
0. Figure 2 shows is the
much larger than between
comparison
the initial thred1,
output then α ≈ 1; and if E(Ip(x)) is small enough, then α ≈ 0. Figure 2 shows the comparison between
of the lower surface D by α fusion (with Y(x) = Ip(x)) and the result of guided
the initial output of the lower surface D by α fusion (with Y(x) = Ip(x)) and the result of guided filter
filter [24]. As
[24].can be seen,
As can be seen, thetheresult
result of
of α fusionis issimilar
α fusion similar to guided
to the the guided filterbut[24],
filter [24], somebut some
details stilldetails still
need further needrefinement.
further refinement.

(a) (b)
Figure 2. Comparison of α fusion and guided filter: (a) the α fusion, with thred1 = 10, the radius of
Figure 2. Comparison
the mean filter of fusion
α (b)
is 40; filter. filter: (a) the α fusion, with thred1 = 10, the radius of the
and guided
the guided
mean filter is 40; (b) the guided filter.
Information 2017, 8, 135 8 of 20

Information 2017, 8, 135 8 of 20

4.2. Mapping Model
4.2. Mapping Model
The edge-preserving smoothing using PL is not a regression process; the output and input of the
PL are notedge-preserving
The a one-to-one linear relationship.
smoothing usingImage
PL is smoothing is an operator
not a regression process;thatthe blurs
output theand
details
inputand
of
textures.
the PL areTherefore, for PL, alinear
not a one-to-one number of differentImage
relationship. adjacent inputs should
smoothing output approximate
is an operator that blurs theor even
details
equal pixel values.
and textures. Therefore,
Therefore, weaestablish
for PL, number aofmapping
differentmodel
adjacentbased on the
inputs look-up
should table
output I2Y instead or
approximate of
the polynomial model in machine learning. The idea is based on an assumption
even equal pixel values. Therefore, we establish a mapping model based on the look-up table I2Y of the image’s spatial
similarity
instead ofthatthethe points with
polynomial the same
model pixel values
in machine are clustered
learning. The idea nearby. Thus,
is based oninantheassumption
scanning process
of the
of f (I(x))spatial
image’s = P(x), similarity
the same pixel value
that the of I(x)
points may
with thebesame
calculated from several
pixel values different
are clustered valuesThus,
nearby. of P(x);
in
whereas, these values are changed smoothly. For this reason, to illustrate,
the scanning process of f(I(x)) = P(x), the same pixel value of I(x) may be calculated from severalthe size of our mapping
model
differentI2Yvalues
is 256 of
in P(x);
an eight-bit
whereas,image
theseand the initial
values values as
are changed its index.For
smoothly. Therefore, aftertoinitializing
this reason, illustrate,
the mapping model, we perform the PL by:
size of our mapping model I2Y is 256 in an eight-bit image and the initial values as its index.
Therefore, after initializing the mapping model, we perform the PL by:
P( x ) = I2Y [Ip( x )] × α + (1 − α) × Ibm( x )
P( x) = I 2Y [Ip( x)] × α + (1 − α ) × Ibm( x) , (13)
I2Y [Ip( x )] = P( x ) , (13)
I 2Y [Ip( x)] = P( x)

where I2Y
where I2Yisisour
ourmapping
mapping model, which
model, whichis aislook-up table,
a look-up and and
table, we calculate the latest
we calculate prediction
the latest result
prediction
of Ip(x)ofand
result Ip(x)theand
output. Then, we
the output. update
Then, the latest
we update themapping of pixelofvalues
latest mapping from Ip(x)
pixel values fromtoIp(x) I2Y.
in P(x)
P(x) to
The mapping model mainly plays the role of logical classification rather
in I2Y. The mapping model mainly plays the role of logical classification rather than linearthan linear transformation,
so we obtain a more
transformation, so we accurate matting
obtain a more resultmatting
accurate near theresult
depth discontinuities.
near Figure 3 shows
the depth discontinuities. Figurethe3
comparison among theamong
shows the comparison iterative
theresult of PL,
iterative α fusion
result of PL,and guided
α fusion andfilter [24].filter
guided It can be It
[24]. seen
canthat the
be seen
iterative result of pixel learning smoothed more textures of the background than
that the iterative result of pixel learning smoothed more textures of the background than the initial the initial output of
α fusionofand
output α the guided
fusion andfilter. However,
the guided there
filter. are somethere
However, sharpare edges
somethatsharp
have been
edgessmoothed
that havedue to
been
the excessively smooth label we have set near the edges of depth. To this end,
smoothed due to the excessively smooth label we have set near the edges of depth. To this end, we we modify the label,
which
modifyisthe thelabel,
low-frequency image
which is the Ibm.
low-frequency image Ibm.

(a) (b) (c)

Figure 3.
Figure 3. Comparison
Comparisonofof refinement:
refinement: (a) pixel
(a) the the pixel learning,
learning, with=thred1
with thred1 10; (b) =the
10; (b) thewith
α fusion, α fusion,
thred1 with
= 10;
thred1 = 10; (c) the
(c) the guided filter. guided filter.

4.3. Learning Label

4.3. Learning Label
Considering that Ib = darkb(Ip) (or Ib = brightb(Ip)) is a low-frequency image, which is
Considering that Ib = darkb (Ip) (or Ib = brightb (Ip)) is a low-frequency image, which is
disconnected near the depth edges, we try to refine the image D by learning from Ib and compare
disconnected near the depth edges, we try to refine the image D by learning from Ib and compare the
the two refinement results in Figure 4.
two refinement results in Figure 4.
Information 2017, 8, 135 9 of 20
Information 2017, 8, 135 9 of 20

(a) (b)
Figure 4. Comparison of refinement from different labels: (a) the label is Ibm, with thred1 = 10; (b)
Figure 4. Comparison of refinement from different labels: (a) the label is Ibm, with thred1 = 10; (b) the
the label is Ib, with thred1 = 10.
label is Ib, with thred1 = 10.
As can be seen, the bright region of Figure 4b leaves some block artifacts, but sharp edges near
As can be seen, the bright region of Figure 4b leaves some block artifacts, but sharp edges near
the depth discontinuities. Thus, we need the model to learn from Ibm for large pixel values, and the
the depth
small discontinuities. Thus, we
pixel values are learned fromneed the model we
Ib. Concretely, to learn fromand
fuse Ibm IbmIbfor
by large pixelbased
α fusion values, and the
on the
smallpixel
pixelvalue,
valuesso are learned
for the fusionfrom
weight, Concretely,
Ib. we have: we fuse Ibm and Ib by α fusion based on the pixel
value, so for the fusion weight, we have:
Ib.* Ibm
βD = 2
, (14)
∗ + thred 2
Ib.* Ibm
Ib. Ibm
β = , (14)
where β D is the fusion weight of αD fusion
Ib.∗for
IbmD.+It thred2
is worth
2 mentioning that the estimation of
upper surface U should be the opposite of lower surface D, which is:
where β D is the fusion weight of α fusion for D. It is worth mentioning that the estimation of upper
(255 − Ib).* (255 − Ibm )
surface U should be the opposite of βlower
U =
surface D, which is: 2
, (15)
(255 − Ib).* (255 − Ibm ) + thred 2
∗
where thred2 is set to 20 by experience.
βU = In this−
(255 Ib)we
way, can −
. (255 the ) α fusion
useIbm , to combine Ib with Ibm (15)
∗ 2
to obtain a new label image by: (255 − Ib). (255 − Ibm) + thred2

where thred2 is set to 20 by experience. In F = Ibm

this .* βwe
way, − β ).*
+ (1can useIb ,the α fusion to combine Ib with
(16)
Ibm to
obtain a newβ label
where is β Dimage
or β Uby:depending on Ib. Finally, our iteration of pixel learning is:
F = Ibm.∗ β + (1 − β).∗ Ib, (16)
P( x) = I 2Y [Ip( x)] × α + (1 − α ) × F( x) , (17)
where β is β D or β U depending on Ib. Finally, our iteration of pixel learning is:
It is important to note that the cost function should be rewritten as E (Ip( x)) = I 2Y (Ip( x)) − F( x) 2

due to the changes of the input

P( x )data
= I2Yand[Ip
the
( xlearning
)] × α + label.
(1 − αFinally,
) × F( xthe
), refined image is shown in (17)
Figure 5. It can be seen that the PL algorithm is smoother in detail than the guided filter and
preserves
It the sharp
is important edges
to note that near the function
the cost depth discontinuities, so that as
should be rewritten theEhalo
(Ip(xartifacts can(Ip
)) = k I2Y be(reduced
x)) − F(x)k2
visibly.
due to the changes of the input data and the learning label. Finally, the refined image is shown in Figure 5.
It can be seen that the PL algorithm is smoother in detail than the guided filter and preserves the sharp
edges near the depth discontinuities, so that the halo artifacts can be reduced visibly.
Information 2017, 8, 135 10 of 20
Information 2017, 8, 135 10 of 20
Information 2017, 8, 135 10 of 20

(a) (b)
(a) (b)
Figure 5. Comparison of refinement: (a) the pixel learning, where thred1 = 10, thred2 = 20; (b) the
Figure 5. Comparison of refinement: (a) the pixel learning, where thred1 = 10, thred2 = 20; (b) the guided filter.
guided
Figure filter.
5. Comparison of refinement: (a) the pixel learning, where thred1 = 10, thred2 = 20; (b) the
guided filter.
In addition, it is also important to pay attention to the failure of using the priors of the
In addition, it is also important to pay attention to the failure of using the priors of the
dark/bright
dark/brightchannel
channelin in
the sky region
regionoror the extremely darkshadow
shadow region. The failure will result in
In addition, it is the
alsosky
important thepay
to extremely dark
attention region.
to the failure of The failure
using will result
the priors in
of the
overenhancement,
overenhancement,
dark/bright channelwhichwhich can
in thecanstretch
sky stretch
regiontheornontexture
the nontexture
the extremelydetails
details such
dark such
shadowasas
thethe compression
compression
region. information,
information,
The failure will resultasin as
shown in Figure
shown 6. Therefore,
in Figure
overenhancement, 6. Therefore,
which can wewecancan
limit
stretch limit
the the
theestimation
estimation
nontexture of
ofU
details U to
to be
such be notcompression
not
as the toolow
too low(not
(nottoo
toohigh
highforfor
information, asD) by
D)
cutting
by off the
cutting
shown initialization
off the 6.
in Figure of I2Y:
initialization
Therefore, of I2Y:
we can limit the estimation of U to be not too low (not too high for D)
by cutting off the initialization of I2Y: I 2Y (i ) = min( I 2Y (i ), t )
D D D
I2YD (i ) = min( I2YD (i ), tD, ) (18)
II22YYUD((ii)) == max(
min( I 2YDU (i ), tDU ) , (18)
I2YU (Ii2)Y=(i)max ( I2Y (i ), t , )
= max( I 2YU(i ), t )U
(18)
where i is the index of the look-up table Uand tD and UtU are U
the cutoff thresholds for D and U,
where i is thei index
respectively.
where is the of the look-up
Empirically,
index we set
of the table and
tD = 150
look-up tD and
and
table and tDU and
tU = t70 aredefault
as the cutoff
tU are thethresholds
values. thefor
Oncethresholds
cutoff D and
value forUU,
of Dis respectively.
not
andtoo U,
small
Empirically, and
respectively.
wethe value
tD =of150
Empirically,
set D isand
notset
we ttoo
U t=Dlarge,
asthe
= 150
70 andskytU =region
default 70 as and
values. the values.
default
Once extremelyOnce
the value dark Ushadow
ofthe value region
is notoftooU is of the
not
small too
and the
valueimage
small
of D andwill
is notnottoo
the be large,
stretched
value of Dthe too
is not much,
skytoo soand
large,
region that the region
the sky
the useless
extremely details
and the
darkwill not bedark
extremely
shadow enhanced offor
shadow
region the the
regiondisplay.
image of the
will not
The
image result
will after
not using
be Equation
stretched too (18)
much, with
so default
that the truncation
useless are
details shown
will notin
be
be stretched too much, so that the useless details will not be enhanced for the display. The result afterFigure
enhanced6c. As
for can
the be seen,
display.
Figure 6b shows that the result (18)without
with adefault
truncation is strange in the in
prior failure regions.
can beUsing
usingThe result
Equation after using
(18) with Equation
default truncation are showntruncation are shown
in Figure Figure
6c. As can 6c. As
be seen, Figure seen,
6b shows
Equation
Figure 6b(18) showsto cut offthe
that the result
initialization
withoutmay lead to a more
a truncation comfortable
is strange in the results for theregions.
prior failure human visualUsing
that the result without a truncation is strange in the prior failure regions. Using Equation (18) to cut off
system.
Equation (18) to cut off the initialization may lead to a more comfortable results for the human visual
the initialization
system.
may lead to a more comfortable results for the human visual system.

(a) (b) (c)

Figure 6. Overenhancement
(a) and truncation: (a) the
(b)original image; (b) the result(c)
without truncation
by tD = 255,
Figure tU = 0; (c) the result
6. Overenhancement andwith truncation
truncation: (a)by tDoriginal
the = 150, tUimage;
= 70. (b) the result without truncation
by6.tDOverenhancement
Figure and truncation:
= 255, tU = 0; (c) the result (a) the
with truncation by toriginal
D = 150, timage;
U = 70. (b) the result without truncation by

tD = 255, tU = 0; (c) the result with truncation by tD = 150, tU = 70.

Information 2017, 8, 135 11 of 20

Information 2017, 8, 135 11 of 20

5. Color Casts and Flowchart
5. Color Casts and Flowchart
In this section, we will discuss the white balancing and time complexity. Besides, a flowchart of
the methodIn this section,
is also we will discuss the white balancing and time complexity. Besides, a flowchart of
summarized.
the method is also summarized.
5.1. White Balancing
5.1. White Balancing
After the U and D are refined by the PL algorithm, a clear image can be obtained by Equation (6).
After the U and D are refined by the PL algorithm, a clear image can be obtained by
However, image enhancement will aggravate color casts of the input image, even though it is hard to
Equation (6). However, image enhancement will aggravate color casts of the input image, even
perceive by the human visual system, as shown in Figure 7b.
though it is hard to perceive by the human visual system, as shown in Figure 7b.

(a) (b) (c)

Figure 7. Color casts: (a) the original image needs white balance; (b) the result without white balance;
Figure 7. Color casts: (a) the original image needs white balance; (b) the result without white balance;
(c) the result with white balance.
(c) the result with white balance.
Though the white balance algorithms may discard unwanted color casts, there are quite a few
Though the white balance algorithms may discard unwanted color casts, there are quite a few
images that do not need a color correction. If so, the result will be strange tones. Hence, we make a
images that dotonot
judgment need a whether
determine color correction. If so, requires
the input image the result will be
a white strange
balance tones. Hence,
processing by: we make a
judgment to determine whether the input image requires a white balance processing by:
thredWB < | AR − AG | + | AB − AG | + | AB − AR | , (19)

where A is the atmospheric light, <|

thredWB AR −
which canAbeG |+| A B − Aby
estimated G |+| B − A R |,algorithm, and we apply (19)
theAdehazing
that of Kim et al. [29] A typical value of thredWB is 50, and then, we use the gray world [34] for white
A is the atmospheric
wherebalancing. The gray world light, which
assumes can
that thebebrightest
estimated byofthe
pixel thedehazing
grayscalealgorithm, andon
image is white weitsapply
that of Kimimage.
color et al. Coincidentally,
[29] A typical value of thredWB
the atmospheric is 50,
light and then,
A, which we use by
is estimated thethegray world algorithm
dehazing [34] for white
is alsoThe
balancing. one gray
of theworld
brightest points.that
assumes In fact, Equation (19)
the brightest pixelisof
used
theto determine
grayscale whether
image the ambient
is white on its color
light of the input image is white, so that it is possible to settle whether
image. Coincidentally, the atmospheric light A, which is estimated by the dehazing algorithm the inputs need whiteis also
one ofbalancing.
the brightest points. In fact, Equation (19) is used to determine whether the ambient light of the
input image is white, so that it is possible to settle whether the inputs need white balancing.
5.2. Flowchart of the Proposed Methods
5.2. Flowchart of thecomplexity
The time Proposed Methods
of each step of the proposed algorithm is O(n). Concretely, we make a
preliminary estimate of U and D through the max/minimum filter [35] to obtain the block
The time complexity of each step of the proposed algorithm is O(n). Concretely, we make a
bright/dark channel image; then, we use mean filtering (with a wide variety of O(n) time methods).
preliminary
The alphaestimate
fusion of U and
is used forDPLthrough
iterationthe
andmax/minimum filterimage
generating the label [35] to obtainEquations
through the block(11),
bright/dark
(13)
channel
and (16), which requires just a few basic matrix operations (addition, subtraction, The
image; then, we use mean filtering (with a wide variety of O(n) time methods). pointalpha
fusion is used for PL
multiplication, iteration
point and
division), andgenerating
so does thethe label
fusion image
weight throughthrough
calculated Equations (11), (13)
Equations (12), and
(14) (16),
whichandrequires justwe
(15). Next, a few
applybasic
the matrix operations
PL to refine the image,(addition, subtraction,
which needs pointofmultiplication,
only one time scanning. Finallypoint
we stretch
division), thedoes
and so imagethebyfusion
Equation (6), which
weight needs athrough
calculated few matrix operations.
Equations For (14)
(12), white balancing,
and we we
(15). Next,
use the gray world [34], which is also an O(n) time algorithm. Furthermore, in
apply the PL to refine the image, which needs only one time of scanning. Finally we stretch the imagethe step of image
refinement, due to the spatial smoothness of the estimated components U and D, it is possible to
by Equation (6), which needs a few matrix operations. For white balancing, we use the gray world [34],
introduce the up-/down-sampling technique to reduce the input scale of the algorithm to save
which is also an O(n) time algorithm. Furthermore, in the step of image refinement, due to the spatial
processing time. The overall flow diagram of the proposed algorithm is shown in Figure 8.
smoothness of the estimated components U and D, it is possible to introduce the up-/down-sampling
technique to reduce the input scale of the algorithm to save processing time. The overall flow diagram
of the proposed algorithm is shown in Figure 8.
Information 2017, 8, 135 12 of 20
Information 2017, 8, 135 12 of 20

8. Flow diagram of the proposed

Figure 8. proposed algorithm:
algorithm: for the the gray
gray scale,
scale, bright
brightpp and
and dark
darkpp should be
be
which means
skipped, which means the
the input
input gray
gray image U =
image II == IpU =Ip
IpDD. .

6. Experiment
6. Experiment and
and Discussion
Discussion
The method
The methodwe weproposed
proposedwaswas implemented
implemented withwith
C++ inC++theinMFC
the (Microsoft
MFC (Microsoft
FoundationFoundation
Classes)
Classes) framework. A personal computer with an Intel Core i7 CPU at 2.5 GHz
framework. A personal computer with an Intel Core i7 CPU at 2.5 GHz and 4 GB of memory and 4 GB of memory
was
was used.
used. Experiments
Experiments werewere conducted
conducted usingusing four kinds
four kinds of lowofcontrast
low contrast
images images including
including thekinds
the three three
kinds mentioned
mentioned before before
and anand an additional
additional one forone for comparison.
comparison. We compared
We compared our approach
our approach with thewith the
typical
typical algorithms
algorithms and studies
and the latest the latest studies
on each kindonof each kind ofimage
low contrast low tocontrast image
verify the to verify and
effectiveness the
effectiveness and superiority
superiority of ours. of ours.

6.1. Parameter Configuration

Most ofof the
the parameters
parameters have have an an optimal
optimal value
value for for the
the human
human visual
visual system.
system. Including
Including the
the
large-scale max/min
max/minfilter filterand
andmean
meanfilter
filterdue
duetotothethelearning
learning label should
label should reduce
reducethe the
texture of the
texture of
input
the image.
input The The
image. default setting
default for for
setting thetheparameters
parameters in in our
ourexperiments
experimentsisisshown showninin Table
Table 1.
Empirically, we set the radius
Empirically, radius of
of the
the max/min
max/minfilterfiltertoto2020and andthethemean
mean filter with
filter with a radius of of
a radius 40.40.
A
key
A key parameter
parameterininour
ourapproach
approachisisthred1,
thred1,which
whichisisvery
verysensitive
sensitive forfor the
the outputs.
outputs. Large
Large thred1 makes
the output converge quickly to label F, leading to a smooth U or D, which may produce the halo.
Small thred1
thred1 can reduce
reduce the the halo,
halo, but also result in unwanted saturation saturation in some regions,
regions, asas shown in
Figure 9. Unlike thred1, thred2 has less effect on the result. A small thred2 makes label F similar to Ib.
Empirically, we
Empirically, we set
set itit to
to 20.
20. For
For the
the cutoff
cutoff values,
values, ttDD and ttUU are used to preserve the sky region and and
the extremely dark shadow region from being being stretched
stretched by by the
the enhancement.
enhancement. According to Ju’s [27]
statistics, we set tDD ==150 150and
andtUtU= =70.
70.Finally, wewe
Finally, setsetthredWB
thredWB to to
50 50
based
based ononthetheobservation
observationof 20
of
images
20 images that need
that a white
need a white balance
balanceorornot;
not;however,
however,this thisisisjust
justfor
foreight-bit
eight-bit RGB
RGB images,
images, and images
with different
with different bit
bit depths
depths shouldshould have
have another
anothervalue.
value.

Table 1.
Table 1. Default configuration
configuration of
of the
the parameters.
parameters.

Parameters
Parameters
Values
Values
Equations
Equations
Radius of max/min filter 20 Equation (13)
Radius of max/min filter 20 Equation (13)
Radius
Radius of mean
of mean filter filter 4040 Equation
Equation(13)(13)
thred1
thred1 2020 Equation
Equation(12)(12)
thred2
thred2 2020 Equations (14)(14)
Equations and (15)
and (15)
tD t D 150
150 Equation
Equation (18) (18)
tU 70 Equation (18)
tU 70 Equation (18)
thredWB 50 Equation (19)
thredWB 50 Equation (19)
Information 2017, 8, 135 13 of 20
Information 2017, 8, 135 13 of 20
Information 2017, 8, 135 13 of 20

(a) (b) (c) (d)

Figure(a) (b) (a) the original hazy image;

9. Results with different thred1: (c) (d) = 10;
(b) thred1 = 1; (c) thred1
Figure 9. Results with different thred1: (a) the original hazy image; (b) thred1 = 1; (c) thred1 = 10;
(d) thred1 = 20.
(d) Figure
thred1 =9.20.
Results with different thred1: (a) the original hazy image; (b) thred1 = 1; (c) thred1 = 10;
(d) thred1 = 20.
6.2. Scope of Application
6.2. Scope of Application
First
6.2. Scope of of all, Figure 10 shows the results of our algorithm for color images including a hazy day,
Application
underwater,
First a nighttime
of all, Figure and even
10 shows thearesults
multidegraded image. As can
of our algorithm for be seen,
color our approach
images can a
including stretch
hazy day,
theFirst
underwater,
of all,
details Figure
clearly
a nighttime
10
andand shows
recover the colors
vivid
even
resultsinofunderwater
a multidegraded
our algorithm
image.
for color
or As
heavily
can hazy
images including a hazy day,
regions.
be seen, our approach can stretch
underwater, a nighttime and even a multidegraded image. As can be seen, our approach can stretch
the details clearly and recover vivid colors in underwater or heavily hazy regions.
the details clearly and recover vivid colors in underwater or heavily hazy regions.

(a) (b)
Figure 10. Results of enhancement for color images: (a) the original hazy image; (b) our results.

(a) (b)
Figure 10. Results of enhancement for color images: (a) the original hazy image; (b) our results.
Figure 10. Results of enhancement for color images: (a) the original hazy image; (b) our results.
Information 2017, 8, 135 14 of 20
Information 2017, 8, 135 14 of 20

Besides,
Information
Besides, our
2017,
our8, approach
approachalso
135 alsoworks
worksforforgrayscale
grayscaleimages, such
images, as infrared
such as infraredimages. According
images. to20the
14 of
According to
different bit depths, we give another set of parameters. Figure 11 shows the video
the different bit depths, we give another set of parameters. Figure 11 shows the video screenshots screenshots from a
Besides,
theodolite our
of the naval approach
base. also works
Webase.
stretch for grayscale images, such as infrared images. According to
from a theodolite of the naval Wethe infrared
stretch the images
infraredbyimages
a linearbymodel with
a linear the max
model value
with and
the max
minthe different
value, which bitisdepths,
the wewidely
most give another
used set of parameters.
model to display Figure 11
infrared shows the video screenshots
images.
value and min value, which is the most widely used model to display infrared images.
from a theodolite of the naval base. We stretch the infrared images by a linear model with the max
value and min value, which is the most widely used model to display infrared images.

Figure 11. Results of enhancement for infrared images (14 bits): thred1 = 800, thred2 = 1600, tD = 4000,
Figure
Figure 11.11. Results
Results ofof enhancementfor
enhancement forinfrared
infraredimages
images (14
(14 bits): thred1
thred1 ==800,
800,thred2
thred2= =1600,
1600,tDtD
= 4000,
= 4000,
tU = 14,000.
tU tU = 14,000.
= 14,000.
6.3. Haze Removal
6.3.6.3. Haze Removal
Haze Removal
Next,
Next,wewecompare
compareour ourapproach
approach with
with Zhang’s [28], He’s
Zhang’s [28], He’s [23],
[23], Kim’s
Kim’s[29]
[29]and
andTan’s
Tan’s[21]
[21]
Next,
approaches we compare our approach with Zhang’s [28], He’s [23], Kim’s [29] and Tan’s [21] approaches
approachesforforhazehazeremoval.
removal. InIn Figure
Figure 12,
12, the depth edges
the depth edges between
between thetheforeground
foregroundand and
for haze removal.
background may In Figure haloes
produce 12, the by
depth
manyedges between
dehazing the foreground
algorithms. As and
can bebackground
seen, the may produce
results of other
background may produce haloes by many dehazing algorithms. As can be seen, the results of other
haloes
methods by usually
many dehazing algorithms. As can be seen, thethe results
halo,ofexcept
other methods usually look dim,
methods usuallylook
lookdim,
dim,and
and most
most ofof them
them restore
restore the halo, except Zhang’smethod.
Zhang’s method. OnOn thethe
and most
contrary,
contrary,
of them
our restore
ourapproach
the halo,
approachcompensates
except
compensates for
Zhang’s
for the
method.
the illumination
On
illumination and
the contrary,
and has
hasnot our approach
notintroduced
introducedany compensates
anysignificant
significant
for the illumination and has not introduced any significant haloes.
haloes.
haloes.

(a)
(a)
Figure 12. Cont.
Information 2017, 8, 135 15 of 20
Information 2017, 8, 135 15 of 20
Information 2017, 8, 135 15 of 20

(b) (c) (d) (e) (f)

(b) (c) (d) (e) (f)
Figure 12. Comparison of haze removal: (a) input image; and results from using the methods by (b)
Figure 12.12.Comparison
Figure Comparison of of haze removal:(a)(a)input
haze removal: input image;
image; andand results
results fromfrom using
using the methods
the methods by (b)by
Zhang’s [28]; (c) He’s [23]; (d) Kim’s [29]; (e) Tan’s [21]; (f) ours; our parameters were set as: for D
(b)Zhang’s
Zhang’s[28];
[28];(c)(c)
He’s [23];
He’s (d) (d)
[23]; Kim’s [29]; [29];
Kim’s (e) Tan’s [21]; (f)
(e) Tan’s ours;
[21]; (f) our parameters
ours; were setwere
our parameters as: for
setDas:
thred1 = 1, for U thred1 = 20.
forthred1 = 1, =
D thred1 for1,Ufor
thred1 = 20. = 20.
U thred1

6.4. Lightning Compensation

6.4.6.4. Lightning
Lightning Compensation
Compensation
We also compare our method with MSR [5], Wang’s [10] and Lin’s [11] approaches for
WeWe
lightningalsoalso compare
compare
compensation ourour method
method
in FigureswithwithMSR
13
MSRWang’s
and[5],
[5], Wang’sand
14. Figure[10]
[10]Lin’s
13 shows
and [11] Lin’s
an image
[11] approaches
approaches for
for lightning
with nonuniform
lightning
compensation compensation in Figures 13 and 14. Figure 13 shows an image with nonuniform
illumination,in Figures
which has13 and 14. region.
a shadow Figure We13 shows
can seean image
that bothwith MSRnonuniform illumination,
and Wang’s approach which
increase
illumination, which has a shadow region. We can see that both MSR and Wang’s approach increase
hasthe
a shadow
exposureregion.
of the Weboycan see the
inside thattire;
both MSR and
however, inWang’s approach
some regions increase
there, the exposure of
is an overexposure, theasboy
such
the exposure of the boy inside the tire; however, in some regions there, is an overexposure, such as
inside the tire;
the letters onhowever,
the tire. in some regions there, is an overexposure, such as the letters on the tire.
the letters on the tire.
In Incontrast,
contrast, no no matter
matter in shadow
in the the shadow regionsregions (boy’s
(boy’s face) or theface) or the
sunshine sunshine
regions regions
(tires, boy’s
In contrast, no matter in the shadow regions (boy’s face) or the sunshine regions (tires, boy’s
(tires, boy’s our
clothing), clothing),
approach ourgives
approach gives clear
clear details. Figuredetails. Figure 14 image,
14 is a nighttime is a nighttime image,some
which includes which
clothing), our approach gives clear details. Figure 14 is a nighttime image, which includes some
non-detail
includes some information
non-detail in the dark region.
information in the The
dark result of MSR
region. The drawsof
result outMSR
non-detail information in the dark region. The result of MSR draws out the compressed information,
the draws
compressed
out information,
the compressed
which is regarded
information, which isasasregarded
noise. On the
asthe other
noise. Onhand, Lin’s result Lin’s
the other is much better, which increases exposure
which is regarded noise. On other hand, Lin’s hand,
result is much result is much
better, whichbetter, which
increases increases
exposure
and
exposure suppresses the
and suppresses noise.
theInIn this
noise. experiment, our
In this experiment, approach is
our approach similar to
is similarLin’s. Both
toBoth Lin’s
Lin’s.Lin’s [11]
Both[11] and
Lin’s [11]
and suppresses the noise. this experiment, our approach is similar to Lin’s. and
andourour approach
approach compensate
compensate forfor
thetheillumination
illumination and andsuppress
suppress thetheuseless
useless information
information in in
thethedark
dark
our approach compensate for the illumination and suppress the useless information in the dark
region.
region. Ours
Ours restores moredetails
restores details than Lin’s
Lin’s [11], such as the leaves atatthe top left corner and more
region. Ours restoresmore
more detailsthan than Lin’s [11], such as
[11], such as the
theleaves
leavesat thethetop
topleft
leftcorner
corner andand more
more
textures
textures of
of of the
thethe road.
road. However,
However, Lin’s
Lin’s [11]
[11] result
result reduces
reduces thethe glow
glow of
ofof light
light at the side of the road, and
textures road. However, Lin’s [11] result reduces the glow lightatatthetheside
sideofofthe
theroad,
road,andandhas
has a more comfortable
a more color than ours.
has acomfortable
more comfortable color color
than ours.
than ours.

(a)
(a)

(b) (c) (d)

(b) (c) (d)
Figure 13. Comparison of the nonuniform illumination image: (a) input image; and results from
Figure 13. Comparison of the nonuniform illumination image: (a) input image; and results from
Figure theComparison
using13. methods by of
(b)the nonuniform
multiscale illumination
retina-cortex image:
(Retinex) (a) input
(MSR) [5]; (c)image;
Wang’sand results
[10]; from using
(d) ours.
using the methods by (b) multiscale retina-cortex (Retinex) (MSR) [5]; (c) Wang’s [10]; (d) ours.
the methods by (b) multiscale retina-cortex (Retinex) (MSR) [5]; (c) Wang’s [10]; (d) ours.
Information 2017, 8, 135 16 of 20
Information 2017, 8, 135 16 of 20

Information 2017, 8, 135 16 of 20

(a)
(a)

(b) (c) (d)

(b)
Figure 14. Comparison the approaches for the(c) (d)
Figure 14. Comparison ofofthe nighttime image: (a) input image;
approaches for the nighttime image: (a) input image;and the results
and the results
from using
Figure the
14.the methods
Comparison by (b)
of (b) MSR
theMSR [5];
approaches(c) Lin’s
for the[11]; (d) ours.
nighttime image: (a) input image; and the results
from using methods by [5]; (c) Lin’s [11]; (d) ours.
from using the methods by (b) MSR [5]; (c) Lin’s [11]; (d) ours.
6.5. Underwater Enhancement
6.5. Underwater Enhancement
6.5. Underwater
Most of theEnhancement
underwater enhancements need a white balancing to discard unwanted color casts,
Most
due to of the underwater
various illuminations. enhancements
In Figure 15, we need a white balancing
compare to discard etunwanted color [5],
casts,
Most of the underwater enhancements need a whiteour methods
balancing to with
discard Heunwanted
al.’s [23], MSR
color casts,
due to
Ancuti various illuminations.
et al.’s [32] and Zhang In In Figure
et al.’s 15,
[8] 15, we
approachescompare our
for enhancingmethods with
thewith He
underwater et al.’s
images. [23], MSR be[5],
It can[5],
due to various illuminations. Figure we compare our methods He et al.’s [23], MSR
Ancuti
seen et al.’sHe
[32]etand Zhang et al.’s [8] and
approaches
MSR [5]for enhancing the underwater images. It can be
Ancutithatet al.’s [32] al.’s approach
and Zhang [23]
et al.’s [8] approaches forhave significant
enhancing contrast stretching
the underwater images. It of canthebe
seen that
underwaterHe et al.’s
image, approach [23]
but approach and
retain the [23] MSR
colorand [5]
casts.MSRhave
Ancutisignificant contrast stretching of the underwater
seen that He et al.’s [5] ethave
al.’s significant
approach [32] applied
contrast the gray-world
stretching of the
image,
methodsbut retain
for the
white color
balancingcasts.
and Ancuti
restored et aal.’s approach
comfortable [32]
color; applied
however,
underwater image, but retain the color casts. Ancuti et al.’s approach [32] applied the
the gray-world
their results methods for
oversaturated
gray-world
white
the balancing
region
methods forwithandbrilliant
white restored
balancing a and
comfortable
illumination.
restored color;
Zhang ethowever,
al. proposed
a comfortable theirhowever,
color; results
an oversaturated
improvedtheir MSR the regionthe
resultstooversaturated
enhance with
underwater
brilliant image
illumination. and obtained
Zhang et al. a result
proposed withan suitable
improved color,
MSR buttothe contrast
enhance
the region with brilliant illumination. Zhang et al. proposed an improved MSR to enhance the stretching
the underwater is very
imagelittle.
and
Our
obtained approach
a result
underwater image restored
withand a
suitable vivid
obtainedcolor, color
but the
a result owing
with to
contrast
suitable the gray-world
stretching
color, butisthe
very [34]; moreover,
little. stretching
contrast Our approach there
is very is
restoredno
little. a
oversaturation
vivid in our result, and the details of the reef are clearer than
Our approach restored a vivid color owing to the gray-world [34]; moreover, there is nothe
color owing to the gray-world [34]; moreover, there is no other
oversaturation methods.
in our result, and
details of the reefin
oversaturation are
ourclearer
result,than
and other methods.
the details of the reef are clearer than other methods.

(a)
(a)

(b) (c) (d) (e) (f)

Figure(b)
15. Comparison of the (c)approaches for the underwater
(d) image: (a) (e)
input image; and the(f) results
form using
Figure the methods by (b) He’s [23]; (c) MSR [5]; (d) Ancuti’s [32]; (e) Zhang’s [8]; (f) ours.
Figure 15.15. Comparisonofofthe
Comparison theapproaches
approachesforfor the
the underwater
underwater image:
image:(a)(a)input
inputimage;
image;andand the results
the results
form using the methods by (b) He’s [23]; (c) MSR [5]; (d) Ancuti’s [32]; (e) Zhang’s [8]; (f) ours.
form using the methods by (b) He’s [23]; (c) MSR [5]; (d) Ancuti’s [32]; (e) Zhang’s [8]; (f) ours.
Information 2017, 8, 135 17 of 20
Information 2017, 8, 135 17 of 20

6.6. Multidegraded Enhancement

6.6. Multidegraded Enhancement
Figure
Figure 1616 presents
presents the the results
results of
of aa multiple
multiple degraded
degraded image,
image, which
which includes
includes color
color casts,
casts, low
low
illumination and a hazy environment. We compare our results with
illumination and a hazy environment. We compare our results with He et al. [23], Li et al. He et al. [23], Li et al. [30],
[30],
Lin
Lin et
et al.
al. [11]
[11] and
and Ancuti
Ancuti et et al.
al. [32]. As can
[32]. As can be
be seen,
seen, all
all of
of them
them can
can handle
handle aa part
part ofof the
the degradation.
degradation.
He et al.’s method removed most of the haze, but kept the red color
He et al.’s method removed most of the haze, but kept the red color of the inputs, as well of the inputs, as well
as theas
the restoration looked too dim. Though Li et al.’s algorithm achieved a
restoration looked too dim. Though Li et al.’s algorithm achieved a better result on color recovery,better result on color
recovery,
haze removal hazeandremoval and illumination
illumination compensation,compensation, the retained
the result still result still
theretained
originaltheredoriginal
hue; Linred et
hue; Lin et al.’s nighttime enhancement compensated for the low illumination
al.’s nighttime enhancement compensated for the low illumination such as the plants in the such as themiddle
plants
in the road;
of the middle of the
Ancuti et road; Ancuti
al.’s result et al.’sallresult
handled of thehandled all of the
degradations, degradations,
in spite in spite of the
of the overcompensation
overcompensation
for illumination, which for illumination,
resulted in which
a lack resulted
of colorsin(the
a lack of colors
bushes (the bushes
and leaves and to
are hard leaves are hard
recognize as
to recognize
green). as green).

(a)

(b) (c) (d) (e) (f)

Figure 16.
Figure 16. Comparison
Comparison of
of the approaches for
the approaches for the
the underwater
underwater image:
image: (a) input image;
(a) input image; and
and the
the results
results
from using the methods by (b) He’s [23]; (c) Li’s [30]; (d) Lin’s [11]; (e) Ancuti’s [32]; (f) ours.
from using the methods by (b) He’s [23]; (c) Li’s [30]; (d) Lin’s [11]; (e) Ancuti’s [32]; (f) ours.

6.7. Quantitative
6.7. Quantitative Comparison
Comparison
Based on the
Based on the above above results
results onmultidegraded
on the the multidegraded image
image in Figurein16,Figure 16, we aconducted
we conducted quantitativea
quantitative using
comparison comparison
the MSE using
(meanthesquared
MSE (mean
error)squared
and the error) and the SSIM
SSIM (structural (structural
similarity similarity
index). Table 2
shows the quantitative comparisons, in which the MSE represents the texture details of an details
index). Table 2 shows the quantitative comparisons, in which the MSE represents the texture image,
of an image, and the SSIM is used for measuring the similarity
and the SSIM is used for measuring the similarity between two images. between two images.

Table 2.
Table 2. Quantitative
Quantitativecomparison of of
comparison Figure 16 based
Figure on the
16 based on MSE and the
the MSE andStructural Similarity
the Structural index
Similarity
(SSIM).
index (SSIM).
He’s [23] Li’s [30] Lin’s [11] Ancuti’s [32] Avg Changes Ours
He’s [23] Li’s [30] Lin’s [11] Ancuti’s [32] Avg Changes Ours
MSE 0.066 0.019 0.023 0.071 0.0283 0.073
MSE
SSIM 0.066
0.48 0.019
0.73 0.023
0.88 0.071
0.56 0.0283
−0.4325 0.073
0.23
SSIM 0.48 0.73 0.88 0.56 −0.4325 0.23

From Table 2, we can observe that the MSE is inversely proportional to SSIM, meaning that the
From
greater theTable 2, webetween
difference can observe that the
the results MSE
and theisinput
inversely proportional
image, to SSIM,
the more details are meaning
restored. that
The
the greater the difference between the results and the input image, the more
Avg changes column shows the average changes in MSE and SSIM between the proposed method details are restored.
and Avg
The thosechanges
from column
the other shows the He’s
studies. average
[23]changes in MSE[32]
and Ancuti’s andapproaches
SSIM betweenhadthe proposed
a higher MSE method
and a
and
lower SSIM than Li’s [30] and Lin’s [11] approaches; in other words, He’s [23] and Ancuti’slower
those from the other studies. He’s [23] and Ancuti’s [32] approaches had a higher MSE and a [32]
SSIM than Li’s
approaches [30]less
were andsimilar
Lin’s [11] approaches;
to the in other
input image, words,
as well He’s [23]
as restored and details
more Ancuti’s
of [32]
the approaches
image. Our
were less similar
approach had thetohighest
the input
MSEimage,
and theas lowest
well asSSIM,
restored more details
meaning that theofproposed
the image. Our approach
method obtained
more texture information than the other methods. From the average changes, we can see that our
Information 2017, 8, 135 18 of 20

had the highest MSE and the lowest SSIM, meaning that the proposed method obtained more texture
information than the other methods. From the average changes, we can see that our method improves
the MSE and reduces the SSIM more than other methods. That means our approach is better suited for
this kind of low contrast images.

7. Conclusions
Image quality can be affected by the shooting of certain scenes, such as hazy, night or underwater,
which will degrade the contrast of images. In this paper, we have proposed a generic model for
enhancing the low contrast images based on the observation of the 3D surface graph, called the effective
space. The effective space is estimated by the dark and bright channel priors, which are patch-based
methods. In order to reduce the artifacts produced by the patches, we have also designed the pixel
learning for edge-preserving smoothing, which was inspired by online learning. Combining the
model with the pixel learning, low contrast image enhancement becomes simpler and has fewer
artifacts. Compared with a number of competing methods, our method shows more favorable results.
The quantitative assessment demonstrates that our approach can provide an obvious improvement to
both traditional and up-to-date algorithms. Our method has been applied to a theodolite of the naval
base and can enhance a 720 × 1080 video stream by 20 ms/f (50 fps).
Most of the parameters of our method were set empirically, which depends on some features
of an image such as the size or depth. To make a better choice of the parameters, more complex
features [36–40] should be studied, which will be figured out systematically in future work.
Besides, since the look-up table model replaced the values one by one, the spatial continuity of
the image will be destroyed and some noise will be introduced. This is a challenging problem, and an
advanced model is needed to keep the memory of spatial continuity. We leave this for future studies.

Acknowledgments: This work is supported by the National Natural Science Foundations of China (Grant No. 61401425
and Grant No. 61602432).
Author Contributions: Gengfei Li designed the algorithm, edited the source code, analyzed the experiment
results and wrote the manuscript; Guiju Li and Guangliang Han made contribution to experiments design and
revised the paper.
Conflicts of Interest: The authors declare no conflicts of interest.

References
1. Land, E.H. Recent advances in Retinex theory and some implications for cortical computations: Color vision
and the natural image. Proc. Natl. Acad. Sci. USA 1983, 80, 5163–5169. [CrossRef] [PubMed]
2. Land, E.H. An alternative technique for the computation of the designator in the Retinex theory of color
vision. Proc. Natl. Acad. Sci. USA 1986, 83, 3078–3080. [CrossRef] [PubMed]
3. Land, E.H. Recent advances in Retinex theory. Vis. Res. 1986, 26, 7–21. [CrossRef]
4. Jobson, D.J.; Rahman, Z.; Woodell, G.A. Properties and performance of a center/surround Retinex.
IEEE Trans. Image Process. 1997, 6, 451–462. [CrossRef] [PubMed]
5. Jobson, D.J.; Rahman, Z.; Woodell, G.A. A multiscale Retinex for bridging the gap between color images and
the human observation of scenes. IEEE Trans. Image Process. 1997, 6, 965–976. [CrossRef] [PubMed]
6. Kimmel, R.; Elad, M.; Shaked, D.; Keshet, R.; Sobel, I. A Variational Framework for Retinex. Int. J. Comput. Vis.
2003, 52, 7–23. [CrossRef]
7. Ma, Z.; Wen, J. Single-scale Retinex sea fog removal algorithm fused the edge information. Jisuanji Fuzhu
Sheji Yu Tuxingxue Xuebao J. Comput. Aided Des. Comput. Graph. 2015, 27, 217–225.
8. Zhang, S.; Wang, T.; Dong, J.; Yu, H. Underwater Image Enhancement via Extended Multi-Scale Retinex.
Neurocomputing 2017, 245, 1–9. [CrossRef]
9. Si, L.; Wang, Z.; Xu, R.; Tan, C.; Liu, X.; Xu, J. Image Enhancement for Surveillance Video of Coal Mining Face
Based on Single-Scale Retinex Algorithm Combined with Bilateral Filtering. Symmetry 2017, 9, 93. [CrossRef]
10. Wang, Y.; Wang, H.; Yin, C.; Dai, M. Biologically inspired image enhancement based on Retinex.
Neurocomputing 2016, 177, 373–384. [CrossRef]
Information 2017, 8, 135 19 of 20

11. Lin, H.; Shi, Z. Multi-scale retinex improvement for nighttime image enhancement. Opt. Int. J. Light
Electron Opt. 2014, 125, 7143–7148. [CrossRef]
12. Xie, S.J.; Lu, Y.; Yoon, S.; Yang, J.; Park, D.S. Intensity variation normalization for finger vein recognition
using guided filter based singe scale Retinex. Sensors 2015, 15, 17089–17105. [CrossRef] [PubMed]
13. Lan, X.; Zuo, Z.; Shen, H.; Zhang, L.; Hu, J. Framelet-based sparse regularization for uneven intensity
correction of remote sensing images in a Retinex variational framework. Opt. Int. J. Light Electron Opt. 2016,
127, 1184–1189. [CrossRef]
14. Wang, G.; Dong, Q.; Pan, Z.; Zhang, W.; Duan, J.; Bai, L.; Zhanng, J. Retinex theory based active contour
model for segmentation of inhomogeneous images. Digit. Signal Process. 2016, 50, 43–50. [CrossRef]
15. Jiang, B.; Woodell, G.A.; Jobson, D.J. Novel Multi-Scale Retinex with Color Restoration on Graphics Processing
Unit; Springer: New York, NY, USA, 2015.
16. Shi, Z.; Zhu, M.; Guo, B.; Zhao, M. A photographic negative imaging inspired method for low illumination
night-time image enhancement. Multimedia Tools Appl. 2017, 76, 1–22. [CrossRef]
17. Wang, Y.; Zhuo, S.; Tao, D.; Bu, J.; Li, N. Automatic local exposure correction using bright channel prior for
under-exposed images. Signal Process. 2013, 93, 3227–3238. [CrossRef]
18. Shwartz, S.; Namer, E.; Schechner, Y.Y. Blind Haze Separation. In Proceedings of the IEEE Computer
Society Conference on Computer Vision and Pattern Recognition, New York, NY, USA, 17–22 June 2006;
pp. 1984–1991.
19. Shen, X.; Li, Q.; Tian, Y.; Shen, L. An Uneven Illumination Correction Algorithm for Optical Remote Sensing
Images Covered with Thin Clouds. Remote Sens. 2015, 7, 11848–11862. [CrossRef]
20. Kopf, J.; Neubert, B.; Chen, B.; Cohen, M.; Cohen, D.; Deussen, O.; Uyttendaele, M.; Lischinski, D. Deep photo:
Model-based photograph enhancement and viewing. ACM Trans. Graph. 2008, 27, 1–10. [CrossRef]
21. Fattal, R. Single image dehazing. ACM Trans. Graph. 2008, 27, 1–9. [CrossRef]
22. Tan, R.T. Visibility in bad weather from a single image. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, CVPR 2008, Anchorage, AK, USA, 23–28 June 2008; pp. 1–8.
23. He, K.; Sun, J.; Tang, X. Single Image Haze Removal Using Dark Channel Prior. IEEE Trans. Pattern Anal.
Mach. Intell. 2011, 33, 2341–2353. [PubMed]
24. He, K.; Sun, J.; Tang, X. Guided Image Filtering. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 1397–1409.
[CrossRef] [PubMed]
25. Meng, G.; Wang, Y.; Duan, J.; Xiang, S.; Pan, C. Efficient Image Dehazing with Boundary Constraint and
Contextual Regularization. In Proceedings of the IEEE International Conference on Computer Vision, Sydney,
Australia, 1–8 December 2013; pp. 617–624.
26. Jo, S.Y.; Ha, J.; Jeong, H. Single Image Haze Removal Using Single Pixel Approach Based on Dark Channel
Prior with Fast Filtering. In Computer Vision and Graphics; Springer: Cham, Switzerland, 2016; pp. 151–162.
27. Ju, M.; Zhang, D.; Wang, X. Single image dehazing via an improved atmospheric scattering model.
Vis. Comput. 2017, 33, 1613–1625. [CrossRef]
28. Zhang, S.; Yao, J. Single Image Dehazing Using Fixed Points and Nearest-Neighbor Regularization.
In Proceedings of the Asian Conference on Computer Vision, Taipei, Taiwan, 20–24 November 2016; Springer:
Cham, Switzerland, 2016; pp. 18–33.
29. Kim, J.-H.; Jang, W.-D.; Sim, J.-Y.; Kim, C.-S. Optimized contrast enhancement for real-time image and video
dehazing. J. Vis. Commun. Image Represent. 2013, 24, 410–425. [CrossRef]
30. Li, Y.; Tan, R.T.; Brown, M.S. Nighttime Haze Removal with Glow and Multiple Light Colors. In Proceedings
of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; IEEE
Computer Society: Washington, DC, USA, 2015; pp. 226–234.
31. Ji, T.; Wang, G. An approach to underwater image enhancement based on image structural decomposition.
J. Ocean Univ. China 2015, 14, 255–260. [CrossRef]
32. Ancuti, C.; Ancuti, C.O.; Haber, T.; Bekaert, P. Enhancing underwater images and videos by fusion.
In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence,
RI, USA, 16–21 June 2012; pp. 81–88.
33. Ma, C.; Ao, J. Red Preserving Algorithm for Underwater Imaging. In Geo-Spatial Knowledge and Intelligence;
Springer: Singapore, 2017; pp. 110–116.
34. Ebner, M. Color constancy based on local space average color. Mach. Vis. Appl. 2009, 20, 283–301. [CrossRef]
Information 2017, 8, 135 20 of 20

35. Van Herk, M. A fast algorithm for local minimum and maximum filters on rectangular and octagonal kernels.
Pattern Recognit. Lett. 1992, 13, 517–521. [CrossRef]
36. Le, P.Q.; Iliyasu, A.M.; Sanchez, J.A.G.; Hirota, K. Representing Visual Complexity of Images Using a 3D
Feature Space Based on Structure, Noise, and Diversity. Lect. Notes Bus. Inf. Process. 2012, 219, 138–151.
[CrossRef]
37. Gu, K.; Zhai, G.; Lin, W.; Liu, M. The Analysis of Image Contrast: From Quality Assessment to Automatic
Enhancement. IEEE Trans. Cybernet. 2017, 46, 284–297. [CrossRef] [PubMed]
38. Hou, W.; Gao, X.; Tao, D.; Li, X. Blind image quality assessment via deep learning. IEEE Trans. Neural Netw.
Learn. Syst. 2017, 26, 1275–1286.
39. Gu, K.; Li, L.; Lu, H.; Lin, W. A Fast Computational Metric for Perceptual Image Quality Assessment.
IEEE Trans. Ind. Electron. 2017. [CrossRef]
40. Wu, Q.; Li, H.; Meng, F.; Ngan, K.N.; Luo, B.; Huang, C.; Zeng, B. Blind Image Quality Assessment Based on
Multichannel Feature Fusion and Label Transfer. IEEE Trans. Circuits Syst. Video Technol. 2016, 26, 425–440.
[CrossRef]

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).