Improved Bilateral Filtering for a Gaussian Pyramid Structure-Based Image Enhancement Algorithm

Abstract

To address the problem of unclear images affected by occlusion from fog, we propose an improved Retinex image enhancement algorithm based on the Gaussian pyramid transformation. Our algorithm features bilateral filtering as a replacement for the Gaussian function used in the original Retinex algorithm. Operation of the technique is as follows. To begin, we deduced the mathematical model for an improved bilateral filtering function based on the spatial domain kernel function and the pixel difference parameter. The input RGB image was subsequently converted into the Hue Saturation Intensity (HSI) color space, where the reflection component of the intensity channel was extracted to obtain an image whose edges were retained and are not affected by changes in brightness. Following reconversion to the RGB color space, color images of this reflection component were obtained at different resolutions using Gaussian pyramid down-sampling. Each of these images was then processed using the improved Retinex algorithm to improve the contrast of the final image, which was reconstructed using the Laplace algorithm. Results from experiments show that the proposed algorithm can enhance image contrast effectively, and the color of the processed image is in line with what would be perceived by a human observer.

1. Introduction

With the development of intelligent monitoring technology, a large number of cameras have been deployed in intelligent cars, traffic monitoring, and military investigation. However, weather conditions have a great influence on the quality of video images. In particular, some foggy weather may need image enhancement rather than proceeding to the next step such as image monitoring [1,2,3]. The scattering of atmospheric particles in the fog introduces defects such as uneven exposure, low contrast, and serious color distortion in images captured by a Charge-coupled Device(CCD) camera. Since it is not possible to engineer these devices to eliminate the influence of atmospheric scattering due to the physical limitations of CCD sensors, the development of algorithms that reduce the effect of fluctuations in incident light—to improve the quality of images—is of great significance.

Retinex theory was originally developed by Edwin Land as a model for explaining human color perception [4,5,6]. Jobson et al. [7,8] subsequently extended this theory into a general-purpose image enhancement algorithm to address problems caused by changing the lighting and atmospheric conditions inherent to the process of acquiring images of Earth from space. The single-scale Retinex (SSR) algorithm provides the dynamic range compression, color constancy, and sharpening that are required to alleviate these problems. In essence, based on convolution with a Gaussian function with a specific scale factor, it operates by separating incident light components from reflected light components to improve the contrast in an image. As such, it is suitable for processing multispectral satellite imagery, and applicable in diverse areas such as aerospace exploration [9], aviation safety [10,11], medical radiography [12], underwater photography [13,14], forensic investigations [15], and general-purpose photography [16,17].

Several improvements have been made to the SSR algorithm since its initial development, in order to achieve a balance between dynamic range compression and tonal rendition. As the SSR’s image enhancement effect depends on the scale factor of the Gaussian function, the multi-scale Retinex (MSR) algorithm [8] was developed to combine dynamic range compression, color consistency, and tonal rendition, producing images that compare favorably with human visual perception. Here, the input image is processed using three different scale factors (small, intermediate, and large), to ensure that the color fidelity and resolution of the image are retained. In the multi-scale Retinex with color restoration (MSRCR) algorithm, a supplementary color restoration process is included with the MSR algorithm to compensate for the color desaturation inherent in its operation. The resulting algorithm yields images with good color rendition even for severe gray-world violations. In Refs. [18] and [19], a different image enhancement framework based on the Retinex algorithm was proposed, in which the Gaussian filter (GSF) used in the traditional Retinex algorithm is replaced with a region covariance filter (RCF). In contrast to GSFs, for each pixel, the output of an RCF depends on the covariance between local image features. Therefore, an RCF is self-adapting to different pixels in an image, and it can thus estimate the nature of incident illumination more accurately than a GSF. Because of its ability to enhance contrast, eliminate noise, and enhance details in an image, the RCF–Retinex algorithm can be considered to have the best overall performance of the Retinex-based enhancement methods. Zhang et al. [20] proposed a mixed multi-Retinex method combined with fractional differentiation that can adjust its ambient illumination processing capacity. Hence, the dynamic range of an image captured in low light conditions can be modified to reduce noise caused by poor lighting and enhance the quality of nighttime images. A new technique, similar to conventional enhancement methods in which the illumination component of the processed image is manipulated, was introduced in [21]. The technique makes use of an augmented Lagrange multiplier based algorithm for optimization of simultaneous estimation of a smoothed illumination component and the reflectance component of the image.

While Retinex-based algorithms are able to enhance most images adequately, the output image can sometimes be optimized through visual analysis, depending on the user’s experience. As previously stated, with traditional Retinex methods, the image enhancement effect depends on the scale factor of the Gaussian function, leading to some limitations. Hence, we propose an improved Retinex image enhancement algorithm based on Gaussian pyramid transformation with bilateral filtering to eliminate the impact of outliers in the histogram of the image and further improve its contrast, thus improving the overall visual performance of the Retinex algorithm.

The rest of the paper is structured as follows. The improved Retinex model is briefly described in Section 2. Details of the improved bilateral filtering function are discussed in Section 3. An overview of the Gaussian–Laplacian multi-scale pyramid algorithm is given in Section 4. Analysis of the results of experiments is given in Section 5, and conclusions are presented in Section 6.

2. Improved Retinex Model

2.1. Single-Scale Retinex Model

The SSR model is a color constant visual image enhancement method based on Land’s theory of human visual perception. In this model, the color of an object is considered to be fixed, while the corresponding image obtained by the camera is composed of a low-frequency incident light component and a high-frequency component resulting from the reflections of light on the surface of the object, as shown in Figure 1.

For a given image, the relationship between incident and reflected light components can be expressed as

$$S(x,y)=L(x,y)\cdot R(x,y),$$

(1)

where $S(x,y)$, $R(x,y)$, and $L(x,y)$ denote the full color component, the reflected light component, and the incident light component of a given image, respectively.

The principle of traditional Retinex image enhancement is to remove or reduce the impact of incident light components on an image by deconstructing this into separate incident and reflection components. The contrast of the reflection image is consequently enhanced, the texture information of the image is compressed within a certain range, and at the same time, the basic tone of the original image is retained. For ease of calculation, logarithms of both sides of (1) are taken, yielding the following:

$$\ln (S(x,y))=\ln (L(x,y))+\ln (R(x,y)).$$

(2)

The mathematical form of the reflection component of the image can be obtained by rearranging (2) as follows:

$$\ln (R_i(x,y))=\ln (S_i(x,y))-\ln (L_i(x,y))$$

(3)

where $S_i(x,y)$, $R_i(x,y)$, and $L_i(x,y)$ denote the color component, the reflection component, and the incident light component of the $i$-th color channel, respectively.

Inspection of (3) shows that the reflection component of the image can be obtained by deducing the incident light component. Hence, the accuracy of incident light estimation directly affects the fog removal effect. Since $L$ is low-frequency information, a Gaussian convolution function can be used to better estimate the incident light component from the image, i.e.,

$$L(x,y)=G(x,y)\ast S(x,y),$$

(4)

where $\ast$ represents the convolution operation, G(x,y) is a Gaussian function, defined as $G(x,y)=\frac{1}{2\pi {\sigma }^2}\exp (-\frac{x^2+y^2}{2{\sigma }^2})$, and $\sigma$ is the standard deviation of the Gaussian function, also known as the scale factor.

2.2. HSI Color Space

The HSI color space is an alternative to the RGB color space that better reflects human perception of color information. With this model, color is encoded using three different attributes: Hue, saturation, and intensity. Here, hue and saturation are related to human perception of color, while, in contrast, intensity reflects the apparent brightness of an image, determined by the lighting conditions. Intensity is calculated as follows:

$$I(x,y)=\raisebox{1ex}{$R(x,y)+G(x,y)+B(x,y)$}\!\left/ \!\raisebox{-1ex}{$3$}\right.$$

(5)

where $R$, $G$, and $B$ are the red, green, and blue components of an RGB-encoded image, respectively.

2.3. The Improved Retinex Algorithm

The dependence of the SSR’s enhancement effect on the Gaussian scale factor, $\sigma$, can be illustrated as follows. When the value of $\sigma$ is large, the color fidelity of the processed image is good. However, details in the image become difficult to identify. Conversely, when this value is reduced such that the details are highlighted after enhancement, the color fidelity of the processed image is poor. In particular, in areas where the local color changes significantly, there is a “halo” effect after SSR enhancement (Figure 2b).

To address the defects of a traditional Retinex algorithm, we proposed the use of Gaussian pyramid transformation and an improved bilateral filtering function that increases edge retention and improves noise removal to replace the Gaussian filter function. The mathematical model for this improved bilateral filter function is deduced by improving the spatial kernel function and using a rotating window function to determine the pixel difference scale parameters.

The operation of our technique proceeds as follows. The color space of the image is transformed from RGB to HSI in order to separate brightness from color information. Improved bilateral filtering of Retinex enhancement is subsequently performed on the intensity component of the HSI image in order to obtain a reflection image whose edges are maintained and are unaffected by changes in brightness. This enhanced image is then returned to the RGB color space, following which a Gaussian–Laplacian pyramid is generated by successively applying a Gaussian filter to an input image and scaling down the resolution of the resulting image. Finally, sub-images in the pyramid are enhanced using the improved Retinex algorithm for multi-scale processing before Laplacian reconstruction is applied to complete image enhancement. A flowchart detailing the operation of this improved Retinex algorithm is shown in Figure 3.

3. Improved Bilateral Filtering Function

3.1. Traditional Bilateral Filtering Functions

In a traditional Retinex algorithm, the central-surround filter function is typically a Gaussian function. Hence, once operation of the algorithm is complete, the edges of the image are blurred, and details in the image disappear. To address this, in our algorithm we have adopted a bilateral filter [22] as an “edge maintenance–noise reduction” filter, which is defined mathematically as

$$f^{\prime }(i,j)=\frac{{\displaystyle \sum _{(m,n)\in {\mathsf{\Omega }}_{p,i,j}}{w}_d(m,n)w_r(m,n)f(m,n)}}{{\displaystyle \sum _{(m,n)\in {\mathsf{\Omega }}_{p,i,j}}{w}_d(m,n)w_r(m,n)}},$$

(6)

where

$$w_d(m,n)=\exp (-\frac{{(i-m)}^2+{(j-n)}^2}{2{\sigma }_d^2})$$

(7)

$$w_r(m,n)=\exp (-\frac{{(f(i,j)-f(m,n))}^2}{2{\sigma }_r^2}).$$

(8)

$f(m,n)$ and $f^{\prime }(i,j)$ are respectively the input and filtered images, $w_d$ and $w_s$ are kernel functions in the space and range domains, and ${\sigma }_d$ and ${\sigma }_r$ are spatial distance and pixel difference scale parameters, respectively. In addition, ${\mathsf{\Omega }}_{p,i,j}$ is the set of pixel points in the input image, with $(i,j)$ defined as the center and $2p+1$ as the radius, and $p$ is the radius of the filter. With this algorithm, a larger filter radius corresponds to a larger filter interval. However, calculation is made correspondingly more complex.

From (6), (7), and (8), it can be noted that the efficiency of a bilateral filter is determined by ${\sigma }_d$ and ${\sigma }_r$. Experimentation has revealed [23] that increasing the value of ${\sigma }_d$ leads to a blurrier image after filtering. Similarly, increasing the value of ${\sigma }_r$ increases the size of the $2p+1$ pixel interval and reduces the correlation between pixels. In typical situations, the values of these two parameters are manually adjusted according to the requirements of the particular application. Hence, in this paper, we propose an adaptive adjustment method for selecting the values of ${\sigma }_d$ and ${\sigma }_r$.

3.2. Improved Spatial Domain Kernel Function

To improve the efficiency of the $w_d$ kernel function, we considered the influence of different pixels on noise smoothing. In the spatial domain, the $2p+1$ filter window parameter is set with $(i,j)$ defined as the central filter point. Inspection of (7) shows that pixels located closer to the central filter point have a greater influence on noise smoothing in the spatial domain. Hence, to address this, we defined our improved spatial domain kernel function as follows:

$$w_d(m,n)=(1-\sqrt{\frac{{(i-m)}^2+{(j-n)}^2}{2p+1}})\cdot \exp (-\frac{{(i-m)}^2+{(j-n)}^2}{2{\sigma }_d^2})$$

(9)

where the spatial distance difference scale parameter is defined as ${\sigma }_d=\frac{p}{2}$, since 95% of possible outcomes from the Gaussian function are concentrated in the $[-2{\sigma }_d,2{\sigma }_d]$ range. Thus, in essence, ${\sigma }_d$ is a spatial standard deviation, and after the experiment, we chose $p=5$.

3.3. Improved Selection of Pixel Difference Scale Parameters

In the ${\mathsf{\Omega }}_{p,i,j}$ point set, which is used as a sample to find the standard deviation (${\sigma }_n$) of pixel intensities in an image, the gray value of each pixel is defined as $[a_{(2p-1,2p-1)},a_{(2p-1,2p)}\dots a_{(2p+1,2p+1)}]$. However, unlike with the spatial domain, selection of this standard deviation as the scale parameter leads to an excessively large value for ${\sigma }_d$, as shown in [24]. Hence, a rotating window function for determining the value of ${\sigma }_d$ is proposed as follows. A $\left(2p+1\right)\times \left(p+1\right)$ rectangular window centered on the $\left(x,y\right)$ pixel is selected, with an offset angle defined as $2\pi /K$. The standard deviation of pixels in this rectangular window is then calculated, following which the window is rotated by the offset angle. This calculation and rotation process is successively iterated for a complete revolution (i.e., $K$ rotations should be completed). Finally, the minimum pixel standard deviation $\min ({\sigma }_n)$ computed is selected as ${\sigma }_d$; the window in which this value was calculated is used in subsequent filtering. ${\sigma }_d$ is defined as follows:

$${\sigma }_d=\min ({\sigma }_n)=\underset{k=0,1,\dots K-1}{\min }\{std[f{(x,y)}_{2\pi k/K}]\}$$

(10)

where $f{(x,y)}_{2\pi k/K}$ is the value of a pixel at a given location in the window, $K=8$.

4. Gaussian–Laplacian Multi-Scale Pyramid Algorithm

Gaussian–Laplacian pyramid [24] generation is a multi-scale image decomposition method with a sampling rate of 2, i.e., the resolution of a resulting image is halved in each successive decomposition. The resulting pyramid consists of layers of decomposed images decreasing in resolution from bottom to top, with the original image defined as the base level of the pyramid (level 0). In detail, beginning from level 0, decomposition consists of convolution of the input image with a Gaussian low-pass filter followed by a down-sample operation in order to halve the horizontal and vertical resolution of the processed image. The filtered down-sampled image is subsequently defined as the image at level 1 of the pyramid ($F_1$). This operation is repeated, as defined by the number of levels required in the Gaussian pyramid. Mathematically, this process is defined as

$$F_l={\displaystyle {\sum }_{m=-2}^2{\displaystyle {\sum }_{n=-2}^{2}G(x,y)}}{F}_{l-1}(2i+m,2j+n),$$

(11)

where $G(x,y)$ is a low-pass filter, l is the level of the pyramid, N is the number of layers in the pyramid, $0\le l\le N,i\le R_l,j\le C_l$, and $R_l,C_l$ represent the number of rows and columns in $F_l$ and $N=3$, respectively.

As each image in the pyramid, $F_l$, is of a different resolution, iterative image generation and boundary closure may be affected by noise and dispersed sampling. To ensure the structural information in the highest resolution image is retained in subsequent pyramid levels, image enhancement is required. To do this, we employed the improved Retinex algorithm to recover details in images lost in processing. Enhanced pyramid images are subsequently termed $F_l^{\prime }$.

To complete the image enhancement process, the modified Gaussian pyramid is reconstructed to yield a single image with the same resolution as the original input. Reconstruction is completed by generating a Laplacian pyramid from the image-enhanced Gaussian pyramid. This reconstruction process is detailed as follows:

(1)

The topmost image in the Gaussian pyramid, $F_{l+1}^{\prime }$, is interpolated to obtain an image termed $D_l^{}$, which has the same resolution as the image in the preceding layer ($F_l^{\prime }$).

(2)

$D_l^{}$ is subtracted from $F_l^{\prime }$, with the difference, $F_N^{}$, stored in the Laplace residual set. $F_N^{}$ is subsequently added to $F_l^{\prime }$ to yield $F_N^{\prime }$, which is interpolated for reconstruction of the image in the preceding layer.

(3)

Computation of the Laplacian and image interpolation continues iteratively, until the reconstructed image is of the same resolution as the original input. Using the terminology described above, the process can be described as (12). A Gaussian–Laplacian multi-scale pyramid from intermediate results of different pyramid levels is presented in Figure 4. In our experiment, the sampling value of the pyramid was 3.

$$\{\begin{array}{c}{F}_N=F_l{}^{\prime }-D_l\\ F_N^{\prime }=F_N+F_l{}^{\prime }\end{array}.$$

(12)

5. Experiments and Results

To verify the effectiveness of our algorithm, we compared its operation in experiments with those of the original SSR, MSRCR, Non-local retinex(NLR) [25], and Illumination Invariant Face Recognition (INF) [26,27] algorithms based on a database [28] evaluating their performance using objective and subjective measures of image perception. The proposed operator was implemented in MATLAB 2014b on a computer running the Windows 7 (64 bit) operating system.

5.1. Results of Improved Bilateral Filtering

In this experiment, we selected the database [29] which includes 24 .png images (typically 768 × 512 or similar size) for processing. Gaussian white noise with mean value $\mathrm{u}=0$ and the standard deviation $\sigma =0.01$, $\sigma =0.02$,$\sigma =0.05$,$\sigma =0.1$, and $\sigma =0.2$ was added to these images—an example of which is shown in Figure 5—to degrade their quality such that the performance of the different filtering algorithms could be observed. We note that with the original bilateral filter, edges in the images were lost, as they had all been smoothed. In contrast, Gaussian noise is few perceptible following operation of our algorithm, as shown in Figure 5d. Moreover, the image performance following operation of our algorithm is a better improvement, shown in Figure 5d. The edges in the image are retained after denoising, as shown in Figure 6c.

For additional verification, we compared the two filtering algorithms with respect to Peak Signal to Noise Ratio (PSNR), Mean Squared Error (MSE), Structural Similarity (SSIM) [30], Multi-scale structural similarity (MS-SSIM) [31], Visual Information Fidelity (VIF) [32], Full Refer- ence Visual Information Fidelity (FR-VIF) [33], and visual signal-to-noise ratio (VSNR) [34], shown in

Table 1. Evaluation of bilateral filtering algorithms based database [29] (24 images).

Gaussian White Noise	Algorithm	PSNR	MSE	SSIM	MSSIM	VIF	FR-VIF	VSNR
$\underline{\sigma =0.01}$	Original algorithm	22.493	376.772	0.542	0.912	0.361	0.231	18.221
	Modified algorithm	27.034	129.231	0.799	0.964	0.566	0.379	26.314
$\underline{\sigma =0.02}$	Original algorithm	20.505	587.780	0.488	0.898	0.327	0.205	17.375
	Modified algorithm	28.414	93.752	0.928	0.993	0.794	0.463	34.386
$\underline{\sigma =0.05}$	Original algorithm	18.014	1035.936	0.451	0.881	0.284	0.182	16.101
	Modified algorithm	27.220	123.450	0.946	0.991	0.691	0.469	30.743
$\underline{\sigma =0.1}$	Original algorithm	16.222	1560.708	0.431	0.869	0.257	0.170	15.226
	Modified algorithm	27.507	115.525	0.970	0.995	0.713	0.502	32.404
$\underline{\sigma =0.2}$	Original algorithm	16.127	1610.014	0.418	0.856	0.249	0.165	14.992
	Modified algorithm	22.286	481.923	0.860	0.952	0.446	0.365	21.867

. This MATLAB wrapper code is presented in [35]. In this case, the objective measure scores are directly averaged; we highlight the best result of the two algorithms with boldface. We note that the intensity of noise in the images following processing is higher with the traditional bilateral filtering algorithm than it is with our modified solution, based on a comparison of the values of seven Image Quality Assessment (IQA) measures.

5.2. Subjective Analysis of Image Enhancement Algorithms

To verify the effectiveness of our improved image enhancement technique, we conducted experiments comparing its performance to those of the SSR, MSRCR, NLR, and INF algorithms based on a database [28] with 4322 images, which is a real foggy test dataset provided by Gac Research Institute. The partial results of these experiments are shown in Figure 6, with Column 1 depicting original images, Column 2 depicting images processed using the SSR algorithm, Column 3 depicting images processed using the MSRCR algorithm, Column 4 depicting images processed using the NLR algorithm, Column 5 depicting images processed using the INF algorithm, and Column 6 depicting images processed using our improved image enhancement technique.

There are degradation and dark regions in the original images in Figure 7, which are characteristic of uneven lighting, making them suitable for image enhancement. We note that with all five algorithms considered, most of the foreground details are preserved, and the color in the resulting images is enhanced following processing. However, with the SSR, MSRCR, and NLR algorithms, the “halo” effect can also be observed, as seen most obviously in Figure 7(a-2), Figure 7(d-3), Figure 7(e-3), Figure 7(f-3), and Figure 7(f-4). The MSRCR and INL algorithms darken the image and lose the details, as shown in Figure 7(a-3–f-3) and Figure 7(a-5–f-5). While details in the blurred areas of the original images can be enhanced using the other four algorithms, there is a deviation between the color in the input and output images. Hence, the color of the imaged object is not restored, and the visual representation of this object is thus distorted. In addition, the MSRCR and NLR algorithm was not effective in removing the fog from the foreground of the images, as shown in Figure 7(a-4–f-4).

In contrast, the texture and color contrast of images processed using our technique are improved, and enhancement is effective for images in different low light situations. In addition, following processing, the deviation of color between the input and output images is small. As color fidelity is better, the overall visual representation of the image is retained. Finally, details in the image are more clearly identified, as can be seen from Figure 7(a-6–f-6).

For a more quantitative representation of the effect of the different algorithms, histograms of the grayscale intensity of the images in Row (a) of Figure 6 are shown in Figure 8. From this, it can be seen that the distribution of pixel intensities is more uniform and the average grayscale intensity of the image is reduced. This is indicative of the fog removal effect obtained with our algorithm; fog is observed as a brighter area in an image, with a correspondingly large grayscale intensity. A reduction in the average grayscale intensity of an image thus suggests the removal of fog.

5.3. Objective Evaluation of Image Enhancement Algorithms

For objective evaluation of the different image enhancement algorithms, we calculated the Peak Signal to Noise Ratio (PSNR), Mean Squared Error (MSE), SSIM, MS-SSIM, VIF, Information content weighting Structural SIMilarity (IWSSM) [35], and VSNR of the images in the database, as shown in

Table 2. Objective comparison of the performance of different enhancement algorithms based on information Peak Signal to Noise Ratio (PSNR), Mean Squared Error (MSE), SSIM, MS-SSIM, VIF, IWSSIM, and VSNR.

Algorithm	PSNR	MSE	SSIM	MSSIM	VIF	IWSSIM	VSNR
OI	N/A	0.000	1.000	1.000	1.000	1.000	N/A
SSR	18.584	1097.904	0.734	0.761	1.192	0.728	7.194
MSRCR	12.034	4438.279	0.547	0.761	0.826	0.731	12.376
NLR	13.751	3424.179	0.767	0.799	0.867	0.662	6.932
INF	10.357	6132.800	0.632	0.480	0.355	0.736	7.956
Our	24.071	333.602	0.851	0.914	0.994	0.894	13.224

. In this case, the correlation scores are directly averaged and we highlight the best result with boldface, as each of these parameters characterizes a different aspect of the image.

Table 2 shows that images processed using our algorithm had the best values, suggesting that more high-frequency information is processed using our technique, and details in the image are subsequently enhanced. Similarly, the PSNR and SSIM values were maximized using our algorithm, indicating that the contrast of the images is improved with our technique, and details are more identifiable. Then, MSSIM and VIF value of the image were maximized with our technique, indicating its improved reconstruction quality. Finally, the other values also improved using our algorithm, demonstrated that our algorithm leads to significant and consistent performance of image enhancement.

6. Conclusions

In this paper, we proposed a new Retinex-based image enhancement algorithm to address the problem of image blurring caused by uneven illumination. The technique features an improved bilateral filtering function, addressing the problem of an over-enhanced luminance image and the associated loss of texture. In addition, we combine our modified Retinex model with Gaussian pyramid down-sampling for multi-scale processing, in order to eliminate blurring problems and increase the contrast of the final image. We compared the performance of our technique with those of the SSR, MSRCR, NLR, and INF algorithms in experiments, with the results highlighting its improved image enhancement. However, the combination of the improved bilateral filtering function and the Gaussian pyramid transformation increases the time complexity of the technique, a problem which will be addressed in future studies.