4-Band Multispectral Images Demosaicking Combining LMMSE and Adaptive Kernel Regression Methods
Abstract
:1. Introduction
2. Design of the Four-Band MSFA and Application Assumptions of the LMMSE Method
2.1. Design of the 4-Band MSFA
2.2. Application Assumptions of LMMSE Method
3. Overview of LMMSE and Kernel Regression Methods
3.1. LMMSE Demosaicking Method
3.2. Kernel Regression Method
3.3. Adaptive Kernel Regression
4. Proposed Multispectral Demosaicking Method
4.1. Blue Band Estimation by LMMSE Method
4.2. Orange Band Estimation at Red and Green Pixels
4.3. Green Band Estimation at Red Pixels and Vice Versa
4.4. Red and Green Bands Estimation at Orange Pixels
4.5. Red, Green, and Orange Bands Estimation at Blue Pixels
4.6. Estimated Bands Enhancement Using Adaptive Kernel Regression
5. Experimental Results
5.1. Visual Performance Evaluations
5.2. Quantitative Performance Evaluations
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Miao, L.; Qi, H. The design and evaluation of a generic method for generating mosaicked multispectral filter arrays. IEEE Trans. Image Process. 2006, 15, 2780–2791. [Google Scholar] [CrossRef] [PubMed]
- Aggarwal, H.K.; Majumdar, A. Compressive Sensing Multi-Spectral Demosaicing from Single Sensor Architecture. In Proceedings of the 2014 IEEE China Summit & International Conference on Signal and Information Processing (ChinaSIP), Xi’an, China, 9–13 July 2014; pp. 9–13. [Google Scholar]
- Monno, Y.; Tanaka, M.; Okutomi, M. Multispectral demosaicking using adaptive kernel upsampling. In Proceedings of the 2011 18th IEEE International Conference on Image Processing, Brussels, Belgium, 11–14 September 2011; pp. 3218–3221. [Google Scholar]
- Jaiswal, S.J.; Fang, L.; Jakhetiya, V.; Pang, J.; Mueller, K.; Au, O.C. Adaptive multispectral demosaicking based on frequency-domain analysis of spectral correlation. IEEE Trans. Image Process. 2017, 26, 953–968. [Google Scholar] [CrossRef] [PubMed]
- Mihoubi, S.; Losson, O.; Mathon, B.; Macaire, L. Multispectral demosaicing using pseudo-panchromatic image. IEEE Trans. Comput. Imaging 2018, 3, 982–995. [Google Scholar] [CrossRef] [Green Version]
- Sun, B.; Yuan, N.; Cao, C.; Hardeberg, J.Y. Design of four-band multispectral imaging system with one single-sensor. Future Gener. Comput. Syst. 2018, 86, 670–679. [Google Scholar] [CrossRef]
- Sun, B.; Zhao, Z.; Xie, D.; Yuan, N.; Yu, Z.; Chen, F.; Cao, C.; de Dravo, V.W. Sparse spectral signal reconstruction for one proposed nine-band multispectral imaging system. Mech. Syst. Signal Process. 2020, 141, 106627. [Google Scholar] [CrossRef]
- Lapray, P.J.; Wang, X.; Thomas, J.B.; Gouton, P. Multispectral filter arrays: Recent advances and practical implementation. Sensors 2014, 14, 21626–21659. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Miao, L.; Qi, H.; Ramanath, R.; Snyder, W.E. Binary tree-based generic demosaicking algorithm for multispectral filter arrays. IEEE Trans. Image Process. 2006, 15, 3550–3558. [Google Scholar] [CrossRef] [PubMed]
- Monno, Y.; Kikuchi, S.; Tanaka, M.; Okutomi, M. A practical one-shot multispectral imaging system using a single image sensor. IEEE Trans. Image Process. 2015, 24, 3048–3059. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Monno, Y.; Kiku, D.; Kikuchi, S.; Tanaka, M.; Okutomi, M. Multispectral demosaicking with novel guide image generation and residual interpolation. In Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France, 27–30 October 2014; pp. 645–649. [Google Scholar]
- Monno, Y.; Kiku, D.; Tanaka, M. Adaptive residual interpolation for color and multispectral image demosaicking. Sensors 2017, 17, 2787. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhang, L.; Wu, X. Color demosaicking via directional linear minimum mean square-error estimation. IEEE Trans. Image Process. 2005, 14, 2167–2178. [Google Scholar] [CrossRef] [PubMed]
- Bayer, B. Color Imaging Array. U.S. Patent 3971065, 20 July 1976. [Google Scholar]
- Amba, P.; Dias, J.; Alleysson, D. Random color filter arrays are better than regular ones. J. Imaging Sci. Technol. 2016, 60, 50406-1. [Google Scholar] [CrossRef] [Green Version]
- Amba, P.; Thomas, J.B.; Alleysson, D. N-LMMSE demosaicing for spectral filter arrays. J. Imaging Sci. Technol. 2017, 61, 40407-1–40407-11. [Google Scholar] [CrossRef] [Green Version]
- Takeda, H.; Farsiu, S.; Milanfar, P. Kernel regression for image processing and reconstruction. IEEE Trans. Image Process. 2007, 16, 349–366. [Google Scholar] [CrossRef] [PubMed]
- Multispectral Image Dataset. Available online: http://www.cs.columbia.edu/CAVE/databases/multispectral/ (accessed on 25 February 2022).
- Available online: https://www.photonics.com (accessed on 25 February 2022).
- Péguillet, H.; Thomas, J.B.; Gouton, P.; Ruichek, Y. Energy balance in single exposure multispectral sensors. In Proceedings of the 2013 Colour and Visual Computing Symposium (CVCS), Gjovik, Norway, 5–6 September 2013; pp. 1–6. [Google Scholar]
- Monno, Y.; Tanaka, M.; Okutomi, M. Multispectral demosaicking using guided filter. In Digital Photography VIII; SPIE: Burlingame, CA, USA, 24 January 2012; Volume 8299, pp. 82990O–1–82990O–7. [Google Scholar]
- Wang, C.; Wang, X.; Hardeberg, J. A linear interpolation algorithm for spectral filter array demosaicking. In International Conference on Image and Signal Processing; Springer: Cherbourg, France, 2014; Volume 8509, pp. 151–160. [Google Scholar]
Images | ||||||
---|---|---|---|---|---|---|
Beads | 0.5469 | 0.4267 | 0.2773 | 0.6887 | 0.8381 | 0.2498 |
Balloons | 0.8000 | 0.6647 | 0.6529 | 0.9602 | 0.9666 | 0.8968 |
Pompoms | 0.6138 | 0.1869 | 0.0152 | 0.7798 | 0.9021 | 0.5075 |
Cloth | 0.9700 | 0.9222 | 0.6599 | 0.9654 | 0.8180 | 0.6679 |
Statue | 0.9818 | 0.9413 | 0.8688 | 0.9859 | 0.9797 | 0.9356 |
Face | 0.9817 | 0.9530 | 0.8665 | 0.9896 | 0.9637 | 0.9248 |
Food | 0.9882 | 0.9008 | 0.6753 | 0.9362 | 0.9168 | 0.7244 |
Feathers | 0.8995 | 0.8398 | 0.7235 | 0.9733 | 0.9046 | 0.8378 |
Flowers | 0.9247 | 0.7965 | 0.6781 | 0.9438 | 0.9317 | 0.7952 |
Beans | 0.9454 | 0.9136 | 0.8465 | 0.9652 | 0.9576 | 0.8772 |
Painting | 0.9610 | 0.8192 | 0.6973 | 0.9348 | 0.9674 | 0.8332 |
Thread | 0.9002 | 0.8467 | 0.7339 | 0.9518 | 0.9326 | 0.7982 |
Clay | 0.6538 | 0.5807 | 0.2737 | 0.7703 | 0.7579 | 0.200 |
Superballs | 0.6448 | 0.7475 | 0.5906 | 0.7482 | 0.8802 | 0.3602 |
Toys | 0.9685 | 0.8780 | 0.6065 | 0.9441 | 0.8567 | 0.6685 |
Glass | 0.7431 | 0.4076 | 0.2074 | 0.8667 | 0.8476 | 0.5632 |
CD | 0.828 | 0.7183 | 0.7070 | 0.8780 | 0.7761 | 0.5789 |
Hairs | 0.9814 | 0.9524 | 0.8965 | 0.9919 | 0.9862 | 0.9591 |
Peppers | 0.9049 | 0.7097 | 0.5233 | 0.9265 | 0.8626 | 0.6935 |
Sponges | 0.5476 | 0.3080 | 0.0675 | 0.9068 | 0.8371 | 0.5745 |
Paints | 0.9744 | 0.9525 | 0.9001 | 0.9892 | 0.9583 | 0.9219 |
Beers | 0.9461 | 0.8127 | 0.6932 | 0.9524 | 0.9772 | 0.8751 |
Chart_Toy | 0.9951 | 0.9866 | 0.9674 | 0.9964 | 0.9906 | 0.9792 |
Sushi | 0.9813 | 0.9559 | 0.7866 | 0.9804 | 0.8892 | 0.7964 |
Lemons | 0.8715 | 0.7262 | 0.6711 | 0.9658 | 0.9897 | 0.9325 |
Slices | 0.9443 | 0.8963 | 0.8522 | 0.9838 | 0.9679 | 0.9287 |
Images | PSNR | ||||
---|---|---|---|---|---|
BTES | DFWF | ASCD | N-LMMSE | Ours | |
Beads | 30.7458 | 33.2131 | 30.2940 | 28.1932 | 29.8430 |
Balloons | 42.0289 | 46.9371 | 39.2510 | 38.5023 | 40.0571 |
Pompoms | 38.4598 | 41.2875 | 35.1001 | 31.7721 | 33.7244 |
Cloth | 28.5308 | 31.3640 | 28.9022 | 33.1767 | 34.1697 |
Statue | 40.6305 | 44.1420 | 31.5929 | 39.8950 | 41.2060 |
Face | 38.2092 | 40.2888 | 35.9277 | 38.1173 | 41.1619 |
Food | 40.0772 | 43.2572 | 37.0315 | 40.0100 | 40.0189 |
Feathers | 35.1460 | 39.4372 | 33.0949 | 31.2218 | 34.8913 |
Flowers | 39.1085 | 38.4263 | 33.0538 | 33.6341 | 38.6609 |
Beans | 32.6284 | 36.9307 | 32.2185 | 29.4240 | 34.0663 |
Painting | 30.8851 | 34.8590 | 28.4571 | 30.9910 | 34.7533 |
Thread | 36.3351 | 41.3007 | 31.6812 | 35.3227 | 39.5297 |
Clay | 32.2509 | 36.1485 | 34.3987 | 31.2567 | 34.4575 |
Superballs | 41.7985 | 44.9294 | 36.3779 | 34.7720 | 37.0399 |
Toys | 42.7080 | 43.4266 | 36.7039 | 35.6316 | 38.8146 |
Glass | 26.4927 | 31.1506 | 31.3545 | 30.8802 | 33.5763 |
CD | 36.4992 | 34.8518 | 37.8778 | 36.2179 | 39.8332 |
Hairs | 32.9339 | 36.8394 | 36.2247 | 36.8732 | 39.9698 |
Peppers | 35.0235 | 33.4378 | 36.0790 | 34.4836 | 36.8838 |
Sponges | 30.5707 | 25.5476 | 31.2702 | 29.3916 | 30.1159 |
Paints | 27.2903 | 28.2379 | 32.4600 | 33.0013 | 33.6658 |
Beers | 36.1153 | 29.1305 | 33.6159 | 30.5370 | 33.5116 |
Chart_Toy | 28.0560 | 31.1273 | 32.6595 | 34.7302 | 37.9766 |
Sushi | 37.2125 | 38.8466 | 39.4100 | 40.0039 | 42.3519 |
Lemons | 31.9442 | 35.3157 | 38.6506 | 32.8326 | 36.7108 |
Slices | 31.0185 | 35.2776 | 35.3538 | 38.1607 | 40.1883 |
Average | 34.7192 | 36.7581 | 34.1939 | 34.1935 | 36.8146 |
Images | SSIM | ||||
---|---|---|---|---|---|
BTES | DFWF | ASCD | N-LMMSE | Ours | |
Beads | 0.8719 | 0.7840 | 0.8368 | 0.8073 | 0.8756 |
Balloons | 0.9903 | 0.9398 | 0.9449 | 0.9561 | 0.9748 |
Pompoms | 0.9549 | 0.8606 | 0.8956 | 0.9001 | 0.9046 |
Cloth | 0.8476 | 0.9155 | 0.7864 | 0.9153 | 0.9278 |
Statue | 0.9413 | 0.9739 | 0.9354 | 0.9419 | 0.9797 |
Face | 0.9718 | 0.9830 | 0.9471 | 0.9528 | 0.9881 |
Food | 0.9749 | 0.9667 | 0.9649 | 0.9675 | 0.9804 |
Feathers | 0.9480 | 0.9197 | 0.8924 | 0.9208 | 0.9482 |
Flowers | 0.9519 | 0.9165 | 0.8917 | 0.9394 | 0.9575 |
Beans | 0.9524 | 0.9019 | 0.8836 | 0.9190 | 0.9356 |
Painting | 0.8798 | 0.8887 | 0.7743 | 0.8985 | 0.9127 |
Thread | 0.9208 | 0.9561 | 0.8757 | 0.9577 | 0.9731 |
Clay | 0.9780 | 0.9017 | 0.8837 | 0.9216 | 0.9455 |
Superballs | 0.9807 | 0.9156 | 0.9299 | 0.9399 | 0.9548 |
Toys | 0.9701 | 0.9513 | 0.9269 | 0.9522 | 0.9769 |
Glass | 0.9145 | 0.8843 | 0.8681 | 0.9148 | 0.9392 |
CD | 0.9791 | 0.9378 | 0.9470 | 0.9560 | 0.9741 |
Hairs | 0.9536 | 0.9717 | 0.9127 | 0.9654 | 0.9785 |
Peppers | 0.9842 | 0.9058 | 0.8879 | 0.9132 | 0.9478 |
Sponges | 0.9693 | 0.8927 | 0.8862 | 0.8890 | 0.9014 |
Paints | 0.9380 | 0.9159 | 0.9195 | 0.9300 | 0.9690 |
Beers | 0.9766 | 0.9490 | 0.9026 | 0.9565 | 0.9705 |
Chart_Toy | 0.9338 | 0.9538 | 0.9131 | 0.9696 | 0.9740 |
Sushi | 0.9749 | 0.9443 | 0.9726 | 0.9698 | 0.9817 |
Lemons | 0.9609 | 0.9353 | 0.9518 | 0.9355 | 0.9578 |
Slices | 0.9455 | 0.9278 | 0.9329 | 0.9549 | 0.9751 |
Average | 0.9486 | 0.9228 | 0.9025 | 0.9325 | 0.9542 |
Images | RMSE | ||||
---|---|---|---|---|---|
BTES | DFWF | ASCD | N-LMMSE | Ours | |
Beads | 0.0407 | 0.0398 | 0.0347 | 0.0470 | 0.0340 |
Balloons | 0.0164 | 0.0160 | 0.0162 | 0.0191 | 0.0140 |
Pompoms | 0.0279 | 0.0273 | 0.0237 | 0.0288 | 0.0257 |
Cloth | 0.0342 | 0.0315 | 0.0408 | 0.0201 | 0.0197 |
Statue | 0.0121 | 0.0118 | 0.0262 | 0.0113 | 0.0096 |
Face | 0.0118 | 0.0114 | 0.0178 | 0.0169 | 0.0100 |
Food | 0.0137 | 0.0134 | 0.0173 | 0.0171 | 0.0109 |
Feathers | 0.0227 | 0.0218 | 0.0286 | 0.0229 | 0.0184 |
Flowers | 0.0176 | 0.0189 | 0.0247 | 0.0175 | 0.0139 |
Beans | 0.0245 | 0.0232 | 0.0279 | 0.0255 | 0.0209 |
Painting | 0.0207 | 0.0163 | 0.0396 | 0.0208 | 0.0189 |
Thread | 0.0199 | 0.0186 | 0.0303 | 0.0182 | 0.0113 |
Clay | 0.0101 | 0.0315 | 0.0320 | 0.0290 | 0.0208 |
Superballs | 0.0217 | 0.0212 | 0.0205 | 0.0168 | 0.0152 |
Toys | 0.0181 | 0.0180 | 0.0239 | 0.0157 | 0.0132 |
Glass | 0.0228 | 0.0322 | 0.0314 | 0.0236 | 0.0218 |
CD | 0.0076 | 0.0202 | 0.0168 | 0.0127 | 0.0115 |
Hairs | 0.0117 | 0.0156 | 0.0183 | 0.0140 | 0.0103 |
Peppers | 0.0087 | 0.0244 | 0.0236 | 0.0163 | 0.0152 |
Sponges | 0.0172 | 0.0679 | 0.0445 | 0.0441 | 0.0401 |
Paints | 0.0210 | 0.0395 | 0.0287 | 0.0280 | 0.0216 |
Beers | 0.0120 | 0.0363 | 0.0260 | 0.0277 | 0.0225 |
Chart_Toy | 0.0209 | 0.0277 | 0.0269 | 0.0153 | 0.0129 |
Sushi | 0.0107 | 0.0127 | 0.0123 | 0.0108 | 0.0082 |
Lemons | 0.0142 | 0.0207 | 0.0144 | 0.0176 | 0.0155 |
Slices | 0.0135 | 0.0175 | 0.0190 | 0.0126 | 0.0100 |
Average | 0.0179 | 0.0244 | 0.0256 | 0.0211 | 0.0171 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Hounsou, N.; Mahama, A.T.S.; Gouton, P. 4-Band Multispectral Images Demosaicking Combining LMMSE and Adaptive Kernel Regression Methods. J. Imaging 2022, 8, 295. https://doi.org/10.3390/jimaging8110295
Hounsou N, Mahama ATS, Gouton P. 4-Band Multispectral Images Demosaicking Combining LMMSE and Adaptive Kernel Regression Methods. Journal of Imaging. 2022; 8(11):295. https://doi.org/10.3390/jimaging8110295
Chicago/Turabian StyleHounsou, Norbert, Amadou T. Sanda Mahama, and Pierre Gouton. 2022. "4-Band Multispectral Images Demosaicking Combining LMMSE and Adaptive Kernel Regression Methods" Journal of Imaging 8, no. 11: 295. https://doi.org/10.3390/jimaging8110295