Novel Infrared Terahertz Fusion 3D Non Invasive Imagi 2022 Composites Scienc
Novel Infrared Terahertz Fusion 3D Non Invasive Imagi 2022 Composites Scienc
Novel Infrared Terahertz Fusion 3D Non Invasive Imagi 2022 Composites Scienc
A R T I C L E I N F O A B S T R A C T
Keywords: The reinforcement of polymer composites using plant fibres has received vast and growing attention for inves
Plant fibre tigation owing to their capability to serve as semi-structural materials, especially as a substitute for fiberglass.
Infrared However, in this type of research, the structural complexity, due to geometrical mismatches and inherent
Terahertz
variability to the properties of vegetable fibres and plant fibre composites would benefit from more effective non-
Fusion
invasive inspection techniques. In this work, a novel infrared-terahertz fused approach was proposed, and a new
NDT
unsupervised deep residual fusion algorithm was developed in order to combine the outputs from two different
physical mechanisms. Specifically, infrared imaging provided the details of the defects and the morphological
information from the surface and subsurface, while terahertz imaging recorded the distribution of the fibres and
the resin. Finally, the fused infrared-terahertz slices were reconstructed into a form of 3D exhibition for in
spection and evaluation of two plant fibre-reinforced polymer composites in first time using multiphysical im
aging and fusion.
1. Introduction price of the equipment still restrict the application [7]. Ultrasonic
scanning is another common approach for composite inspection when
In recent years, plant fibre-reinforced polymer (PFRP) composites considering safety and low cost while the dependence on the coupling
have received vast and growing attention for investigation owing to the medium such as water limits the application of this inspection technique
potential of serving as semi-structural materials. PFRP has many ad to waterproof materials [8]. In the last few decades, infrared thermog
vantages including the fact that it is environmentally friendly, light raphy (IRT) has been developed as a non-invasive and non-contact in
weight and low cost, etc. Several investigations of PFRP such as coir, spection method [9,10]. Several techniques such as optical excitation
hemp, jute and bamboo have already been presented to reveal the me thermography (OET) [11,12], laser excitation thermography [13], ul
chanical properties and the possible role and use of non-destructive trasonic excitation thermography [14] and eddy current excitation
inspection (NDI) on manufactured composites [1]. However, internal thermography (ECT) [15] have been applied in the evaluation of com
manufacturing inspection is still a problem. Therefore, NDI needs to be posites. Nevertheless, IRT is not ideal for transparent and
conducted in order to broaden the field for the application of PFRP. semi-transparent materials like glass fibre reinforced composites [8]. In
Advances in NDT inspection on composite structures have been addition, limited penetration capability restricts the inspection depth of
reviewed by several researchers [2–5]. In composite structure inspec IRT [16], which actually provides the surface and the sub-surface in
tion, X-ray is widely adapted for computed tomography (CT) [6]. This formation for evaluation. Terahertz time-domain spectroscopy
technique has the ability to provide promising results with the 3D (THz-TDS) has attracted more attention for composite evaluation in
structure of the specimen. However, the ionizing radiation and high recent years, since it is safer than X-rays and offers a stronger capability
* Corresponding author. Centre for Composite Materials and Structures, Harbin Institute of Technology, Harbin 150001, China.
E-mail address: [email protected] (H. Zhang).
https://doi.org/10.1016/j.compscitech.2022.109526
Received 22 October 2021; Received in revised form 14 March 2022; Accepted 11 May 2022
Available online 14 May 2022
0266-3538/© 2022 Elsevier Ltd. All rights reserved.
J. Hu et al. Composites Science and Technology 226 (2022) 109526
Fig. 1. Structure of specimens used in this study: (a) Jute composite; (b) Kenaf composite.
2
J. Hu et al. Composites Science and Technology 226 (2022) 109526
Fig. 3. Schematic diagram of the proposed multi-modality spatial-spectral data fusion technique.
3
J. Hu et al. Composites Science and Technology 226 (2022) 109526
Fig. 4. Schematic and experimental setup of the THz-TDS system used herein.
adapted to record the temperature profile. The cooling time was set at fixed the terahertz images, and then registered the infrared images to the
10 s for flashes. The camera spatial resolution was 640 × 512 pixels (25 terahertz images. The details of the alignment process can be described
μm × 25 μm of detector size) and a 50 mm lens was employed to provide as follows. First, we selected several CPs manually from the contours of
a field of view (FOV) of 18.2◦ (horizontal) × 14.6◦ (vertical). the samples on the images provided by THz and IRT systems. Then, we
calculated the transform from the selected CPs on IRT to the selected CPs
3.3. Unsupervised deep residual fusion algorithm on fixed THz imaging, by using the operators such as stretching, rotating
and scaling. Finally, we adapted the transform calculated from the
As illustrated in Fig. 6, the analytical results from THz-TDS and IRT selected CPs to the whole IRT images. The color bar was then matched to
were pre-processed to confirm that the images for fusion were in strict obtain the intensity alignment image for fusion. The above pre-
geometrical alignment and intensity matching. In this study, the processing procedures were conducted using Matlab software. In the
geometrical alignment was conducted by matching the control points fusion stage, an unsupervised encoder-decoder network architecture
(CPs) from the contours of the sample. During the alignment process, we was introduced. The encoder including a convolutional layer and a
4
J. Hu et al. Composites Science and Technology 226 (2022) 109526
dense-connected block extracted high-dimensional features on images train the weights of the convolutional layers. This process is aimed at
from different modalities. Then, these high-dimensional features were ensuring that the Encoder-Decoder network can extract salient features
fused with an l1 -norm based fusion rule and reconstructed by the of the input images and use these features to reconstruct the images. To
decoder, which is composed of convolutional layers. achieve this, the total loss function formulated in Eq. (4) needs to be
The encoder illustrated in Fig. 7 combines dense connections with minimized, which is a weighted combination of the structural similarity
residual modules to extract high-dimensional features from input im loss LSSIM and pixel loss LP .
ages. Residual learning can optimize several convolutional layers as
Ltotal = λLSSIM + LP (4)
identity mapping which eases the training of the deep neural network
and benefits from increasing depth [23]. Dense connections between where, λ refers to the magnitude difference between the structural
residual blocks ensure that the input of each residual block is the similarity loss and the pixel loss. λ = 1 is adopted in this work.
concatenation of all of the former modules [24]. This effectively protects The pixel loss is calculated using l2 -norm to measure the difference
the information in the middle layer from vanishing and sufficiently ex between the input and output matrix as follows:
tracts salient features.
The fusion layer is constructed with an operator based on l1 -norm. LP = ‖O − I‖2 (5)
First, the l1 -norm map of each point (x, y) is calculated on the features
extracted by the encoder. The activity level map Mi (x, y) for each source where I indicates the raw image and O refers to the image reconstructed
is calculated using a blocked average operator rolling on the l1 -norm by the Encoder-Decoder network.
map: The structural similarity loss is calculated as in [25]:
x∑
+1 ∑y+1 ⃦ ⃦ LSSIM = 1 − SSIM(O − I) (6)
⃦ 1:N ⃦
b=y− 1 Фi (a, b) 1
Mi (x, y) = a=x− 1
(1) where, SSIM(O − I) refers to the structural similarity operator which is a
9
perceptual measurement of the similarity of the fused image and input
where Фni (i = 1, ⋯, k) refers to the feature map calculated by the image. It aims at enabling the network to extract the representation and
encoder of the i-th input source, k ≥ 2 indicates the total number of the to reconstruct the image considering both the local structure and
input modalities, n = 1, 2, ⋯, N (N = 64) indicates the index of the spe contrast. The network in the Encoder-Decoder structure is trained on the
cific feature belonging to the feature map, and N denotes the total dataset containing 142,776 grayscale images of MS-COCO [26].
number of the features generated by the encoder per input.
Then, the weighted mask Wi (x, y) for each source is calculated by: 4. Experimental results and discussion
Mi (x, y)
Wi (x, y) = ∑k (2) Fig. 8 illustrates the THz-TDS analysis process of the raw data
j=1 Mj (x, y) collected from the experimental system working in transmission mode.
The analysis includes both time-domain imaging and frequency-domain
while, the fused feature map Фnf (x, y) is obtained as follows: analysis. In the jute fibre composite illustrated in Fig. 8 (a), the time-
domain curve of spatial location on the chopped glass fibre shows a
∑
k
Фnf (x, y) = Wi (x, y)Фni (x, y) (3) larger time delay. In order to utilize this, the arrival time (38.73 ps) of
i=1 the peak on the time-domain curve of chopped glass fibre is chosen for
imaging on the chopped glass fibre layer. The frequency-domain curve
The fused feature map is then processed by the decoder to recon
reflects the different characteristics in the absorption of variant mate
struct the fusion image, which effectively contains the information
rials. The woven roving appears to have a higher absorption at 0.44 THz
provided by different sensors. The decoder in a concise and efficient
due to the high absorption coefficient of woven roving material at this
architecture includes four convolutional layers with a decreasing num
point in the spectrum. Therefore, the spectral amplitude image at this
ber of output features as 64, 32, 16 and 1.
point is adapted for imaging on the woven roving layer. In the kenaf
In the training process, the fusion layer based on the l1 -norm oper
fibre composite illustrated in Fig. 8 (b), the time-domain curve of the
ator would be removed. The encoder and decoder are directly stacked to
5
J. Hu et al. Composites Science and Technology 226 (2022) 109526
Fig. 8. THz-TDS analysis: (a) Jute composite sample; (b) Kenaf composite sample.
6
J. Hu et al. Composites Science and Technology 226 (2022) 109526
spatial location on kenaf yarn reveals a larger time delay than that of from different views of the 3D reconstruction model. A more intuitive
other materials. With this in mind, the corresponding arrival time (78.43 representation can be found in the video in the supplementary materials.
ps) of the peak on the time-domain curve of kenaf yarn is selected for In the top row of the figure, the 3D reconstruction result is cut on the
imaging on the kenaf yarn layer. The chopped strand matrix shows chopped strand mat layer and the kenaf yarn layer, respectively. Fig. 12
obvious absorption at 0.48 THz in the spectrum. This motivates us to use (a) is cut on the chopped strand mat layer and compared with the CT
the spectral amplitude image at this point for imaging on the chopped slice at a depth of 0.8 mm from the back panel. The low-velocity impact-
strand matrix layer. induced damage and banded dent observed in IRT imaging are merged
In order to obtain further complementary information on different with the glass fibre content distribution viewed in THz imaging. All of
depths, we also carried out experiments in the THz reflection mode. As these patterns are presented on the fusion imaging result and can be
illustrated in Fig. 9, THz waves produce a reflection on the boundary of mapped by the referred X-ray CT slice. Fig. 12(b) is a cut on the kenaf
different media. The first reflection wave generated at the junction of air yarn layer, which refers to the CT slice obtained at a depth of 1.6 mm
and the surface veil layer carries the information of the glass fibre mesh from the back panel. It is worth mentioning that, as an internal layer, the
on this layer. The second reflection wave generated on the boundary of kenaf yarns in this layer are observed clearly with the impact-induced
the surface veil and chopped strand mat layers contains the information damage in the 3D fusion result. In the bottom row of the figure, we
of the fibre distribution on the chopped strand mat layer. The mask box further present the cross-section imaging and horizontal-section imag
in Fig. 9 shows that a time-gated Fourier transform [27] was carried out ing of the 3D reconstruction result. Fig. 12(c) shows the cross-section
for amplitude imaging on the reflected waves. The curves used here are imaging cut at the position of 71.5 mm and 88.5 mm from the upper
collected from the THz experiment [28] on the reflection mode of jute edge of the sample. Interestingly, the position of 71.5 mm from the
fibre composite on the backside view. The same processes are repeated upper edge is exactly the location of the impact-induced damage.
on the front view data of the jute fibre composite, including both back Therefore, the morphology of the impact can be observed on this cross-
and front view data of the kenaf fibre composite. section imaging. Fig. 12(d) presents the horizontal-section of the kenaf
The analysis of the raw data collected from IRT is introduced in fibre yarns layer. As an internal layer in such a 3D structure with
Fig. 10. With the thermal diffusion process, IRT can present information complicated fibre content, the kenaf yarn layer with rich plant fibre
at different depths. In this work, optical excitation thermography is information is chosen for the detailed comparison with the CT slice. The
conducted on both sides of the specimens. The view of the thermal kenaf yarns present on the cropped horizontal-section of the 3D fusion
sequence collected from both sides is flipped on the back-side view. imaging and the cropped CT slice are matched with a red arrow. A total
Readers should note that the second frame was selected as the surface of 26 kenaf yarns can be mapped into the X-ray CT inspection result one-
information, while the hundredth frame was chosen as the subsurface by-one. This illustrates that the proposed 3D fusion imaging technique
information. In the jute composite, surface information includes the generates high quality images on the internal structures. The fusion of
impact-induced damage and the air holes existing in cured resin in the the information from complementary depths of different imaging sys
form of hot spots. Sub-surface information mainly contains impact- tems paves the way for conducting high contrast and high resolution
induced damage and slight traces of woven roving. In the kenaf com imaging on the hybrid laminates.
posite, surface information includes impact-induced damage, air holes In addition to the qualitative comparison, a quantitative evaluation
and the mark created by ink, while subsurface information mainly with the signal-to-noise ratio (SNR) [8] was also carried out to charac
contains the impact area. terize the fusion performance. On each sample, we calculated SNR at
In Fig. 11, the 3D fusion imaging results on the jute fibre composite various depths and then obtained the averaged SNR. The results are
are shown. The left part shows the 3D reconstruction result with the shown in Table 2.
impact region and air holes on the surface. The surface veil made of glass It can be observed that, in both the qualitative and quantitative
fibre mesh can also be seen on the surface of the reconstruction result. comparison, the proposed technique as a non-invasive method led to a
On the right part, the 3D reconstruction model is cut at the chopped performance which was comparable to the performance obtained with
strand mat layer, and then compared with the X-ray CT inspection at a the X-ray CT. In general, the proposed fusion imaging method can be a
depth of 0.4 mm from the backside of the sample. It is worth noting that, potential substitute for ionizing X-ray analysis in nondestructive eval
other than the impact region, the chopped glass fibre distribution on the uation of plant fibre reinforced composites.
CT inspection result can also be mapped by the chopped strand mat layer
on the 3D fusion imaging. 5. Conclusions
The 3D reconstruction results of the kenaf fibre composite are shown
in Fig. 12. In the middle row of the figure, we list the images observed In this work, a multi-modality unsupervised spatial-spectral data
Fig. 9. THz-TDS analysis in the reflection mode using time-gated Fourier transform on the backside view of the jute composite sample.
7
J. Hu et al. Composites Science and Technology 226 (2022) 109526
Fig. 10. IRT results: (a)–(d) Jute composite sample; (e)–(h) Kenaf composite sample.
Fig. 11. 3D fusion imaging of the jute fibre composite and the comparison with the X-ray CT slice.
fusion technique was developed for plant fibre composite inspection different techniques. The fusion results are reconstructed in a 3D form to
using data from THz-TDS and IRT. The analysis process on raw data from show the through-depth exhibition of the materials.
THz-TDS modality includes time-domain and frequency-domain anal Due to the complex structure of the plant fibre composites, an
ysis to obtain images by layers owing to the time and spectral charac effective high-resolution imaging technique is still a key research topic
teristics on different materials, while information from different depths in the composite research area. The main contribution of this study is the
on the specimens is collected with the thermal diffusion process on proposal of a novel imaging technique based on THz-TDS and IRT data
optical excitation thermography. Furthermore, an unsupervised deep fusion. A spatial resolution of 0.5 mm has been achieved in the final
residual fusion algorithm is adapted to merge information from two output results. In addition, an unsupervised deep residual fusion
8
J. Hu et al. Composites Science and Technology 226 (2022) 109526
Fig. 12. 3D fusion imaging results on the kenaf fibre composite. Images in the middle row refer to the 3D model from different observation views. The top row
illustrates the comparison of the 3D reconstruction results cut from different layers and the X-ray CT imaging. The bottom row shows the cross-section and horizontal
section of the kenaf fibre composite.
algorithm introduced here can effectively merge the salient features This research was supported by the Natural Sciences and Engineering
from different imaging techniques. Furthermore, the 3D reconstruction Research Council (NSERC) Canada through Discovery and CREATE ‘oN
result offers a better understanding of the characteristics of the com DuTy!’ programs as well as Canada Research Chair in Multipolar
posites. The results and discussion present the 3D fusion imaging results Infrared Vision (MiViM)
with the X-ray CT inspection, which proves that the proposed 3D fusion This research was also supported by the >National Key Laboratory of
imaging technique offers a promising performance for the imaging of Science and Technology on Advanced Composites in Special Environ
different layers in such complex composite structures. ments of China through Open-end Research Fund program.
Jue Hu: Methodology, Software, Formal analysis, Investigation, Supplementary data to this article can be found online at https://doi.
Writing - Original draft preparation. Hai Zhang: Conceptualization, org/10.1016/j.compscitech.2022.109526.
Validation, Investigation, Writing - Reviewing and Editing, Supervision,
Project administration. Stefano Sfarra: Investigation, Resources, Visu References
alization, Writing - Review & Editing. Carlo Santulli: Resources, Writing-
Reviewing and Editing. Guiyun Tian: Writing-Reviewing and Editing, [1] I.M. De Rosa, C. Santulli, F. Sarasini, Acoustic emission for monitoring the
Supervision. Xavier Maldague: Writing-Reviewing and Editing, Super mechanical behaviour of natural fibre composites: a literature review, Compos Part
a Appl Sci Manuf 40 (2009) 1456–1469.
vision, Project administration, Funding acquisition.
[2] P. Cawley, Inspection of Composites-Ccurrent Status and Challenges, Proc 9th
ECNDT, Berlin, 2006.
[3] S. Sfarra, C. Ibarra-Castanedo, C. Santulli, A. Paoletti, D. Paoletti, F. Sarasini, et al.,
Falling weight impacted glass and basalt fibre woven composites inspected using
non-destructive techniques, Compos. B Eng. 45 (2013) 601–608.
9
J. Hu et al. Composites Science and Technology 226 (2022) 109526
[4] S. Gholizadeh, A review of non-destructive testing methods of composite materials, [15] Y. He, G. Tian, M. Pan, D. Chen, Impact evaluation in carbon fiber reinforced
Procedia Struct. Integr. 1 (2016) 50–57. plastic (CFRP) laminates using eddy current pulsed thermography, Compos. Struct.
[5] P. Duchene, S. Chaki, A. Ayadi, P. Krawczak, A review of non-destructive 109 (2014) 1–7.
techniques used for mechanical damage assessment in polymer composites, [16] C. Ibarra-Castanedo, X.P.V. Maldague, Infrared Thermography. Handb. Tech.
J. Mater. Sci. 53 (2018) 7915–7938. Diagnostics, Springer, 2013, pp. 175–220.
[6] M.A. Ali, R. Umer, K.A. Khan, W.J. Cantwell, Application of X-ray computed [17] T. Chady, P. Lopato, Testing of glass-fiber reinforced composite materials using
tomography for the virtual permeability prediction of fiber reinforcements for terahertz technique, Int. J. Appl. Electromagn. Mech. 33 (2010) 1599–1605.
liquid composite molding processes: a review, Compos. Sci. Technol. (2019) [18] J. Zhang, C. Shi, Y. Ma, X. Han, W. Li, T. Chang, et al., Spectroscopic study of
107828. terahertz reflection and transmission properties of carbon-fiber-reinforced plastic
[7] C. Jördens, S. Wietzke, M. Scheller, M. Koch, Investigation of the water absorption composites, Opt. Eng. 54 (2015) 54106.
in polyamide and wood plastic composite by terahertz time-domain spectroscopy, [19] J. Zhang, J. Chen, J. Wang, J. Lang, J. Zhang, Y. Shen, et al., Nondestructive
Polym. Test. 29 (2010) 209–215. evaluation of glass fiber honeycomb sandwich panels using reflective terahertz
[8] H. Zhang, S. Sfarra, A. Osman, K. Szielasko, C. Stumm, M. Genest, et al., An imaging, J Sandw Struct \& Mater 21 (2019) 1211–1223.
infrared-induced terahertz imaging modality for foreign object detection in a [20] Z. Liu, D.S. Forsyth, J.P. Komorowski, K. Hanasaki, T. Kirubarajan, Survey: state of
lightweight honeycomb composite structure, IEEE Trans. Ind. Inf. 14 (2018) the art in NDE data fusion techniques, IEEE Trans Instrum Meas 56 (2007)
5629–5636. 2435–2451.
[9] R. Yang, Y. He, Optically and non-optically excited thermography for composites: a [21] J. Wang, T. Xu, L. Zhang, T. Chang, J. Zhang, S. Yan, et al., Nondestructive damage
review, Infrared Phys. Technol. 75 (2016) 26–50. evaluation of composites based on terahertz and X-ray image fusion, NDT & E Int
[10] F. Ciampa, P. Mahmoodi, F. Pinto, M. Meo, Recent advances in active infrared (2022) 102616.
thermography for non-destructive testing of aerospace components, Sensors 18 [22] H.M. Akil, I.M. De Rosa, C. Santulli, F. Sarasini, Flexural behaviour of pultruded
(2018) 609. jute/glass and kenaf/glass hybrid composites monitored using acoustic emission,
[11] H. Zhang, F. Robitaille, C.U. Grosse, C. Ibarra-Castanedo, J.O. Martins, S. Sfarra, et Mater Sci Eng A 527 (2010) 2942–2950.
al., Optical excitation thermography for twill/plain weaves and stitched fabric dry [23] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, Proc.
carbon fibre preform inspection, Compos Part A Appl Sci Manuf 107 (2018) IEEE Conf. Comput. Vis. pattern Recognit. (2016) 770–778.
282–293. [24] H. Li, X.-J. Wu, Densefuse: a fusion approach to infrared and visible images, IEEE
[12] H. Zhang, P. Verberne, S.A. Meguid, C. Ibarra-Castanedo, X.P.V. Maldague, Trans Image Process 28 (2018) 2614–2623.
Autonomous high resolution inspection of kiss-bonds skins of carbon nanotube [25] Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, others, Image quality
reinforced nanocomposites using novel dynamic line-scan thermography approach, assessment: from error visibility to structural similarity, IEEE Trans Image Process
Compos. Sci. Technol. 192 (2020) 108111. 13 (2004) 600–612.
[13] H. Zhang, L. Yu, U. Hassler, H. Fernandes, M. Genest, F. Robitaille, et al., An [26] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, et al., Microsoft
experimental and analytical study of micro-laser line thermography on micro-sized Coco: Common Objects in Context, Eur. Conf. Comput. Vis., 2014, pp. 740–755.
flaws in stitched carbon fiber reinforced polymer composites, Compos. Sci. [27] A. Redo-Sanchez, B. Heshmat, A. Aghasi, S. Naqvi, M. Zhang, J. Romberg, et al.,
Technol. 126 (2016) 17–26. Terahertz time-gated spectral imaging for content extraction through layered
[14] X. Zhang, Y. He, T. Chady, G. Tian, J. Gao, H. Wang, et al., CFRP impact damage structures, Nat Commun 7 (2016) 1–7.
inspection based on manifold learning using ultrasonic induced thermography, [28] Cyndie Poulin, Meriam Triki, Karim Bousmaki, Alexandre Duhant, Hervé Louche &
IEEE Trans. Ind. Inf. 15 (2018) 2648–2659. Bertrand Wattrisse Pages. Terahertz thermometry system to measure temperature
in the thickness of a solid polymer. 37-53. Published online: 14 Aug 2017 https://
doi.org/10.1080/17686733.2017.1362791.
10