1330 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 68, NO.

4, APRIL 2021

Tongue Tumor Detection in Hyperspectral

Images Using Deep Learning
Semantic Segmentation
Stojan Trajanovski , Caifeng Shan , Senior Member, IEEE, Pim J. C. Weijtmans,
Susan G. Brouwer de Koning, and Theo J. M. Ruers

Abstract—Objective: The utilization of hyperspectral I. INTRODUCTION

imaging (HSI) in real-time tumor segmentation during a
ORLDWIDE, head and neck cancer accounts for more
surgery have recently received much attention, but it re-
mains a very challenging task. Methods: In this work,
we propose semantic segmentation methods, and com-
W than 650,000 cases annually [1]. Tongue cancer is a
specific case of head and neck cancer, and patients suffering
pare them with other relevant deep learning algorithms for from tumors in tongue tissue are generally treated by removing
tongue tumor segmentation. To the best of our knowledge,
this is the first work using deep learning semantic seg- the cancer surgically. Accurate segmentation of the tumor tissue
mentation for tumor detection in HSI data using channel from the healthy part is challenging; besides tumor tissue, the
selection, and accounting for more spatial tissue context, surgeon removes a margin of non-tumorous tissue around the
and global comparison between the prediction map, and tumor, which is called a resection margin in surgical oncol-
the annotation per sample. Results, and Conclusion: On ogy [2]. Moreover, literature stresses the importance of adequate
a clinical data set with tongue squamous cell carcinoma,
our best method obtains very strong results of average dice resection margins, as it is a powerful predictor of the 5-year
coefficient, and area under the ROC-curve of 0.891 ± 0.053, survival rate [3]–[5].
and 0.924 ± 0.036, respectively on the original spatial im- Smits et al. [6] conducted a clinical review regarding resection
age size. The results show that a very good performance margins in oral cancer surgery and found that 30% to 65% of
can be achieved even with a limited amount of data. We the resection margins in surgical results are inadequate. The
demonstrate that important information regarding tumor
decision is encoded in various channels, but some channel removal of an oral tumor and determining the resection margins
selection, and filtering is beneficial over the full spectra. are challenging procedures, since they rely on palpation. This
Moreover, we use both visual (VIS), and near-infrared (NIR) entails that the surgeon classifies tumor tissue based experience
spectrum, rather than commonly used only VIS spectrum; and on the sense of touch and sight. The accuracy of resection
although VIS spectrum is generally of higher significance, margins is assessed by pathologists and they analyze the re-
we demonstrate NIR spectrum is crucial for tumor captur-
ing in some cases. Significance: The HSI technology aug- moved tissue after a surgery. This is time-consuming process,
mented with accurate deep learning algorithms has a huge and it is less effective since it provides surgeons with information
potential to be a promising alternative to digital pathology after the surgery instead of during the procedure. From this
or a doctors’ supportive tool in real-time surgeries. perspective, there is an opportunity for technological solutions
Index Terms—Tumor segmentation, hyperspectral imag- to assist surgeons in the assessment of tumor tissue and to
ing, deep learning. determine adequate resection margins, by providing real-time
feedback during oral cancer surgery. Moreover, there are no
commonly accepted clinical standards or adopted techniques
Manuscript received April 16, 2020; revised July 29, 2020 and for this purpose.
September 14, 2020; accepted September 19, 2020. Date of publication Hyperspectral imaging (HSI), originally developed for remote
September 25, 2020; date of current version March 19, 2021. The
work of Caifeng Shan was supported by the National Natural Science sensing [7], has been successfully used in many fields such
Foundation of China under Grant 61972188. This research was done as food quality and safety, resource control, archaeology and
while Stojan Trajanovski and Caifeng Shan were with Philips Research, biomedicine [8]. With advancements in hardware and compu-
Eindhoven, The Netherlands. (Stojan Trajanovski and Caifeng Shan
contributed equally to this work.) (Corresponding author: Caifeng Shan.) tational power, it has become an emerging imaging modality
Stojan Trajanovski is with the Microsoft Inc. for medical applications [9]. HSI has the potential advantages
Caifeng Shan is with the Shandong University of Science, and Tech- of low cost, relatively simple hardware and ease of use. This
nology, Qingdao, China (e-mail: [email protected]).
Pim J. C. Weijtmans was with the Eindhoven University of makes HSI a candidate for intra-operative support of a surgeon.
Technology. Compared to regular RGB images it is challenging to process
Susan G. Brouwer de Koning and Theo J. M. Ruers are with the the HSI data due to the size of the data: hundreds of color bands
Netherlands Cancer Institute, Amsterdam, The Netherlands, and also
with the fMIRA Institute, University of Twente. for each pixel in a patient image with a large spatial size results
Digital Object Identifier 10.1109/TBME.2020.3026683 in large files with varying amounts of redundant information.

0018-9294 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: Indira Gandhi Delhi Technical University for Women. Downloaded on August 18,2023 at 08:51:04 UTC from IEEE Xplore. Restrictions apply.
 TRAJANOVSKI et al.: TONGUE TUMOR DETECTION IN HYPERSPECTRAL IMAGES USING DEEP LEARNING SEMANTIC SEGMENTATION 1331

Previous studies have mostly focused on tumor classification and second for correctly segmenting the tumor tissue
tasks [10]–[13] in HSI images. Fei et al. [11] have evaluated the in some cases, although VIS channels remain the most
use of HSI (450–900 nm) on specimen from patients with head significant ones for the performance on average.
and neck cancer. They achieved an area under the ROC-curve r We have proposed a novel channel selection/reduction
(AUC) of 0.94 for tumor classification with a linear discriminant technique, rather than standard reduction techniques that
analysis on a data set of 16 patients in which 10 were verified to eliminate pure channel information (e.g., PCA), and we
have squamous cell carcinoma (SCCa). However, their testing have demonstrated that there is a selected set of number
was done on specimens from the same patient as the classifier of channels that leads to optimal performance; meaning
was trained on. Lu et al. [14] and Halicek et al. [12] acquired that with smaller number of channels part of the signal
multiple specimen from 50 head and neck cancer patients with is lost and higher number of channels brings extra noise,
26 having squamous cell carcinoma (SCCa) in the same visual both contributing to reduced performance.
spectral range 450-900 nm. They [12] did a classification task The remainder of the paper is organized as follows. Details
with deep convolutional neural networks on 25 × 25 patches of the clinical dataset are given in Section II. The proposed
with leaving-one-patient-out cross-validation and reported ac- methods and the obtained results are provided in Section III
curacy of 77% for the SCCa group. Animal study on mice [15] and Section IV, respectively. We conclude in Section V.
with induced tumors was conducted by Ma et al. [10], achieving
an accuracy of 91.36% with convolutional neural networks in
a leave-one-out cross-validation also for a classification task. A II. CLINICAL DATASET
similar animal study on prostate cancer and on head and neck The clinical data was collected at the Netherlands Cancer
cancer in mice were conducted in [16] and [17]. Ravì et al. [18] Institute (NKI) - Antoni van Leeuwenhoek Hospital in Amster-
use random forest-based approaches on hyperspectral images for dam, the Netherlands. Hyperspectral images of specimens from
brain tumor segmentation. Other machine learning techniques, 14 patients undergoing surgery for the removal of squamous cell
minimum spanning trees [19], support vector machines [13], carcinoma of the tongue were acquired. All ethical guidelines
[20], k-nearest neighbors algorithm [21], [22], naïve [22] Bayes, required for ex vivo human studies were followed.
gaussian mixture models [23] and well-performing deep learn- Directly after resection, the specimen was brought to the
ing architectures [24] (e.g., inception [25]), have also be used pathology department, where the resection margins were inked
for hyperspectral images, mostly for classification tasks. In order according to the routine clinical workflow. The pathologist
to simplify and reduce the large number of channels, standard localized the tumor by palpation and subsequently cut the spec-
techniques such as tensor decomposition [26] or principal com- imen through the middle of the tumor. First, a RGB image
ponent analysis (PCA) [27] are applied. In all mentioned studies was taken from the cut surface with a regular photo camera.
the focus lies entirely on spectral information in the visible range Immediately after and without touching the specimen, the cut
around 450–900 nm. surface of the specimen was imaged with two hyperspectral
All of the above research works: (i) have utilize only the cameras (Spectral Imaging Ltd., Oulu, Finland), one operating
visible part of the spectra; (ii) have focused mainly on animal in the visible wavelength range (VIS) and the other in the
cases; (iii) have mostly utilize PCA or other well-established near-infrared wavelength range (NIR). The instrumentation with
techniques for channels/dimensionality reduction; or (iv) have the hyperspectral cameras and the supporting equipment for data
mostly focused on classification task as a global malignancy as- acquisition are shown in Fig. 1. After HSI imaging, the specimen
sessment per patient. However, the need of real-time and precise was subjected to further routine pathological processing, and the
intraoperative feedback requires accurate segmentation between pathologist annotated different tissue types on the histopathol-
the tumor and non-tumor in human tissues. In this work, we have ogy slide. Additional details on the data acquisition can be found
examined several structural, spectral and semantic segmentation in [32].
deep learning models (such as U-Net [28] variants) taking Both HSI cameras are push broom line scan system. The VIS
patches of images with all or predefined channels selection, camera captures 384 wavelengths in the 400 nm-1000 nm range
but assessing the global performance on a per patient/specimen with 1312 samples per line, for 612 lines. The NIR camera
base. Compared to the previous work, we have used much captures 256 wavelengths in the 900 nm-1700 nm range with
broader spectra utilizing both the visible (VIS) and near-infrared 320 samples per line, for 191 lines.
(NIR) spectral ranges [29] that has been rarely employed [30], In order to label the HSI data, a histopathological slide is taken
[31], especially in the context of deep learning methods. The from the surface that has been scanned. The slide is digitized and
contributions of the paper are the following: delineated to mark the tumor (red), healthy muscle (green) and
r With the best semantic segmentation method, we have epithelium & non-tumor tissue (blue). This is the first step shown
achieved competitive performance of average dice coef- in Fig. 2(a). From the delineation a mask is created. During
ficient and area under the ROC-curve of 0.891 ± 0.053 histopathological processing the specimen was deformed and to
and 0.924 ± 0.036, respectively, in the leave-patients-out correct this, a non-rigid registration algorithm is used. Obvious
cross-validation with on a clinical data set of 14 patients. matching points in the histopathological and RGB images were
r We have demonstrated that the (often omitted) near- visually selected. Using these points, the mask is transformed to
infrared spectra are crucial: first for spotting/classifying; match the RGB picture. This is depicted in Fig. 2(a) in middle

Authorized licensed use limited to: Indira Gandhi Delhi Technical University for Women. Downloaded on August 18,2023 at 08:51:04 UTC from IEEE Xplore. Restrictions apply.
 1332 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 68, NO. 4, APRIL 2021

III. METHODS
Having the annotation masks along with the hyper-cube al-
lows for supervised machine learning. With the annotated data
at hand, we consider two types of methods: 1) pixel-wise classi-
fication: spectral, structural, hybrid, 2) semantic segmentation;
and compare their performance. It is also important to stress that
we have a prediction for two classes: healthy1 and tumor tissues.

A. Channel Selection, Spectral, Structural and Hybrid

Approaches
Using small patches spanning all bands the spectral infor-
mation can be captured. By selecting bigger patches, structural
information becomes available. By combining both inputs, the
full spectral and some structural information are available for
the network.
1) Channel Selection: The channel selection gives the most
important channels of a dataset. This step is also important as
it ranks/separates more important channels containing signal
from the less important ones containing noise, which improves
Fig. 1. (a) A table shifts tissue under the camera. Lines are recorded the performance. It is used in both pixel-wise and semantic
and assembled into a data volume. (b) The imaging system used to segmentation. The selection method is based on l1 regularization
collect the HSI data, where a tissue is placed on the table in the bottom.
concept. By adding an additional layer between the input and the
first layer of the model, it is possible to examine what weights are
given to the channels. The patches we use are three-dimensional
row as transformation T1. The point-selection [33] is done again and we want to find a subset of channels, so a two-dimensional
on the RGB and HSI data to derive transformation T2, which depth-wise convolution [35] is used, with a 1 × 1 kernel and 1
is used to transform the mask again to match the HSI data. The × 1 stride (see Fig. 3(a)). The weights of this selection layer are
VIS and NIR datasets contain the same patients, therefore the constrained to range from 0 to 1, to disable and enable channels
data cubes could be combined into a broad spectrum ranging respectively. To encourage channels selection, an incentive is
from 400 nm to 1700 nm. During the registration process, the added. Without this, all weights would simply be 1 and there
points that were used to determine the transformations were would be no effective change. l1 weight regularization is con-
stored, which could then be used to align the VIS and NIR data sidered for this incentive. With l1 the regularization parameter,
cubes. Transformation T2 (NIR data) was inversed to transform wi is the layer weight of matching channel i and n the number
the HSI NIR to the RGB shape, and subsequently transform of channels. The result J is added to the loss during training,
T2 (VIS data) was applied to the VIS data. The NIR data so there is a penalty on having non-zero weights. However, now
had a lower resolution, therefore it was up sampled during the all weights will go to zero, some faster than others. Ideally,
transformation. At this point, the data volumes had the same our aim is that the important channels go to one and all others
shape and could be concatenated along the spectral axis. Due to go to zero. The formula is adapted by applying a function to
the overlap in spectral range, which was approximately 120 nm, the weights before summing them and this is added to the

half of 120 nm from VIS cube and the other half from NIR loss l1 ni=1 (1 − ( |wpi | − 1)2 ). This function is a second degree
were removed. Because the first and last bands of the data polynomial and has maximum 1 at p. The value of p is set at.
cube were noisy, bands from both sets were removed instead 99 creating a small pocket at (0.99, 1) where the loss slightly
of removing from only the NIR or VIS cube. The reflectance as increases while the weight is reduced from 1 to. 99. After that
a function of the wavelength depicted in Fig. 2(b) demonstrates small pocket, the loss will decrease until the weight is zero as
that a merger/stack of VIS & NIR (top and bottom subfigures demonstrated in Fig. 3(b). More details of the selected channels
if summed up) well captures the full spectra. See [32] for more are given in Section IV.
details on the data registration and preprocessing. 2) Spectral Neural Network Architectures: To exploit the
The number of pixels for the two datasets are shown in Tables I full extent of the spectral information, a network is designed for
and II. There is a small difference between the VIS and NIR sets, using 1 × 1 patches including the full spectrum of the HSI data.
despite they are made with the same tissue and annotations. This Five fully connected (FC) layers were used. The first layer has
could be explained in several ways. For instance, the tissue could as many units as wavelengths in the input data (640 for stacked,
have moved between measurements, causing a deformation of where the contributions of VIS and NIR are 384 and 256,
the tissue, or that the tissue was imaged from a different angle. respectively). The last layer is the output layer, and therefore has
Another explanation might be that the difference is a result of an
error while transforming the histopathological image to match 1 As mentioned earlier healthy tissue accounts for epithelium, muscle and
the HSI data. other non-tumor tissue

Authorized licensed use limited to: Indira Gandhi Delhi Technical University for Women. Downloaded on August 18,2023 at 08:51:04 UTC from IEEE Xplore. Restrictions apply.
 TRAJANOVSKI et al.: TONGUE TUMOR DETECTION IN HYPERSPECTRAL IMAGES USING DEEP LEARNING SEMANTIC SEGMENTATION 1333

Fig. 2. (a) Annotation and registration of the hyperspectral data: tumor (red), healthy tongue muscle (green), and healthy epithelium or other
non-tumor tissue (blue). (b) Reflectance of the acquired data for VIS and NIR. (Blue curves, labeled as “unknown,” represent epithelium or other
non-tumor tissue and it has been used with term “non-tumor,” interchangeably.)

TABLE I
THE NUMBER OF PIXELS PER PATIENT IN THE SPECIM-VIS DATASET. PATIENT 30 WAS THE ONLY PATIENT WITHOUT NON-TUMOR TISSUE IN THE
ANNOTATION. THE TERM NON-TUMOR REFERS TO EPITHELIUM OR OTHER ORDINARY NON-TUMOR TISSUE

TABLE II
THE NUMBER OF PIXELS PER PATIENT FOR THE SPECIM-NIR DATASET

Authorized licensed use limited to: Indira Gandhi Delhi Technical University for Women. Downloaded on August 18,2023 at 08:51:04 UTC from IEEE Xplore. Restrictions apply.
 1334 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 68, NO. 4, APRIL 2021

Fig. 3. (a) Depth-wise convolutions outcome. (b) Behavior for the channel selection loss function.

only two units. The hidden layers in between had 320, 160 and is designed, using a 1 × 1 patch that covers all wavelengths to
80 units, and those layers were followed by batch normalization incorporate spectral features and a bigger patch with selected
to stabilize the training process. The neural network is visualized channels for the structural information. The network is visu-
in Fig. 4(a). Alternatively, instead of fully connected layers, alized in Fig. 4(c). A spectral patch is selected from the data
convolutional neural networks [36] (i.e. shareable weights and volume, and fed into four fully connected layers. The first of
translation-invariant filters) have been also explored, but these those layers has the same number of units as the amount of
resulted in slightly worse or comparable performance. The next wavelengths in the spectral sample. The following hidden layers
thing worth considering is expanding to the spatial context of have 256, 128 and 64 units respectively. Additionally, a structural
the data cube. However, taking more pixels in a combination patch is selected from the data using the channel ranking as
to having all channels is becoming computationally challenging discussed in the channel selection section. The patch is flattened
(e.g., significantly slower or memory demanding) and it also and followed by four hidden dense layers with 512, 256, 126
brings extra redundancy and possible conflicting context in some and 64 units. After concatenation of the spectral and structural
of the channels with the annotation. Some channel importance branches, four additional dense layers follow with 64, 32, 16 and
prioritization and filtering are in order as elaborated in the 2 output units. After every layer, batch normalization is applied
following sections. to stabilize the training process.
3) Structural Neural Network Architectures: By increas-
ing the patch size, morphological features in the HSI data can
be used to classify the tissue. With the most important channels B. Semantic Segmentation Based on U-Net Neural
known either reduced by the channels selection (as described Network Variants
earlier) or alternatively a principle component analysis (PCA), a Based on the hyper-cubes and corresponding annotations,
model that focuses on structure can be trained. Data with a high we create HSI input and annotation patches with wider spatial
resolution will benefit the most from this approach, as it will context. The reason of using patches, with a fixed size per
contain more structural information compared to low resolution experiment, is two-fold: (i) we have a limited data from 14
data. With 45 VIS channels selected and spatial patch of 21 × 21, patients and in this way, we create a train and validation cohort of
input data cubes are defined. The number of VIS channels used patches that can lead to reasonable results and (ii) in the semantic
have been selected empirically. We conducted experiments with segmentation, the spatial dimensions (length and width) of the
different number of channels, and found that (i) initially the HSI data cubes are different, thus some patching is needed in
performance improved when more channels were included; (ii) order to have a unique input data shape for any neural network.
the performance did not get any better when using more than Moreover, having convolutional layers at the initial layer still
45 VIS channels [37, Page 19, Figure 16]. This is followed by allows to examine the test performance on the full spatial size
flattening which results in 19,845 input units. The network is for different patients with one predict forward pass in the neural
similar to the spectral model, with fully connected hidden layers network without changing it at all. We use leave-patients-out
and it is shown in Fig. 4(b). The difference lies in the input cross validation, thus there are never patches from the same
shape and the first fully connected layer. Instead of matching patient in both train and validation sets.
the number of units of the flattened channel-filtered input, 1024 In our approach, we use 100 random patches of size 256 × 256
units are used. This is done to reduce the number of parameters for each patient with the central pixels being 50/50 tumor
in the model. As in the spectral neural networks, convolutional and healthy classes and appropriate channel selection of the
neural blocks are also tried, but these result in worse or similar most significant channels as explained in Section III-A1. This
performance over the fully connected layers. means that we do 7-fold cross-validation with 1200 patches
4) Hybrid Spectral & Structural Neural Network Archi- (12 patients) of size 256 × 256× #channels for training and
tectures: To utilize both spectral and structural contexts, we 200 patches (2 patients) for validation. (We have conducted
derive a dual-stream spectral and structural model. The model additional experiments having standard train/validation/hold out

Authorized licensed use limited to: Indira Gandhi Delhi Technical University for Women. Downloaded on August 18,2023 at 08:51:04 UTC from IEEE Xplore. Restrictions apply.
 TRAJANOVSKI et al.: TONGUE TUMOR DETECTION IN HYPERSPECTRAL IMAGES USING DEEP LEARNING SEMANTIC SEGMENTATION 1335

Fig. 4. Benchmark neural networks, visualized using [34]. (a) Spectral neural network. (b) Structural neural network with channels filtering.
(c) Dual-stream spectral and structural neural network.

set data partition or having nested cross-validation – thus leaving Although, this loss could be promising, it also brings two addi-
less patients for reporting results, but the results are similar tional parameters that have to be tuned. With these input patches
to those with this partition scheme.) Each patch contains both and annotations (size 256 × 256 × 2 tumor/no tumor), we train
tumor and healthy tissue and in fact, covers most of the spatial a U-Net neural network [28] variant with batch normalization.
part of each HSI cube. It depends on the tissue, but in each The architecture is visualized in Fig. 5(a).
patch for training both healthy and tumor tissues are decently In order to separately examine the contribution of the VIS and
represented. To achieve better generalization, we apply standard NIR parts of the data cube, we derive dual-stream U-Net based
data-augmentation techniques such as rotation and flipping of neural network. As visualized in Fig. 5(b), there is a separate
the patches. As a loss function dice coefficient is used for for stream/branch of the VIS and NIR accepting the corresponding
training and validation that compares the overlap of the full-size parts of the data cube, followed by appropriate averaging or
prediction map with the annotation map. Additionally, we have maximization for the final prediction.
experimented with alternative losses like the focal loss, but 1) Details on the Training Process: We have experi-
comparable (or worse) and less stable results were obtained. mented with different batch size and, as expected, reasonable

Authorized licensed use limited to: Indira Gandhi Delhi Technical University for Women. Downloaded on August 18,2023 at 08:51:04 UTC from IEEE Xplore. Restrictions apply.
 1336 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 68, NO. 4, APRIL 2021

Fig. 5. U-Net based neural network architectures for HSI data. The visualization is drawn using PlotNeuralNet software [34] (https://github.
com/HarisIqbal88/PlotNeuralNethttps://github.com/HarisIqbal88/PlotNeuralNet). (a) Singe-stream U-Net for stacked (VIS and NIR) patches. (b)
Dual-stream U-Net based with separate VIS and NIR streams.

batch size of 16 or 32 leads to more stable validation curves

compared to smaller batch size (e.g., batch size of 4). This is
demonstrated in the Fig. 6 where the train and validation learning
curves per epochs are show for two experiments that only differ
in the batch size. We have experimented with different loss
optimizers like Adam [38], standard SGD (Stochastic gradient
descent), RMSprop [39] etc., but it seems the performance is not
affected much by this choice, thus we decided for the former in
most of the experiments. We have experimented with different
values of the learning rate and values in the range 10−4 or smaller
leads to slow validation loss improvement per epoch and it might
require many epochs to achieve a decent validation loss. On the
other hand, learning rates in the range of 10−3 or bigger leads to
having a perfect train loss in the earlier epoch that often leads to
overfitting in some of the folds. Therefore, a moderate learning Fig. 6. The influence of the batch size on the training.

Authorized licensed use limited to: Indira Gandhi Delhi Technical University for Women. Downloaded on August 18,2023 at 08:51:04 UTC from IEEE Xplore. Restrictions apply.
 TRAJANOVSKI et al.: TONGUE TUMOR DETECTION IN HYPERSPECTRAL IMAGES USING DEEP LEARNING SEMANTIC SEGMENTATION 1337

TABLE III
RESULTS OF THE EXPERIMENTS WITH DIFFERENT DEEP NEURAL NETWORKS

to have this selection done a-priory (and not repeating it over

again for data that might not be present anymore); and it is
crucial for deployment in practice as band selection is time
consuming so inference phase is more feasible in this way. A
general conclusion is that the very initial and very end bands are
the least significant; while various bands are quite important in
the middle of the range as well as reasonably at the beginning
and close to the end in the range. After the important channels
are selected, we can proceed with smaller size cube patch input
with noise channels filtered for the proposed neural networks.
Fig. 7. Channel selection results.
B. Performance Results and Comparison
We evaluate the results of the proposed semantic segmentation
−4
rate of 0.5 × 10 is often the best choice although this can vary method and compare it with the spectral, structural, dual-stream
per fold. pixel-wise approaches proposed. We also want to stress that
we have evaluated several other alternatives for: U-Net depth,
IV. RESULTS AND DISCUSSION the number of input channels, regularization techniques, hyper-
parameter optimization, but due to space limitation, we show
A. Channel Selection the best and most representative results in Table III as the
The proposed method starts with a selection of important others are worse or with comparable performance and properties.
channels. Fig. 7 (left) shows the weights of the selection layer as We use several performance metrics like the dice coefficient,
the training progresses. The choice of channels does not change the area under the ROC-curve (AUC), accuracy, sensitivity,
over the epochs, but does become more distinct as shown by the and specificity for evaluating the validation results. Using all
increasing weights of selected channels. Fig. 7 (right) shows these performance metrics allows to evaluate the performance
the weights at epoch 30 and illustrates how selected channels reasonably, even when the tumor/healthy presence in the data is
are clustered together. It also shows that no channels are given imbalanced without the need to choose a classification threshold.
the maximum weight in all folds indicating that selected chan- The validation results of these metrics are given in Table III,2
nels are given low weights in some folds. From Fig. 7 (right), we showing that semantic segmentation by HSI U-Net variant with
can see that important channels lie both in the VIS and NIR part 128 input channels that can capture both spectral and spatial
of the spectra, and it will be demonstrated later (see Fig. 10) that
both are crucial for spotting the tumor. There is a difference in the 2 (i) “HSI U-Net Stacked Xch” refers to all 640 channels (384 VIS and 256

selection band for a single patient and the conclusion on which NIR) are stacked in a cube, channel importance selection algorithm is applied and
the X most significant are selected; (ii) “HSI U-Net Stacked X + Y ch central”
channels are more important is made based on all patients thus means the central X VIS channels are selected (from 384 in total), the central
those having recognized as the most important in the majority Y NIR channels are selected (from 256) and then they are stacked together in a
of the patients being ranked higher as a cumulative contribution HSI cube; (iii) “HSI U-Net Stacked X + Y ch most important” means channel
importance is separately applied for VIS and NIR, such that the most important
across all patients. This is the only (i) fair and general approach, X VIS channels are selected, the most important Y NIR channels are selected
(ii) it is important if we bring additional data from a new patient and then they are stacked together in a HSI cube.

Authorized licensed use limited to: Indira Gandhi Delhi Technical University for Women. Downloaded on August 18,2023 at 08:51:04 UTC from IEEE Xplore. Restrictions apply.
 1338 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 68, NO. 4, APRIL 2021

Fig. 8. Validation performance represented per patient by the dice coefficient, AUC, accuracy, sensitivity and specificity. On the x-axis are the
patient ID. (The ID values are not of particular meaning or importance.)

Fig. 9. A VIS (1st column) and NIR (2nd column) HSI bands, the RGB representations (3rd column), ground truth (4th column; green is for healthy,
red is for tumor tissue) and hard predictions of our method (5th column; predicted pixels are green for values smaller than 0.5, or red for values at
least 0.5) for 4 patients. (For the two HSI slices (first two columns), standard viridis color map has been used, see e.g., [40].)

Authorized licensed use limited to: Indira Gandhi Delhi Technical University for Women. Downloaded on August 18,2023 at 08:51:04 UTC from IEEE Xplore. Restrictions apply.
 TRAJANOVSKI et al.: TONGUE TUMOR DETECTION IN HYPERSPECTRAL IMAGES USING DEEP LEARNING SEMANTIC SEGMENTATION 1339

Fig. 10. An exceptional case. Ground truth (green for healthy, red for tumor) and hard predictions (threshold 0.5) by VIS and NIR U-Net.

aspects has the best performance (in bold). It is important to In this figure, for this particular patient, we can see that by using
stress that by both using less and more than 128 channels NIR channels, the algorithm captures the tumor tissue, while
the performance is degraded (Table III) as either some tumor this is not a case by using VIS channels only. Although there
information is ignored or some noisy channels dominate in the are such cases as in Fig. 10, which highlights the importance of
decision, respectively. It is also interesting to mention that the the NIR channels, in the majority of the cases VIS channels are
proposed HSI U-Net variant still works well, although starting those contributing the most in spotting the tumor tissue.
with less initial filters compared to the input channels, opposite
to standard U-Net for RGB images (where 3 channels are sig- V. CONCLUSION
nificantly less than the initial number of filters), thus realizing
Real-time tumor segmentation during surgery is an impor-
an immediate pooling/selection effect. With the best performing
tant and challenging task. On the other hand, recent hyper-
HSI U-Net, we achieve a mean dice coefficient and AUC vali-
spectral camera developments offer additional possibility for
dation scores of 0.891 ± 0.053 and 0.924 ± 0.036, respectively.
better quality and more insights that can lead to more accurate
In Fig. 8, the performance on a per-patient base is shown,
segmentation. Several techniques have been proposed in the
demonstrating the algorithm works well (with some reasonable
past, mostly based on standard machine learning and pixel-wise
variance) in all cases. In addition, it is interesting to mention
approaches. To the best of our knowledge, this is the first work
U-Net experiments with RGB patches show significantly worse
using deep learning U-Net [28] semantic segmentation for tumor
performance (around 0.8 for both dice and AUC) than those with
detection and trainable channels selection for both NIR and
multiple HSI channels, which suggests that the HSI channels
VIS HSI spectra. The proposed semantic segmentation shows
richness is important for improved precision. On the other hand,
superior performance over the other alternatives (average dice
Halicek et al. [41] conducted research focusing on this aspect
coefficient and area under the ROC-curve of 0.891 ± 0.053 and
and found HSI did not provide significant advantages over the
0.924 ± 0.036, respectively). We also demonstrate that channel
identification of tumor margins in ex-vivo tissue compared to
selection and filtering is beneficial over the full spectra for
RGB imagery. Therefore, we admit more in-depth research is
achieving better performance, that both VIS and NIR channels
needed in this direction. To better illustrate the accuracy of the
are important and very good performance can be achieved even
prediction compared to the ground truth labels, we depicted the
with a limited amount of data. Moreover, we have shown that
hard prediction map and the ground truth maps for 4 patients in
the often omitted near-infrared (NIR) spectra is crucial for
Fig. 9. It is also important to mention that although the network
detecting the tumor in some cases. The hyperspectral cameras
is trained on fixed patches, the reported test results are based on
have been demonstrated to be powerful tools in many fields;
the original HSI dimension (that is different for each HSI image)
and the technology, augmented with accurate algorithms, has a
directly using the obtained model, because the convolutional
huge potential in biomedical engineering and medicine and with
kernels of the initial layers can take arbitrary input size with a
promising results available could be a doctors’ supportive tool
single pass. We have done experiments with U-Net networks of
for real-time surgeries and an alternative to digital pathology.
different number of layers, different units per layer, different
dropout values or other regularizers, but the performance is
worse or comparable to those reported in Table III. REFERENCES
[1] F. Bray et al., “Global cancer statistics 2018: Globocan estimates of
incidence and mortality worldwide for 36 cancers in 185 countries,” CA:
C. The Importance of VIS and NIR Channels A Cancer J. Clin., vol. 68, no. 6, pp. 394–424, 2018.
[2] E. Healthcare, “Tumor resection,” [Online]. Available: https://www.
From the results in Table III, we can see VIS channels are emoryhealthcare.org/orthopedic-oncology/tumor-resection.htm% l
generally more significant that NIR for tumor segmentation. [3] T. Yee Chen et al., “The clinical significance of pathological findings
in surgically resected margins of the primary tumor in head and neck
However, there are rare cases where the NIR channels are the carcinoma,” Int. J. Radiat. Oncol. Biol. Phys., vol. 13, no. 6, pp. 833–837,
most crucial and decisive for tumor segmentation, while the 1987.
tumor is not spotted by VIS channels. The importance of the NIR [4] T. R. Loree and E. W. Strong, “Significance of positive margins in
oral cavity squamous carcinoma,” The Amer. J. Surger., vol. 160, no. 4,
spectral alone or even better in a combination with VIR in a dual pp. 410–414, 1990, papers of the Society of Head and Neck Surgeons
stream U-Nets architecture (Fig. 5(b)) is visualized in Fig. 10. presented at the 36th Annual Meeting.

Authorized licensed use limited to: Indira Gandhi Delhi Technical University for Women. Downloaded on August 18,2023 at 08:51:04 UTC from IEEE Xplore. Restrictions apply.
 1340 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 68, NO. 4, APRIL 2021

[5] A. Binahmed et al., “The clinical significance of the positive surgical [25] M. Halicek et al., “Hyperspectral imaging of head and neck squamous
margin in oral cancer,” Oral Oncol., vol. 43, no. 8, pp. 780–784, 2007. cell carcinoma for cancer margin detection in surgical specimens from
[6] R. W. Smits et al., “Resection margins in oral cancer surgery: Room for 102 patients using deep learning,” Cancers, vol. 11, no. 9, 2019, Art. no.
improvement,” Head Neck, vol. 38, no. S1, pp. E2197–E2203, 2016. 1367.
[7] A. F. Goetz, “Three decades of hyperspectral remote sensing of the earth: [26] G. Lu et al., “Spectral-spatial classification for noninvasive cancer de-
A personal view,” Remote Sens. Environ., vol. 113, pp. S5–S16, Sep. 2009. tection using hyperspectral imaging,” J. Biomed. Opt., vol. 19, no. 10,
[8] G. Lu and B. Fei, “Medical hyperspectral imaging: A review,” J. Biomed. pp. 1–18, 2014.
Opt., vol. 19, no. 1, 2014, Art. no. 010901. [27] H. Chung et al., “Superpixel-based spectral classification for the detection
[9] M. A. Calin et al., “Hyperspectral imaging in the medical field: Present of head and neck cancer with hyperspectral imaging,” in Proc. Med. Imag.:
and future,” Appl. Spectrosc. Rev., vol. 49, no. 6, pp. 435–447, 2014. Biomed. Appl. Mol., Struct., Functional Imag., B. Gimi and A. Krol, Eds.,
[10] L. Ma et al., “Deep learning based classification for head and neck cancer vol. 9788, International Society for Optics and Photonics. SPIE, 2016,
detection with hyperspectral imaging in an animal model,” in Proc. SPIE, pp. 260–267.
2017, Art. no. 10137. [28] O. Ronneberger et al., “U-net: Convolutional networks for biomedical
[11] B. Fei et al., “Label-free reflectance hyperspectral imaging for tumor image segmentation,” in Proc. Med. Image Comput. Comput.-Assisted
margin assessment: A pilot study on surgical specimens of cancer patients,” Intervention, ser. LNCS, vol. 9351. Springer, 2015, pp. 234–241.
J. Biomed. Opt., vol. 22, no. 8, pp. 1–7, 2017. [29] S. G. Brouwer de Koning et al., “Near infrared hyperspectral imaging
[12] M. Halicek et al., “Deep convolutional neural networks for classifying to evaluate tongue tumor resection margins intraoperatively (Conference
head and neck cancer using hyperspectral imaging,” J. Biomed. Opt., vol. Presentation),” in Proc. Opt. Imag., Therapeutics, Adv. Technol. Head Neck
22, no. 6, 2017. Surgery Otolaryngol., Art. no. 104690G, doi: 10.1117/12.2289166.
[13] E. Kho et al., “Broadband hyperspectral imaging for breast tumor detection [30] E. Kho et al., “Hyperspectral imaging for resection margin assessment
using spectral and spatial information,” Biomed. Opt. Express, vol. 10, during cancer surgery,” Clin. Cancer Res., 2019.
no. 9, pp. 4496–4515, Sep. 2019. [31] H. Akbari et al., “Cancer detection using infrared hyperspectral imaging,”
[14] G. Lu et al., “Detection of head and neck cancer in surgical specimens Cancer Sci., vol. 102, no. 4, pp. 852–857, 2011.
using quantitative hyperspectral imaging,” Clin. Cancer Res.: An Official [32] S. G. Brouwer de Koning et al., “Toward assessment of resection margins
J. Amer. Assoc. Cancer Res., vol. 23, no. 18, pp. 5426–5436, 2017. using hyperspectral diffuse reflection imaging (400–1,700 nm) during
[15] G. Lu et al., “Detection and delineation of squamous neoplasia with tongue cancer surgery,” Lasers Surgery Med., vol. 52, no. 6, pp. 496–502.
hyperspectral imaging in a mouse model of tongue carcinogenesis,” J. [33] F. L. Bookstein, “Principal warps: Thin-plate splines and the decomposi-
Biophoton., vol. 11, no. 3, 2018, Art. no. e201700078. tion of deformations,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 11,
[16] H. Akbari et al., “Hyperspectral imaging and quantitative analysis for no. 6, pp. 567–585, Jun. 1989.
prostate cancer detection,” J. Biomed. Opt., vol. 17, no. 7, 2012. [34] H. Iqbal, “Harisiqbal88/plotneuralnet v1.0.0,” Dec. 2018. [Online]. Avail-
[17] L. Ma et al., “Adaptive deep learning for head and neck cancer detection able: https://doi.org/10.5281/zenodo.2526396
using hyperspectral imaging,” Vis. Comput. Industry, Biomed., Art, vol. 2, [35] F. Chollet, “Xception: Deep learning with depthwise separable convo-
no. 1, Nov. 2019, Art. no. 18. lutions,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017,
[18] D. Ravì et al., “Manifold embedding and semantic segmentation for pp. 1800–1807.
intraoperative guidance with hyperspectral brain imaging,” IEEE Trans. [36] I. Goodfellow et al., Deep Learn. Cambridge, MA, USA: MIT Press, 2016,
Med. Imag., vol. 36, no. 9, pp. 1845–1857, Sep. 2017. [Online]. Available: http://www.deeplearningbook.org
[19] R. Pike et al., “A minimum spanning forest-based method for noninvasive [37] P. Weijtmans, “From hyperspectral data cube to prediction with deep
cancer detection with hyperspectral imaging,” IEEE Trans. Biomed. Eng., learning,” Master’s thesis, Eindhiven Univ. Technol. (TU/e), Eindhoven,
vol. 63, no. 3, pp. 653–663, Mar. 2016. The Netherlands, Mar./Apr. 2019. [Online]. Available: https://research.tue.
[20] H. Fabelo et al., “Spatio-spectral classification of hyperspectral images for nl/nl/studentTheses/from-hyperspectral-data-cube-to-tumor-prediction
brain cancer detection during surgical operations,” PLOS ONE, vol. 13, [38] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in
no. 3, pp. 1–27, 03 2018. Proc. (Int. Conf. Learn. Representations), ser. ICLR ’15, San Diego, CA,
[21] G. Florimbi et al., “Accelerating the k-nearest neighbors filtering algo- USA, 2015.
rithm to optimize the real-time classification of human brain tumor in [39] T. Tieleman and G. Hinton, “Lecture 6.5—RmsProp: Divide the gradient
hyperspectral images,” Sensors, vol. 18, no. 7, 2018. by a running average of its recent magnitude,” COURSERA: Neural
[22] G. Lu et al., “Hyperspectral imaging of neoplastic progression in a mouse Networks for Machine Learning, 2012.
model of oral carcinogenesis,” in Proc. Med. Imag.: Biomed. Appl. Mol., [40] Matplotlib, “Perceptually uniform sequential colormaps,” [Online].
Structural, Functional Imag., B. Gimi and A. Krol, Eds., vol. 9788, Available: https://matplotlib.org/3.2.1/tutorials/colors/colormaps.
International Society for Optics and Photonics. SPIE, 2016, pp. 252–259. html#miscellane% ous
[23] B. Regeling et al., “Hyperspectral imaging using flexible endoscopy for [41] M. Halicek et al., “Tumor detection of the thyroid and salivary glands
laryngeal cancer detection,” Sensors, vol. 16, no. 8, 2016, Art. no. 1288. using hyperspectral imaging and deep learning,” Biomed. Opt. Express,
[24] S. Ortega et al., “Detecting brain tumor in pathological slides using vol. 11, no. 3, pp. 1383–1400, Mar. 2020.
hyperspectral imaging,” Biomed. Opt. Express, vol. 9, no. 2, pp. 818–831,
Feb. 2018.

Authorized licensed use limited to: Indira Gandhi Delhi Technical University for Women. Downloaded on August 18,2023 at 08:51:04 UTC from IEEE Xplore. Restrictions apply.