National University Of Computer and Emerging

Sciences

Skin Lesion Detection and Classification using Deep

Learning Techniques
Authors:
Zara Ahmad 15L-4351
Fatima Tahira 15L-4169
Hafiz Ahmad Hassan 15L-4359
Muhammad Adeel Khan 15L-4006

Supervisor: Mr. Bismillah Jan

B.S. Computer Science

Final Year Project: 2018

Department of Computer Science

FAST NU, Lahore, Pakistan
 2

Anti-Plagiarism Declaration

This is to declare that the above publication produced under the:

Title: Skin Lesion Detection and Classification using Deep Learning Techniques
is the sole contribution of the author(s) and no part hereof has been reproduced on as it is basis
(cut and paste) which can be considered as Plagiarism. All referenced parts have been used to
argue the idea and have been cited properly. I/We will be responsible and liable for any
consequence if violation of this declaration is determined.

Student 1
Name: Zara Ahmad

Signature: ___________________

Student 2
Name: Fatima Tahira

Signature: ___________________

Student 3
Name: Muhammad Adeel Khan

Signature: ___________________

Student 4
Name: Hafiz Ahmad Hassan

Signature: ___________________

Date: 30/10/2018
Skin Lesion Detection and Classification using Deep Learning Techniques 3

Table of Contents
Anti-Plagiarism Declaration .................................................................................................. 2
LIST OF TABLES ..................................................................................................................... 5
LIST OF FIGURES ................................................................................................................... 6
ABSTRACT ............................................................................................................................... 7
1. CHAPTER 1: INTRODUCTION ...................................................................................... 8
1.1 Goals and Objectives ................................................................................................... 8
1.2 Project Scope ............................................................................................................... 8
1.3 Work done so far ......................................................................................................... 8
1.4 Elaboration of Problem ............................................................................................. 10
1.5 Conclusion................................................................................................................. 11
1.6 Outline ....................................................................................................................... 11
2. CHAPTER 2: LITERATURE REVIEW ......................................................................... 12
2.1 Introduction ............................................................................................................... 12
2.2 Detailed Literature Review ....................................................................................... 12
2.2.1 Definitions.......................................................................................................... 12
2.2.2 Classification...................................................................................................... 13
2.2.3 Summary of Papers ............................................................................................ 13
2.2.4 Content Representation ...................................................................................... 18
2.2.5 Evaluation Measures .......................................................................................... 18
2.3 Literature Review Summary ..................................................................................... 20
2.4 Conclusion................................................................................................................. 23
3. CHAPTER 3: REQUIREMENTS AND DESIGN .......................................................... 24
3.1 Functional Requirements........................................................................................... 24
3.1.1 Data Preparation and Data Pre-Processing Requirements ................................. 24
3.1.2 Data Set Splitting Requirements ........................................................................ 24
3.1.3 Modelling Requirements:................................................................................... 24
3.1.4 Data Evaluation and Parameter Tuning Requirements: ..................................... 24
3.1.5 Data visualization............................................................................................... 24
3.1.6 Analysis and Validation Requirements .............................................................. 24
3.2 Non-Functional Requirements .................................................................................. 25
3.2.1 Usability ............................................................................................................. 25
3.2.2 Reliability........................................................................................................... 25
3.2.3 Extensibility ....................................................................................................... 25
3.2.4 Compatibility ..................................................................................................... 25
3.2.5 System Visualization ......................................................................................... 25
3.3 Hardware Requirements ............................................................................................ 25
3.4 Software Requirements ............................................................................................. 25
3.5 System Architecture .................................................................................................. 26
3.6 Use Case .................................................................................................................... 26
3.7 Graphical User Interface (GUI)................................................................................. 27
3.8 Activity Diagram ....................................................................................................... 28
3.8.1 Classification Problem ....................................................................................... 28
3.8.2 Detection Problem ............................................................................................. 29
4. CHAPTER 4: IMPLEMENTATION .............................................................................. 30
4.1 Classification Problem .............................................................................................. 30
4.1.1 Initial Implementation ........................................................................................ 30
4.1.2 Final Implementation ......................................................................................... 31
4.2 Detection Problem ..................................................................................................... 32
 4

5. CHAPTER 5: EXPERIMENTAL RESULTS AND ANALYSIS ................................... 33

5.1 Classification Problem .............................................................................................. 33
5.1.1 Overview of Model A and Model B implementation ........................................ 33
5.1.2 Evaluation Results ............................................................................................. 34
5.2 Detection Problem ..................................................................................................... 35
5.2.1 Evaluation Measures .......................................................................................... 35
5.2.2 Evaluation Results ............................................................................................. 36
6. CHAPTER 6: CONCLUSION ........................................................................................ 37
6.1 Results and Challenges – Classification Problem ..................................................... 37
6.2 Results and Challenges – Detection Problem ........................................................... 37
7. REFERENCES ................................................................................................................ 38
8. APPENDIX ...................................................................................................................... 39
Skin Lesion Detection and Classification using Deep Learning Techniques 5

LIST OF TABLES
Table 1: Confusion Matrix [3] ................................................................................................. 18
Table 2: Summary of Research Papers I .................................................................................. 21
Table 3: Summary of Research Papers II ................................................................................ 22
Table 4: Residual Network Results.......................................................................................... 37
Table 5: Research Papers Details ............................................................................................. 40
 6

LIST OF FIGURES

Figure 1: System Architecture ................................................................................................. 26

Figure 2: Use Case Diagram .................................................................................................... 26
Figure 3: Prototype for Classification Problem ....................................................................... 27
Figure 4: Prototype for Detection Problem .............................................................................. 27
Figure 5: Activity Diagram for Classification Problem ........................................................... 28
Figure 6: Activity Diagram for Detection Problem ................................................................. 29
Figure 7: Loss against learning rate ......................................................................................... 31
Figure 8: Model A - ResNet34................................................................................................. 34
Figure 9: Model B - ResNeXt50 .............................................................................................. 34
Figure 10: Actual Mask ........................................................................................................... 35
Figure 11: Predicted Mask ....................................................................................................... 35
Skin Lesion Detection and Classification using Deep Learning Techniques 7

ABSTRACT
Deep Neural Networks have made advancements in a plethora of fields in recent years. One
important field where these techniques can be applied is digital image processing in medical
domain. To detect and classify skin lesion in images using machine learning techniques is a
crucial task to assist physician in decision making related to patient health.
This work focus on skin lesion classification and detection in images using deep learning
techniques. Cancerous tumour images are classified as being benign or malignant using
Convolution Neural Network (CNN) in the first stage. In second stage we will use the same
technique to detect cancerous lesion in melanoma images. This Tumour detection model is
basically an expert system built to detect skin cancer in patients. The model is trained on images
using deep learning techniques. Initial results obtained using above mentioned technique has
78% accuracy.
Introduction 8

1. CHAPTER 1: INTRODUCTION
Deep learning is a piece of a more extensive group of machine learning. It uses
techniques to learn features directly from data which can be in any form including
images and sound. In Image Processing different algorithms are applied to images to
analyse and extract the required results.
The purpose of this project is to build a model that aid doctors in diagnostics. It can be
quite difficult to detect a disease as positive or negative and being humans, doctors
naturally require an additional opinion at times to help or confirm their diagnostic. The
model is hence intended to initially serve as a second opinion to a doctor’s diagnosis.
To build our expert system, we will be using a Convolution Neural Network and some
deep learning techniques. The model takes an image as input and detects skin lesion
area. Upon positive results, the model further classifies the tumor as ‘Benign’ or
‘Malignant’.
Models detecting and classifying Skin Tumor detection have been built previously but
we aim to use better and more efficient algorithms in our model to improve the accuracy
that has been achieved up till now.
The dataset that is used for classification is taken from ISIC Archive [1]. Models
detecting and classifying Skin Tumour detection have been built previously but we aim
to use better and more efficient algorithms in our model to improve the accuracy that
has been achieved up till now.

1.1 Goals and Objectives

1. Classify positive skin cancer results as ‘Benign’ or

‘Malignant’ using CNN
2. Skin Lesion Detection using CNN
3. Make our own neural network using hyperparameters

1.2 Project Scope

In first phase we will train convolution neural network to classify skin melanoma
images into ‘Benign’, ‘Malignant’ or ‘None’. In second phase our model will detect
skin cancerous lesion using the same approach, to train the convolution neural network.
In the end we will try to make our own neural network using hyperparameter.

1.3 Work done so far

The research papers we read had used insufficient data which is not enough for an
unbiased analysis and can lead to model under fitting. In paper [2] we found out that
some machine learning models implemented before have classification accuracy of
85% using Support Vector Machine and 95% using Back-Propagation Neural Network.
Skin Lesion Detection and Classification using Deep Learning Techniques 9

They have trained the model on only 100 images and their accuracy results too are
based on a very small chunk of test data i.e. one hundred images.
In research paper [2] hybrid (combination of BNN and SVM) model approach had been
used to classify skin cancer as benign or malignant only. They had not handled
unlabelled during training and testing phase. By using hybrid approach and increasing
feature vector size twice to improve accuracy from 70% to 95%, they had also increased
the complexity of the Neural Network Model. Under fitting has also resulted due to
small size of dataset used for training and testing. Moreover, it was also stated in the
paper that these results are initial and not final.
Although a high accuracy had been achieved in [2], paper [3] mentions that accuracy
may not always be a good measure of evaluating a models performance. Other factors
need to be considered too. A model for example trained on uneven data with a majority
of one class maybe correctly able to identify only that class. Now when the test data
containing a higher proportionality of the same class comes, with minimal or no test
data belonging to other classes, the model will give a very high accuracy rate since the
test data is mostly composed of the class it classifies correctly.

In [4] Image processing, techniques are applied in which images were enhanced using
Gabor filter and FFT. Then it was segmented using threshold and Watershed
segmentation approach. As a result, features are extracted using binarization and
masking approach. True Acceptance Rate of this method is (85.7%) and conversely
false acceptance rate is (14.3%) without using any deep learning techniques.

Keeping in view the potential issues that could arise in view of our previous research
we have worked on the classification problem using resNet34 and resNet50
architecture. Through resNet34 we were able to achieve 89.9% classification accuracy
and through resnet50, we were able to achieve 99.9% accuracy.
Introduction 10

1.4 Elaboration of Problem

Our problem is to train a model to detect skin cancerous lesions and to further classify
them into three classes, ‘Benign’ or ‘Malignant’. Benign represents a class of tumor
composed of non-cancerous cells. Malignant represents tumor composed of cancerous
cells. They are invasive, composed of abnormal cells and do not stop growing.

1. Dataset
Data is acquired from ISIC [1] which contains 23887 images on which the model is
trained. These are skin melanoma images.

2. Image Pre-processing
Dataset acquired consists of images of differing dimensions so all of them are
resized to have the same dimensions 300*300. The dimension size is chosen
keeping in view the limited runtime space available with Google Colab.
Also, the RGB images are converted to gray level images.

3. Image Segmentation
The lesion in the images is segmented from the surrounding skin. This is done using
thresholding since there is a clear distinction of color between the lesion and the
surrounding skin

4. Convolution Neural Network (CNN)

Model is built using CNN which is trained on the processed dataset. The trained
model then classifies the images as ‘Benign’ or ‘Malignant’.

5. Evaluation Protocols
Result are evaluated based upon the features (weights) extracted using CNN.
For classification, cross entropy function will be used and for detection Jaccard
index will be used as evaluation function.
Cross Entropy is used to quantify the difference between two probability
distributions. Cross entropy loss determines how far away the prediction is from
true distribution. Following is the formula [9] to compute cross entropy:
𝐇(𝐩, 𝐪) = − ∑ 𝒑(𝒙) 𝐥𝐨𝐠 𝒒(𝒙)
𝒙

Jaccard index computes the overlapping region between two sets, in our case these
will be predicted segmented lesion and actual threshold lesion.
Finally several calculations are also done to further evaluate a model. They include
accuracy, precision, recall and specificity. These terms are elaborated in the
upcoming section.
Skin Lesion Detection and Classification using Deep Learning Techniques 11

1.5 Conclusion

This section elaborates the problem statement and explains what we want to achieve.
Our model will be able to detect and classify skin lesions into ‘Benign’ and ‘Malignant’.
We will train our own Neural Network using hyperparameters to achieve high accuracy.

1.6 Outline

This document focuses on the analysis of the literature, design modules,

implementation details and finally results of our experiments. The literature review
focuses on the techniques already used for tumor detection and classification, issues
with the previous approaches and an analysis of their pros and cons. Design modules
describe the steps involved in our system design. Implementation details focus on how
we used convolutional neural networks in detecting and classifying the tumors, along
with the elaboration of Colab as our starting platform and GPU dependencies. Lastly
our experimental results starting from 78% to improvements with 99.2% accuracy is
shown.
Literature Review 12

2. CHAPTER 2: LITERATURE REVIEW

2.1 Introduction

Literature review is the first step in any research based project, helping us identify the
issues with the current methods, paving way for future work and possible problems that
researchers can face. A significant amount of research is being done every year in this
domain. Various ISIC challenges on tumor detection and classification are held every
year but there is yet to develop a model that accurately classifies the tumors given a
large dataset, given that it’s also tested using suitable accuracy measures. If done
properly, this could save millions of doctors and patients around the world valuable
resources and time.
We analyzed [2] [5] [3] [4] [6] [7] [8] papers, from as early as 2007 to as recent as 2018,
which include three papers related to lung cancer and three related to skin cancer tumor
detection and classification. This literature review focuses on the of pros and cons of
various machine learning approaches and image processing techniques used for
classification and detection analyzing them from different angles including unlabeled
data handling, dataset size, biasness in data and the accuracy measures employed to test
theirs results.

2.2 Detailed Literature Review

This section provides a detailed review of the literature we read to gain a deeper
insight into the subject and related work.

2.2.1 Definitions

 Lesion: Infected area of skin

 Benign: Represents a tumor that does not spread through other vital organs or
tissues near it.
 Malignant: Represents a tumor that spreads through vital organs and tissues.
 Thresholding: Method of image segmentation
 SVM: Support Vector Machines which are used to analyze data for classification
 Jaccard Index: It is a statistic used to compute similarity/diversity between sample
sets
Skin Lesion Detection and Classification using Deep Learning Techniques 13

2.2.2 Classification

Various techniques are used to classify the tumor images.

 Back Propagation Neural Network: The errors calculated at hidden layers and
output layers help in updating the current weights such that the resultant output
is close to the targeted output and the overall network error is minimum. [2]

 Support Vector Machine: It uses hyperplane (decision boundary) to separate

data into different categories. For tweaking the model’s performance we can
fine tune the “kernel” ,”gamma” and “C” parameters. [2]

 Bayesian Classifier: In Paper [7] Bayesian classifier was used to classify lesion
images.

 Convolution Neural Network: In paper [8], Convolution Neural Network is

used for skin lesion segmentation. The architecture uses 27 convolution blocks,
5 up-convolution layers, 5 max-pooling layers and 1 output layer.

2.2.3 Summary of Papers

Different researchers have applied different techniques to classify the tumor images.
These papers are summarized below

1. Paper: A Novel Hybrid System for Skin Lesion Detection [2]

Authors: Andy Chiem, Adel Al-Jumaily, Rami N. Khushaba
Classification Techniques: Back Propagation using WPT features extraction
Neural Network and Support Vector Machine
Accuracy: 95.1% (Neural Network) and 85% (SVM)

Neural Networks and SVM were used for lesion detection. Primarily, the images
are acquired using the Dermatoscopy technique which uses oil to make the area
translucent aiding in picturing the skin structures. Secondly, the images are
processed using median filter which helps in reducing noise caused by fine hair..
Thresholding followed by a boundary tracing algorithm was used for segmenting a
region with tumor from normal skin region. Fourthly, wavelet algorithm was used
for feature extraction. This algorithm works by providing detailed information
about an image on different scale stressing upon different features in each scale. It
presents information such as the texture and granularity of the image. Finally, the
images are classified as either malignant or benign lesions. Back-propagation
Neural Network gave an accuracy of 95%. Support Vector gave an 85%
respectively.

2. Paper: Skin Lesion Segmentation and Classification for ISIC 2018 Using
Traditional Classifiers with Hand-Crafted Features [7]
Authors: Russell C. Hardie, Redha Ali, Manawaduge Supun De Silva, and
Temesguen Messay Kebede
Detection Techniques: RGB Bayes Classifier guided by regression network
Literature Review 14

Jaccard Index: 0.701

Classification Technique: Hand crafted features
Recall (Sensitivity): 0.7303

In this paper they have presented their results of lesion segmentation and
classification using hand crafted features. Distinction between normal and lesion
area was done in RGB space using RGB color vectors. Gaussian mixture models
(GMMs) are used to give an estimate as to whether a pixel is a lesion pixel. SVM
was used to set the threshold for image segmentation. With the help of 200 hand
crafted features and SVM, lesions were classified. Features are computed from the
RGB image and the segmentation mask obtained in phase 1.
Phase 1, Lesion Detection, results in a mean overlap score of 0.701. In phase 2, the
average class recall on lesion classification was 0.7303.

3. Paper: Automatic Skin Lesion Segmentation Using Deep Fully Convolutional

Networks [8]
Authors: Hongming Xu and Tae Hyun Hwang
Classification Techniques: Skin Lesion segmentation using Convolution Network
Jaccard index: 0.7344

In this paper, convolutional network was used to segment the lesion skin region. A
deep fully convolutional network is trained the architecture of which has 27
convolution blocks, 5 up-convolution layers, 5 max-pooling layers and 1 output
layer. Sigmoid activation function was used after 1x1 convolution operation. The
network is trained using Adam optimization with a learning rate of 10-4. Binary
cross entropy was used as a loss function. Dual threshold technique was used to
segment the lesion area from the normal skin after obtaining a validation image
from the model. The average segmentation score obtained is 0.7344.

The following Papers are based on lung cancer dataset. Although they are not on
Skin Lesion but they play an important role in our research based project by giving
an insight on the different methods implemented to process images or detecting lung
cancer which are very much related to our field of research.

4. Paper: Lung Cancer Detection Using Image Processing Techniques [4]

Authors: Mokhled S. AL-TARAWNEH
Techniques: Gabor Filter, Segmentation, Normality comparison

Image processing techniques have been implemented in this paper. The research
encompasses stages like image refinement, segmentation and extraction of features.
Frequency and Spatial domain techniques are used for image refinement utilizing
Gabor filter, Fast Fourier transform and auto enhancement. Image segmentation
was done through Thresholding approach or Marker-Controlled Watershed
Segmentation approach. Thresholding converts and image into a binary image
representing parts of an image above or below a threshold. Distinction between the
background and the object is done through Marker-driven segmentation. In feature
Skin Lesion Detection and Classification using Deep Learning Techniques 15

extraction phase, algorithms and techniques are used to isolate desired portions of
an image, usually done by Binarization or Masking approach.
Pixel percentage and mask labeling is used for image comparison.

5. Paper: Lung Nodule Classification Combining Rule-based and SVM [6]

Authors: Zhang Jing, Li Bin, Tian Lianfang
Classification Techniques: Combination of Rule Based and SVM
Accuracy: 84.39%

A combination of SVM and rule-based technique was used in this paper. To do this
there are three possible methods, first is rule based, second being SVM and third is
the combination of both.
Initially the features are calculated. Thirteen features were extracted including gray,
variance and texture features.
Secondly, if all the 13 features are taken into consideration then the computation
can be wrong and some abundant blood vessels may take longer time in
classification thus decreasing the efficiency of SVM. As a result some non-nodule
candidates can be ignored by following the simple rules which are:

i. Exclude blood vessels if Rectangle degree >=1;

ii. Exclude blood vessels if Slenderness <= 0.4;
iii. Exclude blood vessels if ellipticity > =4.
Finally all the extracted features are taken as an input to the SVM in order to classify
the candidate. Before feeding the data into SVM, it is normalized in 0 to 1 range.
There are two output classes nodule class if out =1 and non-nodule if out = -1.
The accuracy obtained by combining Rule Based and SVM is 84.39%.

6. Paper: Performance comparison of artificial neural network and logistic regression

model for differentiating lung nodules on CT scans [5]
Authors: Hui Chen, Jing Zhang, Yan Xu, Budong Chen, Kuan Zhang
Classification Technique 1: Artificial Neural Network
Accuracy: 90%
Classification Technique 2: Logistic Regression
Accuracy: 86.9%

This paper analyzes feed forward ANN and LR model and their differences when it
comes to detection. There were 135 malignant and 65 benign images in the dataset..
A 0.5 threshold was set for ANN for classification.The performance metrics used
in this paper include calibration, clinical usefulness and discrimination (accuracy of
tumor classification). Training and testing data was very less for unbiased analysis
of data. Therefore bootstrap resampling approach was used. ANN performed better
than the LR model. In addition to that the overall accuracy rate was also higher for
ANN’s. When it comes to discrimination and clinical usefulness ANN performed
better. LR models performed better in calibration.

7. Paper: Skin Lesion Classification using Hybrid Deep Neural Networks [9]
Authors: Amirreza Mahbod, Rupert Ecker, Isabella Ellinger
Literature Review 16

Classification Technique: Convolutional Neural Network

Accuracy: 84.7%
In this paper, they have used pre-trained CNN model for classification of three
classes melanoma, seborrheic keratosis and benign. They have trained the model
with 2000 images from dataset provided by ISIC. They have used features for
classification like texture structure, color and other morphological features. After
training they have tested the model with around 150 images and evaluated results
using evaluation method. AOC is 84.8% and the result of binary classification for
Melanoma and seborrheic keratosis is 93.6%.
The dataset had 374 malignant tumor images, 1372 benign and 254 seborrheic
keratosis images, totaling to about 2000 colored dermoscopic images. After that
they have applied the pre-processing techniques. From 2000 images they flip each
image 0, 90, 180, 270 and create an artificial dataset for training model. For feature
extraction they used AlexNet and VGG-16 architecture. They have used 2 fully
connected layers and one detector layer with dimension 1000. After feature
extraction they have used SVM for multiclass classification. For evaluating SVM
result they have mapped the probabilities from SVM to logistic regression.
8. Paper: Skin Lesion Classification Via Combining Deep Learning Features and
Clinical Criteria Representations [10]
Authors: Xiaoxiao Li, Junyan Wu, Hongda Jiang, Eric Z. Chen4, Xu Dong5 and
Ruichen Rong
Classification Technique: Convolutional Neural Network
Accuracy: 85.3%

In this paper neural networks were used for tumor detection from a dataset
containing 10015 images. Image features are encoded using ResNet50 and
Dense201.UNet model is used to generate lesion masks.
.In order to combine traditional features and CNN features Feature Fusion
LightGBM was used. It provides multiple hyper-parameters for achieving best
performance.
Then LightGBM combines the features obtained after applying pre-trained models
and selects the best ensemble parameters used to train LightGBM. Evaluation
measures are applied on training dataset. Afterwards ResNet, ResNet + Crop,
DenseNet + Crop, Clinical Feature and AllFusion model results are obtained on
validation and training images. Then it is analyzed that fusion model achieved better
and more stable performance on normalized multi-class accuracy i.e. 0.853.

9. Paper: Detection and Analysis of Skin Cancer from Skin Lesions [11]
Authors: Nidhal K. EL Abbadi and Zahraa Faisal
Segmentation Techniques: Skin Lesion segmentation using Markov and Laplace
Filers and ABCD rule Accuracy: 85.3%
Results: Specificity: 97.08%, Sensitivity: 94.02% and Accuracy: 95.45%
Skin Lesion Detection and Classification using Deep Learning Techniques 17

This paper implements the Detection and analysis of detected regions of lesion.
Firstly, there’s filtering for removing unnecessary elements from the images we can
say pre-processing. The images are converted in to another color space called YUV,
the reason for using this space is removing thick hair from images. The
segmentation of the region done by Markov and Laplace Filers. Additional thing is
being used in this paper is ABCD rule consist of 4 things Asymmetry, Borders,
Color and Diameter. These parameters being used for computing TDS (Total
Dermoscopic Scope) which is also used for classification of lesion. Then features
being extracted from the images and used in SVM for classifying (analysis) the type
of lesion it is. These papers are implemented on 220 images and achieved an
accuracy of 95.45%.

10. Paper: A multi-task framework for skin lesion detection and segmentation [12]
Authors: Sulaiman Vesal, Shreyas Malakarjun Patil, Nishant Ravikumar and
Andreas K. Maier
Techniques: Skin Lesion segmentation and detection joint using Fast CNN (image
localization), Skinet(Croping images) and softmax (classifier)
Results: Accuracy: 96%, Jacard Index: 88% and Dice Co-efficiant: 93%

This paper implements the Detection and segmentation together using Fast-CNN.
First the dataset is pre-processed for removing unnecessary elements. The dataset
used for these papers included 2000 images. Models used included PH2 and ISBI.
Region Proposal Network, a region based convolutional network was used to
determine the region of interest. After feature extraction the data is passed through
Softmax function. For getting the actual region of the lesion localized images being
cropped by the SkinetIn result, they got Jaccard index of 88% and accuracy 96%.

11. Paper: Skin Lesion Analysis towards Melanoma Detection Using Deep Learning
Network [13]
Authors: Yuexiang Li, Linlin Shen
Techniques: Deeply Fully Connected Residual Network, lesion index calculation
unit (For Segmentation and Classification), CNN (For Feature extraction),
Weighted Softmax and Batch Normalization

In this paper deep learning techniques used separate networks for classification and
segmentation. Lesion Index Calculation Unit was used for segmentation as well as
classification and CNN was used for feature extraction. . The Techniques are
implemented on 2000 images from ISIC (2017).
Secondly simple Convolutional Neural network is used for feature extraction from
the images. Weighted SoftMax and batch normalization for improving our results.
Evaluation measures used in this paper include Jaccard index, precision, accuracy,
dice coefficient and sensitivity
Literature Review 18

2.2.4 Content Representation

Upon obtaining positive results of skin lesion, we further classify the image into two
types:

 Benign: Tumor composed of non-cancerous cells

 Malignant: Tumor composed of cancerous cells

2.2.5 Evaluation Measures

Different performance measuring metrics exist and based on the model, the most
suitable measure is chosen. Some of the ways to evaluate performance are discussed
below [3].

i. Jaccard Index:
It is a statistic used to compute similarity/diversity between sample sets.

ii. Confusion Matrix:

Describes the correctness and accuracy of the model in a form of a table and is used
in classification problems which have outputs with two or more types of classes.
Confusion Matrix is a matrix with predicted and actual results across rows and
columns. The table below is a good representation of confusion matrix.

Disease No Disease

Positive
True Positive False Positive
Test Result

Negative
False Negative True Negative
Test Result

Table 1: Confusion Matrix [3]

False positive and false positives are the incorrect result and it is our goal to
minimize these to make our model a better classifier.
Skin Lesion Detection and Classification using Deep Learning Techniques 19

iii. Accuracy:
It is the correct predictions made by our model over all the predictions made.
𝑻𝑷+𝑻𝑵
Accuracy = 𝑻𝑷+𝑭𝑷+𝑭𝑵+𝑻𝑵 [3]

iv. Precision:
It tells us the proportion of objects we classified correctly over the total number of
objects that belonged to that class. Describes a models overall performance with
respect to false positives.
𝑻𝑷
Precision = [3]
𝑻𝑷+𝑭𝑷

v. Recall or Sensitivity:
Recall tells what proportion of objects that belonged to a particular class were
classified correctly by our model. Describes a models overall performance with
respect to false negatives.
𝑻𝑷
Recall = 𝑻𝑷+𝑭𝑵 [3]

vi. Specificity:
Specificity is the exact opposite of recall. It tells what proportion of objects that did
not belong to a particular class were predicted correctly as not belonging to that
class by our model.
𝑻𝑵
Specificity = 𝑻𝑵+𝑭𝑷 [3]

vii. F1 Score:
This represents both Precision (P) and Recall (R). One way of getting this score is
through taking the arithmetic mean of both the values (P+R)/2. But harmonic mean
is a better way of computing the score as it gives better performance measures in
most cases. The formula is written below.
𝟐∗𝑷∗𝑹
F1 Score = 𝑷+𝑹 [3]
Literature Review 20

2.3 Literature Review Summary

Paper Author Approach Score Description

A Novel Hybrid Andy Chiem, Classification Accuracy: Through WPT, features are
System for Skin Adel Al- using Back- 95.1% extracted and fed into
Lesion Jumaily, Rami Propagation Back-Propagation Neural
Detection [2] N. Khushaba Neural Network with 2 hidden
Network layers are used to classify
images as either benign or
malignant
A Novel Hybrid Andy Chiem, Classification Accuracy: 85% For classification into
System for Skin Adel Al- using Support benign or malignant N-
Lesion Jumaily, Rami Vector Machine dimensional hyperplane is
Detection [2] N. Khushaba constructed using SVM.
SVM models in such a way
that different categories of
target variable fall on
different side of the plane
in clusters.
Skin Lesion Russell C. i. Detection i. Jaccard First segmentation is done
Segmentation Hardie, Techniques: Index: 0.701 to detect skin lesion area
and Redha Ali, RGB Bayes ii. Recall/ then SVM is used for
Classification for Manawaduge Classifier Sensitivity: classification
ISIC 2018 Using Supun De guided by 0.7303
Traditional Silva, and regression
Classifiers with Temesguen network
Hand-Crafted Messay
Features [7] Kebede ii. Classification
Technique:
Hand crafted
features
Automatic Skin Hongming Xu Skin Lesion Jaccard index: Test problem involved
Lesion and Tae Hyun segmentation 0.7344 segmenting the image to
Segmentation Hwang using detect skin lesion
Using Deep Fully Convolution
Convolutional Network
Networks [8]
Skin Lesion Amirreza Convolutional 84.7% In this paper, they have
Classification Mahbod, Neural used the pre-trained CNN
using Hybrid Rupert Ecker, Network model for classification of
Deep Neural Isabella three classes melanoma,
Networks [9] Ellinger seborrheic keratosis and
benign.
Skin Lesion Detection and Classification using Deep Learning Techniques 21

Skin Lesion Xiaoxiao Li, Convolutional 85.3% Deep convolutional neural

Classification Junyan Wu, Neural networks are used to solve
Via Combining Hongda Jiang, Network skin lesion analysis
Deep Learning Eric Z. Chen4, towards melanoma
Features and Xu Dong5 and detection problem.
Clinical Criteria Ruichen Rong
Representations
[10]
Detection and Nidhal K. EL Skin Lesion Specificit: This paper implements the
Analysis of Skin Abbadi and segmentation 97.08%, Detection and analysis of
Cancer from Zahraa Faisal using Markov Sensitivit: detected regions of lesion.
Skin Lesions and Laplace 94.02% and
[11] Filers and Accuracy:
ABCD rule 95.45%
A multi-task Sulaiman Skin Lesion Accuracy: This paper implements the
framework for Vesal, segmentation 96%, Jacard Detection and
skin lesion Shreyas and detection index: 88% segmentation together
detection and Malakarjun joint using Fast and Dice Co- using Fast-CNN
segmentation Patil, Nishant CNN (image efficiant: 93%
[12] Ravikumar, localization),
and Andreas Skinet (Croping
K. Maier images) and
softmax
(classifier)
Skin Lesion Yuexiang Li, Deeply Fully Classification This paper was not only
Analysis Linlin Shen Connected and focused on Detection and
towards Residual Segmentation Segmentation, the
Melanoma Network , Results: AUC: additional thing was
Detection Using lesion index 0.912, Jaccard accurate feature extraction.
Deep Learning calculation unit, Index: 0.753
Network [13] CNN,
Feature
Weighted
Extraction
Softmax and
Results:
Batch
Precision:
Normalization
0.422,
Sensitivity:
0.693

Table 2: Summary of Research Papers I

Literature Review 22

Papers not directly related to our research project but which certainly play an important role in
extending our research are summarized below.

Paper Author Approach Score Description

Lung Cancer Mokhled Gabor filter, Accuracy: Image processing
Detection S. AL- Segmentation 85% techniques have
Using Image TARAWN using been researched
Processing EH watershed, upon in this paper
Techniques normality which are the first
[4] comparison step in medical
diagnosis
Lung Nodule Zhang Combination Accuracy: The paper proposes
Classification Jing, Li of Rule Based 84.39% an approach
Combining Bin, Tian and SVM combining rule-
Rule-based Lianfang based and SVM to
and SVM [6] classify lung
nodules. Some
nodules are ignore
on basis of rules.
Finally features
calculated are
inputted to SVM
and results
obtained.
Performance Hui Chen, i. Artificial i. This paper analyzes
comparison Jing Neural Accuracy ANN and LR models
of artificial Zhang, Network 1: 90% for classifying lung
neural Yan Xu, nodules. Three
ii. Logistic ii.
network and Budong aspects of model
Regression Accuracy
logistic [5] Chen, performance are
2 : 86.9%
Kuan considered;
Zhang calibration, clinical
usefulness and
discrimination.
Study reveals that
ANN’s
outperformed LR
models in
discrimination and
clinical usefulness
but did not
outperform in
calibration.

Table 3: Summary of Research Papers II

Skin Lesion Detection and Classification using Deep Learning Techniques 23

2.4 Conclusion

The work that we have analyzed so far employed various techniques including back-
propagation neural network, SVM, ANN. Giving accuracy of 95% with back-
propagation neural network [2], 85% with SVM [2] and about 90% with ANN [5]. We
have found that most papers did not handle unlabeled data in ISIC archive. In addition
to this small chunk of data is used for training and testing mostly 200 images. The
studies conducted so far lacked proper accuracy measures. In paper [5] unbalanced data
has been used to train and test their model, using 135 malignant and 65 benign, putting
a question mark on their accuracy claims. In reference paper [2] it is mentioned that
their results are initial and not final. Our aim was to achieve more than 90% accuracy
using CNN and we successfully achieved it. Details are given in the implementation
chapter.
Requirements and Design 24

3. CHAPTER 3: REQUIREMENTS AND DESIGN

3.1 Functional Requirements

The functional requirements of our system include:

3.1.1 Data Preparation and Data Pre-Processing Requirements

 System shall provide features for labelling the dataset.

 System shall provide features for data formatting to make the dataset consistent.
 System shall provide features for noise removal and features for handling data
inconsistencies.
 System shall provide features for data scaling so that the data can converted into a form
appropriate for feature extraction.

3.1.2 Data Set Splitting Requirements

System shall provide appropriate tools for splitting data into training, testing and
validation sets.

3.1.3 Modelling Requirements:

Our System shall provide features for feeding the hyper-parameters into our model.
System shall have required libraries for training deep learning models.

3.1.4 Data Evaluation and Parameter Tuning Requirements:

System shall allow to test the data against data that has not been used for training. In
order to improve the accuracy, system shall have features for tuning parameters.

3.1.5 Data visualization

System shall provide features for visualizing the data and results.

3.1.6 Analysis and Validation Requirements

System shall provide features to test the efficiency of our model based on different
evaluation criteria.
Skin Lesion Detection and Classification using Deep Learning Techniques 25

3.2 Non-Functional Requirements

3.2.1 Usability
 The systems interface shall be easy to understand and consistent.
 The system shall be user friendly for admins.

3.2.2 Reliability
Dataset given to the system will be used to compare the results of our model and to
measure the reliability of the results.

3.2.3 Extensibility
Modular approach will be used so that adding and removing components is easier.

3.2.4 Compatibility
System will be compatible for Mac, Windows and Unix Platforms.

3.2.5 System Visualization

System capable of visualizing machine behaviour so that even a person with little
knowledge can interpret the machines health.

3.3 Hardware Requirements

 Graphics Processing Unit (GPU)
 Unix/ Windows/ Mac
 Internet Connectivity

3.4 Software Requirements

 COLAB (Google Drive) is used for this model
 Compatibility with other Python Platforms (like Jupyter Notebook)
 Chrome/ Firefox/ Opera/ Internet Explorer/ Safari
Requirements and Design 26

3.5 System Architecture

Figure 1: System Architecture

3.6 Use Case

Figure 2: Use Case Diagram

Skin Lesion Detection and Classification using Deep Learning Techniques 27

3.7 Graphical User Interface (GUI)

Figure 3: Prototype for Classification Problem

Figure 4: Prototype for Detection Problem

Requirements and Design 28

3.8 Activity Diagram

3.8.1 Classification Problem

Figure 5: Activity Diagram for Classification Problem

Skin Lesion Detection and Classification using Deep Learning Techniques 29

3.8.2 Detection Problem

Figure 6: Activity Diagram for Detection Problem

Implementation 30

4. CHAPTER 4: IMPLEMENTATION
4.1 Classification Problem
4.1.1 Initial Implementation

Our goal till now was to improve accuracy for skin lesion classification problem which
required predicting whether a tumor belonged to benign or malignant class.
Initially, we achieved an accuracy of 73% that we improved to 78%.
Skin Melanoma images are acquired from ISIC [1]. We used 900 images and reduced
the image size to 300x300 keeping into consideration Google Colab’s limited run time
space available.
We used convolutional neural network to accomplish this task. Our convolutional
neural network used 32 3x3 convolutional filters followed by max pooling on
300x300x1 image in the first layer. After which we were left with 150x150x32 (Stride
set to 2). After applying 64 3x3 convolutional filters in the second layer, we got 150 x
150 x (32x64) shape, which was reduced to 75 x 75 x (32 x 64) after max pooling. This
process was repeated for 128 3x3 filters, followed by flattening and the processing
through 128 dense layer units.
To evaluate our model and to check how far our prediction was from true distribution,
cross entropy was used.
We then started focusing on further improving this accuracy by finding the underlying
errors and abnormalities in behavior of our model. Finally we were able to achieve our
desired accuracy by implementing ResNext algorithm for Skin Lesion Classification.
 Skin Lesion Detection and Classification using Deep Learning Techniques 31

4.1.2 Final Implementation

4.1.2.1 Dataset Used

Dataset was taken from ISIC-Archive [1], comprising of half benign and half malignant
tumors, ignoring the corrupted images. This balance of images was maintained to avoid
any biasedness in dataset. We implemented two models. Model A used a total of 1990
images from which 599 were used for testing. Model B used 2014 images from which
628 were for testing.

4.1.2.2 Methodology
GPU (Quadro K4000M), python version 2.7 with libraries PyTorch[20] and fast.ai were
used in the classification process. ResNet34 and ResNetXt50 were used for
classification. In order to improve our classification results, image augmentation,
varying learning rates and stochastic gradient descent were used.
Image Augmentation was applied to cater to the loss of ROI (Region of Interest)
because of image cropping in testing phase. Averaged version of an image generated
from different versions was used in training the models.
In order to determine appropriate learning rates for our models, different learning rates
were passed to ResNet34 and ResNetXt50 layers. (1e-2)/2, (1e-2)/6 and (1e-2)/18 were
chosen as learning rates for different layers. Figure [7] shows graph for loss against
different learning rates.

Figure 7: Loss against learning rate

 Implementation 32

4.2 Detection Problem

4.2.1.1 Dataset Used
Dataset was taken from ISIC-Archive [1], comprising of half benign and half malignant
tumors, ignoring the corrupted images. This balance of images was maintained to avoid
any biasedness in dataset. 6000 images were used in total of which 30% were used for
testing and remaining 70% for training.

4.2.1.2 Methodology

GPU Nvidia GTX 1070 8GB was used to implement U-Net Architecture [14]. After
Preprocessing by apply Power law transformation techniques, images and encoding list
of each image is produce. These are then fed to neural network and results were
computed against different size of images. With size 256 x 256, jaccard index value
came out to be 77. But when image dimensions were reduced to 512x512, Jaccard index
value of 74 was achieved. For improving results, same accuracy improvement
techniques were applied as described in above.
Skin Lesion Detection and Classification using Deep Learning Techniques 33

5. CHAPTER 5: EXPERIMENTAL RESULTS AND ANALYSIS

5.1 Classification Problem

5.1.1 Overview of Model A and Model B implementation

Pre-Processing
For model A, images were resized to 299x299 and for model B, 256x256.

Power Law Transformation

Before feeding data to model B, images were enhanced further using a gamma value of
1.5.

Training
After feeding the images and labels to the model. Model A was trained on 15 epochs
whereas model B was trained on 45 epochs.
Experimental Results and Analysis 34

5.1.2 Evaluation Results

The confusion matrix Figure [8], generated through model A shows that 64 images out
of a total of 599 were incorrectly classified.

Figure 8: Model A - ResNet34

For model B, Figure [9] shows that only 2 out of a total of 627 images were incorrectly
classified.

Figure 9: Model B - ResNeXt50

Skin Lesion Detection and Classification using Deep Learning Techniques 35

5.2 Detection Problem

5.2.1 Evaluation Measures

Two techniques were used as evaluation measures. One was Dice which is computed as
follows.
2 ∗ |𝑋 ∩ 𝑌|
|𝑋| + |𝑌|
The value obtained with dice for 512*512 image dimension is 83.7%

The other evaluation measure used is Jaccard Index. Images are fed into the model and a
mask is computed highlighting the lesion area. This predicted mask is compared against the
actual mask and Jaccard index is computed. When images of dimension 512*512 are fed into
the model, the jaccard index value is 77. Following images show the actual and predicted
mask for an image.

Figure 10: Actual Mask

Figure 11: Predicted Mask

Experimental Results and Analysis 36

5.2.2 Evaluation Results

Following are the results when model was fed images of dimensions 256*256

epoch trn_loss val_loss <lambda> dice jaccardIndex

0 0.109912 0.135662 0.943683 0.8641 0.766968
1 0.106406 0.136117 0.943394 0.863633 0.766275
2 0.109768 0.135443 0.944138 0.865063 0.768383
3 0.105541 0.135437 0.944346 0.864283 0.769008
4 0.105394 0.135775 0.944214 0.865269 0.768685
5 0.106734 0.13571 0.944383 0.865605 0.769175
6 0.106785 0.135663 0.944524 0.865905 0.769622
7 0.107212 0.135556 0.94467 0.866227 0.770098
8 0.104676 0.135411 0.944817 0.866493 0.770498
9 0.107288 0.135625 0.94469 0.866262 0.770151

Below are the results when model was fed images of dimensions 512*512
epoch trn_loss val_loss <lambda> dice jaccardIndex
0 0.134988 0.149975 0.944588 0.836694 0.738154
1 0.144968 0.149766 0.944581 0.836871 0.738393
2 0.117624 0.149591 0.944571 0.837023 0.738602
3 0.111463 0.149463 0.944564 0.837134 0.738756
4 0.124255 0.149372 0.944559 0.837209 0.734459
5 0.130772 0.149316 0.944556 0.837264 0.738935
6 0.143931 0.149291 0.944555 0.837282 0.738959
7 0.117556 0.149282 0.944555 0.837289 0.738969
Skin Lesion Detection and Classification using Deep Learning Techniques 37

6. CHAPTER 6: CONCLUSION
6.1 Results and Challenges – Classification Problem
Cross Entropy is used as evaluation measure in training and testing phase. We have
successfully achieved a high accuracy for our classification problem that is 99.5%. The
goal has been achieved but it could have been even more authentic but the hardware
constraints limited us to use a smaller dataset of up to 2000 images. Unavailability of
GPU and lack of space on Google’s Colab presented us with runtime issues when
increasing the batch size and image dimensions, so we had to settle for a smaller dataset.

Table [4] below shows the results for the two models implemented.

Model Training Testing MCC Accuracy

Loss Loss

Resnet34 0.11508 0.31392 0.000155 89.5%

ResNext50 0.03842 0.02658 0.000179 99.2%

Table 4: Residual Network Results

In the previous years, researchers have used various models such as SVM, Linear
Regression and many more to approach the same tumor detection and classification
problem but a further analysis into their techniques highlighted that their
experimentation was based on small dataset which was in some cases also biased data.
We employed CNN keeping in view the large dataset and correspondingly large feature
vector sizes that our model has to deal with. The goal of improving accuracy has been
successfully achieved.

6.2 Results and Challenges – Detection Problem

For detection, U-Net Convolution Network is used. The challenge presented in this
problem too was unavailability of GPU. To overcome that, images dimensions were
reduced to 512*512. The results achieved were a jaccard value of 74% and dice value
of 84%.
Appendix 38

7. REFERENCES
[1] "ISIC Archive," [Online]. Available: https://www.isic-archive.com. [Accessed 2018].
[2] A. Chiem, A. Al-Jumaily and R. N. Khushaba, "A Novel Hybrid System for Skin
Lesion Detection," in Intelligent Sensors, Sensor Networks and Information,
Melbourne, Qld., Australia, 2007.
[3] M. Sunasra, "Performance Metrics for Classification problems in Machine Learning,"
11 11 2017. [Online]. Available: https://medium.com/greyatom/performance-metrics-
for-classification-problems-in-machine-learning-part-i-b085d432082b. [Accessed 13 09
2018].
[4] M. S. AL-TARAWNEH, "Lung Cancer Detection Using Image Processing," Leonardo
Journal of Practices and Technologies, vol. 20, pp. 147-158, 2012.
[5] H. Chen and J. Zhang, "Performance comparison of artificial neural network and
logistic regression model for differentiating lung nodules on CT scans," Expert Systems
with Applications, vol. 39, pp. 11503-11509, 2012.
[6] Z. Jing, L. Bin and T. Lianfang, "Lung nodule classification combining rule-based and
SVM," in Bio-Inspired Computing: Theories and Applications (BIC-TA), Changsha,
China, 2010.
[7] C. R. Hardie, "Skin Lesion Segmentation and Classification for ISIC 2018 Using
Traditional Classifiers with Hand-Crafted Features," 2018.
[8] Y. Yuan, M. Chao and Y.-C. Lo, "Automatic Skin Lesion Segmentation Using Deep
Fully Convolutional Networks With Jaccard Distance," IEEE Transactions on Medical
Imaging, p. 99, 2017.
[9] A. Mahbod, R. Ecker and I. Ellinger, "Skin Lesion Classification Using Hybrid Deep
Neural Networks," ArXiv, 2017.
[10] X. Li, J. Wu, H. Jiang and E. Z. Chen, "Skin Lesion Classification Via Combining Deep
Learning Features and Clinical Criteria Representations," 2018.
[11] Z. Faisal and N. Khdhair El abbadi, "Detection and analysis of skin cancer from skin
lesions," International Journal of Applied Engineering Research, vol. 12, pp. 9046-
9052, 2017.
[12] S. Vesal, S. M. Patil, N. Ravikumar and A. Maier, "A multi-task framework for skin
lesion detection and segmentation," in Workshop on skin image analysis, Granada,
Spain, 2018.
[13] Y. Li and L. Shen, "Skin Lesion Analysis towards Melanoma Detection Using Deep
Learning Network," Sensors, vol. 18, no. 2, p. 556, February 2018.
[14] "Wikipedia," [Online]. Available: https://en.wikipedia.org/wiki/Cross_entropy.
[Accessed October 2018].
 Skin Lesion Detection and Classification using Deep Learning Techniques 39

8. APPENDIX

Paper Training Testing Data Source Year Conference

Number data Size / Publisher

[1] 100 total Not De Department of 2007 Intelligent Sensors,

specified Dermatology
Sensor Networks and
University Information, 2007. ISSNIP
2007.
lowa College of
3rd International
Medicine
Conference on,
and Derm Net
2007, pp. 567–572.

[4] 254 total Not 3A grade hospital in 2010 Bio-Inspired Computing:

specified
Guangzhou. Theories and Applications
(BIC-TA), 2010 IEEE Fifth
International Conference on,
2010, pp. 1033–1036.

[2] 200 200 Not specified 2012 Leonardo Electron.

J. Pract. Technol.,
vol. 20, pp. 147–58.

[5] 254 total Not Not specified 2012 Expert Syst. Appl.,
specified
vol. 39,
no. 13, pp. 11503–11509,
2012.

[8] 2590 100 ISIC[6] 2017 IEEE Transactions on Medical

validation, Imaging ( Volume: 36 , Issue:
9 , Sept. 2017 )

1000 for
test
 Appendix 40

[7] not not ISIC[6] 2018 ArXiv

specified specified

[10] 2000 150 ISIC[6] 2017 ArXiv

[11] 70011 validation ISIC[6] 2018 BiorXiv

1001

Testing
2003

Table 5: Research Papers Details