CRANFIELD UNIVERSITY

F. FEDERIGHI

ASSISTANCE FOR RADIOLOGISTS

SCHOOL OF AEROSPACE, TRANSPORT AND

MANUFACTURING
Computational Software Techniques in Engineering

MSc
Academic Year: 2017–2018

Supervisor: Dr T. Gilbert Supervisor

August 2018
 CRANFIELD UNIVERSITY

SCHOOL OF AEROSPACE, TRANSPORT AND

MANUFACTURING
Computational Software Techniques in Engineering

MSc

Academic Year: 2017–2018

F. FEDERIGHI

Assistance for radiologists

Supervisor: Dr T. Gilbert Supervisor

August 2018

This thesis is submitted in partial fulfilment of the

requirements for the degree of MSc.

c Cranfield University 2018. All rights reserved. No part of

this publication may be reproduced without the written
permission of the copyright owner.
Contents
List of Figures 4

List of Tables 4

1 Abstract 5

2 Acknowledgments 6

3 Introduction 7

4 Literature Review 7

5 Aims and Objectives 9

6 Proposed Method 10
6.1 First Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.2 Inconvenient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
6.3 Improvements and Artificial Intelligence . . . . . . . . . . . . . . . . . . . . . . . . 13

7 Algorithm and Software Development 14

7.1 Image Recognition Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
7.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
7.2 Convolutional Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.3 Scraping Google Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

8 Conclusion 20

References 21

Appendices 22

3
List of Figures
1 MRI of a knee, axial section . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 Example of using reverse image search . . . . . . . . . . . . . . . . . . . . . . . . . 12
3 Example of using reverse image search with a keyword entered manually . . . . . . 13
4 Imagined approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5 Desired algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6 Scraping example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

List of Tables

4
1 Abstract
This thesis proposes a new approach to help radiologists in their jobs through artificial intelligence
technologies.

This new concept is based on the Google tool called Reverse Image Search and a Deep Learn-
ing algorithm. This tool allows to search information or images on a pathology from a photo of a
medical image (MRI, radio, scan).

The deep learning algorithm was used to identify the human body part of a medical image. The
algorithm used is a convolutional neural network based on the InceptionV3 architecture.

The use of such an algorithm made it possible to use google reverse Image Search to search
for images similar to a medical image, which was not originally possible.

5
2 Acknowledgments
First of all, I would like to thank Mr Raymond AUPHAN for allowing me to do this internship.
I would also like to thank my Director Mr Cédric CAVIN, and my academic supervisor Mr Rémi
PREVOT for the trust they have placed in me throughout my mission.

I also thank Gabriel DIEZ, Yoann BENTZ and Benjamin DE BERNARDI for helping me during
the last part of my internship.

Finally, I would like to thank all the people at HCUBE for their welcome and their kindness
towards me.

6
3 Introduction
I did my end of study internship in a start-up called HCube Conseil, located near Aix-en-Provence,
FRANCE. With the company Com Network, HCube Conseil is part of a group, DPR industry,
which is the fund of investment with the aim of moving towards artificial intelligence and digital
transformation. HCube Conseil wishes to meet the needs of companies wishing to acquire skills
in algorithmics, Big Data, strategy, to be always more competitive. The goal for a company isto
evolve at the pace of technology, not to be subjected to it. This internship was carried out within
the framework of obtaining a Master’s degree from Cranfield University.

The objective of this internship is to think and design a system to assist the medical profession
and more specifically radiologists in their jobs using new technologies such as artificial intelligence
and image processing.

During this project I worked with 2 radiologists to try to better understand their profession and
especially to propose a system that really meet the expectations of health professionals.

4 Literature Review
Today, many people are interested in how to facilitate and improve the work of health profession-
als, particularly radiologists.

The radiologist is the person who makes a medical diagnosis from medical images such as X-
rays, MRIs, scans or ultrasounds.

A great deal of research has been carried out or is being carried out to arrive at an automatic
diagnosis based on medical images. It is one of the most popular fields at the moment combining
artificial intelligence and medicine.

A lot of research has been conducted such as research on the automatic detection of tumours
in the brain, on the automatic detection of cardiovascular diseases, on the detection of fractures
or sprains... This work is based on medical images (MRI, radio, CT, ultrasound) to automatically
establish a diagnosis.

Before arriving at an automatic diagnosis, the researchers concentrated on the automatic clas-
sification of medical images into 2 parts: normal or abnormal. For this the researchers used Deep
Learning/Machine Learning and image processing techniques. Most of this research followed the
same protocol.

The first step consists in acquiring data. This involves acquiring for example a number of brain
images illustrating a brain with tumour and a brain without tumour.

The second step consists in improving the quality of the collected images, to standardize these
images by applying for example a Histogram Equalization. This allows to have images covering
all shades of gray (from 0 to 255) and thus allows to have a homogeneous collection of images.

7
The third step is to segment the scene. This is an optional step. In the context of brain tu-
mour classification, for example, it’s about separating the tumour from other elements of the brain
such as the skull, the grey matter, etc., in order to classify it as a brain tumour. To do this some
researchers have decided to use discrete wavelet transform to segment the image of the brain.
Other researchers have decided to apply morphological operations.

The fourth step consists in extracting information from the images in order to classify them into
2 categories: normal and abnormal. This step is called features extraction. To do this we can find
different methods in the literature such as the use of the grey level co-occurence matrices or the
use of the Discrete wavelet transform.

The fifth step consists in classifying the images into 2 cagoteries: normal or abnormal. To do
this several machine learning algorithms were used. Some researchers have chosen to simply use
the KNN algorithm while other researchers have decided to use neural networks or SVM.

The last one is to train the algorithms and test them. To train the algorithms most researchers
used a k-fold cross validation. Finally, it is a question of measuring the performance of the method
used. The most common method is to calculate the confusion matrix.

A similar approach is applied to make a complete diagnosis and thus to diagnose what a patient
was suffering from. Instead of classifying the images into 2 categories, the researchers classified
them into several categories each representing a pathology. Very good results have been obtained
in the detection and classification of tumours. The author of this article was able to detect [8] and
classify tumours into 4 categories using Deep Learning.

One of the disadvantages of these advances is that researchers have focused on one part of the
human body and on a specific pathology. For example the researcher presented above focused on
tumor classification. The human body being vulnerable to thousands of pathogens, it should be
possible to extend the work presented above to all known pathologies but this requires a lot of
work and data. Indeed, obtaining annotated data from the whole human body and its pathologies
is extremely difficult.

To overcome this problem and thus to be able to offer a tool capable of helping the whole medical
profession and not only neurologists, it would be necessary to widen the field of action and reduce
the accuracy of the diagnosis. We would then be talking about diagnostic aid and no longer auto-
mated diagnosis.

For example, researchers[3][6] have proposed a Deep Learning algorithm to automatically rec-
ognize the part of the human body from medical images. The real interest of this approach will
be discussed in the following paragraphs.

The author of this paper chose to use Deep Learning and more precisely a network of convo-
lutional neurons to recognize 5 parts of the human body: lung, liver, heart, kidney, and lumbar
spine. The functioning of a convolutional neural network will be presented in the following para-
graphs. The author first recovered data, i.e. medical images, from the U.S. National Library of
Medicine. The author then compared three known architectures (LeNet, AlexNet and GoogLeNet)
and proposed his own convolutional neural network to solve this problem.

8
The author of this paper has achieved a success rate of 80% by proposing his own architecture. Its
architecture showed better results than the other 3 architectures (LeNet, AlexNet and GoogLeNet).

This success rate is impressive but needs to be improved in order to offer a real solution to health
professionals. Finally, it is important to raise the last point: all this research and these advances
do not take into account the industrial dimension and never talk about the calculation time needed
to classify new data. This is not insignificant information when a product is to be industrialised.

5 Aims and Objectives

Let us state clearly in this chapter what the objectives and purpose of this project are.

It is important to recall that the primary goal of this project is to propose a tool to assist ra-
diologists in their jobs. This tool must also be able to help other health professionals such as
neurologists, but also medical students. More generally, our system must be usable by all health
professionals or students using medical imaging.

The work to be carried out during these 6 months is to reflect and propose a new approach
to assist radiologists and other health professionals in establishing a medical diagnosis. This new
approach should not be reserved for a part of the human body such as the brain or a specialist
such as the neurologist.

The radiologist is the specialist in medical imaging. He is a doctor specialized in medical imaging
techniques: X-rays, ultrasounds, scans and MRI. In particular, it detects fractures, tumours or
infections on all parts of the body. It monitors fetal growth in pregnant women and makes visible
any pathology or fracture that a general practitioner may suspect. As such, it plays an essential
role by issuing a diagnosis that the practitioner, with image in support, can interpret.

It is important to offer a tool that can be easily applied to the whole human body and not to
offer an extremely sophisticated tool for a particular part of the human body. The system to
imagine must be simple and quick to use. The user experience should not be neglected.

The first step that is expected is to carry out a feasibility study first and to outline the sub-
ject. It is important to note that the solution to be imagined must be industrialisable and must
not be a simple concept. Through this internship it is important that this project is concretized
by the writing of a specification and that all the functionalities are clearly stated. For example, an
application or a website can be made concrete by the design of the different screens that make it up.

Finally, the last stage of this internship consists of a proof of concept. The proof of concept
must be as close as possible to the solution imagined. Indeed, this should make it possible to eval-
uate the technical solution (accuracy, relevance, speed, etc.) but it should also make it possible to
evaluate the user experience.

9
6 Proposed Method
6.1 First Approach
The first weeks of the internship were devoted to thinking about a new concept to help different
health professionals. The purpose of this project is to propose a new tool or application to assist
radiologists in establishing a diagnosis. The system to be designed must add value to all the ex-
pertise held by a radiologist.

This project was carried out in collaboration with 2 radiologists. During one meeting a point
was raised. Radiologists sometimes need to search for information on a pathology or to search for
images similar to the case before them.

It is necessary to know that today, that there is really no atlas or collection of images, regrouping
all the pathologies of the human body. For example, the entire anatomy of the human body can
be found online on the e-anaotmy website, but only medical images from a healthy human body
can be found. A specialist rarely needs to consult images of a healthy patient to reassure himself
in his diagnosis. A specialist rather needs to consult medical images on a very precise pathology
to comfort himself in his diagnosis or quite simply by simple intellectual curiosity. For that a
specialist or a radiologist is led to consult specialized sites like www.sfrnet.org or medical social
networks like Figure1. These sites have valuable information but this information is not always
easy to find and takes time.

To overcome this problem, radiologists use Google Image to search for information on a pathology
or images similar to a case they are working on. Google is the most used search engine in the world
and the most powerful and allows to obtain very easily relevant results from a simple keyword.
Google does not replace the radiologist at all but allows him to comfort himself or to help him in
establishing his diagnosis.

When a radiologist does a search, they use Google Image using keywords and then only look
at results (images) that look like the mri or scan they are looking at. The new concept I’m work-
ing on comes from here.

The idea is to automate this Google image search and search from a photo and not from a keyword.
More generally, it is a question of designing a application making it possible to search on Internet
for similar images from a photo and not from a keyword. The radiologist or the health professional
would just have to take a picture of an unknown medical image then identify it using the developed
tool.

This new concept is therefore based on Internet research and image comparison. The goal is
to combine the power of a search engine like Google with image comparison to get better results
and more easily.

The first thing to do was to learn about the different search engines that search images based
on an image. The two best known are Google Reverse Image Search and TinEye. To compare
them, the relevance of the results was compared. So a search was done with the same photo and
the results were compared. After several it was clear that Google was clearly superior to TiniEye

10
in searching for similar images. In most cases TiniEye was not able to find results.

In summary Google Reverse Image Search seems to be a tool perfectly suited to the initial need.
This tool offers the possibility to search images on the Internet from photos or from images and
to use all the power of Google in the information search.

6.2 Inconvenient
Despite all the positive points of Google’s Reverse Image Search, this application has 2 major
disadvantages in the realization of our project. The first disadvantage concerns the development
of the application. We want to create an application using Google’s image search system. To do
this Google provides an API to retrieve the results obtained during a search. The problem is that
this API, called CSE (Custom Search Engine), does not work for searching images based on an
image or a photo. This API only allows you to retrieve the results from a keyword search. This
is a very important point since we do not want our future users to use Google but rather our
application. You should also know that browsing through Google Reverse Image Search is not
very well thought out, this is one of the reasons why we chose to develop our own search system.

The second disadvantage concerns the use of Google Reverse Image Search. You should know
that when you search from an image, Google automatically associates a keyword with the photo.
This keyword is very important in the search since Google always looks for similar images in rela-
tion to the original photo and in relation to this famous keyword. The problem is that if Google
can’t identify the photo, then the results will be completely wrong. To better understand this
phenomenon here is an example:

Figure 1: MRI of a knee, axial section

If a search is made from this image, here is the keyword Google associates and the results it finds :

11
 Figure 2: Example of using reverse image search

Google has identified the photo as a ”silver”. This misidentification leads to erroneous comple-
mentary results as shown in this photo (similar images are at the bottom of the image).

To remedy this problem it is quite possible to change the keyword manually. By replacing the
keyword ”silver” by ”Knee” here is what you get:

12
 Figure 3: Example of using reverse image search with a keyword entered manually

By specifying to Google from which part of the human body the photo is taken, we can see that
the results are much more convincing. It’s important to note that the photo (Figure 1) that led
to the search is not a photo found on the internet but a photo taken with a phone in a doctor’s
office. The real problem was not the keyword assignment but the fact that it was a user who had
to manually enter the keyword to get relevant results. Knowing this, the application would lose
its interest since the user would have to enter a keyword in addition to importing a photo.

6.3 Improvements and Artificial Intelligence

A solution had to be found to solve the problems listed above if we wanted to have a competitive
and innovative application.

13
At First a completely different solution was thought of . The idea was no longer to use Google to
search for images, but to build up one’s own database of medical images. I quickly realized that
this was going to be difficult since there is no database referencing all the pathologies of the human
body. The problem was not to develop an image search algorithm similar to Google’s but to ac-
quire data. It would have been possible to acquire annotated data for a specific part of the human
body such as the brain, for example, but not for the whole human body. We would then have
ended up with an application that could only be used on a very specific case, that was not the goal.

So the original solution using Google’s tool was rethought. To solve the first problem and get
the results of a google search, it was discovered that it was possible to use scraping. Scraping is
generally a technique to extract content (information) from one or more websites fully automati-
cally. These are scripts, computer programs, which are responsible for extracting this information.
So with scraping the idea was to rebuild a Google-based search engine and build your own API.

To solve the second problem, an idea was to use a generic keyword that would allow Google
to identify any medical image. Several keywords were tried as medical imaging, but this was not
conclusive. The results were not accurate enough.

To gain precision and obtain images similar to the original photo, it was necessary to at least
specify the part of the human body as it was done in Figure 3. To be able to identify the part
of the human body automatically, an image recognition algorithm had to be set up. Here is an
explanatory diagram summarizing the imagined approach.

Figure 4: Imagined approach

7 Algorithm and Software Development

7.1 Image Recognition Algorithm
7.1.1 Introduction
It is important to recall that the imagined system, intended to help radiologists in their jobs, is
based on a Google search from a photo. The photo search system (google reverse image search)
does not work for medical images. Google is not able to identify medical images as it can identify
animals or common objects. Without this first identification Google is therefore not able to find
images similar to the original photo. Searching for images similar to a photo on the web is the

14
heart of the imagined application.
To solve this problem, an algorithm had to be developed to identify the part of the human body of
the medical image sought and thus use this information to search for approaching images on Google.

The algorithm to be designed must be able to recognize parts of the human body from medi-
cal images (MRI, scan, radio).

Figure 5: Desired algorithm

The algorithm to design is in fact an image classification algorithm. Classification problems are
solved pretty well today using Machine Learning and Deep Learning. Machine Learning is an
artificial intelligence technology that allows computers to learn without having been explicitly pro-
grammed for this purpose. To learn and develop, however, computers need data to analyze and
train on.

Deep Learning is a subcategory of Machine Learning. Deep Learning, like Machine Learning,
solves classification problems. The difference lies in the input data. In a Machine Learning algo-
rithm, entries are raw data such as tables filled with numbers, while in a Deep Learning algorithm
entries can be objects such as images.

In short, the learning machine can be used to classify images on the condition that the char-
acteristics of the images (color, shape, etc.) have been extracted beforehand. Deep Learning can
be used to classify images directly from the images, without having to extract their characteristics.
Both methods could have been used to solve the classification problem stated above but Deep
Learning was chosen.

15
7.2 Convolutional Neural Network
To classify images using Deep Learning there is a well-known algorithm called convolutional neural
network that has proven itself many times.

Convolutional neural networks are to date the most powerful models for classifying images. Known
by the acronym CNN, they have two distinct parts. As input, an image is provided in the form
of a pixel matrix. It has 2 dimensions for a grayscale image. The color is represented by a third
dimension, depth 3 to represent the fundamental colors (Red, Green, Blue).

The first part of a CNN is the convolutional part itself. It functions as an image feature ex-
tractor. An image is passed through a succession of filters, or convolution cores, creating new
images called convolution cards. Some intermediate filters reduce the image resolution by a local
maximum operation. In the end, the convolution cards are flattened and concatenated into a
vector of characteristics, called CNN code.

This CNN code at the output of the convolutional part is then connected at the input of a second
part, consisting of fully connected layers (multilayer perceptron). The role of this part is to com-
bine the characteristics of the CNN code to classify the image

• Data Collection

The first thing to do, before designing a classification algorithm, is to acquire data to train the
artificial intelligence algorithm. To simplify the initial problem, the problem was restricted to
the MRI classification of three parts of the human body: the knee, brain and lumbar spine. The
images used to train the algorithm have been downloaded from https://radiopaedia.org/. A total
of 1600 images were downloaded.

– 200 images for the knee axial section.

– 200 images for the knee sagittal section.

– 200 images for the knee coronal section.

– 200 images for the brain axial section.

– 200 images for the brain sagittal section.

– 200 images for the brain coronal section.

– 200 images for the lumbar rachis axial section.

– 200 images for the lumbar rachis sagittal section.

The algorithm developed has 8 possible outputs.

16
 • Data preparation

Once the data were downloaded, the images were randomly divided into 3 subparts: one part
reserved for training, one part reserved for evaluation and one part reserved for testing. 80% of
the images were used for testing, 10% were used for evaluation and 10% were used for testing. The
images were randomly split using a Python script (see appendix).

• Selected architecture and transfert learning

Once the data were downloaded, the architecture constituting the convolutional neural network
had to be chosen. Creating a new convolutional neural network is costly in terms of expertise,
hardware and the amount of annotated data required. However, it is possible to harness the power
of CNNs, with accessible hardware and a reasonable amount of annotated data. All the complex-
ity of creating CNN can be avoided by adapting publicly available pre-trained networks. These
techniques are called transfer learning because the knowledge gained on a general classification
problem is used to apply it again to a particular problem.

The architecture that has been chosen to perform a transfert learning is called InceptionV3. In-
ceptionV3 has been trained with several million images and is capable of classifying images into
1000 categories. Among these 1000 categories there are cats, dogs, houses, etc but no medical cat-
egory. However, the power of such a trained architect allows relevant features to be automatically
extracted from medical images. Once the characteristics have been extracted, two fully-connected
layers are added to classify the images into 8 categories.

The algorithm was developed in Python using the Keras library. Below is part of the algorithm
used to classify images into 8 categories (Knee, Brain, Lumbar Spine).
1 l o c a l w e i g h t s f i l e = ’C: \ \ U s e r s \\ F l o r i a n FEDERIGHI\\ Downloads \\
i n c e p t i o n v 3 w e i g h t s t f d i m o r d e r i n g t f k e r n e l s n o t o p . h5 ’
2 pre trained model = InceptionV3 (
3 i n p u t s h a p e =(150 , 1 5 0 , 3 ) , i n c l u d e t o p=F a l s e , w e i g h t s=None )
4 pre trained model . load weights ( l o c a l w e i g h t s f i l e )
5
6 for layer in pre trained model . layers :
7 layer . trainable = False
8
9 l a s t l a y e r = p r e t r a i n e d m o d e l . g e t l a y e r ( ’ mixed7 ’ )
10 p r i n t ( ’ l a s t l a y e r output shape : ’ , l a s t l a y e r . o u t p u t s h a p e )
11 l a s t o u t p u t = l a s t l a y e r . output
12
13 # F l a t t e n t h e o u t p u t l a y e r t o 1 dimension
14 x = layers . Flatten () ( last output )
15 # Add a f u l l y c o n n e c t e d l a y e r w i t h 1 ,024 h i d d e n u n i t s and ReLU a c t i v a t i o n
16 x = l a y e r s . Dense ( 1 0 2 4 , a c t i v a t i o n= ’ r e l u ’ ) ( x )
17 # Add a d r o p o u t r a t e o f 0 . 2
18 x = l a y e r s . Dropout ( 0 . 2 ) ( x )
19 # Add a f i n a l s i g m o i d l a y e r f o r c l a s s i f i c a t i o n
20 x = l a y e r s . Dense ( 8 , a c t i v a t i o n= ’ softmax ’ ) ( x )
21
22 # C o n f i g u r e and c o m p i l e t h e model

17
23 model = Model ( p r e t r a i n e d m o d e l . input , x )
24
25
26 model . c o m p i l e ( l o s s= ’ c a t e g o r i c a l c r o s s e n t r o p y ’ ,
27 o p t i m i z e r=RMSprop ( l r =0.0001) ,
28 m e t r i c s =[ ’ a c c ’ ] )

Lines 1 to 4 load the selected architecture and the weights of the pre-trained network, in this
case InceptionV3. The input image is resized to 150 by 150 pixels to accelerate the speed of the
algorithm. The images are always resize.

Lines 6-7 allow to freeze the whole network in order to converse the loaded weights.

Lines 9-14 allow to extract the characteristics of an image. It is important to note that the
extracted characteristics come from the ”mixed7” layer and not from the last layer. When per-
forming trasnfert learning to classify images completely different from the images used to train the
network, it is important not to go too far in the network. Indeed because more one advances in
the network, more the algorithm extracts characteristics specific to the classes having entrained
the algorithm.

Finally the lines 15-20 allow to build a neural network of 1024 hidden units with for input the
extracted characteristics and for output the 8 categories.

• Performance Evaluation

The last step consisted in evaluating the model developed on the images reserved for the test set.
To really measure the accuracy of the algorithm, a 5 cross validation was performed.

92 percent accuracy on average was achieved with the model developed. However this figure
does not reflect reality because the images that participated in the testing set are perfect. This
means that these images, for example, do not show any deformations or have uniform illumination.
In reality this will not be the case because when picture is taken of a screen, the picture becomes
distorted and the illumination is often not uniform.

7.3 Scraping Google Results

An API had to be developed to retrieve the results of a Google Image search. Indeed Google
does not have an API making it possible to recover results coming from a search starting from a
photograph. To do this, the best solution seemed to be the scraping method. Scraping consists in
extracting information in a page. It is possible to develop a scraping script with a large number of
programming languages but for simplicity reasons, the script in question was developed in python
language because it is the language that was used to develop deep learning script. It therefore
seemed obvious to continue with this language.

There are 2 known libraries for scraping in python: BeautifulSoup and Scrapy. These 2 libraries
are equivalent and offer more or less the same thing. Scrapy also offers the possibility to process

18
 several requests at the same time. This means that it is possible to extract information from sev-
eral web pages simultaneously. On the other hand, BeautifulSoup is compatible with the python
multiprocessing module. BeautifulSoup offers the possibility to create a script to extract informa-
tion from a page and perform other actions in the same time.

Extracting information from several pages simultaneously is not useful in API design, that’s why
BeautifulSoup was chosen to develop the API. In addition, compatibility with the multiprocessing
python module can be useful for the future.

Scraping was chosen to extract information and images from a Google Image search. Below is
a sample page where scraping was used:

Figure 6: Scraping example

Le contenu de cette page à été récupéré de la façon suivante :

1 response = r e q u e s t s . g e t ( l i n k , h e a d e r s=USER AGENT)
2 html = response . text
3 soup = B e a u t i f u l S o u p ( html , ’ lxml ’ )
4 data = soup . f i n d a l l ( ’ d i v ’ , a t t r s ={ ’ c l a s s ’ : ’ rg meta n o t r a n s l a t e ’ } , l i m i t=l i m i t )

First, the page is parsed (lines 1-2) i.e. the python script interprets the HTML content of the
page. Then the information is extracted using the BeautifulSoup library (lines 3-4). This script
provided extracts only image links and related metadata (title, domain name, source) using the
f ind all(0 div 0 , attrs = 0 class0 :0 rg metanotranslate0 , limit = limit) function. This function ex-
tracts all the information in the”<div>” called ”rg meta notranslate”. This”<div>” contains the
image link and associated metadata. The extracted information is then stored in a ”data” variable.

19
8 Conclusion
A new approach to assist radiologists in their work has been proposed. This new method has the
particularity of being able to help not only radiologists but also other health professionals using
medical imaging. This new tool could even be used by medical students. This new approach is
based on using Google Reverse Image search to search for images similar to a medical image. This
new approach allows the user to search information on a pathology from a photo of a medical
image (MRI, scan, Radio). This new approach brings more value to the whole human body and
not only to a particular part.

This new solution involves the use of artificial intelligence to identify the human body part of
a medical image. Thanks to a Deep Learning algorithm and the power of Google, a user will able
to obtain images similar to the MRI or radio he has in front of him.

A proof of concept has been developed ensuring the feasibility of this new concept. The ap-
plication developed is now able to identify 3 parts of the human body. This application will have
to be extended to the whole human body and validated by health professionals before being really
industrialized.

20
References
[1] https://developers.google.com/machine-learning/crash-course/ml-intro.

[2] https://keras.io/.

[3] Leigh Ellen Christine Potter, Jake Araullo and Lewis Carter. A Deep Learning Architecture for
Classifying Medical Images of Anatomy Object. Proceedings of APSIPA Annual Summit and
Conference 2017.

[4] https://developers.google.com/machine-learning/practica/image-classification/exercise-3.

[5] Mohamed El-Rashidy 1, Taha Taha 2, Nabil Ayad 3 and Hoda Sroor 4. An Effective Intelligent
Model for Medical Diagnosis. September 2011.

[6] Holger R. Roth, Christopher T. Lee, Hoo-Chang Shin, Ari Seff, Lauren Kim, Jianhua Yao,
Le Lu, Ronald M. Summers. ANATOMY-SPECIFIC CLASSIFICATION OF MEDICAL IM-
AGES USING DEEP CONVOLUTIONAL NETS. April 2015.

[7] Marcelo Gagliano, John Van Pham, Boyang Tang,Hiba Kashif,James Ban. Applications of
Machine Learning in Medical Diagnosis. 2017.

[8] Heba Mohsen a,*, El-Sayed A. El-Dahshan b,c, El-Sayed M. El-Horbaty d, Abdel-Badeeh M.
Salem d. Classification using deep learning neural networks for brain tumors. December 2017.

21
 Appendices
A. Appendix: Image Recognition
1 # −∗− c o d i n g : u t f −8 −∗−
2 ”””
3 Created on Sun J u l 22 2 1 : 5 3 : 3 8 2018
4
5 @author : F l o r i a n
6 ”””
7
8 import o s
9 from t e n s o r f l o w . python . k e r a s import l a y e r s
10 from t e n s o r f l o w . python . k e r a s import Model
11 from t e n s o r f l o w . python . k e r a s . o p t i m i z e r s import RMSprop
12 from t e n s o r f l o w . python . k e r a s . p r e p r o c e s s i n g . image import ImageDataGenerator
13 from t e n s o r f l o w . python . k e r a s . a p p l i c a t i o n s . i n c e p t i o n v 3 import I n c e p t i o n V 3
14
15 import os , s i g n a l
16
17
18
19
20
21
22 b a s e d i r = ’C: \ \ U s e r s \\ F l o r i a n FEDERIGHI\\ Documents \\HCUBE\\ D a t a s e t s ’
23 t r a i n d i r = o s . path . j o i n ( b a s e d i r , ’ Train ’ )
24 v a l i d a t i o n d i r = o s . path . j o i n ( b a s e d i r , ’ V a l i d a t i o n ’ )
25
26
27 l o c a l w e i g h t s f i l e = ’C: \ \ U s e r s \\ F l o r i a n FEDERIGHI\\ Downloads \\
i n c e p t i o n v 3 w e i g h t s t f d i m o r d e r i n g t f k e r n e l s n o t o p . h5 ’
28 p r e t r a i n e d m o d e l = I n c e p t i o n V 3 (
29 i n p u t s h a p e =(150 , 1 5 0 , 3 ) , i n c l u d e t o p=F a l s e , w e i g h t s=None )
30 p r e t r a i n e d m o d e l . l o a d w e i g h t s ( l o c a l w e i g h t s f i l e )
31
32 f o r l a y e r i n p r e t r a i n e d m o d e l . l a y e r s :
33 layer . trainable = False
34
35 l a s t l a y e r = p r e t r a i n e d m o d e l . g e t l a y e r ( ’ mixed7 ’ )
36 p r i n t ( ’ l a s t l a y e r output shape : ’ , l a s t l a y e r . o u t p u t s h a p e )
37 l a s t o u t p u t = l a s t l a y e r . output
38
39 # F l a t t e n t h e o u t p u t l a y e r t o 1 dimension
40 x = l a y e r s . F l a t t e n ( ) ( l a s t o u t p u t )
41 # Add a f u l l y c o n n e c t e d l a y e r w i t h 1 ,024 h i d d e n u n i t s and ReLU a c t i v a t i o n
42 x = l a y e r s . Dense ( 1 0 2 4 , a c t i v a t i o n= ’ r e l u ’ ) ( x )
43 # Add a d r o p o u t r a t e o f 0 . 2
44 x = l a y e r s . Dropout ( 0 . 2 ) ( x )
45 # Add a f i n a l s i g m o i d l a y e r f o r c l a s s i f i c a t i o n
46 x = l a y e r s . Dense ( 8 , a c t i v a t i o n= ’ softmax ’ ) ( x )
47
48 # C o n f i g u r e and c o m p i l e t h e model
49 model = Model ( p r e t r a i n e d m o d e l . input , x )
50
51
52 model . c o m p i l e ( l o s s= ’ c a t e g o r i c a l c r o s s e n t r o p y ’ ,

22
53 o p t i m i z e r=RMSprop ( l r =0.0001) ,
54 m e t r i c s =[ ’ a c c ’ ] )
55
56 t r a i n d a t a g e n = ImageDataGenerator (
57 r e s c a l e =1./255 ,
58 r o t a t i o n r a n g e =40 ,
59 w i d t h s h i f t r a n g e =0.2 ,
60 h e i g h t s h i f t r a n g e =0.2 ,
61 s h e a r r a n g e =0.2 ,
62 zoom range =0.2 ,
63 h o r i z o n t a l f l i p=True )
64
65 # Note t h a t t h e v a l i d a t i o n d a t a s h o u l d not be augmented !
66 t e s t d a t a g e n = ImageDataGenerator ( r e s c a l e =1./255)
67
68 train generator = train datagen . flow from directory (
69 t r a i n d i r , # This i s t h e s o u r c e d i r e c t o r y f o r t r a i n i n g images
70 t a r g e t s i z e =(150 , 1 5 0 ) , # A l l images w i l l be r e s i z e d t o 150 x150
71 b a t c h s i z e =5,
72 # S i n c e we use b i n a r y c r o s s e n t r o p y l o s s , we need b i n a r y l a b e l s
73 c l a s s m o d e= ’ c a t e g o r i c a l ’ )
74
75 # Flow v a l i d a t i o n images i n b a t c h e s o f 20 u s i n g t e s t d a t a g e n g e n e r a t o r
76 validation generator = test datagen . flow from directory (
77 validation dir ,
78 t a r g e t s i z e =(150 , 1 5 0 ) ,
79 b a t c h s i z e =5,
80 c l a s s m o d e= ’ c a t e g o r i c a l ’ )
81
82 h i s t o r y = model . f i t g e n e r a t o r (
83 train generator ,
84 s t e p s p e r e p o c h =120 ,
85 e p o c h s =20 ,
86 v a l i d a t i o n d a t a=v a l i d a t i o n g e n e r a t o r ,
87 v a l i d a t i o n s t e p s =10 ,
88 v e r b o s e =2)
89 # s e r i a l i z e model t o YAML
90 model yaml = model . t o y a m l ( )
91 with open ( ” model . yaml ” , ”w” ) a s y a m l f i l e :
92 y a m l f i l e . w r i t e ( model yaml )
93 # s e r i a l i z e w e i g h t s t o HDF5
94 model . s a v e w e i g h t s ( ” model hcube . h5” )
95
96 model . s a v e ( ” model hcube . h5” )
97 o s . k i l l ( o s . g e t p i d ( ) , s i g n a l . SIGKILL
98 \ newpage

B. Appendix: Scraping Script

1 d e f getHtml ( u r l , keyword ) :
2 g o o g l e u r l = ’ h t t p : / / images . g o o g l e . com/ s e a r c h b y i m a g e ? i m a g e u r l= ’+u r l+ ’&q= ’+
keyword
3 r e s p o n s e = r e q u e s t s . g e t ( g o o g l e u r l , h e a d e r s=USER AGENT)
4 response . r a i s e f o r s t a t u s ()
5 return response . text
6
7

23
 8
9 def scrapSImgsUrl ( g o o g l e u r l ) :
10 s i m i l a r I m g s U r l= ’ ’
11 r e s p o n s e = r e q u e s t s . g e t ( g o o g l e u r l , h e a d e r s=USER AGENT)
12 #p r i n t ( r e s p o n s e )
13 #r e s p o n s e . r a i s e f o r s t a t u s ( )
14 html = r e s p o n s e . t e x t
15 soup = B e a u t i f u l S o u p ( html , ’ lxml ’ )
16 l i n k = soup . f i n d ( ’ a ’ , a t t r s ={ ’ c l a s s ’ : ’ iu−card−h e a d e r ’ } , h r e f=True )
17 #p r i n t ( l i n k )
18 i f link :
19 s i m i l a r I m g s U r l = ’ h t t p s : / / g o o g l e . com ’+l i n k [ ’ h r e f ’ ]
20 return similarImgsUrl
21
22
23 d e f scrapImg ( l i n k , l i m i t ) :
24 r e s p o n s e = r e q u e s t s . g e t ( l i n k , h e a d e r s=USER AGENT)
25 html = r e s p o n s e . t e x t
26 soup = B e a u t i f u l S o u p ( html , ’ lxml ’ )
27 data = soup . f i n d a l l ( ’ d i v ’ , a t t r s ={ ’ c l a s s ’ : ’ rg meta n o t r a n s l a t e ’ } , l i m i t=l i m i t )
28 r e t u r n data
29
30
31 d e f e x p o r t R e s u l t s T o T x t F i l e ( data ) :
32 path=” . / images / t e s t . t x t ”
33 output = open ( path , ”wb” )
34 f o r i i n r a n g e ( l e n ( data ) ) :
35 d i c t i o = a s t . l i t e r a l e v a l ( data [ i ] . s t r i n g )
36 output . w r i t e ( d i c t i o [ ” pt ” ] . encode ( ” u t f 8 ” ) )
37 #s t r=d i c t i o [ ” p t ” ]
38 output . w r i t e ( ” | | ” . encode ( ” u t f 8 ” ) )
39 output . w r i t e ( d i c t i o [ ” i s u ” ] . encode ( ” u t f 8 ” ) )
40 output . w r i t e ( ” | | ” . encode ( ” u t f 8 ” ) )
41 output . w r i t e ( d i c t i o [ ” tu ” ] . encode ( ” u t f 8 ” ) )
42 output . w r i t e ( ” | | ” . encode ( ” u t f 8 ” ) )
43 output . w r i t e ( d i c t i o [ ” ru ” ] . encode ( ” u t f 8 ” ) )
44 output . w r i t e ( ( o s . l i n e s e p ) . encode ( ” u t f 8 ” ) )
45 output . c l o s e ( )
46
47
48 d e f e x p o r t R e s u l t s T o S t r i n g ( data ) :
49 metadata=” ”
50 f o r i i n r a n g e ( l e n ( data ) ) :
51 d i c t i o = a s t . l i t e r a l e v a l ( data [ i ] . s t r i n g )
52 metadata=metadata+d i c t i o [ ” pt ” ]
53 metadata=metadata+” | | ”
54 metadata=metadata+d i c t i o [ ” i s u ” ]
55 metadata=metadata+” | | ”
56 metadata=metadata+d i c t i o [ ” tu ” ]
57 metadata=metadata+” | | ”
58 metadata=metadata+d i c t i o [ ” ru ” ]
59 metadata=metadata+” \n”
60 r e t u r n metadata
61
62
63

24
64 d e f searchByImageUpload ( img , keyword ) :
65 f i l e P a t h = img
66 s e a r c h U r l = ’ h t t p : / /www. g o o g l e . hr / s e a r c h b y i m a g e / upload ’
67 m u l t i p a r t = { ’ encoded image ’ : ( f i l e P a t h , open ( f i l e P a t h , ’ rb ’ ) ) , ’ q ’ : keyword }
68 r e s p o n s e = r e q u e s t s . p o s t ( s e a r c h U r l , f i l e s =m u l t i p a r t , a l l o w r e d i r e c t s=F a l s e )
69 fetchUrl = response . headers [ ’ Location ’ ]
70 return fetchUrl

25