INTRODUCTION TO TENSORFLOW,

DEEP LEARNING & APPLICATION

(COMPUTER VISION –
OBJECT DETECTION)
MODULE 10
 • WHAT IS OBJECT DETECTION
• DEEP LEARNING
• WHAT IS DEEP LEARNING
• DEEP LEARNING VS MACHINE LEARNING

OUTLINE • DEEP LEARNING APPROACHES

• TENSORFLOW OBJECT DETECTION API

• PREPARING DATA
• TRAINING & EVALUATING
WHAT IS OBJECT
DETECTION
 OBJECT DETECTION =
OBJECT CLASSIFICATION + OBJECT LOCALIZATION
 ONE MODEL FOR TWO TASKS?

Po - is object
exists
bx1
- bounding box
bx2 coordinates
Object detection - output is the one number (index) of a class
by1

c1
by2

c2 - object’s
variables
c3
…

Object localization - output is the four numbers - cn

coordinates of bounding box.
DEEP LEARNING
WHAT IS DEEP LEARNING

• DEEP LEARNING IS AN AI FUNCTION THAT MIMICS THE WORKINGS OF THE

HUMAN BRAIN IN PROCESSING DATA FOR USE IN DETECTING OBJECTS,
RECOGNIZING SPEECH, TRANSLATING LANGUAGES, AND MAKING
DECISIONS.
• DEEP LEARNING AI CAN LEARN WITHOUT HUMAN SUPERVISION, DRAWING
FROM DATA THAT IS BOTH UNSTRUCTURED AND UNLABELED.
• DEEP LEARNING, A FORM OF MACHINE LEARNING, CAN BE USED TO HELP
DETECT FRAUD OR MONEY LAUNDERING, AMONG OTHER FUNCTIONS.
DEEP LEARNING VS MACHINE LEARNING
 • CLASSICAL APPROACH
(HAAR FEATURES) - FIRST
OBJECT DETECTION REAL
TIME FRAMEWORK (
VIOLA-JONES)
• DEEP LEARNING APPROACH
- NOW STATE OF THE ART IN
OBJECT DETECTION
APPROACHES • OVERFEAT
• R-CNN
• FAST R-CNN
• YOLO
• FASTER R-CNN
• SSD AND R-FCN
 DEEP LEARNING
APPROACH
OverFeat - published in 2013, multi-scale
sliding window algorithm using Convolutional
Neural Networks (CNNs).

C.NN - Regions with CNN features. Three stage

approach:
- Extract possible objects using a region
proposa method (the most popular one being l
Selective Search).
- Extract features from each region using a
CNN.
- Classify each region with SVMs.
 DEEP LEARNING APPROACH
Fast R-CNN - Similar to R-CNN, it used Selective
Search to generate object proposals, but instead of
extracting all of them independently and using SVM
classifiers, it applied the CNN on the complete
image and then used both Region of Interest (RoI)
Pooling on the feature map with a final feed
forward network for classification and regression.

YOLO - You Only Look Once: a

simple convolutional neural
network approach which has
both great results and high
speed, allowing for the first
time real time object
detection.
 DEEP LEARNING APPROACH
Faster R-CNN - Faster R-CNN added what
they called a Region Proposal Network
(RPN), in an attempt to get rid of the
Selective Search algorithm and make the
model completely trainable end-to-end.

SSD and R-FCN

Finally, there are two notable papers, Single Shot
Detector (SSD) which takes on YOLO by using
multiple sized convolutional feature maps achieving
better results and speed, and Region-based Fully
Convolutional Networks (R-FCN) which takes the
architecture of Faster R-CNN but with only
convolutional networks.
TENSORFLOW OBJECT
DETECTION API
 • OPEN SOURCE FROM 2017-07-
15
• BUILT ON TOP OF
TENSORFLOW
•
TF OBJECT CONTAINS TRAINABLE
DETECTION MODELS

DETECTION • CONTAINS FROZEN WEIGHTS

• CONTAINS JUPYTER
API NOTEBOOK
• MAKES EASY TO
CONSTRUCT, TRAIN AND
DEPLOY
• OBJECT DETECTION MODELS
• DEPENDENCIES: GETTING STARTED

If model will be trained locally - better

▪ PROTOBUF 2.6 to install tensorflow-gpu.
▪ PYTHON-TK
▪ PILLOW 1.0 Dependencies for tensorflow-gpu:
▪ LXML ▪ NVIDIA GPU with CUDA Compute Capability 3.0
▪ TF SLIM (INCLUDED) (list)
▪ Ubuntu 16.04 at least
▪ JUPYTER NOTEBOOK
▪ CUDA® Toolkit 9.0
▪ MATPLOTLIB ▪ NVIDIA drivers associated with CUDA Toolkit
▪ TENSORFLOW 9.0.
(TENSORFLOW- ▪ cuDNN v7.0
• GPU) ▪ libcupti-dev
Installation
▪ CYTHON instruction
▪COCOAPI Latest version of CUDA Toolkit - 9.1
INSTALLATION not compatible with tensorflow 1.6,
need to install 9.0
INSTRUCTION
CREATING A
DATASET
 • TENSORFLOW OBJECT
DETECTION API USES THE
TFRECORD FILE FORMAT
• THERE IS AVAILABLE THIRD-
PARTY SCRIPTS TO CONVERT

DATASET • PASCAL VOC AND OXFORD PET

FORMAT
• IN OTHER CASE EXPLANATION OF
FORMAT AVAILABLE IN GIT REPO.
• INPUT DATA TO CREATE TFRECORD –
ANNOTATED IMAGE
 GETTING IMAGES
• CREATE OWN IMAGES
Grab from internet ▪ RECORD VIDEO WITH NEEDED
▪ Scrap images from google or • OBJECT/OBJECTS (IN 640X480)
Pixabay or whatever ▪ PROCESS VIDEO AND SPLIT ON
▪ For batch downloading SCREENSHOTS - FFMPEG
- Faktun Bulk Image • TIPS
Downloader
▪ CREATE IMAGES WITH DIFFERENT
▪ For data mining by
multiplying existing images - • LIGHTS, BACKGROUND AND SO ON.
ImageMagic ▪ IF OBJECT IS ABLE TO HAVE
DIFFERENT FORMS - BETTER
TO CATCH THEM ALL.
▪ TRY TO MAKE 30%-50% OF
• IMAGES WITH OVERLAID
OBJECT
▪ TOOL FOR IMAGE
AUGMENTATION
 LABELING (ANNOTATION) AN
IMAGES

Tools

▪ LabelImg ▪ input: images

▪ FIAT (Fast Image ▪ output: .xml files with
Data Annotation bounding boxes
Tool) coordinates
 CREATING TFRECORD

▪ Tensorflow object detection API repo contains folder dataset_tools with scripts
to coverts common structures of data in TFRecord.

▪ If output data has another structure - here is explanation how to convert it

TRAINING
 TENSORFLOW OD API PROVIDES A

SELECTING A COLLECTION OF DETECTION

MODELS PRE-TRAINED ON THE COCO
DATASET, THE KITTI DATASET, AND

MODEL THE OPEN IMAGES DATASET.

• MODEL NAME CORRESPONDS TO

A CONFIG FILE THAT WAS USED
TO TRAIN THIS MODEL.
• SPEED - RUNNING TIME IN MS
PER 600X600 IMAGE
• MAP STANDS FOR MEAN
AVERAGE PRECISION, WHICH
INDICATES HOW WELL THE
MODEL PERFORMED ON THE
COCO DATASET.
• OUTPUTS TYPES (BOXES, AND
MASKS IF
• APPLICABLE)
 CONFIGURING
● Folders structure ● pipeline.config
train_config: {
fine_tune_checkpoint: "<path_to_model.ckpt>"
num_steps: 200000
}
train_input_reader {
label_map_path: "<path_to_labels.pbtxt>"
tf_record_input_reader {
input_path: "<path_to_train.record>"
}
}
eval_config
{ num_examples:
8000
● label.pbtxt max_evals: 10
use_moving_averages: false
}
eval_input_reader {
label_map_path: "<path_to_labels.pbtxt>"
shuffle: false
num_readers: 1
tf_record_input_reader {
input_path:
"<path_to_test.record>
"
}
}
 TRAINING &
EVALUATING
# From the tensorflow/models/research directory
python object_detection/train.py
--logtostderr
--
pipeline_config_path=/tensorflow/models/object_detection/samples/configs/ssd_mobilenet_v1_p
ets.config
--train_dir=${PATH_TO_ROOT_TRAIN_FOLDER}

# From the tensorflow/models/research directory

python object_detection/eval.py \
--logtostderr \
--pipeline_config_path=$
{PATH_TO_YOUR_PIPELINE_CONFIG} \
--checkpoint_dir=${PATH_TO_TRAIN_DIR} \
--eval_dir=${PATH_TO_EVAL_DIR}