AW S A c a d e m y M a c h i n e L e a r n i n g F o u n d a t i o n s

Module 5: Introducing Computer Vision

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module overview

Sections Demonstrations
1. Introducing computer vision 1. Introducing Amazon Rekognition
2. Analyzing images and videos 2. Labeling Images with Amazon
SageMaker Ground Truth
3. Preparing custom datasets for computer
vision Lab
4. Module wrap-up Guided Lab: Facial Recognition

Knowledge check
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 2
Module objectives

At the end of this module, you should be able to:

• Describe the use cases for computer vision
• Describe the Amazon managed machine learning (ML) services for image and video
analysis
• List the steps required to prepare a custom dataset for object detection
• Describe how Amazon SageMaker Ground Truth can be used to prepare a custom
dataset
• Use Amazon Rekognition to perform facial detection

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 3
M o d u l e 5 : I n t ro d u c i n g C o m p u t e r Vi s i o n

Section 1: Introducing computer vision

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Computer vision overview

Computer vision is the automated extraction of

information from digital images.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 5
Computer vision applications

Public safety Authentication and enhanced Content management

and home security computer-human interaction and analysis

Autonomous driving Medical imaging Manufacturing process control

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 6
Computer vision problems

Content recognition
Image analysis
• Object classification
Food?
Breakfast?
Lunch?
Dinner?

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 7
Computer vision problems
Confidence
Content recognition
Image analysis Milk 97.1
• Object classification
Peaches 92.3
• Object detection
Ice Cream 97.1
Salad 69.5
Nuggets 77.5
Bread Roll 94.5
Bounding boxes
(top,left,width,height)
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 8
Computer vision problems

Content recognition
Image analysis Milk
• Object classification Peaches
• Object detection Ice Cream
• Object segmentation
Salad
Nuggets
Bread Roll

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 9
Computer vision use cases

Content recognition
Video analysis
• Instance tracking

Pathing – You can capture the

path of people in the scene. For
example, you can use the
movement of athletes during a
game to identify plays for post-
game analysis.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 10
Computer vision use cases

Content recognition
Video analysis
• Instance tracking
• Action recognition

Analyze shopper behavior and density in

your retail store by studying the path that
each person follows

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 11
Computer vision use cases

Content recognition
Video analysis
• Instance tracking
• Action recognition
• Motion estimation

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 12
 • Computer vision is the automated
Section 1 key extraction of information from
takeaways images
• Image analysis includes object
classification, detection, and
segmentation
• Video analysis includes instance
tracking, action recognition, and
motion estimation

13 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
M o d u l e 5 : I n t ro d u c i n g C o m p u t e r Vi s i o n

Section 2: Analyzing images and videos

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Rekognition

• Managed service for image and

video analysis
• Types of analysis
• Searchable image and video
libraries
• Face-based user verification
• Sentiment and demographic
analysis
• Unsafe content detection
• Text detection
• Security and compliance
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 15
Amazon Rekognition

Can add powerful visual analysis to your application

Is highly scalable and continuously learns

Integrates with other AWS services

Languages supported by the Amazon Rekognition SDKs:

JavaScript Python PHP .NET Ruby Java Go Node.js C++

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 16
Use case 1: Searchable image library

1 2 3 4

Mobile App Amazon S3

AWS Lambda Amazon Rekognition

6 5

Mobile App Amazon Elasticsearch

Service

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 17
Use case 2: Image moderation

1 2 3

Users Amazon S3
AWS Lambda Amazon Rekognition
6
Inappropriate Appropriate
content content
detected detected

5 4

Manual Pictures
Review Posted

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 18
Use case 3: Sentiment analysis

1 2 3

In-store camera Amazon Kinesis Amazon Rekognition Amazon Kinesis

Video Streams Data Streams

4
7 6 5

Amazon QuickSight Amazon Redshift Amazon S3

AWS Lambda

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 19
Using Amazon Rekognition

API operations JSON results

Detect Labels
S3 bucket Detect Faces "Name": "Car",
Recognizing Celebrities "Confidence": 96.9476089477539,
Detecting Unsafe Images "Instances": [
{
"BoundingBox": {
01100110 "Width": 0.07903634756803513,
Byte stream "Height": 0.08699918538331985,
Amazon Rekognition "Left": 0.7790974974632263,
"Top": 0.6977462768554688
},
"Confidence": 96.9476089477539
Image.jpg }

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 20
Image analysis: Object and scene detection

Person Architecture Car

98.9% 92.2% 97.9%

Pedestrian Building Downtown

90.6% 97.6% 95.6%

City
97.6%

Road Street Urban

93.0% 93.0% 97.6%
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 21
 Image analysis: Object and scene detection

"Name": "Car",
"Confidence": 96.9476089477539,
"Instances": [
x = imageHeight * BoundingBox[“Left”] {
y = imageWidth * BoundingBox[“Top”] "BoundingBox": {
"Width":
0.07903634756803513,
"Height":
0.08699918538331985,
"Left": 0.7790974974632263,
"Top": 0.6977462768554688
},
"Confidence": 96.9476089477539
}

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 22
Face detection

• Bounding box
• Attributes
• Emotions
• Facial landmarks
• Quality
• Pose
• Confidence score

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 23
Using facial recognition

• Compare source with target

• Face match –
• Bounding box and confidence
• Similarity score
• Facial landmark locations
• Source face information –
• Bounding box and confidence
• Facial landmarks
• Unmatched face –
• Bounding box and confidence
• Facial landmarks
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 24
Searching for known faces

• Must train the model

{
• Use images with target faces "FaceModelVersion": "string",
"Faces": [
• Perform facial recognition {
"BoundingBox": {
"Height": number,
• Store facial metadata "Left": number,
"Top": number,
Response "Width": number
},
"Confidence": number,
Create collection
Initialize "ExternalImageId": "string",
"FaceId": "string",
"ImageId": "string"
Perform facial }
Index faces recognition ],
Collection "NextToken": "string"
}
Amazon Rekognition Multidimensional facial features
Metadata
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 25
Searching for Known Faces

Multidimensional facial features

SearchFacesByImage
Collection Metadata

Amazon Rekognition
Response
{ "FaceMatches": [ {
"Face": {
"BoundingBox": { "Height": 0.0633333027, "Left": 0.171851992, "Top":
0.73666697, "Width": 0.1106169968 },
"Confidence": 100,
"ExternalImageId": "input.jpg",
"FaceId": "578e2e1b-d0b0-493c-aa39-ba476a421a34",
"ImageId": "9ba38e68-35b6-5509-9d2e-fcffa75d1653" },
"Similarity": 99.9764175415039 } ],
"FaceModelVersion": "3.0",
"SearchedFaceBoundingBox": { "Height": 0.063333332, "Left": 0.171851858, "Top":
0.73666667, "Width": 0.11061728 },
"SearchedFaceConfidence": 99.99999237060547 }
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 26
Guidelines

• Facial detection
• Bounding box, attributes, emotions, landmarks, quality, pose
• Confidence score
• Detection is based on image data
• Gender based on image, not identity
• Emotions are inferred from physical appearance
• You can use the appropriate confidence score for your use case
• Facial recognition should never be used in a way that violates an
individual’s rights, including the right to privacy, or makes autonomous
decisions for scenarios that require analysis by a human*

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 27
Demonstration:
Introducing
Amazon
Rekognition

28 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Working with stored videos

Process: H.264
MOV/MPEG-4
• Start detection Max 10 GB
• People, faces, labels, celebrities, text, Start detection
inappropriate content
• Monitor Amazon Simple Queue Existing video
Service (Amazon SQS) queue for
completion
Topic Amazon
• Get detection results
Monitor SQS Rekognition

Client Queue

Get detection results

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 29
Working with streaming videos

Application process:
1. Stream video to Amazon Kinesis Video
Streams
Amazon Kinesis
2. Connect Amazon Rekognition Video Video Streams
stream processor
3. Read analysis from the Amazon
Kinesis data stream
Amazon Rekognition

Client
Amazon Kinesis
Data Streams

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 30
 • You can use Amazon Rekognition for
Section 2 Key image and video analysis that uses proven,
Takeaways highly scalable, deep learning technology
that doesn’t require machine learning
expertise
• Provides image and video detection of
faces, sentiment, text, unsafe content, and
library search
• Is integrated into other AWS services

31 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
M o d u l e 5 : I n t ro d u c i n g C o m p u t e r Vi s i o n
Section 3: Preparing custom datasets for computer
vision

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Video analysis, labels, Ben, and Metric

AWS DeepLens Powered Cat Flap

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 33
Models need training on domain

Wood 97%
Canvas 88.7%
Text 84.7%
Envelope 71.7%
Greeting Card 70.4%
Mail 70.4%
Hardwood 60.6%
Plywood 60.4%
Paper 57.7%
Advertisement 55.8%
Poster 55.8%
Art 55.6%

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 34
Amazon Rekognition Custom Labels

• Use for –
• Search for logos
• Identify products
• Identify machine parts
• Distinguish between healthy and infected plants
• Almost all vision solutions start with an existing model
• Benefits –
• Simplified data labeling
• Automated machine learning
• Simplified model evaluation, inference, and feedback

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 35
Custom labeling process

1: Collect images 2: Create training dataset 3: Create test dataset 4: Train the model 5: Evaluate 6: Use Model

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 36
Step 1: Collect images

1: Collect images 2: Create training dataset 3: Create test dataset 4: Train the model 5: Evaluate 6: Use Model

• Typically use a few hundred images

• Build domain-specific models
• Use 10 PNG or JPEG images per label
• Use images similar to the images that you want to detect

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 37
Step 2: Create training dataset

1: Collect images 2: Create training dataset 3: Create test dataset 4: Train the model 5: Evaluate 6: Use Model

• Dataset: Data about images, labels, and bounding box

• Create at least two labels
• Label the images by using the console or Amazon SageMaker Ground Truth

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 38
Image-level versus object-level labels

Scenes and concepts Objects with bounding boxes

Label: beach Label: Echo Dot

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 39
Images need labels

{
"source-ref": "s3://b/…/img_2783.jpg",
"small_beach": 1,
"small_beach-metadata": {
"confidence": 1,
"job-name": "labeling-job/small_beach",
"class-name": “Beach",
"human-annotated": "yes",
"creation-date": "2020-03-02T20:42:03.525Z",
"type": "groundtruth/image-classification"
}
}
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 40
Objects need bounding boxes

"small_BB": { "annotations": [
{ "left": 186,"top": 125,”width": 346,"height": 580,
"annotationType": "bounding-box", "id": 5,"class_id": 0},
{ "left": 732,"top": 445,"width": 270,"height": 192,
"annotationType": "bounding-box", "id": 6,"class_id": 1}],
"image_size": [
{ "width": 1152,"height": 864,"depth": 3 }]},
"small_BB-metadata": {
"job-name": "labeling-job/small_BB",
"class-map": { "0": "Echo", "1": "Echo dot“ },
"human-annotated": "yes",
"objects": [ {"confidence": 1}, {"confidence": 1}],
"creation-date": "2020-03-02T20:42:25.930Z",
"type": "groundtruth/object-detection"

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 41
Amazon SageMaker Ground Truth

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 42
Step 3: Create test dataset

1: Collect images 2: Create training dataset 3: Create test dataset 4: Train the model 5: Evaluate 6: Use Model

Training set Testing set

0% 80% 100%

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 43
Step 4: Train the model

1: Collect images 2: Create training dataset 3: Create test dataset 4: Train the model 5: Evaluate 6: Use Model

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 44
Step 5: Evaluate - Metrics

1: Collect Images 2: Create training dataset 3: Create test dataset 4: Train the model 5: Evaluate 6: Use Model

• Evaluate model performance Confusion Matrix

• Metrics Actual
• Precision Cat Not a Cat
• Recall

Predicted
• Overall model performance Cat TP FP
Not a Cat FN TN
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 45
Step 5: Evaluate – Improve

1: Collect Images 2: Create training dataset 3: Create test dataset 4: Train the model 5: Evaluate 6: Use Model

• Data
• Reducing false positives (better precision)
• Adjust the confidence threshold to improve precision
• Add additional classes as labels for training
• Reducing false negatives (better recall)
• Lower the confidence threshold to improve recall
• Use better data or more precise classes (labels) for training
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 46
Step 6: Use the model

1: Collect Images 2: Create training dataset 3: Create test dataset 4: Train the model 5: Evaluate 6: Use Model

aws rekognition detect-custom-labels --project-version-arn "model_arn"\

--image '{"S3Object":{"Bucket":"bucket","Name":"image"}}'\
--min-confidence 70

Returns array of custom labels:

• Label
• Bounding box for objects
• Confidence
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 47
Demonstration:
Labeling Images
with Amazon
SageMaker Ground
Truth

48 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 • Models must be trained for the
Section 3 key specific domain that you want to
takeaways analyze
• You can set custom labeling for the
specific business case
• Custom labeling workflow
• You must label images and create
bounding boxes for objects
• You can use Amazon SageMaker
Ground Truth to build training
datasets for your models

49 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module 5 – Guided
Lab:
Facial Recognition

50 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
M o d u l e 5 : I n t r o d u c i n g C o m p u t e r Vi s i o n

Module wrap-up

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Module summary

In summary, in this module, you learned how to:

• Describe use cases for computer vision
• Describe the AWS managed machine learning (ML) services for image and video
analysis
• List the steps required to prepare a custom dataset for object detection
• Describe how Amazon SageMaker Ground Truth can be used to prepare a custom
dataset
• Use Amazon Rekognition to perform facial detection

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 52
Complete the knowledge check

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 53
Additional resources

• What is Amazon Rekognition

• Welcoming Amazon Rekognition Video: Deep-Learning Based Video Recog
nition
• Classify a Large Number of Images with Amazon Rekognition and AWS Bat
ch

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 54
Thank you

© 2021 Amazon Web Services, Inc. or its affiliates. All rights reserved. This work may not be reproduced or redistributed, in whole or in part, without prior written permission from Amazon
Web Services, Inc. Commercial copying, lending, or selling is prohibited. Corrections, feedback, or other questions? Contact us at https://support.aws.amazon.com/#/contacts/aws-training.
All trademarks are the property of their owners.