Intel® Distribution of OpenVINO™ Toolkit

Digital Courseware for

Educators
Course: Deploying Deep Learning Applications
Intel® Distribution of OpenVINO™ Toolkit

Deploying Deep Learning

Applications
MODULE 5: Streamlining AI Application Development and
Deployment with Deep Learning Workbench
Notices and disclaimers

Performance varies by use, configuration, and other factors. Learn more at intel.com/PerformanceIndex .
Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. See backup for
configuration details. No product or component can be absolutely secure.
Your costs and results may vary.
Intel® technologies may require enabled hardware, software, or service activation.
Intel® optimizations, for Intel® compilers or other products, may not optimize to the same degree for non-Intel products.
Intel does not control or audit third-party data. You should consult other sources to evaluate accuracy.
Results have been estimated or simulated.
Intel is committed to respecting human rights and avoiding complicity in human rights abuses.
See Intel’s Global Human Rights Principles. Intel® products and software are intended only to be used in
applications that do not cause or contribute to a violation of an internationally recognized human right.
© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries.
Other names and brands may be claimed as the property of others.

3
Module 5
Streamlining AI Application Development and
Deployment with Deep Learning Workbench

4
Table of Contents
• Introduction to Deep Learning Workbench ▪ Hands-on Lab
• How Does the Deep Learning Workbench ▪ Exercise 1 : Tiny YOLO* V3 Object Detection
Work? ▪ Assignment: Safety Gear Detection
• Deep Learning Workbench Workflow
• Step by step procedure to execute and see the
performance of a model using Deep Learning
Workbench in the Intel® Dev Cloud for Edge
Workloads

5
Module 5: Learning Objective

• Get familiar with the Deep Learning Workbench.

• Understand the various features of the Deep Learning Workbench workflow.
• Get introduced to various benchmarking tools present in the Deep Learning Workbench.
• Practically apply knowledge gained from the tools available in the OpenVINO™ toolkit.
• Use the Deep Learning Workbench on Intel® DevCloud for Edge Workloads or locally. Practical
application of knowledge gained from the tools available in OpenVINO™ toolkit.

6
Module 5: Learning Outcomes

After completing this module, students/learners should be able to:

• Use the Deep Learning Workbench to practically apply knowledge from previous modules.
• Begin optimized Deep Learning Inference projects using either the CLI or the GUI-based
approach.
• Develop effective decision-making skills to choose the ideal hardware platforms for a project's
requirements.

7
Module 5: Key Questions Addressed

• What is Deep Learning Workbench?

• How does the Deep Learning Workbench work?
• What types of benchmarking are available on the Deep Learning Workbench?
• How to get started with the Deep Learning Workbench on Intel® DevCloud for Edge Workloads?

8
Deep Learning Workbench
▪ Web-based, UI extension tool of the Intel® Distribution of
OpenVINO™ toolkit
▪ Visualizes performance data for topologies and layers to
aid in model analysis
▪ Automates analysis for optimal performance
configuration (streams, batches, latency)
▪ Experiment with INT8 or Winograd calibration for
optimal tuning using the Post Training Optimization Tool
▪ Provide accuracy information through accuracy checker
▪ Direct access to models from public set of Open Model
Zoo
▪ Enables remote profiling, allowing the collection of
performance data from multiple different machines
without any additional set-up.

Development Guide
https://docs.openvino.ai/latest/workbench_docs_Workbench_DG_Introduction.html 9
Deep Learning Workbench
User Goals
▪ DL Workbench helps achieve your goals depending on the stage of your deep learning journey.
▪ If you are a beginner in the deep learning field, the DL Workbench provides you with learning opportunities:
• Learn what neural networks are, how they work, and how to examine their architectures.
• Learn the basics of neural network analysis and optimization before production.
• Get familiar with the OpenVINO™ ecosystem and its main components without installing it on your system.
▪ If you have enough experience with neural networks, DL Workbench provides you with a convenient web interface
to optimize your model and prepare it for production:
• Measure and interpret model performance.
• Tune the model for enhanced performance.
▪ Analyze the quality of your model and visualize output.

Development Guide
https://docs.openvino.ai/latest/workbench_docs_Workbench_DG_Introduction.html 10
Discussion Points
• Discuss what you've learned so far from the previous modules.
• What steps are involved in optimizing a model using the OpenVINO™ toolkit?
• What are the benefits of the visual workflow included in OpenVINO™ toolkit?
• What are some of the features of the Deep Learning Workbench?

11
How Does the Deep Learning Workbench Work?
To execute the workbench on the Intel® DevCloud for Edge Workloads as depicted in figure below,
refer to the instructions in this link.

For installing the workbench on your local machine follow the instructions on this link 12
Deep Learning Workbench Workflow
The general workflow remains nearly the same as what we have discussed with the OpenVINO™
toolkit

For installing the workbench on your local machine follow the instructions on this link 13
Step by Step Procedure to Execute and See the Performance of a Model using
Deep Learning Workbench on the Intel® Dev Cloud for Edge Workloads

Step 1 : Create a Project

14
1.1 The Model –
In this step, we will Select a Model to be used for performance benchmarking

Deep Learning Workbench Homepage to Select a Model

15
The following step is to import the model from “Open Model Zoo” or your own model using the “Original Model”
tab, as shown in below figure

Deep Learning Workbench Homepage to Import a Model

16
Then, as shown in figure below, after selecting a model from Open Model Zoo or uploading your own Original
Model, click the “DOWNLOAD and IMPORT” button.

Deep Learning Workbench Homepage for Importing a Model

17
1.2 Prepare the Environment -
The next step it starts to Prepare the Environment according to the model which have been
imported and wait for the status to get completed to proceed to the next steps

Deep Learning Workbench Homepage Preparing the Environment

18
1.3. Convert Model to IR - General Parameters
At this stage, you are converting the model to the OpenVINO™ Intermediate Representation (IR).

Deep Learning Workbench Homepage for Converting the Model to IR

19
2. Select Environment
Select the Environment or the system that you want to run your project, as we are running the DL
workbench on Intel® DevCloud for Edge Workloads, we can select the array of options available according
to our use case by using “Target” and “Platform”

Deep Learning Workbench Homepage for Environment Selection through Target

and Platform

20
Before launching the project, you must first select the Hardware platform on which you want your
model to run.

Deep Learning Workbench Homepage to Select the Hardware

platform

21
3. Select the Validation Dataset

Validation of the model is always performed against

specific data combined into datasets. To obtain
trustworthy results, a dataset must satisfy the
following requirements:

Format: the data should be compatible with the

model domain (Computer Vision or Natural
Language Processing).

Content: a dataset must be representative. The data

needs to be aligned with the model use case (for
example, images of people for face detection).

Deep Learning Workbench Homepage to Select Validation Dataset

22
In next step, choose an image dataset.

Image datasets can be either Annotated or Not Annotated.

Not Annotated dataset contains only images and allows using most
of the DL Workbench features- measure performance, optimize, and
visualize the model, etc.

Annotated dataset contains images and information about what

each image is showing. It expands the possibilities of working with a
model and allows measuring accuracy and optimizing the model
within a controllable accuracy drop.

Deep Learning Workbench Homepage to Select Image Dataset

23
After uploading the dataset, click “Create Project,” which will generate a project based on all of the choices
we made in the previous steps.

Deep Learning Workbench Homepage while Inferencing is Being Performed

24
Once completed, click the “Download Report” button to see all of the results for your corresponding
model in terms of Throughput, Latency, and other parameters.

Deep Learning Workbench screen to Download Report

25
Following below is an example of a report generated for the model AlexNet* with a Custom validation
Dataset in terms of Throughput measured in FPS and Latency measured in milliseconds,

26
We can use the accuracy checker to visualize the model predictions and measure their accuracy. Figure
below shows an example of visualization.

Report Visualization in Deep Learning

Workbench

27
We can also generate a Performance Summary Report for a specific model, as shown in Fig below, and compare the
results.

Performance Summary in Deep Learning

Workbench

28
Additionally, a Performance Improvements Suggestion is recommended, which can be done by clicking on
the “Perform INT8 calibration to have layers executed in the 8-bit integer precision.”

Deep Learning Workbench for Performance

Improvement

29
Discussion Points
• How can you use the Deep Learning Workbench?
• What are some of the methods that you can use to optimize your model on the Deep Learning Workbench?
• What are some differences between the CLI-based and GUI-based workflow you found after going through the
Deep Learning Workbench’s workflow?
• What are the benefits of using layer-wise execution time coloring?
• What are the benefits of INT8 quantization?
• What are some visualization techniques present in the Deep Learning Workbench?

30
Hands-on Lab

31
Hands-on Labs
Exercise 1: Tiny YOLO* V3 Object Detection

During this course, you have gone

through the practical considerations
of why model optimization is required.

In addition, you have learned about

the various tools in the OpenVINO™
toolkit and understood how to use
them to solve these challenges
through hands-on exercises and
videos.

Now through the Tiny YOLO* V3

Object Detection application on the
Intel® DevCloud for Edge Workloads,
you will be able to apply this workflow
end-to-end.

32
Hands-on Labs
Assignment: Safety Gear Detection

To put the OpenVINO™ toolkit workflow into practice, you have an interesting project to work on.

Using Intel® hardware and software resources, this sample application demonstrates how to create a
smart video IoT solution for safety gear detection. This solution detects any number of objects within
a video frame, specifically looking for people, safety vests, and hardhats.

You can find the lab exercise here.

33