AI Application Testing

Foreword
⚫ This course describes the main process and methods of AI
application testing.

2
 Objectives
⚫ Upon completion of this course, you will be able to:
 Understand the theories and methods of AI application testing.
 Understand the role of AI application testing in model iteration and
optimization.

3
AI Test Task Layers

AI security testing
6

AI application testing (including data management)

5

3
AI computing platform testing (including toolchain testing)

4
AI Application Test Framework

Key roles:
AI development engineers are responsible for design, development implementation, data acceptance, self-test, and optimization.
Data engineers are responsible for data collection, data cleaning, data integration, data labeling, and data quality check.
Test engineers are responsible for test data collection, test scenario analysis, test solutions, test cases, test strategies, test
automation, test acceptance, and evaluation.
5
Test Process and Activities
Scenario Sort out typical application scenarios, subjects, and features based on the requirement
analysis description, and refine the scenarios.

Based on scenario analysis, analyze feature factors that affect the identification rate of AI models and establish a
Feature factor feature factor library. Different AI scenarios have different requirements and complexity of feature factors. For
analysis example, feature factors in CV application scenarios include shooting factors (exposure and angle), environment
factors (weather and background), and attribute factors (age and skin tone of a person).

Data Build datasets based on application scenarios and feature factors, including but not limited to data types,
construction factor proportion, environment factors, and specific requirements.

Different service scenarios focus on different metrics, and the evaluation results are different. So the metrics and
Evaluation related definitions need to be determined based on the scenarios and focuses. For example, some services focus
metrics on recall for comprehensive results. Some services focus on accuracy. Some services feature imbalanced classes,
so they focus on accuracy of each class. Some services are sensitive to memory, so they focus on model size.

Automation
Automated batch testing and evaluation metric collection
solution

6
Feature Factor Analysis
⚫ Feature factor analysis is to analyze the factors that affect the identification performance of AI models
based on the input data. During AI development, data collection and processing are important
prerequisites. A model is only as good as the data it is fed.

⚫ During AI inference, good input data is also important. Good outputs can be obtained only when
good data is available.

Input data Model Output

7
Example: Factor Library of Facial Features (1)
Typical
Factor Category Factor Subcategory Typical Value
Factor
Within 30 degrees (up and down), above 30
Pitch angle
degrees (up and down)
Within 15, 30, 45, 60 degrees (left and right),
Rotation angle
above 60 degrees (left and right)
Slightly out of focus (perceivable), severely
Not focused
out of focus
Exposure Overexposure, underexposure
Special effect Nostalgic, black and white, whitening
Color saturation Oversaturated, undersaturated
Shooting Shooting Motion blur Smear, light painting
Main body
9 box grid
location
Entity integrity < 30%, 50%, 80%
Slight jitter, severe jitter, horizontal jitter,
Jitter
vertical jitter, tilt jitter
Screen-to-body
20%, 30%, 60%
ratio
Focus Macro and telephoto
Noise Add noise to the image.

8
Example: Factor Library of Facial Features (2)

Typical
Factor Category Factor Subcategory Typical Value
Factor

Time Morning, afternoon, night

Weather Cloudy, sunny, rainy, snowy

Simple background, complex

Background
background
Environment Environment
Light Front light, backlight
Light source Street lamp, point light, surface light
Contrast Obvious, not obvious
Location Indoor, outdoor

9
Example: Factor Library of Facial Features (3)
Typical
Factor Category Factor Subcategory Typical Value
Factor
Gender Male, female
Baby (0-1 year old), toddler (1-4 years
old), child (5-11 years old), teen (12-18
Age years old), young (19-35 years old), adult
(36-59 years old), aged (over 60 years
old)
Skin tone Yellow, white, black
Facial Normal, happy, sad, surprised, angry,
Object People
expression pouted lips, tricky face
Headwear, hat, sunglasses, glasses, mask,
Wearing
scarf, earwear
Appearance Hairstyle, beard, makeup
Number of
1, 2, 3, multiple
targets
Angle Front face, side face, upward, downward

10
Constraints Based on Feature Factors
⚫ In facial recognition scenarios, set constraints to guide users to take high-quality face images
to improve the facial recognition rate.
Look at the screen Do not wear a mask Keep your face in the specified area

When taking a photo, please note

the following:

No glasses No masks Look straight ahead

11
Feature Factor Examples

Overexposure ╳ Underexposure ╳ Blurry image ╳

12
Background interference ╳ Tilted image ╳ Ideal image
√
Data Construction
⚫ Build datasets based on application scenarios and feature factors, including but not limited
to data types, factor proportion, environment factors, and specific requirements. When
building a dataset in the test phase, collect as much data as possible from actual situations.
⚫ The dataset building process includes data collection, data cleansing, and data analysis and
understanding.

Data analysis and

Data collection Data cleansing
understanding

13
Data Construction – Data Collection
⚫ 1. Cold start data collection
 (1) Collect historical data offline.
 (2) Obtain network data using crawlers in compliance with laws and regulations.
 (3) Purchase data from third-party data companies.

⚫ 2. Online data backflow

 After AI applications are published, collect data from real application scenarios in the service backend in
compliance with laws and regulations.

14
Data Construction – Data Cleansing

[Event – Prize Announcement] [Huawei Cloud BBS] Are you

Original text: ready for the gifts in October, Huawei Sound speakers?
\(≧▽≦)/\r\nSame activities, different prizes\r\n

Event | Prize Announcement | Huawei Cloud BBS | Are you

Clean text: ready for the gifts in October, Huawei Sound speakers?
Same activities, different prizes

15
Data Construction – Data Analysis and
Understanding
⚫ Analyze, understand, and select proper data for model training and iteration is
important for AI application development. Understand data by performing the
following steps:
 Sample distribution analysis: analyzes sample distribution and determines whether the constructed
dataset covers comprehensive data.
 Negative sample analysis: analyzes samples with poor test performance and finds feature factors of
AI models to optimize the models.

16
Data Construction – Negative Sample Analysis
⚫ Samples with poor test performance are classified into hard samples and samples of unknown classes.

Two hard sample types Samples of unknown classes refer to samples that
have not been learned by AI models.

car

Classes
learned

bus

Unknown car or
classes bus?
or crane?
Samples that can be Samples that are difficult
identified by people but are to be identified by both
difficult to be identified by people and AI, for
AI, for example, vehicles example, license plate
and people in street views numbers in blurred images

17
Evaluation Metrics
⚫ Evaluation metrics are classified into precision metrics and performance metrics. Precision
metrics vary in different AI technical fields, as shown in the following table.

Technical Field Recognized Metrics

Image
Accuracy, recall, precision, F1 score, ROC curve, PR curve, and confusion matrix
classification
Object detection mAP(mean average precision)
Image
MIoU(Mean Intersection over Union)
segmentation
OCR Character precision rate (CPR), character recall rate (CRR)
Machine Bilingual Evaluation Understudy (BLEU)
translation ROUGE(Recall-Oriented Understudy for Gisting Evaluation)
Speech recognition Word error rate (WER), sentence error rate (SER)

18
Evaluation Metrics – Performance Metrics
⚫ Regular performance metrics are as follows:
 Frames per second (FPS): frequency (rate) at which consecutive images (frames) are
captured or displayed
 Floating point operations per second (FLOPs): number of floating point operations that
can be performed by a computing entity in one second
 GPU memory usage
 Queries per second (QPS): number of requests an AI application API responds to per
second

19
 Section Summary
⚫ This course describes the main process of AI application testing, including
scenario analysis, feature factor analysis, data construction, evaluation
metrics, and automation solutions. Dataset construction is a key step in the
entire process and must consider samples under different feature factors and
scenarios a well as hard samples and unknown class samples after AI
applications are brought online.

20
 Q&A
1. Which of the following are performance evaluation metrics of AI
applications? ()
A.FPS(Frames Per Second)
B. mAP(mean average precision)
C. GPU memory usage
D.QPS(Query Per Second)

21
 Thank You.
Copyright © 2024 Huawei Technologies Co., Ltd. All Rights Reserved.
The information in this document may contain predictive statements including,
without limitation, statements regarding the future financial and operating results,
future product portfolio, new technology, etc. There are a number of factors that
could cause actual results and developments to differ materially from those
expressed or implied in the predictive statements. Therefore, such information is
provided for reference purpose only and constitutes neither an offer nor an
acceptance. Huawei may change the information at any time without notice.

Huawei Cloud | Grow with Intelligence