Sponsored by:

Artificial Intelligence and Machine Learning

Projects Are Obstructed by Data Issues
Global Survey of Data Scientists, AI Experts and Stakeholders

May 2019

1 Sponsored by:
Executive Summary
This research finds Artificial intelligence (AI) and machine learning (ML) projects are no longer novel, as
nearly half of companies surveyed have undertaken 4 or more active projects. But it is still early days, as
7 out 10 have started their project in only the last two years, and just half of those projects have been put
into production. Also, half of those surveyed reported their AI/ML teams have 10 or fewer members.

The nascency of enterprise AI has led more than half of the surveyed companies to label their training
data internally or build their own data annotation tool. Unfortunately 8 out of 10 companies indicate that
training AI/ML algorithms is more challenging than they expected, and nearly as many report problems
with projects stalling. 96% of companies surveyed stated they have run into training-related problems
with data quality, labeling required to train the AI, and building model confidence.

This leads to 7 out of 10 companies utilizing external services for the AI or ML projects with many of
them focusing on data collection, labeling and expertise. With AI/ML talent rare and expensive, this
research suggests that enterprises should consider using external solution providers for critical activities
like data labeling and model scoring. The data provides evidence that such outsourcing leads to
improved outcomes.

This survey reveals that enterprises assign strategic value to their machine learning initiatives. They
expect AI and ML to improve all aspects of their businesses, and potentially to be disruptive in their
industry sectors.

2
Key Findings

Enterprise AI/ML is Young But Growing

• 70% report that first AI/ML investment was within last 24 months
• Over half of enterprises report they have undertaken fewer than 4
AI and ML projects

Numerous AI/ML Project Hurdles

• 78% of AI/ML projects stall at some stage before deployment

Training Data Issues Constrain Project Success

• 96% of enterprises encounter data quality and labeling challenges

• 63% have tried to build their own technology solutions
• 71% of teams ultimately outsource ML project activities
• Teams that outsource data labeling get projects into production faster

3 Sponsored by:
DETAILED FINDINGS

4 Sponsored by:
Over Half of Enterprises Report They Have
Undertaken Fewer than 4 AI or ML Projects

30%

25%
25%

How many AI or 20%

17% 17% 17%
ML projects has 15%
15%

your company 9%
started?
10%

5%

0%
1 2 3 4–5 6 – 10 More than
10

5 Sponsored by:
 70% Report Their First AI/ML Project
Involvement Was Within the Last 24 Months

35%
31% 31% 30%
30%

25%

When did you start 20%

your first AI or ML 15%

project?
10% 8%

5%

0%
2019 2018 2017 2016 or earlier

6 Sponsored by:
The Majority of Enterprise AI/ML Project
Teams Have 10 or Fewer Members

30% 28%

25% 24%
Approximately, 21%

how many people

20%
15%
are involved with 15%
12%
your AI or ML 10%
projects?
5%

0%
Less than 5 6 – 10 11 – 20 20 – 50 More than 50

7 Sponsored by:
 Most AI/ML Projects Involve NLP, CV, or
Entity Resolution

Other
13% Natural language
processing
45%
What AI approach
best describes your
Entity resolution
17%
most recent AI or
ML project?

Computer vision
25%

8 Sponsored by:
 Only Half of Enterprises Have Released An
AI/ML Project Into Production

Have any of your No

49%
Yes
51%
company’s AI or
ML projects moved
into production
yet?

9 Sponsored by:
 65% of Projects Have Reached the Data
Labeling and Algorithm Training Phase

Has your AI or ML No Yes

65%
project reached
35%

the stage where it

is using labeled
training data to
train its algorithm?

10 Sponsored by:
 72% Report that Production-Level Model Confidence
Will Require More than 100,000 Labeled Data Items

45% 43%

40%

Approximately, 35%

how much training 30% 28%

data will be 25%

19%
required to deploy 20%

your model with

15%
10%

confidence?
10%
5%
0%
Fewer than 100,000 100,000 – 1,000,000 1 million – 10 million More than 10 million
data items data items data items data items

11 Sponsored by:
 81% Admit Training AI with Data Is More
Difficult Than Expected

No
Yes
19%
Has training the AI 81%

with data been

more challenging
than expected?

12 Sponsored by:
96% Encounter Challenges with
Training Data Quality and Quantity

Bias or errors in the data 66%

Which problems Not enough data 51%

has your company Data not in a usable form 50%

experienced with
AI training data Don’t have the people needed to label the data 28%

specifically? Don’t have the tools needed to label the data 27%

We have not experienced any data training

4%
problems

0% 10% 20% 30% 40% 50% 60% 70%

13 Sponsored by:
78% of AI or ML Projects Stall at Some
Stage Before Deployment

Proof of concept 33%

Model validation and scoring 13%

Algorithm development 9%
In which phase did Training data preparation 7%
your AI or ML Model deployment 6%
project stall?
Algorithm training 6%

Post-deployment enhancement 4%

Our project has not stalled 22%

0% 5% 10% 15% 20% 25% 30% 35%

14 Sponsored by:
 Companies Pursue a Mix of
Data Labeling Approaches

Labeled and annotated company data internally 76%

What approach is
being used to train
the initial Acquired pre-labeled data 45%

algorithm?

Contracted labeling and annotation of company data 30%

0% 10% 20% 30% 40% 50% 60% 70% 80%

15 Sponsored by:
 Companies Pursue a Mix of
Data Labeling Technology Strategies

In-house solution 63%

What solution was

used in your most
recent project to Commercial solution 34%

label and annotate

your training data?
No solution (manual process) 19%

0% 10% 20% 30% 40% 50% 60% 70%

16 Sponsored by:
71% of Companies Outsource AI/ML Activities
with 3 of the Top 5 Related to Training Data

Data collection 36%

Model development and testing 36%

Which of the Data labeling and annotation* 28%
following external Overall project strategy and objectives 27%
services has your Data specialist personnel 26%
company used for Deployment into production 19%
its AI and ML Measurement and maintenance of production… 18%
projects? Governance and management 15%

None, our project was done entirely with… 29%

0% 5% 10% 15% 20% 25% 30% 35% 40%

*Survey answer was labeled ‘Data Preparation’

17 Sponsored by:
 Utilizing Data Labeling Solution Providers
Moved Projects Into Production Faster

Contracted Labeling and Annotation of Company Data

35% 33%

30%

What approach is 25% 23%

being used to train 20%
the initial
algorithm?
15%

10%

5%

0%
Released
Project into Production
in Production Still in Development
Project not in Production

18 Sponsored by:
METHODOLOGY AND
PARTICIPANTS

19 Sponsored by:
Goals and Methodology

The primary research goal was to estimate the

Research Goal maturity of ML in the enterprise. Additionally, the
research sought to understand today’s ML project
challenges, and the tools and resources used in
these projects.

Data professionals and business stakeholders

Methodology
involved in active AI and ML projects were invited to
participate in a survey on their company’s use and
development of AI and ML projects.
The survey was administered electronically, and
participants were offered a token compensation for
their participation.

Participants A total of 227 data scientists, AI experts and business

stakeholders completed the survey. Participants were
from all 5 continents.

20 Sponsored by:
 Companies Represented

Size Industry
Technology 20%
Healthcare 13%
Financial Services 13%
More than
1,000 – 5,000 5,000 Education 8%
37% 63% Manufacturing 8%
Government 6%
Retail 6%
Services 5%
Telecommunications 4%
Food and Beverage 4%
Media and Advertising 3%
Energy and Utilities 3%
Transportation 2%
Pharmaceutical 1%
Non-Profit 1%
Other 3%

0% 5% 10% 15% 20% 25%

21 Sponsored by:
 Companies Represented

AI / ML Deployment Location
United States or Canada 80%

Yes, we plan to start our

project but more than 6 Yes, we have already
started our AI or ML project Europe 11%
months from now
18% 60%

Mexico, Central America, or

4%
South America

Asia 2%
Yes, we will be starting
our AI or ML project in
the next 6 months
22% Australia or New Zealand 2%

Middle East or Africa 1%

0% 20% 40% 60% 80% 100%

22 Sponsored by:
Individuals Represented

Leadership Role AI / ML Role

Business
stakeholder for
AI or ML
projects Working with AI or ML is a
Team Manager Working with 7%
Executive substantial part of my job
43% AI or ML is
27% 51%
my entire job
18%

Frontline or
Admin Working with
30% AI or ML is a
minor part of
my job
24%

23 Sponsored by:
Individuals Represented

Work Role

Deep learning specialist

Natural language processing 4%
specialist
7%
Computer vision specialist
7%

Data scientist
Machine learning specialist 48%
11%

Product owner
23%

24 Sponsored by:
For more information…

About Dimensional Research

Dimensional Research provides practical marketing research to help technology companies make
smarter business decisions. Our researchers are experts in technology and understand how
corporate IT organizations operate. Our qualitative research services deliver a clear
understanding of customer and market dynamics.
For more information, visit www.dimensionalresearch.com.

About Alegion
Alegion is the ML training data partner of the Fortune 1000. We offload the entire burden of
training data from enterprise data science teams, from delivering custom training datasets to
providing machine- and human-scored model testing and offering post-production exception
handling. Our solution combines a purpose-built technology platform with a global pool of on-
demand data specialists, driven by our own managed service team. We support machine learning
projects broadly , with particular emphasis on Computer Vision, Natural Language Processing
and Entity Resolution, in financial services, retail, defense, technology and manufacturing.

For more information, visit www.alegion.com.

25 Sponsored by:
APPENDIX

26 Sponsored by:
74% Are Using AI and ML Projects to
Disrupt the Market Place

Improved customer experience 66%

How is your Reduced costs 54%

company Competitive differentiation 51%

measuring AI or
ML project Increased sales 45%

success? Develop internal skills 32%

Industry disruption 23%

0% 10% 20% 30% 40% 50% 60% 70%

27 Sponsored by:
 AI and ML Projects Require Continual
Commitment

30%

Approximately, 25%
25%
what percent of
25% 23%
22%

the resources used 20%

to develop and 15%

deploy the AI/ML
solution are still 10%

working on that 5%
5%

project?
0%
All of them More than 75% 50% – 75% 25% - 50% Less than 25%

28 Sponsored by:
 82% Are Satisfied with Project Progress

No Yes
18% 82%
In general, has
your company's
most recent AI or
ML project been
successful thus far?

29 Sponsored by:
Lack of Experience
Impeded Most AI/ML Projects

Project was more complicated than expected 53%

Lack of expertise 53%

In your opinion
why wasn’t the AI Problems with the data 33%

or ML project a Model confidence wasn’t high enough 30%

success?
Budget 27%

Shortage of personnel 23%

0% 10% 20% 30% 40% 50% 60%

30 Sponsored by:
 Majority of ML Projects
Trained on Text Data

Text 77%

Images 42%
What types of data
are being collected Voice/Audio 25%

in your most
recent project?
Video 22%

Other 11%

We haven't start collecting data yet 4%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

31 Sponsored by: