DEVOPS FOR AI

A framework for Artificial Intelligence & Machine Learning Solutions

8. Evaluate 5. Deploy for AI

- Validate business case - Model Management
- Improve solution - Deployment Pipeline for AI
- Improve process - AI Data Storage

1. Prepare 4. Test
- Idea generation - Model testing
- Business case - Wizard of Oz
- Validate idea - Prepare feedback loop

EVAL PLOY FO
U DE R
E
AT

AI
FO R R

E
PREPA
A

T
S
TE

TE
N

RA
O
AI

N
R

T
FO ITOR
PE
AI

D +
ES
LD
O

IGN BUI R
AI

2. AI Design 7. Monitor for AI

- Architecture for AI - AI Metrics
- AI/ML Model choice - Model visualisation
- Specs for AI - Business metrics

3. Build + Train 6. Operate

- AI/ML Model - Human in the loop
- Data pre-processing - Scaling & Performance
- Model tuning - Defect Analysis for AI

For more info: www.infosupport.com/ai

DEVOPS FOR AI - AN INTRODUCTION

DEVOPS FOR AI - AN
INTRODUCTION

There isn’t an unique definition of from the traditional Software

DevOps, but Wikipedia defines it as Engineering DevOps.
this: DevOps (a clipped compound of
"development" and "operations") is a This guide shows the DevOps cycle for
software development methodology AI applications, but does not describe
that combines software development in any way of form how you have to
(Dev) with information technology perform a certain activity. It always
operations (Ops). The goal of DevOps depends on the context of your
is to shorten the systems development product or project. It does provide a
life cycle while also delivering features, high level overview of activities to
fixes, and updates frequently in close perform DevOps for AI. With small
alignment with
business objectives.
The DevOps
approach is to
include automation
and event monitoring
at all steps of the
software build.

We can use this

methodology for AI
applications as well.
Many activities articles we show examples and
needed to create an AI application practical cases. As a reader you can
involves traditional Software judge if the example is usable and
Engineering and these activities applicable for your project or product.
remain the same. In DevOps for AI we If not we hope it gives inspiration to
focus on the main activities that differ

For more info: www.infosupport.com/ai

DEVOPS FOR AI - AN INTRODUCTION

find a good way that works within your Each phase has a detailed model with
context. the most important activities. The
models are not designed to be
The DevOps for AI cycle consists of complete and comprehensive. Any
the following phases: valuable additions are welcomed.

1. Prepare for AI Happy reading and let’s create some

2. AI Design awesome AI.
3. Build + Train
4. Test
5. Deploy for AI
6. Operate
7. Monitor for AI
8. Evaluate

For more info: www.infosupport.com/ai

 1. PREPARE FOR AI
“Success is where preparation
and opportunity meet.”
Bobby Unser

ION BU
AT SIN
R E
NE

SS
GE

CA
AI/ML use cases Value Proposition
A
IDE

Business goals Idea validation

SE
Experiment Canvas Feedback

Data retention Initial AI/ML Model choice

DA
Y

Internal policies Determine data needs

C

T
GDPR Data authorisation
I

A
OL

AV
P

AI
TA LA
DA BIL
ITY

During ‘Prepare for AI’ we try to validate our idea as soon as possible. Is it possible to make
significant impact on the business goals. To start validating we probably need some data
and an idea about an initial AI/ML model. The proposed solution must be feasible within the
boundaries of our company’s data policy.

For more info: www.infosupport.com/ai

 2. AI DESIGN
“Success is where preparation
and opportunity meet.”
Bobby Unser

S AI A
N RC
TIO
CA

H
IT
I
IF

EC
EC

User stories Data architecture

TUR
Usablity requirements AI/ML platform
SP

Hypothesis testing Software architecture

E
LS

Execution qualities Select data source

A

Evolution qualities Data requirements

D
N

A
O

T
Monitoring requirements Locate data
A
TI

SE
NC

LE
FU CT
N- ION
NO

In the phase ‘AI Design’ we draw the first contours for our solutions. We give answers to the
questions like which architectures are feasible, which data is needed and where is the data
located? The functional and non-functional requirements sets a framework for the solution.

For more info: www.infosupport.com/ai

 3. BUILD + TRAIN
“Those who can image anything, can create
the impossible”
Alan Turing

RE E -

PROCESSING
FIN P R
E TA
DA
Metrics Data quality
Algorithm selection Feature engineering
Solution refinement Pre-processing pipeline

Model validation Build AI/ML model

Test automation Build services
Validate metrics ML pipeline

BU
TE ILD
IDA MO
AL DE
V L

During ‘Build + Train’ we create our models and the software to serve the models and
choose the metrics to optimize our models on. For continuous delivery purposes we
instantiate data pre-processing and machine learning pipelines next to our regular build.

For more info: www.infosupport.com/ai

 4. TEST
“The bitterness of poor quality remains long
after the sweetness of low price is forgotten.”
Benjamin Franklin

Model Testing

Test Data Management

Prepare feedback loop

TE
L
DE

ST
MO

ING
G

SO
TIN

FW

Sensitive Data Expore User Testing

S
TE

Model hardening Test Pyramid

AR

Penetration testing Test Automation

E

SECURITY TESTING

During ‘Test’ we test our models for production and make sure the whole solution works as
aspected. This means we perform traditional DevOps software testing with extra focus on
security testing. We don’t want to expose senstive data and protect our AI/ML models
against abusive use.

For more info: www.infosupport.com/ai

TEST - SOFTWARE TESTING

CASE STUDY: USER

TESTING AI MODEL FOR
ABN AMRO BANK

Most of the times when we think of Better, more tags and faster
testing our AI or Machine Learning One of the key elements for our model
model it’s about accuracy or maybe is the use of tags. A tag is a keyword
even precision and recall. We presume or term assigned to a piece of
that these measures tell us the information. This kind of metadata
correctness of our product. But the helps describe an item. The search,
correctness of the model is not the recommendation and profiling
same as a satisfied user. Let me tell algorithms are leaning heavily on
you a short story of how we created a these tags. Long story short, they're
great model for tagging knowledge important for the system, so it's
which resulted in poor results after the important the users add the right tags
first user tests. And by the way, this to the information.
story has a happy end.

Smart product After some test rounds we

We developed a smart product full of found the perfect solution
artificial intelligence and machine
for this problem.
learning for ABN AMRO, a large bank
in The Netherlands. Bank employees
can create Standard Operational To improve the processing of tagging,
Procedures (SOPs) and share them we created a tag recommendation
with co-workers. Co-workers can model. With the help machine learning
search for the right SOPs, but the algorithms and with Natural Language
system recommends the right SOPs at Processing we extract all possible tags
the right moment as well. out of the description and the
procedure. The recommendation

For more info: www.infosupport.com/ai

TEST - SOFTWARE TESTING

model classifies the tags as ‘relevant’

or ‘not relevant'. This has to result in
better tags, more tags and faster
tagging.

User testing
We were as confident as we could be.
This was a neat model. All the
measures were looking good.
Precision high, pretty good recall. The
user interface was slick and during
typing the system recommended tag
after tag after tag. Let the user testing
begin. Nothing can go wrong, right? Figure 1: First UI design

Wrong! Our usability testing expert

came back with the first results and
they were disappointing. Yes, we
collected more tags. Yes, the tags
were more relevant, so better. Yes, the
users liked the idea of tag
recommendations. But no, it wasn't
faster. Even worse it took way more
time for each user in the test to
complete the tagging.

Problem and solution

The problem was that we showed all
Figure 2: Final UI design
relevant tags and sometimes this could
add up to over 10 or 15 tags. The

“Our usability testing expert came back with the first results
and they were disappointing”

For more info: www.infosupport.com/ai

TEST - SOFTWARE TESTING

users read all the tags and had to think 4 we came up with the solution in
for each tag if it was the right tag. So it Figure 2 and now we have better tags,
slowed them down. more tags and the tagging is faster
and easier than ever.
After some test rounds we found the
perfect solution for this problem. We Conclusion
split the selected tags and the So although the recommendation
suggested tags and showed less of the model performed well, it didn't satisfy
tags. It took some testing to find the the user. We had to adjust the model
sweet spot for the number of tags to to create the best solution for the user.
show. It wasn't a game of just picking a We often think we know what our
number. The sweet spot is different for model does, but we never know how
each piece of information. After round users react to it, unless we test it.

For more info: www.infosupport.com/ai

 5. DEPLOYMENT FOR AI
“There are really three parts to the creative
process. First there is inspiration, then there is
the execution, and finally there is the release.”
Eddie van Halen

VERSIONING CO

NT
Model Management

. INTEGR
Data Versioning
Code Versioning Train model for production
Data processing pipeline
Build code

AT
ION

CO
N
T. DEPLOY

Deploy AI/ML models

Deploy data pipeline
Deploy software solution GPU cluster
Public / Private cloud
M
EN Distributed resource management

T
CLUSTER MGMT

During ‘Deployment for AI’ we train our models for production and deploy them to the
production environment. This can be a public or private cloud with a GPU cluster. An
important part of the process is to version the AI/ML models along with the data and the
code.

For more info: www.infosupport.com/ai

 6. OPERATE
“You cannot always control what goes on
outside. But you can always control what goes
on inside.”
Wayne Dyer

Bias monitoring
Model monitoring

SOFTWARE
Metric validation

MO

DE
Performance
Diagnostics

L
Logging

Human-in-the-loop
Data quality
Data flow
R A
INF DATA
Network
Hardware
Capacity

During ‘Operate’ we make sure the system keeps running the way as defined. We operate
the software, the infrastructure, but also the model and the data for the AI part of the
solution. We make sure the model keeps performing as specified even when it keeps
learning and evolving.

For more info: www.infosupport.com/ai

 7. MONITOR FOR AI
“I'm still passionately interested in what my
fellow humans are up to. For me, a day spent
monitoring the passing parade is a day well-
spent.”
Garry Trudeau

VALUE COST OF

OWNERSHIP
BUSINESS

Business metrics
Cost monitoring
Mean Time to Repair
Mean To Repair
Return of Investment

Feedback loops

Real-user information Feature requests

USER
Usability A/B scenario's
Usage information

USAGE FEEDBACK

During ‘Monitor for AI’ we monitor our process as well as our product. We monitor if we
meet the expected business impact and challenge this against the running cost. We also
monitor the usage of our product and collect user feedback to find improvements.

For more info: www.infosupport.com/ai

 8. EVALUATE
“You cannot always control what goes on
outside. But you can always control what goes
on inside.”
Wayne Dyer

T
VO
Pivot Process

PI
Pivot Product

Pivot Strategy

Retrospective

Improve solution
VE

Process improvement
PRO
IM

VA
LI
DA

Product
TE

Process

Business Case

During ‘Evaluate’ we validate our business case based on our learning and outcomes of the
product and process. We determine what to improve and pivot accordingly. We can pivot
our business strategy, product or process but we stay grounded in our vision and learned
facts.

For more info: www.infosupport.com/ai

DEVOPS FOR AI - AN INTRODUCTION

DEVOPS CAN
PREVENT AI
EXPERIMENTS

AI technology is so new and team has the right composition (a

promising that we actually have no combination of developers and end
idea what we can do with it. This users); it is also in the mindset that
combination of characteristics makes belongs to DevOps, in which you
AI as fascinating as it is complicated always try to achieve a better result in
to deal with. small steps.

Although more and more time and DevOps therefore has the potential to
budget is being made available within overcome the main pitfalls of AI
companies and institutions to experiments (and in particular that
investigate the possibilities of AI, the translation into practice). And that's in
actual application is often still in an three success factors.
experimental phase. After that phase,
the moment should follow when a The use of qualitative data
new AI tool is actually taken into use The success of an AI solution
by the business; and that is exactly depends on the data that you have at
where it still lingers. your disposal and the quality of it.
Ideally you as a development team
The reasons why AI experiments still want to have access to many
often end up in sight of the finish are different data sources within the
not new - they are phenomena IT has organisation to test and develop.
been dealing with for years. In practice things go wrong there
Fortunately, there has also been a soon. The owners of databases are
solution for these obstacles for years, often spread over the organisation,
namely DevOps teams. This way of so that a lot of time and energy can
working not only ensures that an AI be spent collecting data. Teams

For more info: www.infosupport.com/ai

DEVOPS FOR AI - AN INTRODUCTION

AI technology is so new and promising translation into practice). And that's in

that we actually have no idea what we three success factors.
can do with it. This combination of
characteristics makes AI as fascinating The use of qualitative data
as it is complicated to deal with. The success of an AI solution depends
on the data that you have at your
Although more and more time and disposal and the quality of it. Ideally
budget is being made available within you as a development team want to
companies and institutions to have access to many different data
investigate the possibilities of AI, the sources within the organisation to test
actual application is often still in an and develop.
experimental phase. After that phase, In practice things go wrong there soon.
the moment should follow when a new The owners of databases are often
AI tool is actually taken into use by the spread over the organisation, so that a
business; and that is exactly where it lot of time and energy can be spent
still lingers. collecting data. Teams engaged in AI
experiments therefore have a
The reasons why AI experiments still tendency to do so with the (limited)
often end up in sight of the finish are data that are easily and quickly
not new - they are phenomena IT has available. And then we have not even
been dealing with for years. talked about checking the data quality,
Fortunately, there has also been a where you as a team will really have to
solution for these obstacles for years, pass IT - voilà, an extra barrier.
namely DevOps teams. This way of
working not only ensures that an AI A DevOps team not only involves
team has the right composition (a developers and end users, but also
combination of developers and end team members who are responsible
users); it is also in the mindset that for the data strategy and quality.
belongs to DevOps, in which you
always try to achieve a better result in
Learning to live with a margin
small steps.
of error
DevOps therefore has the potential to IT is, by definition, quite an exact field
overcome the main pitfalls of AI and that makes it difficult for us if
experiments (and in particular that results are not 100% accurate. This
desire for perfection that typifies the

For more info: www.infosupport.com/ai

DEVOPS FOR AI - AN INTRODUCTION

average IT person can slow down or can be improved, for example from 72
even prevent the putting into practice to 80 percent. That in 20 per cent of
of a new AI solution. cases a customer service
representative is needed, that does not
An example: when a team will develop have to be a problem; if the investment
an AI chat bot that is used to handle only pays for itself in the improvement
customer inquiries, the team usually you have made.
wants a solution that always comes up
with the right answer to questions. If This too is a typical example of a
the accuracy then turns out to be 'only' DevOps approach; an incremental
80 percent, then one conclusion may improvement or 'minimal viable
be that the tool is not working properly. product' also has a value, especially in
What is actually forgotten is that the experiment phase. Learn to live
people are not flawless either; a with small steps; several small steps
customer service representative does eventually lead to a big impact.
not always immediately know the right
answer to a question. In these cases Involving the end user
you have quickly made a business But perhaps the main advantage of
case if the accuracy of an AI chatbot working in DevOps teams is to involve

For more info: www.infosupport.com/ai