State of

ModelOps 2021
How 100 AI-Focused Executives from Top
Global Financial Services Companies are
Developing Practices to Ensure Excellence
as They Govern and Scale AI Initiatives
 State of ModelOps 2021

Contents
Click below to navigate

3 Executive
Summary
4 Methodology

5 Foreword
6 Enterprise AI is
Driving the Rise of
ModelOps

9 Enterprises are
Investing in Their
ModelOps Functions
10 Why ModelOps is an
Enterprise Concern

12 The Enterprise Model

Deployment Landscape 13 Emerging Trends in
ModelOps Optimization

16 AI has an
Operationalization
Problem
17 Who Should Be
Responsible for
ModelOps

19 Conclusion

2
 State of ModelOps 2021

Executive Summary
M
odelOps is a must-have capability ModelOps functions to automate and
for enterprises seeking to streamline their existing processes.
operationalize AI, and this has Most AI-focused executives feel that
become apparent in 2021. Enterprises standardizing ModelOps practices across
are investing in their ModelOps functions the enterprise is essential for managing the
and developing processes for governing, operational challenges and risk that come
monitoring and managing AI models at scale. with AI deployment at scale.
This report summarizes the first Models are not like conventional software.
ever research into the state of model As enterprises increasingly recognize this,
operationalization. Our survey of 100 they are creating new roles, such as the
executives from top global financial ‘model operator’. These will bring focus,
services companies provides a unique flexibility and operational excellence
snapshot of the practices large enterprises to managing models, in the same way
are using to govern and scale mission- that DevOps , ITOps and SecOps do for
critical AI initiatives. conventional software.
Our findings show that AI is already Alongside the survey results, we present
widely deployed in many large highlights from several top executives who
enterprises. To address the operational shared their views regarding the extent of
and compliance challenges that comes the enterprise AI operationalization problem
with this, organizations are investing in and how they’re addressing it.

Key Takeaways

270
is the average number
80%
say difficulty managing risk and
of models respondents’ ensuring compliance is a barrier
enterprises have in production to AI adoption for their enterprise

5-7
is the average number of tools
25%
say their existing processes
respondents’ enterprises use for for inventorying models in
developing AI models production are ‘very effective’

90%
of survey respondents expect
69%
say improving the enforcement of AI
to have a dedicated ModelOps governance processes is a key reason
budget within 12 months to invest in a ModelOps platform

Source: Corinium Intelligence, 2021

3
 State of ModelOps 2021

Methodology
T
his representative survey of of Department, and include VPs
100 AI-focused leaders from of Data Science, Global Heads of
the financial services sector Risk, Heads of AI and other AI-
was conducted in February and focused executives.
March, 2021. We asked them 19 questions
Respondents were selected from about their organizations’ AI
the 100 largest financial services maturity and ModelOps capabilities,
companies in North America and as well as the obstacles to model
Europe, respectively. Of these, 84% operationalization facing their
are from Global 500 or Fortune companies and where they’re
500 organizations, including investing to overcome them.
Charles Schwab, Citigroup, Morgan Then, we combined our findings
Stanley, American Express and with commentary from seven industry
JPMorgan Chase. experts to put these unique insights
They have job titles ranging from into the state of enterprise model
C-level to Directors, VPs or Heads operationalization into context.

Contributors

Stu Bailey
Co-Founder and Chief Skip McCormick
Enterprise AI Architect, Data Science Fellow,
ModelOp BNY Mellon

Eli Bernstein Kathryn Zhao

Head of Data, Global Head of
Wells Fargo Asset Electronic Trading,
Management Cantor Fitzgerald

Glenn Hofmann
Chief Analytics Officer, Scott Zoldi
New York Life Chief Analytics Officer,
Insurance Company FICO

Patricia Flynn
VP, Information
Governance and
Solution Delivery,
State Street

4
 State of ModelOps 2021

Foreword
W
e’re delighted to be the sponsor of this first
annual State of ModelOps survey. The very
fact that we have such a survey speaks to the
speed at which our industry is moving in 2021.
ModelOp was founded five years ago with a sole
focus on ModelOps. But even a year or so ago
the notion of ‘ModelOps’ still wasn’t recognized or
understood by many AI practitioners.
That’s changed as the barriers to gaining value from
AI have shifted from creating models to operating them
in production with full efficiency, visibility, compliance
and fairness, as well as within risk limits.
In recent months, there’s been a rapid increase in
attention paid to the critical role of ModelOps as a core
capability, especially for Fortune 500 and Global 500
enterprises. We’re pleased to have been at the forefront
of this effort and excited to serve our customers’ needs.
I want to thank Corinium Intelligence for their
excellent work, and especially thank the 100 senior
leaders who responded to the survey and the seven
executives who agreed to go ‘on the record’ and
provide candid interviews.
This report reflects the challenges real businesses are
facing today and how they’re addressing them in their own
words. I hope that they, and all readers, find it valuable.

Stu Bailey
Co-Founder and Chief
Enterprise AI Architect,
ModelOp

5
 State of ModelOps 2021

Enterprise AI is Driving
the Rise of ModelOps
KEY FINDING

With the use of AI in

business becoming more
widespread and AI budgets
continuing to rise across
the globe, enterprises are
developing better processes
for monitoring, managing
and governing AI systems
in production

M
odelOps is becoming a
critical business function
as more enterprises
successfully deploy and scale their
use of AI models. learning and AI, it’s not really clear Enterprises are Now
Our survey of 100 AI-focused that our existing process works.” Scaling AI Capabilities
leaders from top financial services The term ‘ModelOps’ generally refers When it comes to AI initiatives,
companies shows that these to the enterprise capability focused on where is your company?
enterprises currently have 270 models governance and lifecycle management
All business
in production each, on average. One of all types of decision models, such units using 19%
in five report that all business units in as traditional statistical models, as AI regularly
their organizations use AI regularly, well as AI models. ModelOps touches
while a further 60% report that at least everything that happens with a model Some business
some business units do so. from the point at which its released units using 60%
AI regularly
This is not entirely surprising, as the into production until (and beyond) its
organizations we surveyed are highly ultimate retirement.
Pilot AI
advanced in their use of AI. However, AI models have unique initiatives and/ 21%
the growing number of models at use characteristics and cannot be or experiments
in these enterprises is creating new managed and governed in the same
challenges for AI-focused executives. way as conventional software. So,

270
“For straight transactional while the enterprises we surveyed
processing, I think the model that have decades of experience with
we have works very well,” says models, the growing number of AI The average number of models
Patricia Flynn, VP, Information models at use in these enterprises our respondents’ companies
Governance and Solution Delivery is placing new strains on their typically have in production
at State Street. “For machine business processes. Source: Corinium Intelligence, 2021

6
 State of ModelOps 2021

AI and ModelOps Investment ModelOps is Seen as a Key Business Capability

is Increasing in 2021 Does your company have a dedicated budget for model operations?
The prevalence of AI in business is
only going to increase in 2021. Our
research shows that 71% of financial
services companies have increased 53% 37% 10%
their AI budgets in 2021, with 13%
reporting an increase of more than
$1 million USD.
At the same time, enterprises are Yes No, but we will No, and that will
also investing in ModelOps. More within 12 months not change in the
than half of our respondents say foreseeable future
their enterprises already have a
Source: Corinium Intelligence, 2021
dedicated budget for ModelOps. A
further 37% expect theirs to have
one within 12 months.
Wells Fargo is one example of a as inventorying the models that “Then, they encounter [ops] guys,
large corporation that is prioritizing are currently deployed within their who are like, ‘Wait, you’re working
aspects of ModelOps such as organizations. Just 25% of survey on a model? We have a whole
monitoring and governance as it respondents say the processes governance process and you’re
scales up its AI initiatives. they have in place for this are ‘very supposed to submit forms to us
“That’s something that was really effective’. Meanwhile, 43% would before you start.’”
heavily invested in over the last four describe theirs as ‘ineffective’. Governance is emerging as
years,” says Eli Bernstein, Head of Data “I think it’s because a lot of AI a key driver for investment in
at Wells Fargo Asset Management. projects are so approachable now,” ModelOps. Our survey respondents
“We are in the middle of building out suggests Skip McCormick, Data cite ‘maintaining model audit
the infrastructure that will scale.” Science Fellow at BNY Mellon. “It’s trails for regulatory compliance’
However, our findings show very easy to get ahead of whatever and ‘ensuring visibility into the
that the maturity of the ModelOps model governance organization status of models’ as two of the
processes enterprises currently you already have in place for more top ModelOps challenges facing
have in place varies greatly from traditional models.” enterprises today. Respectively,
company to company. “You have this Wild West of only 31% and 32% say they have
Many executives are running into people doing exciting things with ‘very effective’ processes for these
trouble with fairly basic tasks, such software they download,” he adds. tasks in place.
The companies we surveyed
had already been working to
address model management
“ModelOps is the logical next step challenges. But less than half rate
their current practices as ‘very
after DevOps. We’re looking for a effective’. Indeed, just 41% claim
systematic way to make sure that they have very effective processes
for standardizing model operations,
the models we’re putting into play 46% say they have very effective
processes for monitoring AI
actually do what they should do” models and 48% believe their
Skip McCormick model governance processes are
Data Science Fellow, BNY Mellon very effective.

7
 State of ModelOps 2021

How AI Adoption is Giving Rise to ModelOps “Few organizations

Of course, enterprises have had ops teams to manage software
and legacy (non-AI) models in production for years. But the are simultaneously
constantly evolving nature of AI models means new processes are
needed to operationalize them effectively. putting sufficient
“We have a very formal and rigorous model governance process resources into the
and model risk management organization,” says McCormick. “What
we’re doing now is extending that to cover these continuously infrastructure they’ll
improving models.”
Our research shows that enterprises are facing a range of operational
need to have beneath
ModelOps challenges as they put more AI models into production. that in a production
Not surprisingly, risk is a key issue in the financial services sector:
80% say that difficulty managing risk and ensuring regulatory environment”
compliance is a barrier to operationalizing AI in their enterprises. Skip McCormick
Additional barriers include ‘integration with existing systems’, Data Science Fellow, BNY Mellon
‘gaps in staff and skills’, ‘process inefficiencies’, ‘lack of
standardization’ and ‘cost’. Respectively, 72%, 70%, 64%, 65% and
62% of respondents cite these five hurdles as barriers to ModelOps
in their organizations.
This reflects in part a historic focus on building models rather
than deploying and operating them, as McCormick points out.
“A lot of AI capabilities are still at the stage where it’s a lot of
potential,” he concludes. “Few organizations are simultaneously
putting sufficient resources into the infrastructure they’ll need to
have beneath that in a production environment.”
The rise of ModelOps can be seen as a response to the
increasing recognition of the unique challenges of operationalizing
models at scale.
Models have long been seen as critical enterprise assets, and AI
models are showing their ability to deliver very significant value.
Enterprises increasingly understand that to continuously capture
this value while managing risk requires ModelOps practices for the
age of AI. As a result, they’re investing in ModelOps.

8
 State of ModelOps 2021

Enterprises are Investing in

Their ModelOps Functions
KEY FINDING

ModelOps is becoming a core business capability, with enterprises investing in

creating more efficient processes and systems for operationalizing AI models

ModelOps Processes are Still Maturing ModelOps Must Integrate with

Please rate the systems and processes your company currently the Full Enterprise AI Stack
uses for the following aspects of model operations?
Which of the following systems or
Not Effective Not Very Effective Somewhat Effective Very effective applications should be integrated
with your company’s model
0% 20% 40% 60% 80% 100% operationalization process?

Already is integrated
Inventorying models
that are deployed Should be integrated

Maintaining a
compliance audit trail
30% 70%
Risk management
application
Model governance

38% 62%
Monitoring of models Business
applications

Standardizing
model operations
41% 59%

Ensuring visibility into IT and security

systems
the status of models

0% 20% 40% 60% 80% 100%

42% 58%
IT service
management
ModelOps is Seen as a Key Business Capability
Does your company have a dedicated budget for model operations?

46% 54%
DevOps
tools
53% 37% 10%
50% 50%
Data science
Yes No, but we will No, and that will not change development tools,
within 12 months in the foreseeable future frameworks or platforms

Source: Corinium Intelligence, 2021

9
 State of ModelOps 2021

Why ModelOps is an
Enterprise Concern
KEY FINDING

Enterprises are moving toward a standardized,

platform-agnostic approach to model operationalization
to manage the realities of AI for business in 2021

A
I tech stacks are constantly premises, 58% execute models “On the production side, we’ve
evolving. Data scientists want to using AWS, 58% do so using Google mostly standardized to one platform,”
be able to use the best tools for Cloud, 27% deploy on Microsoft reports Glenn Hofmann, Chief
the job and enterprises are generally Azure and 12% use other cloud- Analytics Officer at New York Life
happy to accommodate this. As a result, based services. A full 94% use at Insurance Company. “All the models
the ecosystems emerging to develop, least two deployment environments. get deployed on that platform. They
deploy and manage AI in enterprise There are also multiple data science can be consumed by any production
settings have become complex. tools in use at the vast majority of system in the company via an API.”
Our research suggests that multiple companies – 81% use at least two data He adds: “We still have a few
teams are now actively deploying science tools to develop AI models, legacy models on other platforms.
AI models in most large enterprises. with 43% reporting that they use more Maintenance of those gets
Of the 100 AI-focused executives than five. increasingly difficult.”
we surveyed, 81% report that their
organizations have more than five
teams deploying AI models, and 29%
say they have more than 10.
There’s a great variety in production
“We still have a few legacy models
environments, with models deployed on other platforms. Maintenance
across a mix of on-premises and
cloud environments.
of those gets increasingly difficult”
While 48% of respondents say Glenn Hofmann
their models are executed on- Chief Analytics Officer, New York Life Insurance Company

10
 State of ModelOps 2021

“You can’t trust a single person vision of what’s important.

Kathryn Zhao, Global Head

and you can’t trust small groups of of Electronic Trading at Cantor

Fitzgerald, says putting uniform

people with different philosophies” model operationalization processes

in place has helped her business unit
Scott Zoldi to manage risk more effectively.
Chief Analytics Officer, FICO “It’s definitely related to adding
more safety guards and de-risking,”
she says. “Our risk appetite is very
small. That’s the main reason that
Hofmann agrees that standardizing is the only way to ensure AI is used I added those extra steps – just to
ModelOps practices is vital for responsibly. make sure that we stay within the risk
ensuring AI models can be put into “You can’t trust a single person parameters and limit our exposure
production, scaled and managed and you can’t trust small groups of in production.”
over time efficiently. Without this people with different philosophies,” With respect to managing risk
level of consistency, there’s a danger he says. “Centralization or and providing governance, key
that some models an enterprise standardization is a big part of AI focus areas for ModelOps include
develops may become unusable. growing up.” processes for approvals, validations,
“If you have models that are not Of course, standardization is inventorying models in production
really deployed, per se, they’re possible without centralization. and tracking model changes and
just built and live in a development But increasingly, companies are activity throughout the AI lifecycle.
environment, you have a ‘recall ensuring that their AI-focused teams “It’s a very extensive process,”
problem’,” he says. “It can be very are monitoring the same things, Zhao concludes. “But we think it’s
difficult to recall their functionality.” following the same risk management necessary to keep us within the
processes and defining a shared accepted risk parameters.”
ModelOps: Standardization
Versus Centralization
There is no one ‘right’ way to do Enterprises are Favoring ModelOps Standardization
ModelOps. However, the trend Which of the following best describes your company’s model operationalization practice?
is toward standardizing and
centralizing the ModelOps functions
across the enterprise. Even in 44% 17%
enterprises that prefer a more
distributed approach to ModelOps, Centralized Decentralized,
with standard but working
standardizing processes is proving
processes towards
popular, in particular to ensure across the centralization
conformance with compliance and company
risk management policies.
A full 61% of survey respondents
say their companies are working to 39%
centralize or standardize their model
Decentralized
operationalization practices, with and managed
44% reporting that they already have. independently by
FICO Chief Analytics Officer each business unit
Scott Zoldi suggests that putting
enterprise-wide standards in place Source: Corinium Intelligence, 2021

11
 State of ModelOps 2021

The Enterprise Model

Deployment Landscape
KEY FINDING

With many teams executing different types of AI in a range of environments and using
multiple data science tools, standardized processes are needed to tame enterprise ModelOps

Multiple Teams Deploy AI Across an Enterprise Data Science Staff Generally Use a Range of Tools
How many different teams are deploying models in your Across all business units, how many different data science
entire company? tools does your company use for developing models?

15+ 1% 7+ 4%

10-15 28% 5-7 38%

5-10 52% 2-4 39%

2-5 19% 1 19%

0% 10% 20% 30% 40% 50% 60% 0% 10% 20% 30% 40% 50% 60%

Enterprises use Many Types of Models AI Models are Deployed in a

What types of models does your company use for AI initiatives? Range of Environments
Where are your AI models executed?

67%
Machine learning models
AWS 58%

54%
Rule-based or heuristic models

Google Cloud 58%

46%
Graph models

34% On premises 48%

Linguistic models

26% Azure 27%

Agent-based models

Other cloud services 12%

25%
Traditional models
Source: Corinium Intelligence, 2021

12
 State of ModelOps 2021

Emerging Trends
in ModelOps
Optimization
KEY FINDING

Enterprise priorities are shifting from

simply getting models out of the lab and into
production toward maximizing ROI and
ensuring AI is used fairly and compliantly

U
ntil fairly recently, companies have focused on
arming data scientists with the tools they need
for rapid AI development.
But with model development capabilities starting to
scale, a new hurdle has emerged. Enterprises are now
targeting their investments toward getting models into
production in an efficient, compliant and transparent
manner, while maintaining their efficacy and value.
“It’s been a common problem for a while now for “It’s been a common
analytics teams to build models that then sit on a shelf
and never actually get deployed into production,” notes problem for a while now
Glenn Hofmann, Chief Analytics Officer at New York Life
Insurance Company. “We’ve now largely resolved that.
for analytics teams to
But I think it’s a relatively recent phenomenon that this is
getting easier to resolve.”
build models that then
Our research shows that managing risk and ensuring sit on a shelf and never
regulatory compliance is the top barrier to model
operationalization in business today. The 100 AI-focused actually get deployed”
executives we surveyed also cite reducing the cost of Glenn Hofmann
ModelOps and improving model governance as the top Chief Analytics Officer,
two benefits of investing in the function. New York Life Insurance Company

13
 State of ModelOps 2021

Strategic Priorities for ModelOps Can Improve the Efficiency of AI Projects

ModelOps Optimization Please rate the following model operationalization platform benefits according to
A core element of ModelOps is the how important they are to your company

ability to automate and streamline Crucial Very important Fairly important Not very important Not important
processes so that AI systems can 0% 20% 40% 60% 80% 100%
be deployed, scaled, monitored
Improved enforcement of
and maintained. governance and compliance rules
Our research shows that executives
Model operationalization
are becoming aware of the benefits cost reductions
of investing in technology to help
Better visibility into the state
automate and standardize ModelOps and status of models
practices.
Improved collaboration across
One key focus is integrating their data, analytics and AI teams
ModelOps systems with their current
Enhanced ability to measure
software stacks. Of the 100 executives model revenue contributions
we surveyed, 70% say their enterprise
Better model auditability
has integrated data from its risk
and audit reporting
management applications with its
ModelOps processes. Meanwhile, Ability to automate and scale
operationalization process
62% say they’ve integrated data from
their business applications and 59% 0% 20% 40% 60% 80% 100%

report doing the same with IT and Source: Corinium Intelligence, 2021

security systems data.

At the same time, 58% say they’ve
integrated the data from their IT Meanwhile, 69% view improving the Fitzgerald’s electronic trading business
services management systems, 54% enforcement of AI governance and has extensive processes in place to
say they’ve integrated the data from compliance rules as a ‘very important’ manage its models in production.
their DevOps tools and 50% say benefit of investing in a ModelOps “I would say we have a very
they’ve integrated the data from their platform, and 66% say the same about comprehensive and mature ModelOps
data science development tools, improving the scalability of ModelOps governance structure in place,”
frameworks or platforms. processes with automation. say Kathryn Zhao, Global Head of
A full 76% of respondents say It’s easy to see why these things are Electronic Trading at Cantor Fitzgerald.
achieving cost reductions is at least so highly valued, when you consider Zhao’s team monitors all models in
a ‘very important’ benefit of such an what must be done to ensure AI production and reports data on key
investment, with 42% describing it systems in production continue to operational metrics to the firm’s model
as crucial. perform over time. For example, Cantor ‘best execution committee’ to ensure
this data is rigorously appraised and
acted on, where necessary.
“We have a performance quality
“It’s a very extensive process. checklist that I do on a monthly basis,”

But we think it’s necessary to Zhao adds. “On the annual basis,
I also have the ‘algo attestation’.
keep us within the accepted For that, I create a list of things like,
‘What’s the algorithm inventory?
risk parameters” What’s the model inventory? What’s
Kathryn Zhao the algorithm performance quality?
Global Head of Electronic Trading, Cantor Fitzgerald And, are there any red flags?”

14
 State of ModelOps 2021

The Future of Enterprise

ModelOps Functions
“It’s just as hard and it’s essential
because, without it, you cannot
“We are in
The rigorous processes needed claim to make a forecast that can be
relied upon,” says Skip McCormick,
the process
to manage and govern full model
life cycles may become unwieldy Data Science Fellow at BNY of digital
Mellon. “[It] usually means that it’s a
when an enterprise starts to scale
their AI programs from dozens to separate system, because that kind transformation,
hundreds of models in production.
So, it’s not surprising that some
of access to a production system
will oftentimes impact production.”
naturally. So,
of the executives we interviewed In McCormick’s view, this is automating
predict that automation will play a something that many businesses
key role in the future of enterprise are still underinvesting in. But most of the
ModelOps functions.
“We are in the process of digital
our research shows that
enterprises are starting to integrate
pipelines
transformation, naturally,” says Eli essential data from business
systems and applications into
will be the key
Bernstein, Head of Data at Wells
Fargo Asset Management. “So, their ModelOps processes. to success”
automating most of the pipelines Clearly, there’s still a long way Eli Bernstein
will be the key to success.” to go before enterprise ModelOps Head of Data, Wells Fargo
Other executives are calling functions start to mature in most Asset Management
for greater investment into organizations. But the investments
‘ground truth’ systems to allow AI-focused executives are
organizations to monitor and verify prioritizing today show many are
the veracity of model predictions. committed to changing that.

15
 State of ModelOps 2021

AI has an Operationalization Problem

KEY FINDING

Enterprises are racing down the path to AI maturity. But there’s no consensus around
who’s responsible for managing and monitoring models once they’re in production

AI-Focused Executives Face Many ModelOps Challenges

Please rate the following potential barriers to operationalizing models as they apply to your company
Very Challenging Somewhat Challenging Not Very Challenging Not a Challenge for Us

0% 20% 40% 60% 80% 100%

Lack of integration with existing systems and applications

Difficulty managing risk and ensuring regulatory compliance

Lack of standardization of model operations

Lack of staff with the necessary skillsets

The cost of operationalizing models

Lack of efficient processes for operationalizing models

But There’s No Concensus On Who Owns ModelOps

Who is responsible for the 24x7 operations and monitoring of models, in your company? (Select all that apply)

RESPONDENTS

Data or Risk Data AI or AI Center

Analytics IT Management Science of Excellence

DevOps 22% 36% 21% 0% 30%

Model
Operators 63% 71% 71% 60% 50%
or Model
RESPONSES

Engineers

IT Operations 63% 36% 43% 80% 80%

Data
52% 79% 21% 80% 60%
Scientists

Source: Corinium Intelligence, 2021

16
 State of ModelOps 2021

Who Should Be Responsible

for ModelOps
KEY FINDING

Enterprises are establishing

dedicated roles, such
as the ‘model operator’,
to oversee the effective
monitoring, governance
and maintenance of AI
models in production

T
he ModelOps function is
underdeveloped in many
enterprises because it
has initially fallen to established
organizations such as DevOps.
This arrangement is not ideal,
due to factors including the unique
characteristics of models compared
with conventional software, the
internal and external compliance
and regulatory requirements around
AI models and the need to support There are many differing views as
many stakeholders and teams. to where responsibility for ModelOps
“You never let the
Enterprises often lack a dedicated should live. Perhaps not surprisingly, data scientists
team with the knowledge, remit or respondents in IT functions are
authority to orchestrate ModelOps as the least likely to say that they feel create their own
a cross-functional capability. Some
say that data science teams should
responsible for ModelOps. Executives
in these roles believe data scientists,
paper because
be responsible, but others feel that model operators or model engineers they’re too close
this isn’t the way to go. should have this responsibility.
“You have to have this third-party However, some experts suggest to the problem”
that knows what they’re looking that people skilled in ITOps have Skip McCormick
at and knows how to measure it,” a strong foundation for ModelOps Data Science Fellow,
argues BNY Mellon Data Science and may be best positioned to BNY Mellon
Fellow Skip McCormick. “You never become model operators, as they
let the data scientists create their have deep experience keeping
own paper because they’re too close production systems running ‘round
to the problem.” the clock’.

17
 State of ModelOps 2021

The Rise of the Model Operator

Until recently, DevOps professionals
“You should not have data scientists
oversaw model operationalization in
most enterprises. One 2019 Gartner
governing and maintaining their
study found that this was the case
in 61% of organizations. own models. In the past that was
“That’s why my group was
introduced,” recalls Patricia Flynn, often true, and it was not effective”
State Street’s VP for Information Glenn Hofmann
Governance and Solution Delivery. Chief Analytics Officer, New York Life Insurance Company
“I support five businesses within
Global Markets, and I sit in the risk
organization. I’m independent to the
business lines, so I don’t have the “From a monitoring perspective, Yet, despite the fact that data
revenue concerns.” we have a couple of dedicated scientists are generally not skilled
But over the past 12-24 months, people on my team,” says Glenn (or interested) in running 24x7
a new class of AI professional has Hofmann, Chief Analytics Officer at production operations, 62%
emerged: the model operator. New York Life Insurance Company. of respondents say their data
Our research suggests that 60% “All they do is post-deployment scientists should have a role
of enterprises now have model monitoring and reporting.” in ModelOps.
operators or model engineers The creation of this new class For McCormick, this indicates that
overseeing model operations of AI professional reflects the many enterprises still have some
across their businesses. By contrast, growing consensus that managing maturing to do. In particular, he
just 26% of the 100 executives we AI models is different from argues that having data scientists
surveyed say their DevOps teams managing traditional software or manage their own models creates a
are responsible for ModelOps. statistical models. conflict of interests.
At the same time, having data
scientists monitor and manage
their own models is inefficient.
Enterprises are Creating Model Operator Roles
Data scientists have unique
Who is responsible for the 24x7 operations and monitoring of models, in your company?
(Select all that apply)
expertise in model development
and are expensive. So, it makes

64% sense to ensure they can spend

as much of their time as possible
IT operations actively driving business value and
innovating with new ideas.
62% “You should not have data
scientists governing and maintaining
Data scientists
their own models,” Hofmann agrees.

60%
“In the past that was often true, and
it was not effective.”
Model operators or model engineers Today, 40% of enterprises still
lack dedicated model operators
26% to handle day-to-day ModelOps
tasks. But if current trends
DevOps
continue, this should be expected
Source: Corinium Intelligence, 2021 to decrease.

18
 State of ModelOps 2021

Conclusion
KEY FINDING

Enterprises have invested to build their

capabilities for creating AI models in
recent years, with significant success

A
cross many industries extract the full value from their AI
and companies, the investments while limiting risk and
strategic power of AI has maintaining efficacy, transparency
been established thoroughly. and compliance.
This has led to a surge in model Organizations are creating
creation. But investments in the dedicated model operator or model
people, processes and tools for engineer roles to take on day-to-day
operationalizing models – i.e. ModelOps duties.
ModelOps – has lagged behind. On the other hand, fewer
That’s now starting to change. organizations are placing ModelOps
More than half of the enterprises in the DevOps organization.
we surveyed now have a dedicated Meanwhile, our expert contributors
ModelOps budget, and that figure is generally agree that data scientists
set to rise to 90% within 12 months. have highly specialized, valuable
The key drivers for this increased skills and are not best suited
investment are managing risk, or positioned to manage the
reducing costs through automation operationalization of their models.
and providing visibility through In 2021, the state of ModelOps
improved monitoring. is encouraging. There’s a growing
AI-focused executives in recognition of the function, the
financial services, an industry that problems that it addresses, the
is relatively mature in its use of opportunities it creates and the
traditional and AI models, report investments that need to be made
confidence in the effectiveness to support it. Like DevOps, ITOps
of their ModelOps functions. But and SecOps before it, ModelOps
they face challenges and see looks set to grow into a core
significant room for improvement. business function in its own right as
They also understand that these global AI use matures. But there is
challenges limit their ability to still a long way to go.

19
Get Essential Data and Analytics
Insights Direct to Your Inbox
As anyone who has attended our global conferences or
events will know, our 300,000-strong network of data
and analytics leaders boasts many of the most forward-
thinking minds in the business world.

On our new content hub, Business of Data, brings those

same essential insights direct to your inbox in a weekly
newsletter packed with exclusive research, video
podcasts, in-depth articles, interviews and reports.

Discover how other senior data and analytics leaders

are tackling the challenges they’re facing today and are
accelerating their organizations’ digital transformations.

For a limited time, subscribing to Business of Data is

free. So, make sure to do so today for complimentary
access to exclusive insights you just can’t find
anywhere else.

SUBSCRIBE NOW
About ModelOp
ModelOp is a pioneer of ModelOps software. The company enables
large enterprises to address the critical governance and scale challenges
necessary to fully unlock the transformational value of enterprise AI and
machine learning investments.
Core to any AI orchestration platform, Global 2000 companies use
ModelOp Center to govern, monitor and orchestrate models across the
enterprise and deliver reliable, compliant and scalable AI initiatives.

For more information, please visit: https://www.modelop.com/

About the Editor

Solomon Radley is an experienced editor and reporter
with a deep understanding of the data, analytics and
CX space and close relationships with many of the
sectors’ most prominent C-level executives.

He works with data and analytics, learning and

development and customer experience leaders to
champion new innovations and highlight how the
world’s most forward-thinking brands are using data
to fuel their digital transformations. Solomon Radley
Global Content Strategist,
To share your data story or enquire about appearing Corinium Global Intelligence
in a Corinium report, blog post or digital event, contact
him directly at [email protected]
 State of ModelOps 2021

Partner with Business of Data

by Corinium
We’ll develop industry benchmarking research, special
reports, editorial content, online events and virtual summits
to establish your brand as an industry thought leader.

FIND OUT MORE HERE

Connect with Corinium

Join us at our events

Visit our blog

Discover Corinium Intelligence Read our white papers

Corinium is the world’s largest business community of more Follow us on LinkedIn

than 250,000 data, analytics, customer experience and digital
transformation leaders. Follow us on Twitter

We’re excited by the incredible pace of innovation and

Like us on Facebook
disruption in today’s digital landscape. That’s why we produce
quality content, webinars and events to connect our audience
Find us on Spotify
with what’s next and help them lead their organizations into
this new paradigm.
Find us on YouTube

Find out more: www.coriniumintelligence.com Find us on iTunes

22