Original
Original
AI Infrastructure
2024 Edition
2
Table of contents
Research methodology___________________________________________28
3
AI is here.
It’s just the beginning.
Welcome to the new era, where AI is not only intriguing and engaging consumers
but exponentially increasing business productivity, transforming business models,
and reimagining customer experiences. From retail to healthcare, there’s no doubt
AI is making a difference in ways like:
• Powering CT scanners with robust algorithms for more accurate diagnostics and
improved medical care.
• Preventing financial fraud via real-time detection using advanced AI tools and
models.
Collectively, there is a consensus over the importance of AI, not just from an
organizational standpoint, but also from a personal standpoint. This is an important
distinction – these numbers show that it’s not only organizations that are driving
adoption, but that people see the value personally.
Source: A commissioned study conducted by Forrester Consulting on behalf of Microsoft and NVIDIA, May 2023
4
AI importance for success
Total 75%
Finance 73%
Healthcare 72%
Retail 76%
Manufacturing 80%
ISV 75%
Total 66%
Finance 68%
Healthcare 62%
Retail 66%
Manufacturing 73%
ISV 64%
Base: Total (n=900), Finance (n=180), Healthcare (n=180), Retail (n=180), Manufacturing (n=180), ISV (n=180)
Source: A commissioned study conducted by Ipsos on behalf of Microsoft, October 2023
5
Many organizations are at the starting line
More than a third of companies are in the early stages of AI adoption: exploring options,
gathering information, and planning various use-cases to strategize implementation,
while a quarter are in the early pilot testing stage. With a majority of organizations still
figuring things out, business leaders have an opportunity to beat their competition and
gain advantages. But to do so, they’ll need to act quickly in implementing their own AI
strategies.
38%
options, gathering information, and
identifying AI use cases as well as
planning and strategizing their AI
implementation. Organizations in this final stage
have fully implemented and
are both actively utilizing and
optimizing.
6
Businesses are focused on automation
and customers
Those companies that have started integrating AI are focused on supporting customer-
facing applications and increasing efficiency through automation. These use cases
tend to bring higher ROI, as they focus on getting more value out of their workers by
reducing time spent on lower-value tasks. This makes sense, considering respondents,
on average, expect a 34% ROI from their AI platforms.
70% 62%
Conversational AI (e.g.,
Document-process
customer service assistant,
automation (e.g., claims
language understanding,
management and automation)
59%
voice control)
52%
Speech transcription and
analytics (e.g., call transcription
and analysis, multimedia content
What AI workloads captioning, conversation
transcription)
has your organization
deployed? 49%
Knowledge mining (e.g., content
search, product discovery
optimization)
Base: (n=641)
Source: A commissioned study conducted by Forrester Consulting on behalf of Microsoft and NVIDIA, May 2023
7
“Leading-edge organizations” blaze the
path forward
While many organizations are early in their AI journeys, 15% of businesses are advanced
in their AI infrastructure and are considered “leading-edge organizations”. These
leading-edge organizations tend to be early adopters of technology and can provide
valuable learnings on effective AI implementation strategies. Our recommendations are
based on an analysis of leading-edge organizations plus other key insights to provide AI
best practices and recommendations any company can leverage.
Stages of AI readiness
8
AI is challenging…
for everyone
The AI landscape continues to evolve and AI implementation poses many different
challenges and obstacles to overcome. Business leaders are faced with the daunting
task of figuring out the best path forward.
99%
99% 59%
59%
In just a few years, I believe the AI will be much more advanced than what
we have today.
9
The greatest challenge? Tackling the
talent gap
Organizations have an immediate need for AI experience and talent. Addressing this
gap by bolstering employees’ skills and training today is the key to bridging the gap
and moving forward quickly.
39% 33%
Leaders who rank having the skills Leaders who rank having enough
required to develop or customize talent as one of their
AI models as one of their top 3 top 3 organizational challenges
technology challenges (out of 13 items). (out of 13 items).
And along with employee talent, unclear ROI of AI implementation, the right resources
to support AI development and management and collaborating across business
functions are top organizational challenges.
10
Infrastructure challenges remain top of mind
The right infrastructure can make or break your AI projects. Overwhelmingly, the AI
challenges fall within infrastructure (hardware, software, and tools) and remain the most
common roadblock in implementing and leveraging powerful AI tools. Prioritizing the
right AI infrastructure is key to successful AI implementation, scaling, and innovation.
Outdated/legacy systems
11
Leaders are looking to partners for help
For companies not sure how to start leveraging AI, partnering with a solution provider
with deep AI expertise and proven AI solutions can help companies accelerate AI
production and address AI infrastructure challenges. Business leaders are looking to
partners to help with infrastructure design and implementation, training and support,
security and compliance, and strategic planning and consultation.
Where it gets interesting is that as companies move further along their AI journey, they
start to prioritize things like performance, optimization, and cloud provider integration.
Engaging the right partner can help businesses of any size and at any stage of AI
implementation accelerate their AI journey. This is both a huge opportunity for partners,
and a burden. They must make sure their staff is ready to go and able to help with
consulting, strategy, and training.
Leading-edge Non-leading-edge
Performance
Training and
monitoring and 47% 47% support
52% 52%
optimization
Infrastructure Infrastructure
design and 44% 44% design and 47% 47%
implementation implementation
12
AI infrastructure remains
elusive
If you’re not sure how to approach your AI infrastructure, you’re not alone. Robust and
scalable infrastructure specifically built for AI is critical to support the complexities of
new AI-driven workloads and processes, but AI infrastructure is a key challenge for most
leaders, who face many obstacles in implementing and operationalizing AI,
such as:
• Outdated and legacy systems that are not designed to handle the complexity and
volume of AI workloads.
• Data security and privacy concerns, especially sensitive and personal data,
that require robust protection and compliance measures.
• Skills gap, as many organizations lack the talent and expertise to develop, customize,
and deploy AI models and applications.
13
Defining AI infrastructure
Infrastructure challenges are amplified due to diverse interpretations of AI
infrastructure among organizations. These interpretations range from integrating AI
capabilities into the existing IT infrastructure to establishing a dedicated hardware
and network infrastructure and developing a comprehensive tech stack that includes
algorithms, frameworks, and libraries. This can make something as simple as
communicating needs with vendors a real challenge when the same language is not
being used.
Not having a clear definition just adds to the challenges of getting started with AI. At
the most simplistic, AI infrastructure includes the hardware, software, networking, and
tools and services used to develop, implement, and optimize AI. As AI continues to
evolve, it’ll be more important than ever to settle on a standard definition that can be
used across all industries.
14
Making AI infrastructure a priority
Businesses are already understanding the urgency and importance of having a solid
foundation for their AI initiatives. 41% of leaders agree that infrastructure is the area
they need most help with and 39% need help with strategic planning and consulting,
citing specialized components like infrastructure or security as well as broader design
and implementation. Additionally, 43% are predominantly proactive in developing their
AI infrastructure strategy, compared to 16% who are mostly reactive. There’s a clear
opportunity for partners to provide the consultation and expertise companies need to
optimize their infrastructure for AI.
43% 42%
Strategic planning and consulting is desired
by many organizations across all industries.
16% Business leaders that were earlier in their
implementation were more likely to need help
with 42% of AI beginner organizations stating
they needed help. As organizations begin their
journey, they can leverage the consultation of
Proactive A mix of both Reactive partners.
proactive and
reactive
15
Industries bring their own nuances
Manufacturing tends to be the most proactive (51%) in planning for AI infrastructure,
significantly more so than those in healthcare and retail. In such a process-driven
industry, even minor gains in efficiency can bring critical advantages over the
competition. AI technologies, even in its current form, brings giant leaps forward in
operational efficiency.
The entire industry stands to gain significant benefits through process optimizations,
advanced automation, and predictive maintenance. Business leaders across other
industries can follow manufacturing’s example and move quickly to stand up the
right infrastructure for their AI needs. Like any other business strategy decision, these
differences highlight the cross-industry nuances that need to be considered when taking
on any new technology.
Finance 43%
Healthcare 38%
Retail 39%
Manufacturing 51%
Independent 43%
Software Vendors
16
Start with the trifecta:
performance, security,
and cost
There is no doubt there’s a lot to consider when finding the right AI infrastructure a
business needs, and the speed of changes being brought to the market add further
complexities. As businesses start analyzing vendors and setups, they can look to these
top considerations as a starting-off point: performance, security, and cost.
44% 42%
44% 42%
37% 37%
17
Performance and scalability
With 44% of leaders prioritizing performance and
scalability, particularly in industries managing high-volume
and complex AI workloads like retail, manufacturing,
49%
and independent software vendors, the imperative for AI 44% 52%
infrastructure lies in providing swift and reliable computing
resources to optimize resource utilization, reduce latency,
and scale up and out as needed. Focusing on performance
Independent
and scalability means looking beyond the cost to consider
software
all the benefits that come with AI and maximizing the full vendors
impact of AI infrastructure implementations.
“Security and privacy” tops the list for finance and healthcare
49%
49% 44%
44% 52%
52%
Finance Healthcare
18
Cost effectiveness
Not surprisingly, cost effectiveness is a priority, with 37% of business leaders across
industries citing getting the value they need from their AI infrastructure and meeting
ROI goals as important. Cost is even more relevant for retail industries.
Other priorities
19
One size does not fit all
Priorities are fluid, shifting to match an organization’s constantly evolving context.
Factors like industry, market, AI maturity level, and platforms create a constantly
evolving environment to navigate. Leaders need to be able to plan for these shifts
in priorities by understanding the organization’s changing context and impact on
implementations.
Level of AI readiness
Leading-edge organizations face different challenges compared to businesses that
are earlier in their AI infrastructure implementations. As an organization moves further
along, flexibility, data management, maintenance, and support compete with earlier
priorities like performance, security, cost effectiveness, and integration.
1% 20
Workload type
The AI infrastructure capabilities needed vary based on business model and workloads.
One company may need fully customizable software, services, and computing
while off-the-shelf AI models, services and platforms may be fine for another. As
AI continues to evolve, the needs of companies and even the solutions are rapidly
changing. Because of this, providers are ramping up their offerings to reach a wider
range of needs and providing multiple points at which a customer can come into their
system.
AI leaders
‘AI leaders’ have a defined AI strategy and want to lead their market by building their
own innovative, homegrown AI models and applications. They require highly performant
supercomputing infrastructure that can flexibly meet complex storage, network, computing,
and security needs. Their AI workloads are incredibly complex, involve massive models, and
require control at every layer of their AI infrastructure.
Business drivers: Developing an end-to-end AI service, solution or platform, from the ground
up. Require unlimited scalability and the ability to deliver customized user experiences.
AI power users
‘AI power users’ also have a defined AI strategy and are heavily customizing pre-built AI
models, infusing company-specific content and data, and retraining. They need control over
each layer of their AI infrastructure but typically don’t require massive compute power when
working with pre-trained models.
Business drivers: Looking to maximize efficiency and minimize time to market. Save time by
using pre-built models and optimizing for their needs.
AI ready
‘AI ready’ companies want infrastructure that is ready to go so they can focus on defining their
AI strategy. They don’t want to worry about the ins and outs of infrastructure, they are looking
for a scalable, out-of-the-box solution that can support discrete processes now and support
AI growth.
Business drivers: Taking their first steps with AI. Need off-the-shelf solutions.
21
Platform considerations
There is no one-size-fits-all to determining whether a company should be on-premises,
hybrid, or on the cloud – every type of solution has their positives and negatives. With
a myriad of factors at play, ultimately the decision about what is best lies with the
company and their unique situation. For example, on-premises AI infrastructure may
offer more control but require more upfront investments and can be expensive to
maintain, difficult to scale, and hard to keep up to date with the latest technologies.
AI infrastructure in the cloud offers fast deployment, scalability and flexibility, and
typically the best technology available, but some have concerns with security, privacy,
and meeting compliance requirements. Hybrid setups offer the benefits (and drawbacks)
of both but with higher degrees of complexity. Startups may benefit even more from
a cloud set-up due to their small employee size and their naturally increased focus on
getting their product to market as fast as possible. Nonetheless, key themes that came
across for each of these were security and cost effectiveness.
For more advanced models, we need GPUs that will be more useful. But for
now, we have some, light-weight models that can be run in CPUs.
Base: (n=641)
Source: A commissioned study conducted by Forrester Consulting on behalf of Microsoft and NVIDIA, May 2023
22
Half of business leaders said they have a hybrid setup with plans to migrate fully to
the cloud for better IT governance and security, increased productivity and scalability,
more successful deployments, greater innovation, and ROI. For those that are “AI
ready,” a cloud provider may provide a more comprehensive out of the box solution to
get started.
Further than deciding whether to have an on-premises, hybrid, or cloud setup, the
vendor choice is fraught with a range of choices. Key features like having high quality
AI algorithms, ability to manage networking capabilities, integration with open-
source tools, API accessibility, scalability, clear documentation, and multi-cloud/hybrid
enablement came through as the highest requirements.
49%
31%
17% 3%
Better IT governance
Increased productivity
Higher percentage of AI concepts successfully deployed in production
Increased scalability
Increased data security for ML models and data sets
Base: (n=641)
Source: A commissioned study conducted by Forrester Consulting on behalf of Microsoft and NVIDIA, May 2023
23
Industry context
Different industries have different priority nuances. More regulated industries like
finance and healthcare put a higher focus on security and privacy, while manufacturing
and ISVs require strong performance and scalability. Additional factors like solution set-
up and level of AI maturity will also make a difference in both their organizational and
technical priorities.
Independent
Manufacturing Software Vendor
1. P
erformance and 1. Performance and
scalability scalability
AI lets us shorten the time required for so many things. And it helps our
performance and overall activity.
24
Harnessing the power
of AI now
To help businesses move forward in their AI journey, we recommend four actions to
help navigate the challenges and speed AI production and integration.
Most businesses will need to make changes, either overhauling their existing
infrastructure, choosing a solution provider offering a full-stack AI platform, or
something in between. A company’s AI infrastructure strategy can shape the future of
their business, either accelerating their AI journey or blocking their innovation.
25
Make it secure
Security, privacy, and compliance should be at the forefront of any AI and infrastructure
plans. Secure AI is the act of securely designing, developing, and deploying AI and
GenAI capabilities and systems. Follow these best practices:
Find a partner
Across all industries, business leaders expressed a need for help with strategic planning
and consulting as well as training and support from AI solution providers. Leading-
edge organizations partner with AI experts to help plan, build, and integrate AI into
their business. Companies of any size and at any stage can benefit from a strategic AI
solution provider. Forming a partnership with a proven AI solution provider can be key
to accelerating AI production and staying competitive.
26
Take the next steps on your
AI transformation journey
Explore how Microsoft Azure is redefining cloud infrastructure to prepare every
business for AI by providing the world-class technology for AI workloads and doing
so sustainably and responsibly.
Get strategic guidance and insights on AI innovation, tailored for business leaders.
Learn how businesses are balancing performance, efficiency, and cost with Azure AI
infrastructure.
© 2024 Microsoft Corporation. All rights reserved. This document is provided "as-is". Information and views
expressed in this document, including URL and other internet Web site references, may change without
notice. You bear the risk of using it. THis document does not provide you with any legal rights to any
intellectual property in any Microsoft product.
27
Research methodology
In May 2023, we commissioned Forrester Consulting to evaluate the current state of AI
among IT directors and decision makers in North America, Europe, and Asia Pacific.
We further explored this topic through a second study with Ipsos in September 2023
among technical and business leaders, developers, and data professionals who are
early majority adopters of technology across 3 markets (US, Germany, and India) and
5 different industries (finance, healthcare, retail, manufacturing, and independent
software companies).
Participants: n=641 Director and above in IT Participants: n=900 ITDMs who are in the
with responsibilities in AI workloads, cloud early majority of technology adopters at
infrastructure. their organizations.
Company Size: 1000+ FTE in North America, Company Size: ISVs = 50+ employees
500+ in all other countries All other industries = 500+ employees
28