0% found this document useful (0 votes)
858 views20 pages

MIT Technology Review - AI

The MIT Technology Review Insights report, sponsored by Microsoft Azure, explores how technology leaders are customizing generative AI to enhance their enterprise-wide AI strategies. Key findings reveal that customization not only boosts efficiency but also fosters unique solutions, user satisfaction, and innovation, with data integrity being a significant barrier. The report highlights the growing adoption of advanced tools and methods, such as retrieval-augmented generation (RAG), to facilitate effective customization in various industries.

Uploaded by

sunnyporus01
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
858 views20 pages

MIT Technology Review - AI

The MIT Technology Review Insights report, sponsored by Microsoft Azure, explores how technology leaders are customizing generative AI to enhance their enterprise-wide AI strategies. Key findings reveal that customization not only boosts efficiency but also fosters unique solutions, user satisfaction, and innovation, with data integrity being a significant barrier. The report highlights the growing adoption of advanced tools and methods, such as retrieval-augmented generation (RAG), to facilitate effective customization in various industries.

Uploaded by

sunnyporus01
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 20

Produced in partnership with From the field: Top best

practices and techniques

Customizing
generative AI for
unique value
2  MIT Technology Review Insights

Preface
“Customizing generative AI for unique value” is an MIT Technology Review Insights
report sponsored by Microsoft Azure. This report seeks to understand how technology
leaders are customizing generative AI in their businesses and to what extent this
is a priority for their enterprise-wide AI strategy. Denis McCauley was the author
of the report, Laurel Ruma was the editor, and Nicola Crepaldi was the producer.
The research is editorially independent, and the views expressed are those of MIT
Technology Review Insights.

We would like to thank the following executives for their time and insights:

Mark Austin, Vice President, Data Science, AT&T


Eric Boyd, Corporate Vice President, AI Platform, Microsoft
Tanwir Danish, Global Solutions and AI Officer, Data and Technology, Dentsu
Brian Demitros, Innovation Lead, Data and Technology, Dentsu
Ece Kamar, Managing Director of AI Frontiers, Microsoft Research
Gabe Pereyra, President and Co-Founder, Harvey AI
Asha Sharma, Corporate Vice President and Head of Product, AI Platform, Microsoft

Methodology
The survey forming the basis of this report was
conducted by MIT Technology Review Insights in
September 2024. The survey sample consists
of 300 global executives holding senior
technology roles in their organizations: chief
information officers, chief technology officers,
chief data/AI officers, vice presidents
and directors of technology, data,
engineering and other influential roles.
The respondents’ organizations are
primarily large enterprises and
are headquartered in 12 countries
in the Americas, Europe, and
Asia-Pacific. Twelve industries
are represented in the survey,
with the largest contingents
of respondents working in
financial services, technology,
consumer goods and
retail, and manufacturing
businesses.
MIT Technology Review Insights 3

CONTENTS

01 Executive summary.................................................................................................................4
02 Embracing AI customization.............................................................................................5
Harnessing new benefits........................................................................................................5
Tomorrow’s large and customized industry-specific models.........................6
Choosing and evaluating models...................................................................................... 7
Customizing operations at AT&T.......................................................................................8
Agents of change.........................................................................................................................9
03 The quality and performance imperative.............................................................10
Implementing a variety of methods................................................................................10
Customizing for accuracy with RAG..............................................................................11
Customizing efficiency for the legal world at Harvey AI...................................12
The devil’s in the data..............................................................................................................12
04 Risk factors..................................................................................................................................13
Keeping internal data protected.......................................................................................13
Ensuring model and application integrity...................................................................14
Avoiding hallucinations...........................................................................................................14
Responsible AI and media insights at Dentsu.........................................................14
05 Continuous operations........................................................................................................16
Empowering dev teams.........................................................................................................16
Identifying use cases .............................................................................................................. 17
Automation for lifecycle management......................................................................... 17
06 Conclusion....................................................................................................................................18
0
4  MIT Technology Review Insights

01
S
Executive
summary

ince the emergence of enterprise-grade


generative AI, organizations have tapped
into the rich capabilities of foundational
models, developed by the likes of OpenAI,
Google DeepMind, Mistral, and others.
The study’s key findings include:
• Customization brings more than efficiency. Boosting
efficiency is a key motivation for customizing generative AI
models, according to 50% of surveyed executives, but it’s
not the only one. As important, say 49%, is gaining the ability
Over time, however, businesses often found these to create unique solutions, 47% cite better user satisfaction,
models limiting since they were trained on vast troves and 42% seek greater innovation and creativity.
of public data. Enter customization—the practice of
adapting large language models (LLMs) to better suit a • RAG provides the backbone for generative AI
business’s specific needs by incorporating its own data performance. Two-thirds of companies are using or
and expertise, teaching a model new skills or tasks, or exploring RAG as a method of customization. Over
optimizing prompts and data retrieval. half (54%) are also employing fine-tuning techniques,
indicating that these two methods, along with prompt
Customization is not new, but the early tools were engineering, are used most effectively in combination.
fairly rudimentary, and technology and development
teams were often unsure how to do it. That’s • Automated evaluation is gaining traction. Over half
changing, and the customization methods and (54%) of surveyed businesses employ manual methods
tools available today are giving businesses greater to evaluate generative AI models. But 26% are either
opportunities to create unique value from their AI beginning to apply automated methods or are doing so
models. now consistently.

We surveyed 300 technology leaders in mostly large • Data integrity is the biggest barrier to customization.
organizations in different industries to learn how Around half the respondents (52%) cite the need to
they are seeking to leverage these opportunities. ensure data privacy and security as the primary difficulty
We also spoke in-depth with a handful of such they face with customization. Most (86%) say focusing on
leaders. They are all customizing generative AI privacy and security has become more important as they
models and applications, and they shared with us customize more actively. One-third overall (32%), and 57%
their motivations for doing so, the methods and tools of the biggest companies in the survey, deem this “much
they’re using, the difficulties they’re encountering, more important”.
and the actions they’re taking to surmount them.
• Advanced tools are empowering developers and
Our analysis finds that companies are moving ahead facilitating lifecycle management. Over half (53%) of
ambitiously with customization. They are cognizant organizations have adopted telemetry tools for tracing
of its risks, particularly those revolving around data and debugging for their developers. Also widely used
security, but are employing advanced methods and are a simplified playground of tools (by 51%) and prompt
tools, such as retrieval-augmented generation (RAG), development and management (46%) to facilitate better
to realize their desired customization gains. collaboration between engineers.
02
MIT Technology Review Insights 5

“What is it that differentiates


your application? It’s usually
Embracing AI the data and knowledge that
your organization can bring
customization to bear.”
Eric Boyd,
Corporate Vice President,

B
usinesses customize for a range of reasons, AI Platform, Microsoft
but tying them all together is the desire to
integrate the organization’s own knowledge
and expertise into the solutions they wish to
build. Such integration is what enables
businesses to build AI applications that are differentiated
from others in the market. “If you’re using an off-the-shelf
model, your application’s going to look very similar to
everyone else’s,” says Eric Boyd, corporate vice
president, AI platform, at Microsoft. “What is it that
differentiates your application? It’s usually the data and Harnessing new benefits
knowledge that your organization can bring to bear.” The executives we surveyed see three benefits
above all from customization. Half of them (50%) say
Foundational models are not suitable for enterprise it’s important for delivering greater efficiency—for
use, says Brian Demitros, innovation lead for data and example, by automating tasks, streamlining workflows,
technology at advertising network Dentsu. “They’re and optimizing business processes. A similar share
trained on the internet and often contain innacurate (49%) also say customizing is important to gain
or misleading information,” he says. “You can’t competitive advantage from having a unique solution
simply use them out of the box and expect a level of in the market. Almost as many (47%) cite enhanced
accuracy needed to support critical decision-making. user satisfaction—for example, from personalization or
Customization is critical to get value out of them.” greater responsiveness (see Figure 1).

Figure 1: Customizing brings efficiency and more


Competitive advantage, enhanced user satisfaction, innovation and creativity, and accelerated R&D are also top
motivators for AI model customization.

Improved efficiency 50%


Competitive advantage of
a unique solution 49%
Enhanced user satisfaction 47%
Innovation and creativity 42%
Accelerated research
and development 42%
Better decision-making 41%
Flexibility 29%
Source: MIT Technology Review Insights survey, 2025
6  MIT Technology Review Insights

Leveraging AI to uncover white spaces and drive


brand growth is a key benefit for marketing and
“You can’t simply use them
advertising agency Dentsu, according to Tanwir out of the box and expect a
Danish, global solutions and AI officer, data and
technology for Dentsu. level of accuracy needed to
“We have proprietary methodologies for driving
support critical decision-
growth, designing and activating the total experience, making. Customization is
and optimizing marketing campaign performance,”
he explains. “These are specialized fields, where our critical to get value out of
expertise, knowledge bases, proprietary data sources,
and machine learning models empower us to build
them.”
enterprise-grade AI systems that deliver growth.”
Brian Demitros,
Gabe Pereyra, president and co-founder of Harvey AI, Innovation Lead,
an AI solutions provider serving the legal industry, also Data and Technology,
cites enhanced creativity and quality as customization Dentsu
benefits. “Many partners at our law firm clients use
custom models to ask questions, get information, draft
briefs, and develop arguments that others may not have.”

Using customized solutions makes these lawyers


better at their jobs. Among the surveyed companies,
42% also cite innovation and creativity—such
as the creation of new products, services, and
business models—as a key desired outcome from
customization. The same percentage cite accelerated
R&D as a key benefit (see Figure 1).

Tomorrow’s large and customized industry-specific models


Large language models may not lend themselves to This will not be the same as interacting with
specialized, industry-specific purposes today, but today’s foundational model, says Pereyra. “It will be
Gabe Pereyra foresees the future development of customized in two ways. First, it will have access
powerful, all-purpose, and self-learning AI models to all the specialized data,” he says. “Then it will
that can play that role. The company he co-founded, learn from all the interactions of the lawyers using
Harvey AI, is seeking to build one for the legal industry. the model.” Therein will lie the model’s real value,
says Pereyra: “It will imitate all the firm’s internal
“In a couple of years, it will no longer make sense expertise.”
for a law firm to have one system that extracts data
from contracts, another that analyzes transcripts,
and another that reviews the e-Discovery corpus,”
according to Pereyra. “Instead, the firm’s professionals
should be able to access a super powerful model
that’s connected to all this data and can perform
tasks across a wide range of thematic areas.”
MIT Technology Review Insights 7

Figure 2: Key model attributes when selecting are performance, multi-modal capabilities, and flexibility
Emerging capabilities like agentic and multi-agent systems are already of high importance for enterprises.

Model performance

64%
Omni-modal and multi-modal capabilities

56%
Flexible model consumption and payment options

53%
Emerging capabilities such as agentic and multi-agent systems
50%
Openness
40%
Efficiencies of small and medium-sized models
37%
Source: MIT Technology Review Insights survey, 2025

Choosing and evaluating models Just over half (53%) of the surveyed executives place
Achieving these outcomes starts with getting model a priority on flexible model consumption and payment
selection right in the first place. What attributes do options: for example, the ability to reserve capacity or
technology teams want in their generative AI models? access hosted fine-tuning. And 50% already want their
Most of the survey respondents (64%) prioritize models to have agentic capabilities (see Figure 2).
performance: the model must provide information that
is relevant, accurate, and coherent to any who query Agentic systems act as autonomous agents,
it. Omni-modal and multi-modal capabilities are also performing tasks and making decisions without the
a priority for 56% of respondents. “We want end- need for direct human intervention. Pereyra says
users to be able to interact with models in different Harvey AI is starting to do more agentic workflows.
ways,” says Mark Austin, vice president of data “These are systems that can take complex legal
science at AT&T. “Chat is just one interface. We’re also tasks, decompose them, solve the subtasks, put them
experimenting with voice—making sure the latency is together, and produce associated or higher-level
good—as well as human-looking avatars.” product,” he says.

“We want end-users to be able to interact


with models in different ways. Chat is one
interface; we’re also experimenting with voice
and human-looking avatars.”
Mark Austin, Vice President, Data Science, AT&T
8  MIT Technology Review Insights

“Model evaluation should be


Customizing operations at AT&T
a critical application feature.
Rather than evaluating once AT&T is adapting a variety of different generative AI
use cases to enhance operational efficiency1 across
and moving on when models the business. The telecom giant is using Microsoft
Azure OpenAI Service to help it migrate legacy
change, we need to move to code into modern code to accelerate developer
continuous evaluation.” productivity, to allow IT professionals to request
additional resources like virtual machines, and to
enable employees to complete common human
Asha Sharma,
resources tasks by asking ChatGPT a question or
Corporate Vice President giving it a command. The task is then passed on to
and Head of Product, the appropriate person on the employee’s behalf.
AI Platform, Microsoft
It’s all part of AT&T’s plan to streamline rote or
repetitive tasks to enable employees to focus on
more complex, higher value jobs on its mission to
provide better connectivity, service, and value to its
customers. “Using Azure OpenAI Service to help
Having agreed on desired model attributes, technology automate some of these more common tasks will
teams then need to evaluate the model options available be an important change to the way we operate.
to them. Manual methods of evaluation predominate There will be meaningful time and cost savings,”
among the surveyed companies. But a quarter of says Jeremy Legg, chief technology officer at AT&T.
respondents (26%) say they are using automated
methods to a greater or lesser extent. Around one- AT&T deploys a number of Microsoft technologies,
fifth (17%) are starting to employ automated evaluation including Azure OpenAI Service, Azure AI Search,
with large data sets and almost 9% of respondents Azure Databricks, and Azure Compute.
are consistently automating, showing how quickly and
advanced this cohort has become. (see Figure 3).

Figure 3: Automated evaluation gains traction


Most organizations are still relying on manual evaluation for assessing generative AI model choices, but the use
of automation is picking up.

28%
26%
20%
17%

9%

We're doing many manual We're doing some manual We're benchmarking We're beginning to We're automating
evaluations for multiple evaluations with small model performance automate some evaluations consistently
projects data sets (using specific metrics evaluations with large
or standards) data sets

Source: MIT Technology Review Insights survey, 2025


MIT Technology Review Insights 9

“How companies evaluate partly depends on their AI


maturity,” says Boyd. “If they’re just getting started
“At the end of the day, this is
with generative AI, evaluation tends to be manual. As all about people. We are
maturity grows, and they start to evaluate different
data sets or different prompts, we’re seeing a lot more creating these systems to
use of automated evaluation.”
provide real value in the
According to Pereyra, automated methods currently things we care about and
have limitations when evaluating specific use cases.
“Automated evaluation gives you a rough directional doing that in the way
sense,” he says. “[The benchmarks it provides]
separate models into different generations, but they
we want them to.”
don’t help you discriminate much beyond that.” The
optimal approach to evaluating models, says Pereyra,
Ece Kamar,
is to employ different evaluation methods—manual, Managing Director of
automated, and benchmarking—in tandem. AI Frontiers,
Microsoft Research
In addition to this, how teams mentally approach
evaluation needs to change, says Asha Sharma,
corporate vice president and head of product, AI important pieces of code that you ship,” she says.
platform, at Microsoft. “Model evaluation should be a “Rather than evaluating once and moving on when
critical application feature that you experiment with models change, we all need to move to continuous
in production, just like you do with the other most evaluation.”

Agents of change
If generative AI has been a game-changer for AI and logistics, and securities trading, to name a
as a whole, agentic systems could do the same few. The list also includes AI model customization.
for generative AI. Taking generative AI’s unique Kamar foresees them being used to overcome one
capability of creating content in the form of of the biggest difficulties companies encounter with
responses to queries further, agents perform actions customization—data availability and quality.
based on the information the model has gathered.
“Multi-agent technologies will be able to create
“To be really useful, AI systems need to act, perceive high-quality and diverse synthetic data for many
the result of their action and then act again,” says domains,” predicts Kamar. She provides an example:
Ece Kamar, vice president of research and managing “You may lack data for a customer service scenario
director of Microsoft’s AI Frontiers Lab. “I don’t want a you have in mind. With a multi-agent setting, one
system that just tells me available flights, for example. agent can simulate the customer support person
I want a system that goes and books the flight for me.” and another can simulate the customer, asking
all kinds of questions. Another agent will monitor
Single-agent AI systems, in which an intelligent entity the conversation and make sure every piece of
(a bot, for example) acts alone to perform a specific information provided by the customer support agent
task, are already in commercial use today. Now, is grounded in facts, with the help of RAG.”
multi-agent systems—where multiple entities interact
collaboratively (sometimes as checks and balances) In developing these emerging systems, it is critical
to complete complex tasks—are coming to the fore. to prioritize the user experience throughout the
process, says Kamar. “At the end of the day, this is
Multi-agent systems lend themselves to autonomous all about people. We are creating these systems to
problem solving in areas such as supply chain provide real value in the things we care about and
management, manufacturing operations, transport doing that in the way we want them to.”
10 MIT Technology Review Insights

03
R
The quality and
performance
imperative
ealizing the potential of generative AI lies in
its continuous improvement. As models and
applications ingest more data, handle more
queries, and learn, their outputs become
more accurate and relevant. Therefore, the
importance of these models and applications is
Each method performs a distinct role. RAG scours and
retrieves data from external and internal sources to
ensure that model outputs are relevant and based on
the most up-to-date information available. Fine-tuning
is a necessary complement, ensuring that the model
is retrieving the internal data it needs to perform the
increasingly tied to business outcomes that organizations specific tasks set for it. Prompt engineering performs
seek in terms of efficiency, competitive differentiation, a third role, that of guiding the design of instructions,
user satisfaction, innovation, and other areas. or prompts, that users give to a model to obtain the
desired information.
Implementing a variety of methods
We asked technology executives their preferred methods “In effect, RAG gives AI the company’s memory,”
of customizing generative AI models. Their responses says Sharma. “While general purpose models are
make clear that their organizations employ not one but powerful, they’re missing context like product, policies,
a trio of methods. Two-thirds (67%) are implementing and the ways the organization does business.” She
RAG or exploring its use. Over half (54%) also employ observes companies benefitting from RAG today in
(or are exploring) model fine-tuning for this purpose. areas such as: customer support, providing real-time
Prompt engineering, cited by almost 46%, rounds out access to up-to-date product documentation, for
the array of methods employed (see Figure 4). example; employee productivity, by turning thousands

Figure 4: RAG is core to customization, in combination with other methods


Fine-tuning and prompt engineering complement RAG as preferred methods of model customization.

67% 54% 46% 4%

Retrieval augmented Fine-tuning Prompt engineering None of these/Don't know


generation (RAG)

Source: MIT Technology Review Insights survey, 2025


MIT Technology Review Insights 11

“With the help of AI agents and


Customizing for accuracy
a customized RAG framework with RAG
that taps into enterprise-grade When devising an advertising strategy for a
proprietary AI models and client, a vital piece of analysis is finding out what
contributions different media channels make to
data repositories, we can now the client’s sales. Dentsu found that using general

support some of the most purpose LLMs made retrieving this information
simpler and faster than before. However, says
impactful decisions our Brian Demitros, its accuracy left a lot to be
desired. “We were getting 40 to 50% accuracy
brands make to in the answers,” he says. “That’s obviously not

drive growth.” acceptable, so we had to do a lot of custom


development.”

Ece Kamar, Working on a campaign for a retail client,


Managing Director Dentsu created its own guardrails, embeddings,
of AI Frontiers, and vector stores, to harness its institutional
Microsoft Research expertise in data analysis for the retail and
marketing domains, according to Tanwir Danish.
“Our models now accurately answer to retailer-
specific questions on marketing performance and
of scattered documents into an instantly accessible budget allocation.” He continues, “With the help
corpus of expertise and skills; and compliance and risk of AI agents and a customized RAG framework
management, as RAG lets companies control exactly that taps into enterprise-grade proprietary
what information an AI model can access and reference. AI models and data repositories, we can now
support some of the most impactful decisions our
When it comes to customizing generative AI brands make to drive growth.”
applications, RAG is the simplest way to get going,
according to Boyd. “It doesn’t take long to wire a To further enhance decision-making, Dentsu
search engine up to a model on top of an organization’s integrated an agentic decision layer, enabling AI-
internal data,” he says. “Once that happens, a driven recommendations for optimizing marketing
generative AI application can be created very quickly. budget allocation. “We can do this because
That’s why there’s so much enthusiasm for it.” we have a library of AI and machine learning
models that identify key drivers of marketing
From there, RAG use usually requires other methods performance. Our optimization models simulate
to get optimal results. With his firm’s work in the legal business outcomes, such as sales, under different
industry, Pereyra finds RAG intuitive to use for basic scenarios,” Danish adds. This AI-powered
searches but more difficult as the queries become approach is now central to how Dentsu leverages
complex. He finds that fine-tuning works well with generative AI to shape campaign strategies for
models designed to perform specific tasks. clients. “We’ve achieved around 95% accuracy in
retrieving the most relevant data and insights—an
Demitros from Dentsu agrees, “RAG is best used in enormous improvement,” says Demitros. “Using
combination with other methods. We’re also doing fine- generative AI without a customized layer is simply
tuning, prompt engineering, and exploring innovative not viable when supporting business decisions
approaches. And for a lot of what we’re doing, there’s for some of the world’s largest brands.”
a heavy software layer on top, which includes security
and privacy as well as decision-making support.”
12 MIT Technology Review Insights

0
Customizing efficiency for the legal world at Harvey AI
Harvey AI—a legal artificial intelligence platform “We’re now looking at leveraging NVIDIA-accelerated
designed specifically for lawyers and law firms—is computing on Azure to train our own open-source
customizing generative AI for use cases2 including models,” he says. The company can also mix
summarizing and comparing documents, referencing managed compute service and dedicated capacity
case law, and facilitate research and analysis. for both its language models and embeddings in
every region where it operates, helping it more
Deployed across hundreds of law firms and efficiently scale throughout for research and product
legal teams, Harvey’s platform helps lawyers and development. “With our need to collocate compute
professional services providers deliver complex and models, Azure makes that easy,” explains Pereyra.
legal results more efficiently. “The reason it’s been
so hard to build technology for industries like legal is Harvey AI uses a mix of products from Microsoft,
the workflows are so varied and complex and no two including Azure OpenAI Service, Azure Database for
days are the same,” explains Gabe Pereyra, Harvey’s PostgreSQL, Azure Blob Storage, and Azure High
president and co-founder. Performance Computing (HPC).

The devil’s in the data Conveying the value of improvements to AI is a challenge


that’s tied to AI’s growing pervasiveness, says Sharma.
The chief barriers to customization cited in the survey
“When AI is embedded throughout your processes,
revolve, not surprisingly, around data integrity—a
isolating the impact is like trying to measure the ROI
term that takes in its accuracy, its relevance, and its
for electricity.” But organizations are starting to flip the
safeguarding. Just over half the respondents (52%)
measurement question, she says. “Instead of asking what
say their main difficulties lie in ensuring data privacy
AI accomplished, they’re asking what AI-enabled humans
and security. Almost as many (49%) cite data quality
can accomplish. For example, a health care provider we work
and preparation. And 45% report that they don’t have
with doesn’t just track the number of radiology images
the ability to measure the impact of customization on a
their model analyzes; it also tracks the additional patient
model’s output and performance (see Figure 5).
consultations that radiologists have been able to take on.”

Figure 5: Data integrity is the biggest challenge to model customization


The ability to measure ROI and finding the right developer talent and skills are also some of the biggest barriers.

Data privacy and security 52%


Data quality and preparation 49%
Ability to measure the impact
(compared to using 'as is') 45%
Developer talent/skills with
generative AI 41%
Access to real-time data 40%
Complexity of different data types
(structured, unstructured, etc.) 39%
Rapid pace of model changes 34%
Source: MIT Technology Review Insights survey, 2025
MIT Technology Review Insights 13

04
G
Risk factors

enerative AI adoption has brought a


greater focus from organizations on the
privacy, safety, and security of customized
models and their data. Executive concerns
about threats to these come through
clearly throughout the global survey, but the experts we
Keeping internal data protected
Probably the biggest privacy concern is with internal
data finding its way into public foundation models.
Customization increases the risk of this happening.
“As you train information into the models, the models
themselves are not capable of following any sort of
interviewed believe the concerns will begin to recede.
role-based authentication,” says Boyd. “If you put sensitive
information into the model, access needs to be restricted
The vast majority of surveyed executives (86%) say the
to people whom you trust with that information.”
advent of generative AI, and the capability to customize
it, has heightened application safety. Around one-third
(32%) say it has become “much more important.” This is one reason why Dentsu integrates multiple
The bigger the organization, the greater the degree layers of security and privacy safeguards into its
of concern with the potential for breaches. Among models. “For us, first-party data belongs solely to our
respondents from the biggest companies (with at least clients,” says Demitros. “We set a very high bar for its
$50 billion in annual revenue), 57% deem safety to be use and do not leverage it to enhance our proprietary
much more important now, compared with just 14% of those models. Under no circumstances can it be used to
working in the smallest firms in the survey (see Figure 6). benefit another client.”

Figure 6: The bigger the business, the greater the concern for security
Generative AI has made the privacy, safety, and security of custom apps a greater priority for organizations of all
sizes, but especially the largest.
$500 million to $1 billion

25% 62% 14%


$1 billion to $10 billion

8% 66% 25%
$10 billion to $50 billion

13% 39% 48%


$50 billion and above

20% 23% 57%


Total

14% 54% 32%


Not changed its importance Made it more important Made it much more important

Source: MIT Technology Review Insights survey, 2025


14 MIT Technology Review Insights

“There is some tension today with the trade-


off between the value of customization
versus its perceived risk. As AI maturity
grows and more organizations customize,
the perceived risk will decline.”
Gabe Pereyra, President and Co-Founder, Harvey AI

“There are so many moving parts to these models, the most prominent are hallucinations (cited by 60%),
and many of the security concerns are valid if you’re the use of compromised or malicious models (58%), and
deploying them in the wrong way,” says Pereyra. He prompt injection attacks (55%), which attackers use to
believes, however, that most issues with data leakage manipulate a model’s outputs (see Figure 7).
or contamination will eventually go away. “There is
some tension today with the trade-off between the Avoiding hallucinations
value of customization versus its perceived risk,” he Hallucination is actually less a security issue than one of
says. “As AI maturity grows and more organizations algorithm, search tool, or data quality, resulting in a model
customize, the perceived risk will decline.” generating incorrect or misleading information. There
is no silver bullet to eliminate hallucinations, according
Ensuring model and application integrity to Austin of AT&T. “You need a variety of approaches
Generative AI’s risk exposure extends beyond data to check, catch, and minimize them,” he says. “Good
privacy threats. The surveyed companies are being prompting helps. And using RAG to produce citations
proactive against a range of threat vectors, among which for every model answer is proving effective for us.”

Responsible AI and media insights at Dentsu


Media and advertising company Dentsu is speeding application architecture consists of loosely coupled
up the process3 of accessing increasingly complex microservices and multiple generative AI agents
media and consumer analytics with a chat-based that act autonomously but can collaborate using
predictive analytics copilot. The new agent can API-based integration and the GraphQL protocol.
interact with natural language and draws on Dentsu’s “The idea for this copilot was to sit within Dentsu’s
extensive media metrics, including forecasting, suite of applications as another kind of microservice
budgeting, modeled client data, and best practices. that maintains the same look and feel and inherits
What once took a team of data scientists weeks to a shared set of governance modules,” says Callum
sift through multiple systems, can now be done 90% Anderson, global director for DevOps and SRE at
faster, enabling Dentsu to quickly respond to trending Dentsu. “We have to be responsible in how we use
topics and emerging technologies for more innovative AI for all our clients,” he explains. “Everything we did,
campaigns. “Delayed campaigns slow customer we considered through the lens of how we governed
service and may result in missed opportunities,” says and ensured the AI is responsible.”
Becca Kline, senior director of analytics at Dentsu.
Dentsu uses many Microsoft technologies, including
The business is using Microsoft Azure AI Foundry Azure OpenAI Service, Azure API Management,
for the copilot to create a system that interacts Azure Kubernetes Service, and Azure Data Factory.
smoothly with its suite of business apps. The
MIT Technology Review Insights 15

Figure 7: Top generative AI threat vectors


Reducing hallucinations, protecting model integrity, and mitigating prompt injection attacks are the threat vectors
respondents are most focused on.

60% 58% 55%

Mitigating hallucinations Protecting model integrity Mitigating prompt injection attacks

48% 47% 33%

Mitigating AI-generated content Mitigating AI-generated use of Mitigating jailbreak attacks


outputs that are harmful known copyrighted content

Source: MIT Technology Review Insights survey, 2025

Sharma sees a shift occurring in how some companies where model security and privacy become less about
are thinking about model protection. “The purely restrictions and more about enabling innovation. It will
security mindset is giving way to what I call an AI not just be about protecting what we have but also
trust architecture,” she says. “We’re entering an era about securing what we can create.”

“We’re entering an era where model security and privacy


become less about restrictions and more about enabling
innovation. It will not just be about protecting
what we have but also about securing what
we can create.”
Asha Sharma,
Corporate Vice President and Head of Product,
AI Platform, Microsoft
16 MIT Technology Review Insights

05
M
Continuous operations

anaging entire portfolios, with several


dozen models and hundreds of
AI-enabled applications poses a set of
complex challenges. According to
Demitros, generative AI is a double-
edged sword for lifecycle management. One edge is the
The other edge of the sword is a massive opportunity.
Demitros notes that software developers only get to
spend part of their day coding. Much of their time is
spent on administrative work, such as documentation and
meetings, he says. “Generative AI is becoming very
helpful in offloading a lot of those administrative tasks,
challenge of keeping pace with advances in generative freeing up more dev cycles.”
AI capabilities. “When the base functionality changes,
the next iteration blows away all the benchmarks that Empowering dev teams
you’ve customized around, so now your entire product, We asked our global survey respondents how they
or a portion of it, needs to change,” he says. “You have to are empowering their development teams building
be willing and able to scrap product much more rapidly.” with AI applications. We learned that companies are
implementing a range of different tools and techniques.
Just over half (53%) are making streamlined telemetry
tools for tracing and debugging capabilities available to
Figure 8: Tools of the trade for next gen teams. A similar share (51%) of surveyed companies are
software development providing a simplified playground of development tools so
Streamlined telemetry, a simplified playground, teams can get started developing custom AI applications
and prompt development and management are more easily. And 46% are using prompt development
respondent organizations’ top development tools and management features that accelerate the creation,
for generative AI software. evaluation, and deployment of model prompts. This
additionally helps to enhance collaboration between AI
Streamlined telemetry tools for tracing and debugging
engineers and app developers (see Figure 8).
53%
Simplified playground to get started more easily Telemetry tools for tracing and debugging point the way
toward complete observability of code generation when
51% working with AI models. This is the next generation of
Prompt development and management that enhances AI application development, according to Sharma. “AI
collaboration between AI engineers and app developers
debugging shows how reasoning flows through your
46% system from initial prompt through model decision to
Advanced customization and fine-tuning capabilities final output,” she says. “It also enables performance
optimization, showing teams which prompts are most
43% effective and how different approaches impact accuracy
Comprehensive evaluation and testing capabilities and costs.” And tracing enhances transparency. “When
something unexpected happens, these tools lets you
41% trace back through the AI’s decision-making process,”
Source: MIT Technology Review Insights survey, 2025 says Sharma. “That builds trust.”
MIT Technology Review Insights 17

Identifying use cases


For all the internal capabilities that businesses are Automation for lifecycle
creating to manage their generative AI operations,
many will require help in several areas for some time to
management
come. When asked where they currently need support, According to Mark Austin, AT&T has 55
by far the most common response is the identification generative AI use cases in production today. One
of use cases, cited by 76% of survey respondents. of the biggest, he says, is an agentic framework
Scaling (mentioned by 47%), establishing prototypes to automate generative AI across the full lifecycle
(44%), performance and quality monitoring (44%), and of software development.
preparing solutions for deployment (42%) are other major
areas where external support and advice are needed. The initiative evolved from Ask AT&T, a generative
AI application the company launched in 2023 to
help employees interact with data. The first thing
Figure 9: Help needed to make sense of Austin and his team saw was about 40% of the
use cases questions being put to it were about coding. The
The vast majority of respondents say their benefit of this was obvious, he says: “We naturally
organization needs help early in the AI don’t want people pasting their code out on the
development process. internet and asking things like ‘How do I fix this?’
We saw that they were asking such questions
internally, and that was hugely important.”
Identifying the business use case and
success criteria for a custom AI project 76% The internal application has since evolved to become
more than a coding tool, encompassing the full
Scaling a successful generative AI solution
to more users (and with larger data sets) 47% software development lifecycle. Austin explains: “It
starts with someone writing in plain English that they
need to develop or modify code to do something
Monitoring performance and quality
continuously (adapting to ongoing change) 44% specific. The tool takes that and generates a plan
and frameworks for how the application will be
built, whether from scratch or atop an existing one.
Establishing prototypes by adjusting
prompts and swapping models 44% Then it writes the code, writes the test scripts to
test the code, checks for software vulnerabilities,
all the way through to deployment.”
Preparing a generative AI solution for
deployment 42% Austin and his team have found employees are
accepting around one-third of the code that
Evaluating performance and quality of
your AI solution 38% they’re getting from Ask AT&T. “It helps us go
faster,” he says. “It also helps us to develop more
securely by catching software vulnerabilities or
Discovering and selecting the right model
for your use case 37% security issues on the front end. And it’s a big
time saver.”

Connecting your AI model selection with


your organizational data 34%
Managing feedback on a deployed
generative AI solution 24%
Reverting learnings from one generative
AI project into a new idea 3%
Source: MIT Technology Review Insights survey, 2025
18 MIT Technology Review Insights

06
E
Conclusion

stimates vary but, over time, generative AI is


likely to add hundreds of billions of dollars,
or more, to the world’s GDP. If it does,
customization is likely to unleash a
significant portion of that value. As we have
Customization methods work best in combination.
RAG is an effective and widely used method for
improving generative AI models, but its utility
depends on the use case. More often than not, it is
most effective when employed with other methods,
illustrated throughout this report, it is when organizations particularly fine-tuning and prompt engineering, that
can tailor generative AI models and applications to their perform complementary roles.
specific needs, and make the most of their own data and
expertise in doing so, that the technology’s full potential Customization with data security in mind. Leakage
can be realized. As powerful as they are, today’s of sensitive data into public models is a real concern.
one-size-fits-all foundation models cannot achieve this. But strong model and data governance is an effective
safeguard. And teams can act to augment this, such as
Customization is not without its challenges, even by adding more security and privacy capabilities to its
for the biggest of organizations with substantial models and applications.
resources at their disposal. It requires high-quality and
well-governed domain-specific data. It requires deep Embrace the holistic abilities of customization. It’s not
collaboration among teams of specialists across AI, just about tailoring individual models and applications
applications, data, and infrastructure. And it demands to the needs of specific use cases. Generative AI can
confidence that the right safeguards are in place to be customized to design and development processes
protect modified models and applications against data across entire portfolios. Businesses should explore
leakage and malign actors. its advantages for improving applications portfolio
management as a whole.
Our research highlights several aspects of generative
AI customization that businesses—particularly those
with lower levels of AI maturity—should consider
carefully as they customize more of their models and
applications. Prominent among them are the following:

Rigorous evaluation is worth the time spent. Manual


study of available models and applications prior to
selection, along with benchmarking, is the right approach
to ensure a smooth path for later customization. There
is a strong case for automated evaluation as the
organization’s generative AI use cases grow in number
and the data sets they need grow in size.
MIT Technology Review Insights 19

About MIT Technology Review Insights


MIT Technology Review Insights is the custom publishing division of MIT Technology Review, the world’s
longest-running technology magazine, backed by the world’s foremost technology institution—producing
live events and research on the leading technology and business challenges of the day. Insights conducts
qualitative and quantitative research and analysis in the US and abroad and publishes a wide variety of content,
including articles, reports, infographics, videos, and podcasts.

About Microsoft Azure


Microsoft Azure is a leading cloud platform for building, deploying, and managing custom AI applications at
scale. Launched in 2010 as a pivotal shift from on-premises datacenters to the cloud, Azure continues to grow
with extensive capabilities that go far beyond infrastructure. With comprehensive services and tools for
developers, AI, data and apps, Azure delivers a cohesive approach to cloud computing that’s unmatched. Its
open, flexible platform is designed to empower companies of all sizes, across industries, and at any stage of AI
transformation.

Endnotes

1. “AT&T improves operations and employee experiences with Azure and AI technologies,” Microsoft, May 18, 2023,
https://www.microsoft.com/en/customers/story/1637511309136244127-att-telecommunications-azure-openai-service.

2. “Harvey makes lawyers more efficient with Azure AI infrastructure,” Microsoft, December 3, 2024,
https://www.microsoft.com/en/customers/story/19750-harvey-azure-open-ai-service.

3. “Dentsu reduces time to media insights by 90% using Azure AI,” Microsoft, November 19, 2024,
https://www.microsoft.com/en/customers/story/19582-dentsu-azure-kubernetes-service.

Illustrations
Illustrations assembled by Peter Crowther Associates Ltd.

While every effort has been taken to verify the accuracy of this information, MIT Technology Review Insights cannot accept any responsibility or liability for reliance by any person
in this report or any of the information, opinions, or conclusions set out in this report.

© Copyright MIT Technology Review Insights, 2025. All rights reserved.


20 MIT Technology Review Insights

MIT Technology Review Insights

www.technologyreview.com
[email protected]

You might also like