MIT Technology Review - AI
MIT Technology Review - AI
Customizing
generative AI for
unique value
2 MIT Technology Review Insights
Preface
“Customizing generative AI for unique value” is an MIT Technology Review Insights
report sponsored by Microsoft Azure. This report seeks to understand how technology
leaders are customizing generative AI in their businesses and to what extent this
is a priority for their enterprise-wide AI strategy. Denis McCauley was the author
of the report, Laurel Ruma was the editor, and Nicola Crepaldi was the producer.
The research is editorially independent, and the views expressed are those of MIT
Technology Review Insights.
We would like to thank the following executives for their time and insights:
Methodology
The survey forming the basis of this report was
conducted by MIT Technology Review Insights in
September 2024. The survey sample consists
of 300 global executives holding senior
technology roles in their organizations: chief
information officers, chief technology officers,
chief data/AI officers, vice presidents
and directors of technology, data,
engineering and other influential roles.
The respondents’ organizations are
primarily large enterprises and
are headquartered in 12 countries
in the Americas, Europe, and
Asia-Pacific. Twelve industries
are represented in the survey,
with the largest contingents
of respondents working in
financial services, technology,
consumer goods and
retail, and manufacturing
businesses.
MIT Technology Review Insights 3
CONTENTS
01 Executive summary.................................................................................................................4
02 Embracing AI customization.............................................................................................5
Harnessing new benefits........................................................................................................5
Tomorrow’s large and customized industry-specific models.........................6
Choosing and evaluating models...................................................................................... 7
Customizing operations at AT&T.......................................................................................8
Agents of change.........................................................................................................................9
03 The quality and performance imperative.............................................................10
Implementing a variety of methods................................................................................10
Customizing for accuracy with RAG..............................................................................11
Customizing efficiency for the legal world at Harvey AI...................................12
The devil’s in the data..............................................................................................................12
04 Risk factors..................................................................................................................................13
Keeping internal data protected.......................................................................................13
Ensuring model and application integrity...................................................................14
Avoiding hallucinations...........................................................................................................14
Responsible AI and media insights at Dentsu.........................................................14
05 Continuous operations........................................................................................................16
Empowering dev teams.........................................................................................................16
Identifying use cases .............................................................................................................. 17
Automation for lifecycle management......................................................................... 17
06 Conclusion....................................................................................................................................18
0
4 MIT Technology Review Insights
01
S
Executive
summary
We surveyed 300 technology leaders in mostly large • Data integrity is the biggest barrier to customization.
organizations in different industries to learn how Around half the respondents (52%) cite the need to
they are seeking to leverage these opportunities. ensure data privacy and security as the primary difficulty
We also spoke in-depth with a handful of such they face with customization. Most (86%) say focusing on
leaders. They are all customizing generative AI privacy and security has become more important as they
models and applications, and they shared with us customize more actively. One-third overall (32%), and 57%
their motivations for doing so, the methods and tools of the biggest companies in the survey, deem this “much
they’re using, the difficulties they’re encountering, more important”.
and the actions they’re taking to surmount them.
• Advanced tools are empowering developers and
Our analysis finds that companies are moving ahead facilitating lifecycle management. Over half (53%) of
ambitiously with customization. They are cognizant organizations have adopted telemetry tools for tracing
of its risks, particularly those revolving around data and debugging for their developers. Also widely used
security, but are employing advanced methods and are a simplified playground of tools (by 51%) and prompt
tools, such as retrieval-augmented generation (RAG), development and management (46%) to facilitate better
to realize their desired customization gains. collaboration between engineers.
02
MIT Technology Review Insights 5
B
usinesses customize for a range of reasons, AI Platform, Microsoft
but tying them all together is the desire to
integrate the organization’s own knowledge
and expertise into the solutions they wish to
build. Such integration is what enables
businesses to build AI applications that are differentiated
from others in the market. “If you’re using an off-the-shelf
model, your application’s going to look very similar to
everyone else’s,” says Eric Boyd, corporate vice
president, AI platform, at Microsoft. “What is it that
differentiates your application? It’s usually the data and Harnessing new benefits
knowledge that your organization can bring to bear.” The executives we surveyed see three benefits
above all from customization. Half of them (50%) say
Foundational models are not suitable for enterprise it’s important for delivering greater efficiency—for
use, says Brian Demitros, innovation lead for data and example, by automating tasks, streamlining workflows,
technology at advertising network Dentsu. “They’re and optimizing business processes. A similar share
trained on the internet and often contain innacurate (49%) also say customizing is important to gain
or misleading information,” he says. “You can’t competitive advantage from having a unique solution
simply use them out of the box and expect a level of in the market. Almost as many (47%) cite enhanced
accuracy needed to support critical decision-making. user satisfaction—for example, from personalization or
Customization is critical to get value out of them.” greater responsiveness (see Figure 1).
Figure 2: Key model attributes when selecting are performance, multi-modal capabilities, and flexibility
Emerging capabilities like agentic and multi-agent systems are already of high importance for enterprises.
Model performance
64%
Omni-modal and multi-modal capabilities
56%
Flexible model consumption and payment options
53%
Emerging capabilities such as agentic and multi-agent systems
50%
Openness
40%
Efficiencies of small and medium-sized models
37%
Source: MIT Technology Review Insights survey, 2025
Choosing and evaluating models Just over half (53%) of the surveyed executives place
Achieving these outcomes starts with getting model a priority on flexible model consumption and payment
selection right in the first place. What attributes do options: for example, the ability to reserve capacity or
technology teams want in their generative AI models? access hosted fine-tuning. And 50% already want their
Most of the survey respondents (64%) prioritize models to have agentic capabilities (see Figure 2).
performance: the model must provide information that
is relevant, accurate, and coherent to any who query Agentic systems act as autonomous agents,
it. Omni-modal and multi-modal capabilities are also performing tasks and making decisions without the
a priority for 56% of respondents. “We want end- need for direct human intervention. Pereyra says
users to be able to interact with models in different Harvey AI is starting to do more agentic workflows.
ways,” says Mark Austin, vice president of data “These are systems that can take complex legal
science at AT&T. “Chat is just one interface. We’re also tasks, decompose them, solve the subtasks, put them
experimenting with voice—making sure the latency is together, and produce associated or higher-level
good—as well as human-looking avatars.” product,” he says.
28%
26%
20%
17%
9%
We're doing many manual We're doing some manual We're benchmarking We're beginning to We're automating
evaluations for multiple evaluations with small model performance automate some evaluations consistently
projects data sets (using specific metrics evaluations with large
or standards) data sets
Agents of change
If generative AI has been a game-changer for AI and logistics, and securities trading, to name a
as a whole, agentic systems could do the same few. The list also includes AI model customization.
for generative AI. Taking generative AI’s unique Kamar foresees them being used to overcome one
capability of creating content in the form of of the biggest difficulties companies encounter with
responses to queries further, agents perform actions customization—data availability and quality.
based on the information the model has gathered.
“Multi-agent technologies will be able to create
“To be really useful, AI systems need to act, perceive high-quality and diverse synthetic data for many
the result of their action and then act again,” says domains,” predicts Kamar. She provides an example:
Ece Kamar, vice president of research and managing “You may lack data for a customer service scenario
director of Microsoft’s AI Frontiers Lab. “I don’t want a you have in mind. With a multi-agent setting, one
system that just tells me available flights, for example. agent can simulate the customer support person
I want a system that goes and books the flight for me.” and another can simulate the customer, asking
all kinds of questions. Another agent will monitor
Single-agent AI systems, in which an intelligent entity the conversation and make sure every piece of
(a bot, for example) acts alone to perform a specific information provided by the customer support agent
task, are already in commercial use today. Now, is grounded in facts, with the help of RAG.”
multi-agent systems—where multiple entities interact
collaboratively (sometimes as checks and balances) In developing these emerging systems, it is critical
to complete complex tasks—are coming to the fore. to prioritize the user experience throughout the
process, says Kamar. “At the end of the day, this is
Multi-agent systems lend themselves to autonomous all about people. We are creating these systems to
problem solving in areas such as supply chain provide real value in the things we care about and
management, manufacturing operations, transport doing that in the way we want them to.”
10 MIT Technology Review Insights
03
R
The quality and
performance
imperative
ealizing the potential of generative AI lies in
its continuous improvement. As models and
applications ingest more data, handle more
queries, and learn, their outputs become
more accurate and relevant. Therefore, the
importance of these models and applications is
Each method performs a distinct role. RAG scours and
retrieves data from external and internal sources to
ensure that model outputs are relevant and based on
the most up-to-date information available. Fine-tuning
is a necessary complement, ensuring that the model
is retrieving the internal data it needs to perform the
increasingly tied to business outcomes that organizations specific tasks set for it. Prompt engineering performs
seek in terms of efficiency, competitive differentiation, a third role, that of guiding the design of instructions,
user satisfaction, innovation, and other areas. or prompts, that users give to a model to obtain the
desired information.
Implementing a variety of methods
We asked technology executives their preferred methods “In effect, RAG gives AI the company’s memory,”
of customizing generative AI models. Their responses says Sharma. “While general purpose models are
make clear that their organizations employ not one but powerful, they’re missing context like product, policies,
a trio of methods. Two-thirds (67%) are implementing and the ways the organization does business.” She
RAG or exploring its use. Over half (54%) also employ observes companies benefitting from RAG today in
(or are exploring) model fine-tuning for this purpose. areas such as: customer support, providing real-time
Prompt engineering, cited by almost 46%, rounds out access to up-to-date product documentation, for
the array of methods employed (see Figure 4). example; employee productivity, by turning thousands
support some of the most purpose LLMs made retrieving this information
simpler and faster than before. However, says
impactful decisions our Brian Demitros, its accuracy left a lot to be
desired. “We were getting 40 to 50% accuracy
brands make to in the answers,” he says. “That’s obviously not
0
Customizing efficiency for the legal world at Harvey AI
Harvey AI—a legal artificial intelligence platform “We’re now looking at leveraging NVIDIA-accelerated
designed specifically for lawyers and law firms—is computing on Azure to train our own open-source
customizing generative AI for use cases2 including models,” he says. The company can also mix
summarizing and comparing documents, referencing managed compute service and dedicated capacity
case law, and facilitate research and analysis. for both its language models and embeddings in
every region where it operates, helping it more
Deployed across hundreds of law firms and efficiently scale throughout for research and product
legal teams, Harvey’s platform helps lawyers and development. “With our need to collocate compute
professional services providers deliver complex and models, Azure makes that easy,” explains Pereyra.
legal results more efficiently. “The reason it’s been
so hard to build technology for industries like legal is Harvey AI uses a mix of products from Microsoft,
the workflows are so varied and complex and no two including Azure OpenAI Service, Azure Database for
days are the same,” explains Gabe Pereyra, Harvey’s PostgreSQL, Azure Blob Storage, and Azure High
president and co-founder. Performance Computing (HPC).
04
G
Risk factors
Figure 6: The bigger the business, the greater the concern for security
Generative AI has made the privacy, safety, and security of custom apps a greater priority for organizations of all
sizes, but especially the largest.
$500 million to $1 billion
8% 66% 25%
$10 billion to $50 billion
“There are so many moving parts to these models, the most prominent are hallucinations (cited by 60%),
and many of the security concerns are valid if you’re the use of compromised or malicious models (58%), and
deploying them in the wrong way,” says Pereyra. He prompt injection attacks (55%), which attackers use to
believes, however, that most issues with data leakage manipulate a model’s outputs (see Figure 7).
or contamination will eventually go away. “There is
some tension today with the trade-off between the Avoiding hallucinations
value of customization versus its perceived risk,” he Hallucination is actually less a security issue than one of
says. “As AI maturity grows and more organizations algorithm, search tool, or data quality, resulting in a model
customize, the perceived risk will decline.” generating incorrect or misleading information. There
is no silver bullet to eliminate hallucinations, according
Ensuring model and application integrity to Austin of AT&T. “You need a variety of approaches
Generative AI’s risk exposure extends beyond data to check, catch, and minimize them,” he says. “Good
privacy threats. The surveyed companies are being prompting helps. And using RAG to produce citations
proactive against a range of threat vectors, among which for every model answer is proving effective for us.”
Sharma sees a shift occurring in how some companies where model security and privacy become less about
are thinking about model protection. “The purely restrictions and more about enabling innovation. It will
security mindset is giving way to what I call an AI not just be about protecting what we have but also
trust architecture,” she says. “We’re entering an era about securing what we can create.”
05
M
Continuous operations
06
E
Conclusion
Endnotes
1. “AT&T improves operations and employee experiences with Azure and AI technologies,” Microsoft, May 18, 2023,
https://www.microsoft.com/en/customers/story/1637511309136244127-att-telecommunications-azure-openai-service.
2. “Harvey makes lawyers more efficient with Azure AI infrastructure,” Microsoft, December 3, 2024,
https://www.microsoft.com/en/customers/story/19750-harvey-azure-open-ai-service.
3. “Dentsu reduces time to media insights by 90% using Azure AI,” Microsoft, November 19, 2024,
https://www.microsoft.com/en/customers/story/19582-dentsu-azure-kubernetes-service.
Illustrations
Illustrations assembled by Peter Crowther Associates Ltd.
While every effort has been taken to verify the accuracy of this information, MIT Technology Review Insights cannot accept any responsibility or liability for reliance by any person
in this report or any of the information, opinions, or conclusions set out in this report.
www.technologyreview.com
[email protected]