TH

THEE A
AII INDE
IN DE XX REPORT
REPORT

Measuring
trends in AI

AI
AI IN
INDEX
DEX AN
ANNUA
NUALL REPORT
REPORT

Welcome
Welcome toto the
the
2024
2024 AI
AI Index
Index Welcome to the seventh edition of the AI Index report.
The 2024 Index is our most comprehensive to date and

Repor
Reportt arrives at an important moment when AI’s influence
on society has never been more pronounced. This
year, we have broadened our scope to more
extensively cover essential trends such as technical
DOWNLOAD THE FULL REPORT
advancements in AI, public perceptions of the
DOWNLOAD INDIVIDUAL CHAPTERS technology, and the geopolitical dynamics
surrounding its development. Featuring more original
data than ever before, this edition introduces new
ACCESS THE PUBLIC DATA estimates on AI training costs, detailed analyses of the
responsible AI landscape, and an entirely new chapter
dedicated to AI’s impact on science and medicine.

The AI Index report tracks, collates, distills, and

visualizes data related to artificial intelligence (AI). Our
mission is to provide unbiased, rigorously vetted,
broadly sourced data in order for policymakers,
researchers, executives, journalists, and the general
public to develop a more thorough and nuanced
understanding of the complex field of AI.

TOP TAKEAWAYS

1. A
AII beats humans 2. Industry 3. Frontier models
on some tasks, but continues to get way more
not on all. dominate frontier AI expensive
expensive.
AI has surpassed human research
research. According to AI Index estimates, the
performance on several training costs of state-of-the-art AI
In 2023, industry produced 51
benchmarks, including some in models have reached
notable machine learning models,
image classification, visual unprecedented levels. For example,
while academia contributed only 15.
reasoning, and English OpenAI’s GPT-4 used an estimated
There were also 21 notable models
understanding. Yet it trails behind $78 million worth of compute to
resulting from industry-academia
on more complex tasks like train, while Google’s Gemini Ultra
collaborations in 2023, a new high.
competition-level mathematics, cost $191 million for compute.
visual commonsense reasoning and
planning.

4. The United States 5. Robust and 6. Generative AI

leads China, the EU, standardized investment
and the U.K. as the evaluations for LLM skyrockets.
leading source of responsibility are Despite a decline in overall AI private
top AI models. seriously lacking. investment last year, funding for
generative AI surged, nearly
In 2023, 61 notable AI models New research from the AI Index octupling from 2022 to reach $25.2
originated from U.S.-based reveals a significant lack of billion. Major players in the
institutions, far outpacing the standardization in responsible AI generative AI space, including
European Union’s 21 and China’s 15. reporting. Leading developers, OpenAI, Anthropic, Hugging Face,
including OpenAI, Google, and and Inflection, reported substantial
Anthropic, primarily test their fundraising rounds.
models against different responsible
AI benchmarks. This practice
complicates efforts to systematically
compare the risks and limitations of
top AI models.

7. The data is in: AI 8. Scientific 9. The number of AI

makes workers more progress accelerates regulations in the
productive and even further, thanks United States
leads to higher to AI. sharply increases.
quality work. In 2022, AI began to advance The number of AI-related
scientific discovery. 2023, however, regulations in the U.S. has risen
In 2023, several studies assessed AI’s
saw the launch of even more significantly in the past year and
impact on labor, suggesting that AI
significant science-related AI over the last five years. In 2023,
enables workers to complete tasks
applications—from AlphaDev, which there were 25 AI-related regulations,
more quickly and to improve the
makes algorithmic sorting more up from just one in 2016. Last year
quality of their output. These
efficient, to GNoME, which facilitates alone, the total number of AI-related
studies also demonstrated AI’s
the process of materials discovery. regulations grew by 56.3%.
potential to bridge the skill gap
between low- and high-skilled
workers. Still other studies caution
that using AI without proper
oversight can lead to diminished
performance.

10. People across

the globe are more
cognizant of AI’s
potential impact—
and more nervous.
A survey from Ipsos shows that, over
the last year, the proportion of those
who think AI will dramatically affect
their lives in the next three to five
years has increased from 60% to
66%. Moreover, 52% express
nervousness toward AI products and
services, marking a 13 percentage
point rise from 2022. In America,
Pew data suggests that 52% of
Americans report feeling more
concerned than excited about AI,
rising from 38% in 2022.

CHAPTERS

Chapter 1: Research and Development

This chapter studies trends in AI research and development. It begins by examining trends in AI publications and patents, and
then examines trends in notable AI systems and foundation models. It concludes by analyzing AI conference attendance and
open-source AI so#ware projects.

DOWNLOAD CHAPTER 1

1. INDUSTRY CONTINUES TO
DOMINATE FRONTIER AI
RESEARCH.

2. MORE FOUNDATION MODELS

AND MORE OPEN FOUNDATION
MODELS.

3. FRONTIER MODELS GET WAY

MORE EXPENSIVE.

4. THE UNITED STATES LEADS

CHINA, THE EU, AND THE U.K. AS
THE LEADING SOURCE OF TOP AI
MODELS. In 2022, China led global AI patent origins with 61.1%, significantly outpacing the United
States, which accounted for 20.9% of AI patent origins. Since 2010, the U.S. share of AI patents
has decreased from 54.1%.
5. THE NUMBER OF AI PATENTS
SKYROCKETS.

6. CHINA DOMINATES AI PATENTS.

7. OPEN-SOURCE AI RESEARCH
EXPLODES.

8. THE NUMBER OF AI
PUBLICATIONS CONTINUES TO
RISE.

Chapter 2: Technical Performance

The technical performance section of this year’s AI Index offers a comprehensive overview of AI advancements in 2023. It
starts with a high-level overview of AI technical performance, tracing its broad evolution over time. The chapter then
examines the current state of a wide range of AI capabilities, including language processing, coding, computer vision (image
and video analysis), reasoning, audio processing, autonomous agents, robotics, and reinforcement learning. It also shines a
spotlight on notable AI research breakthroughs from the past year, exploring methods for improving LLMs through prompting,
optimization, and fine-tuning, and wraps up with an exploration of AI systems’ environmental footprint.

DOWNLOAD CHAPTER 2

1. AI BEATS HUMANS ON SOME Performanceoftopclosedvs.openmodelsonselectbenchmarks

Source:AlIndex,2024|Chart:2024AlIndexreport
TASKS, BUT NOT ON ALL. 317.71%

300%

2. HERE COMES MULTIMODAL AI. 250%

в200%.
3. HARDER BENCHMARKS
EMERGE. percntage
pen
150%

100%
losed

54.82%
4. BETTER AI MEANS BETTER DATA 50% 39.57%
WHICH MEANS … EVEN BETTER 17.07%
27.54% 20.91%
8.96% 3.97%
AI. 0%
HELM MMLU ChatbotArena HumanEval SWE-bench MATH GSM8K AgentBench
Leaderboard
Generallanguage Coding Mathematicalreasoning Agent-based
behavior
5. HUMAN EVALUATION IS IN. Benchmark

On 10 select AI benchmarks, closed models outperformed open ones, with a median

performance advantage of 24.2%. Differences in the performance of closed and open models
carry important implications for AI policy debates.
6. THANKS TO LLMS, ROBOTS
HAVE BECOME MORE FLEXIBLE.

7. MORE TECHNICAL RESEARCH

IN AGENTIC AI.

8. CLOSED LLMS SIGNIFICANTLY

OUTPERFORM OPEN ONES.

Chapter 3: Responsible AI
AI is increasingly woven into nearly every facet of our lives. This integration is occurring in sectors such as education, finance,
and healthcare, where critical decisions are o#en based on algorithmic insights. This trend promises to bring many
advantages; however, it also introduces potential risks. Consequently, in the past year, there has been a significant focus on
the responsible development and deployment of AI systems. The AI community has also become more concerned with
assessing the impact of AI systems and mitigating risks for those affected.

This chapter explores key trends in responsible AI by examining metrics, research, and benchmarks in four key responsible AI
areas: privacy and data governance, transparency and explainability, security and safety, and fairness. Given that 4 billion
people are expected to vote globally in 2024, this chapter also features a special section on AI and elections and more broadly
explores the potential impact of AI on political processes.

DOWNLOAD CHAPTER 3

1. ROBUST AND STANDARDIZED

EVALUATIONS FOR LLM
RESPONSIBILITY ARE SERIOUSLY
LACKING.

2. POLITICAL DEEPFAKES ARE

EASY TO GENERATE AND
DIFFICULT TO DETECT.

3. RESEARCHERS DISCOVER
MORE COMPLEX
VULNERABILITIES IN LLMS.

4. RISKS FROM AI ARE A

CONCERN FOR BUSINESSES Researchers find a significant bias in ChatGPT toward Democrats in the United States and the
ACROSS THE GLOBE. Labour Party in the U.K. This finding raises concerns about the tool’s potential to influence
users’ political views, particularly in a year marked by major global elections.

5. LLMS CAN OUTPUT

COPYRIGHTED MATERIAL.

6. AI DEVELOPERS SCORE LOW

ON TRANSPARENCY, WITH
CONSEQUENCES FOR RESEARCH.

7. EXTREME AI RISKS ARE

DIFFICULT TO ANALYZE.

8. THE NUMBER OF AI INCIDENTS

CONTINUES TO RISE.

9. CHATGPT IS POLITICALLY
BIASED.

Chapter 4: Economy

Chapter 5: Science and Medicine

Chapter 6: Education

Chapter 7: Policy and Governance

Chapter 8: Diversity

Chapter 9: Public Opinion

PAST REPORTS

2023
2023 2022
2022 202
20211 2019
2019 2018
2018 201
20177

SUBSCRIBE
SUBSCRIBE TO
TO THE
THE
HAI
HAI NEWSLETTER
NEWSLETTER

Email

Artificial Intelligence Index

Stanford Institute for Human-Centered Artificial Intelligence
Cordura Hall
201 Panama Street
Stanford University
Stanford, CA 94305

Stanford Home Maps & Directions Search Stanford Emergency Info

Terms of Use Privacy Copyright Trademarks Non-Discrimination Accessibility

© Stanford University. Stanford, California 94305.