Gartner

Webinars
Gartner delivers actionable, objective
insight, guidance and tools to enable
stronger performance on your
organization’s mission critical priorities
 Enhance your webinar experience

Ask a Download Watch

Question Attachments Again

RESTRICTED DISTRIBUTION
2 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved.
 Connect with Gartner
Exploit Generative AI for Happier,
More Productive Software Developers

Arun Batchu
VP Analyst

RESTRICTED DISTRIBUTION
3 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved.
The Problem of (Very) Limited Working Memory

Cognitive Working Cognitive

Load Memory Fatigue
• Intrinsic Load A Limited-Capacity Cognitive • Reduced
• Extraneous Load System Responsible for Productivity
Temporarily Holding • Increased Errors
• Germane Load
and Processing Information
• Mental Exhaustion
During Various Mental Tasks

4 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Cognitive Load on Developers Is Unsustainable

Enormous amount Increasing number Constant switching

of enterprise code of open-source between coding
• Is causing code dependencies and searching
barriers in • Making problem • Is shattering
comprehension. solving time- attention-span.
consuming.

5 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Developers Need a
Coding Assistant
But don’t they have code completion?

Yes, but current assistants are

insufficient in reducing cognitive load:
• Static local knowledge, no dynamic,
global knowledge.
• Semantic, not learned.
• Single line, not multiline.

6 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Developers Need an
Intelligent Coding Assistant
… That reduces cognitive load
and increase flow:
• By enabling increased code
comprehension.
• Fast and effortless code discovery.
• Recommend correct, quality
code fragments.

7 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Two Vectors of Generative AI Coding Assistants

Plug-ins Conversational

8 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
 Plug-ins

9 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Augment Developers With AI-Powered Plug-ins

A new generation of coding assistants for

professional developers are demonstrating
not only longer and novel completions,
but also being able to use natural language
prompts to generate code.
Software engineering leaders should
exploit these innovative tools to increase the
productivity of their teams.

10 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
How AI Plug-ins Work
Interacts
Plugs In With
ML-Powered Coding Code Editors Developer
Assistant
Code Editors including IDEs Developer prompts the assistant
and command-line terminals. that responds with one or more
code fragments as suggestions.
Developer selects one (or
rejects) to insert code into the
program.

Behind the Scenes

Interacts
With Trained
On
ML-Powered Coding ML Model Repositories
Assistant (mostly) Cloud
Programming and natural
language repositories.
(Javascript, Python, Java, C, etc).

11 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Representative Vendors of AI Plug-ins

Tabnine Amazon GitHub Codeium

CodeWhisperer Copilot

12 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
 Do AI Code
Assistants Boost
Developer
Productivity
and Satisfaction?

13 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Productivity and Satisfaction Gains
(Vendor Reported)

Tabnine GitHub Copilot Amazon CodeWhisperer

“ … Generates about • “74% focus on more • “Users were 27% more

30% of our users’ code.” satisfying work” and likely to complete tasks
• “88% feel more successfully,” and
productive.” • “Users were an average
of 57% faster.”

14 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Productivity and Satisfaction Gains
(User Reported)

Distinguished engineer, large healthcare company

• “The NPS for AI coding assistants is unbelievable — 85.”
• “The productivity gain estimate is massive — 20% more
code per developer.”

NPS = Net Promoter Score

15 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Productivity and Satisfaction Gains
(User Reported)

VP of technology, large Wall Street firm

• “ ... We are way beyond than just thinking about it …
we are piloting, experimenting.”
• “ ... We have seen up to 40% of the code written by the AI
being accepted by the developer.”

NPS = Net Promoter Score

16 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Productivity and Satisfaction Gains
(User Reported)

Andrej Karpathy, prominent engineer

• “Copilot writes around 80% of my code, with around
80% accuracy. I don’t even really code, I prompt & edit.”

NPS = Net Promoter Score

17 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
 Conversational

18 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Conversational LLMs That Generate Code

A new generation of dialog-based coding

assistants for citizen and professional
developers are demonstrating an ability to
generate and analyze not only code fragments,
but entire programs, in an iterative,
conversational manner.
Software engineering leaders should
exploit these innovative tools to increase the
productivity of their teams.

19 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Representative Vendors of Conversational AI (for Code)

ChatGPT (OpenAI) Bard (Google) You Code (You.com)

20 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Differences Between Plug-ins and Conversational

Plug-ins Conversational
• Prompt: Code & Comments • Prompt: Code & Natural Language
• Integration: IDE • Integration: None
• Correctness: Included • Correctness: None
• Filters: Include • Filters: None
• Dialog facility: Emerging • Dialog facility: Mature
• Code size: Fragments • Code size: Whole Program

21 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Challenges With LLM Generated Code and Text

Intellectual Human
Code Quality Text Quality
Property Factors
• Unsecure • Biased • Copyrighted • Blind Trust
• Incorrect • Toxic Code • Code Leak
• Nonperformant • Data Leak
• Nonconformant

22 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Filters Are Critical Attributes

Optional Filters

Security Scan
ML-Powered Code
Coding Recommendations
Assistant Licensing Scan

Bias Avoidance

Uses ML to generate
code recommendations
based on prior code and
comments in the IDE.

23 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Strategic Planning Assumption

By 2027, 70% of Professional Developers Will Use AI-powered

Coding Tools, Up From Less Than 10% in 2023

Why It Will Happen: Why It Won’t Happen:

• Developers Will Demand Use • Inflated Expectations
• The AI Models Will Get Better, Safer • Unchecked Code Creates Problems
• The Development Environment • Leaders Don’t Align With Developers
Will Transform • Organizations Do Not See ROI
• Organizations Are Afraid of AI

24 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Look Beyond the Text Editor/IDE

Terminal Cloud Dev IaC

• Warp • Replit • IBM (Ansible)
• Fig • GitHub • ChatGPT
• Copilot for CLI Codespaces

IDE = Integrated Development Environment; IaC = Infrastructure as Code; CLI = Command Line Interface
25 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
AI Plug-ins and Conversational AI Are Converging

Replit

Copilot X

Codeium

26 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
Recommendations

Software engineering leaders responsible for software engineering technologies should:

Pilot and deploy AI-powered coding assistants,
– with an eye toward fast rollouts to maximize developer productivity.
Handpick software engineers to master and teach the new skill of “prompt design.”
– by using a combination of natural language and coding practices to figure out how
to optimize code generation with minimal effort.
Form and lead a cross-functional team,
– to identify, monitor and solve for emerging challenges in copyright infringement,
code quality.
– to evaluate AI augmentation in all aspects of the software development life cycle:
requirements, design, development, testing, integration, deployment, operations.

27 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.
 Ask the speaker

RESTRICTED DISTRIBUTION
28 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved.
 The World’s Most Important Gathering
of CIOs and IT Executives™
11 – 13 September 2023 | Gold Coast, Australia
16 – 19 October 2023 | Orlando, FL
6 – 9 November 2023 | Barcelona, Spain
13– 15 November 2023 | Tokyo, Japan
28 – 30 November 2023 | Kochi, India

Discuss key topics like executive At this year’s conference, you’ll learn:
leadership, data and analytics, customer Discover tools and techniques to enhance your
experience, cybersecurity and more. At IT and business strategies
Gartner IT Symposium/Xpo™ you’ll
reimagine your approach to leadership, Examine the opportunities and risks in adopting
emerging and innovative technologies
technology and business strategy.
Challenge how you think about leadership and
Learn more: gartner.com/conf/cio discover new approaches to lead

#GartnerSYM

© 2023 Gartner, Inc. and/or its affiliates. All rights reserved.

 Software Engineering
Technology Roadmap

Software engineering leaders must equip their teams

with the latest platforms, practices and tools to build
software at speed and at scale.

→ Download eBook

RESTRICTED DISTRIBUTION
© 2023 Gartner, Inc. and/or its affiliates. All rights reserved.
Gartner for IT on Social Media

Want to stay in-the-know? Connect with us on

LinkedIn and Twitter to receive the latest
Gartner IT insights and updates across
research, events and more.

It’s all curated specifically for IT leaders and

decision-makers.

Follow us on
Looking for insights delivered to your inbox?
Subscribe to our bi-weekly newsletter.

RESTRICTED DISTRIBUTION
© 2023 Gartner, Inc. and/or its affiliates. All rights reserved.
 Become a Client
Clients receive 24/7 access to proven management
and technology research, expert advice,
benchmarks, diagnostics and more.

Fill out the form to connect with a representative

and learn more.

Learn More

Or give us a call: +441784614280 | +1 855 637 0291

8 a.m. – 7 p.m. ET
8 a.m. – 5 p.m. GMT
Monday through Friday

RESTRICTED DISTRIBUTION
32 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved.
 Get more Gartner insights

Download the research slides

View upcoming and on-demand Gartner webinars

at gartner.com/webinars

Rate this session

RESTRICTED DISTRIBUTION
33 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved.
Rate this session

RESTRICTED DISTRIBUTION
34 © 2023 Gartner, Inc. and/or its affiliates. All rights reserved.