The State of AI

Coding Assistants
In Theory, in Practice, and in the Future
Table of Contents

Introduction: Can We Prove the Value of AI? 3

The Current State of Copilot Adoption
5
What about other AI coding tools?
5
The Importance of Measurement
6
Copilot Adoption and Impact in the Real World
8
Copilot in Practice: How Varo Bank Uses Copilot 10
What’s Next: AI Coding Tool Adoption in 2025
14
About Jellyfish 16

2
T H E S TAT E O F A I C O D I N G A S S I S TA N T S

Introduction Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc congue ex

Can We Prove blandit, interdum diam nec, lobortis arcu. Donec nec elit dui. Nam a turpis
leo. Fusce congue urna vel neque fermentum ornare. Donec sagittis enim

the Value of AI? vel neque fringilla, congue bibendum lectus aliquet. Cras mi lectus, posuere
auctor rutrum eu, porttitor in justo. In hac habitasse platea dictumst. Sed
pharetra a justo vitae cursus.

Andrew Lau
Nam sollicitudin accumsan massa quis rutrum. Cras aliquam at orci a efficitur.
CEO & co-founder, Jellyfish
Vivamus eget dui sit amet quam suscipit pretium. Ut vulputate dui justo, et
bibendum lacus facilisis vel. Integer id tellus ut diam congue vulputate. Sed
ligula purus, imperdiet ac tempus eget, eleifend a magna. Donec maximus
non nunc pulvinar placerat. Duis maximus ligula nulla, vel congue urna
commodo id.
Today’s engineering organizations look a lot different than they did before the first
AI coding assistants first broke onto the scene. When GitHub launched Copilot
Donec quis augue ac enim consequat ultrices. Nunc dapibus scelerisque
back in 2021, we all knew it was a game-changing breakthrough not only in terms
feugiat. Maecenas vestibulum, justo nec hendrerit efficitur, augue leo
of available technology, but for the productivity gains it promised.
bibendum orci, ut finibus eros arcu vel arcu. Sed vel tellus nec risus varius
ultrices. Sed diam elit, maximus id porta in, ultricies at mauris. Donec at mattis
So now, three years on, where do we stand? Have these tools lived up to the
nisi. Curabitur elit ex, venenatis sed interdum nec, venenatis vitae eros.
promises and hype? Has AI permeated every aspect of our lives to deliver a more
productive, efficient, unified version of the world?

The simple answer is: not yet. But we have made tremendous gains in certain areas
and, at least for now, the opportunities seem infinite.

According to a recent survey, three out of every four developers are using or are
planning to use AI tools in their development processes. Engineers and the people
who manage them are, on the surface, fundamentally changing the way they think
about, write, and review code.

But this relatively broad adoption papers over the nuance. Larger, enterprise
organizations have always been slower to adopt new technology, and AI is no
different. For instance, a large financial institution, one that employs thousands
of developers, only recently rolled out a Copilot trial to a few hundred engineers.
We’re going to see these large enterprises testing the waters with proof-of-
concept projects – in some cases, testing multiple AI coding tools at once –
without claiming a winner or deciding if AI actually does, in fact, move the needle.

3
T H E S TAT E O F A I C O D I N G A S S I S TA N T S

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc congue ex
blandit, interdum diam nec, lobortis arcu. Donec nec elit dui. Nam a turpis
On the other hand, earlyleo.
adopters have already
Fusce congue moved
urna vel nequewell past first-wave
fermentum ornare. players
Donec sagittis enim
to explore newcomers including
vel nequeCodeium, Cursor,bibendum
fringilla, congue and Augment,
lectusto name Cras
aliquet. just mi lectus, posuere
a few. These early adopters are rutrum
auctor sure these tools willinmake
eu, porttitor justo. aIndifference, they’ve
hac habitasse platea dictumst. Sed
claimed they are in fact the future,
pharetra and, vitae
a justo for this group, there’s no going back.
cursus.

Between the laggards and thesollicitudin

Nam leaders, it’saccumsan
still too early
massato quis
tell where this
rutrum. all ends
Cras up.at orci a efficitur.
aliquam
But one determining factor will beeget
Vivamus whether
dui sitthese
ametAI coding
quam assistants
suscipit canUtactually
pretium. vulputate dui justo, et
deliver real impact and drive true value
bibendum tofacilisis
lacus the organizations that
vel. Integer id roll ut
tellus them out.
diam congue vulputate. Sed
ligula purus, imperdiet ac tempus eget, eleifend a magna. Donec maximus
That’s why, earlier this year, Jellyfish launched a first-of-its-kind solution to
non nunc pulvinar placerat. Duis maximus ligula nulla, vel congue urna
measure the impact of AI coding tools. The platform currently measures more
commodo id.
than 14,000 engineers at nearly 200 companies to provide a detailed look at how
AI coding tools are changing
Donecengineering.
quis augue ac enim consequat ultrices. Nunc dapibus scelerisque
feugiat. Maecenas vestibulum, justo nec hendrerit efficitur, augue leo
So far, we know that very few companies have reached full adoption, but those
bibendum orci, ut finibus eros arcu vel arcu. Sed vel tellus nec risus varius
that are using copilots are 12.6% faster. That said, these tools are also introducing
ultrices. Sed diam elit, maximus id porta in, ultricies at mauris. Donec at mattis
more bugs to the system, potentially offsetting any productivity gains. It’s
nisi. Curabitur elit ex, venenatis sed interdum nec, venenatis vitae eros.
essentially two steps forward, one step back.

To succeed with these tools, engineering leaders need to plan for and measure
their impact. They need to drive home a vision for how AI will reshape everything
from code reviews to team structure, and they need to figure out how to quantify
ROI in order to justify spend.

In this age of uncertainty around AI, how can you harness its power and clear
the roadblocks that stand in the way of broader adoption – while still hitting
your objectives? In this book, we’ll share real-world strategies, new data, and
predictions for how AI coding tools could shape the engineering industry in 2025
and beyond.

Let’s dive in.

4
T H E S TAT E O F A I C O D I N G A S S I S TA N T S

The Current State

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc congue ex
blandit, interdum diam nec, lobortis arcu. Donec nec elit dui. Nam a turpis

of Copilot Adoption
leo. Fusce congue urna vel neque fermentum ornare. Donec sagittis enim
vel neque fringilla, congue bibendum lectus aliquet. Cras mi lectus, posuere
auctor rutrum eu, porttitor in justo. In hac habitasse platea dictumst. Sed
How widespread are AI coding
pharetratools?
a justo vitae cursus.

Nam sollicitudin accumsan massa quis rutrum. Cras aliquam at orci a efficitur.
Vivamus eget dui sit amet quam suscipit pretium. Ut vulputate dui justo, et
bibendum lacus facilisis vel. Integer id tellus ut diam congue vulputate. Sed
ligula purus, imperdiet ac tempus eget, eleifend a magna. Donec maximus
non nunc pulvinar placerat. Duis maximus ligula nulla, vel congue urna
commodo id.
In 2024, Stack Overflow found that 62% of developers are currently using AI
tools in their development process, an increase of 18% over the previous year.
Donec quis augue ac enim consequat ultrices. Nunc dapibus scelerisque
Gartner believes the tools have staying power: they project that 75% of enterprise
feugiat. Maecenas vestibulum, justo nec hendrerit efficitur, augue leo
software engineers will use coding assistants by 2028.
bibendum orci, ut finibus eros arcu vel arcu. Sed vel tellus nec risus varius
ultrices. Sed diam elit, maximus id porta in, ultricies at mauris. Donec at mattis
The most widely known AI coding assistant is GitHub Copilot. Earlier this year,
nisi. Curabitur elit ex, venenatis sed interdum nec, venenatis vitae eros.
Microsoft claimed in an earnings report that there are now 1.3 million paid Copilot
subscriptions representing 50,000 different companies. GitHub’s most recent
data states that one out of every three companies from the Fortune 500 use
Copilot. And according to Wired, GitHub’s CEO Thomas Dohmke reports that
almost half of all code created by Copilot users is AI-generated.

What about other AI coding tools?

GitHub Copilot isn’t the only AI coding tool on the market. In fact, a Stack
Overflow survey found that the most popular AI tool for developers is Chat – likely
because of the tool’s free option. According to Stack Overflow, the five most
widely-used coding assistants are ChatGPT (84%), GitHub Copilot (49%), Visual
Studio Intellicode (11%), Codeium (7%), and Google Gemini (5%).

ChatGPT 84%
GitHub Copilot 49%
Visual Studio Intellicode 11%
Codeium 7%
Google Gemini 5%
0% 20% 40% 60% 80%

5
T H E S TAT E O F A I C O D I N G A S S I S TA N T S

Engineers differ in the way theyipsum

Lorem use AIdolor
coding tools. consectetur
sit amet, Stack Overflow’s 2024elit. Nunc congue ex
adipiscing
Developer Survey found blandit,
that AI tools will be
interdum diamintegrated mostly
nec, lobortis arcu.inDonec
documenting
nec elit dui. Nam a turpis
code (81%), testing codeleo.
(80%), and
Fusce writingurna
congue codevel(76%)
neque– fermentum
demonstrating
ornare. Donec sagittis enim
the appeal of using AI tovel
automate tediouscongue
neque fringilla, manualbibendum
processes. Experienced
lectus aliquet. Cras mi lectus, posuere
developers may be moreauctor
likely to use GenAI
rutrum to getinstarted
eu, porttitor onhac
justo. In writing codeplatea
habitasse beforedictumst. Sed
testing manually. pharetra a justo vitae cursus.

Two years into the AI coding

Nam era, developer
sollicitudin sentiment
accumsan is still
massa quisstrong:
rutrum.72%
Crasofaliquam
Stack at orci a efficitur.
Overflow survey respondents have
Vivamus favorable
eget opinions
dui sit amet quamon AI tools
suscipit for development.
pretium. Ut vulputate dui justo, et
bibendum lacus facilisis vel. Integer id tellus ut diam congue vulputate. Sed
ligula purus, imperdiet ac tempus eget, eleifend a magna. Donec maximus

The Importance of Measurement

non nunc pulvinar placerat. Duis maximus ligula nulla, vel congue urna
commodo id.

Krishna Kannan
Donec quis augue ac enim consequat ultrices. Nunc dapibus scelerisque
Head of Product, Jellyfish
feugiat. Maecenas vestibulum, justo nec hendrerit efficitur, augue leo
bibendum orci, ut finibus eros arcu vel arcu. Sed vel tellus nec risus varius
Earlier this year, Jellyfish released the diam
ultrices. Sed Stateelit,
of Engineering Management
maximus id porta Report
in, ultricies – a Donec at mattis
at mauris.
survey of more than 600nisi.
software engineers
Curabitur and engineering
elit ex, venenatis leaders.
sed interdum The
nec, report vitae eros.
venenatis
found a serious disconnect between business leaders and individual contributors
when it comes to AI. More than three-quarters (76%) of executives stated that
their team has “embraced AI,” while that sentiment was shared by just 52% of
engineers. That qualitative feedback points to a lack of quantitative data: when an
organization judges by sentiment rather than actual performance, they can quickly
become misaligned.

My team has 76% 52%

embraced AI. 61%
R&D Execs Engineers

AI is spreading quickly throughout engineering organizations, but engineering

metrics aren’t always keeping pace. Many organizations rely on surveys to track which
members of their teams are using AI tools – whether that’s a license purchased
for them by the company, or a free tool like ChatGPT used on an individual basis.
In the best case scenario, the team may also include a qualitative question around
productivity, i.e. “Do you feel AI coding tools are helping you work faster?”

6
T H E S TAT E O F A I C O D I N G A S S I S TA N T S

Those attempts at measurement

Lorem ipsumwill always
dolor sitbe imprecise,
amet, and they
consectetur lack elit. Nunc congue ex
adipiscing
connection to actual performance and business
blandit, interdum diam nec,impact. Asarcu.
lobortis companies commit
Donec nec elit dui. Nam a turpis
to significant investments in Fusce
leo. AI coding tools,
congue they
urna vel need
nequea fermentum
clear view of GenAI
ornare. Donec sagittis enim
adoption, impact, and ROI.
vel neque fringilla, congue bibendum lectus aliquet. Cras mi lectus, posuere
auctor rutrum eu, porttitor in justo. In hac habitasse platea dictumst. Sed
At Jellyfish, we recommend that organizations
pharetra use the following questions to
a justo vitae cursus.
frame their impact measurements:
Nam sollicitudin accumsan massa quis rutrum. Cras aliquam at orci a efficitur.
Vivamus eget dui sit amet quam suscipit pretium. Ut vulputate dui justo, et
bibendum
Are you building software lacusefficiently?
more facilisis vel. Integer id tellus ut diam congue vulputate. Sed
ligula purus, imperdiet ac tempus eget, eleifend a magna. Donec maximus
Compare operational metrics like cycle time to quantify the difference between
non nunc pulvinar placerat. Duis maximus ligula nulla, vel congue urna
engineers using AI and those that aren’t. Jellyfish’s own engineering team found
commodo id.
that cycle time for engineers using Copilot was up to 34% faster than those not
using Copilot.
Donec quis augue ac enim consequat ultrices. Nunc dapibus scelerisque
feugiat. Maecenas vestibulum, justo nec hendrerit efficitur, augue leo
Are your initiatives more predictable?
bibendum orci, ut finibus eros arcu vel arcu. Sed vel tellus nec risus varius
ultrices.
Keep track of deliverables Sed diam
in progress andelit, maximus and
completed, id porta in, ultricies
evaluate whetherat mauris. Donec at mattis
nisi. Curabitur
teams are consistently delivering elit ex, venenatis
on AI-assisted sed interdum nec, venenatis vitae eros.
projects.

Are you focusing more on what matters?

Work allocations should always play an essential role in an engineering team’s
planning and evaluation. Dig into whether allocations shifted after the adoption
of AI – using tools like Copilot to eliminate repetitive tasks could shift effort away
from support and keeping the lights on (KTLO) and toward value-driving work on
new features. With some Jellyfish customers, we’ve seen a 17% increase in effort
towards growth work.

How are engineering operations changing?

We’ve seen engineering work shift up to 15% from writing code to reviewing. As a
result, managers need to identify new sources of bottlenecks. Organizations with
visibility into the evolving software development life cycle will be better able to
effectively apply AI tools throughout the process.

Engineering leaders expect AI to make a difference. The SEMR found that 58% of
managers said AI can help alleviate burnout and enhance team well-being. But the
proof will come in the form of real performance metrics. Without measurement,
organizations have no way of knowing whether AI is delivering on its promise.

7
T H E S TAT E O F A I C O D I N G A S S I S TA N T S

Copilot Adoption and Impact in the Real World

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc congue ex
blandit, interdum diam nec, lobortis arcu. Donec nec elit dui. Nam a turpis
leo. Fusce congue urna vel neque fermentum ornare. Donec sagittis enim
Nik Albarranvel neque fringilla, congue bibendum lectus aliquet. Cras mi lectus, posuere
Product Researcher, Jellyfish
auctor rutrum eu, porttitor in justo. In hac habitasse platea dictumst. Sed
pharetra a justo vitae cursus.
We can take several different approaches to measuring the impact of AI coding tools
on engineering work. AreNam sollicitudin
the teams accumsan
moving massa
faster and quis rutrum.
producing moreCras aliquam
code? Is the at orci a efficitur.
Vivamus
quality of their work improving? eget
Are duiable
they sit amet quam
to focus onsuscipit pretium.
tasks that deliverUtmore
vulputate dui justo, et
value to the organization?bibendum lacus
For business facilisisROI
leaders, vel.isInteger id key
another tellus ut diam
factor congue vulputate. Sed
in evaluating
early experiments with AIligula purus,
coding imperdiet
tools. ac tempus
An engineering teameget, eleifend
might a magna.
be moving more Donec maximus
quickly, but is that speed non nunc
worth pulvinar
what they’replacerat.
spending Duis
onmaximus ligula nulla, vel congue urna
Copilot licenses?
commodo id.
As of November 2024, Jellyfish’s Copilot dashboard includes data from 144
Donec
companies with a combined quis
11,139 augue ac
licensed enim users.
Copilot consequat ultrices.referenced
The dataset Nunc dapibus scelerisque
feugiat.
below includes analysis from 6,500Maecenas vestibulum,
developers, comparingjusto necCopilot-assisted
their hendrerit efficitur, augue leo
work to work done without bibendum
Copilot. orci, ut finibus eros arcu vel arcu. Sed vel tellus nec risus varius
ultrices. Sed diam elit, maximus id porta in, ultricies at mauris. Donec at mattis
Here are some of the keynisi.
takeaways from
Curabitur elitour
ex, most recent
venenatis sedanalysis:
interdum nec, venenatis vitae eros.

Few companies have reached full adoption of Copilot:

The average penetration per company using Copilot is 52%
46% – meaning just over half of engineers are given access to the
tool. But those who adopt Copilot use it regularly: their usage
Average Copilot rate is 46%, meaning they rely on Copilot in their workflows
usage rate (% of days
roughly every other day. The low penetration could be
engineers use Copilot)
across the industry explained by unclear ROI measurement, with companies
rolling out the tool to a smaller subset of engineers while
they determine how to measure value and impact.

Developers using Copilot are delivering 12.6% faster:

We observed a 12.6% decrease in cycle time – measuring
the time between when work begins on a task and when it’s
12.6% decrease in cycle
resolved. A decrease in cycle time demonstrates significant
time, representing a acceleration in delivery speed. What makes this particularly
significant acceleration
of delivery speed noteworthy is that this reduction is based on Jira data, not
just Git-based metrics, meaning it captures a broader and
more accurate picture of engineering team performance.

8
T H E S TAT E O F A I C O D I N G A S S I S TA N T S

“EngineeringLorem
isn’t just about
ipsum dolorcoding;
sit amet, it’s about everything
consectetur in Nunc congue ex
adipiscing elit.
between – the blandit, interdum
planning, diam nec,
testing, lobortis arcu.and
collaboration, Donec nec elit dui. Nam a turpis
more.
By combiningleo.theFusce congue
signals from urna
Jiraveland
neque
Git,fermentum
you gain ornare. Donec sagittis enim
visibility
vel neque
into all the work of thefringilla,
lifecyclecongue bibendum
of building lectus aliquet. Cras mi lectus, posuere
software.”
auctor rutrum eu, porttitor in justo. In hac habitasse platea dictumst. Sed
Jellyfish Research
pharetra a justo vitae cursus.

Nam sollicitudin
Copilot accumsan
users are massamore
introducing quis bugs,
rutrum.but
Cras aliquam
they’re at orci a efficitur.
also
resolving
Vivamus bugs
eget duimore quickly:
sit amet quam suscipit pretium. Ut vulputate dui justo, et
bibendum
The data lacus facilisis
isn’t all vel.We
positive. Integer id tellus
observed ut diam
a 9.4% congue vulputate. Sed
increase
9.4% ligula purus,
in the imperdiet
number of bugsaccreated
tempusamong
eget, eleifend
Copilotausers.
magna. Donec maximus
Increase in bugs non nunc pulvinar
However, placerat.
that increase Duis maximus
is offset ligula
by a 27.4% nulla, velincongue
decrease bug urna
created by Copilot
users
commodo
backlog –id.
the number of bugs per week left unresolved.
For code generation, the increase in bugs created could
Donec quis augue ac enim consequat ultrices. Nunc dapibus scelerisque
indicate a decline in code quality; however, the decrease in
feugiat. Maecenas vestibulum, justo nec hendrerit efficitur, augue leo
unresolved bugs points to a healthier environment overall.
bibendum orci, ut finibus eros arcu vel arcu. Sed vel tellus nec risus varius
ultrices. Sed diam elit, maximus id porta in, ultricies at mauris. Donec at mattis
Acceleration of simple tasks leads to significant
nisi. Curabitur elit ex, venenatis sed interdum nec, venenatis vitae eros.
cost savings:
$112,500
Yearly cost savings Our analysis finds that engineers are saving up to one
for a company with
50 engineers hour per week on accomplishing simple tasks, completing
them 15% faster and allowing them to focus on more
complex work. That figure can add up to significant savings,
particularly for larger engineering organizations. According
to our estimates, a company with 50 engineers would save
approximately $112,500 per year – giving business leaders a
key figure to measure against their overall AI spending.

The per-seat cost for Copilot Enterprise is $39 per month.

For an organization with 50 developers, the total yearly cost would be $23,400.

For every organization, the true impact of Copilot and other AI coding tools
depends on how much the company is spending. If a company is spending around
$25,000 annually on Copilot licenses but seeing the $112,500 in savings mentioned
above, the investment is a no-brainer. However, the math may not always be this
clear-cut. The important takeaway is that each organization must determine the
metrics that matter to them as they assess the impact of AI coding tools, then map
that performance against their actual spending.

9
T H E S TAT E O F A I C O D I N G A S S I S TA N T S

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc congue ex
blandit, interdum diam nec, lobortis arcu. Donec nec elit dui. Nam a turpis

Copilot in Practice
leo. Fusce congue urna vel neque fermentum ornare. Donec sagittis enim
vel neque fringilla, congue bibendum lectus aliquet. Cras mi lectus, posuere
auctor rutrum eu, porttitor in justo. In hac habitasse platea dictumst. Sed
pharetra a justo vitae cursus.

Nam sollicitudin accumsan massa quis rutrum. Cras aliquam at orci a efficitur.
Vivamus eget dui sit amet quam suscipit pretium. Ut vulputate dui justo, et
bibendum lacus facilisis vel. Integer id tellus ut diam congue vulputate. Sed
ligula purus, imperdiet ac tempus eget, eleifend a magna. Donec maximus
non nunc pulvinar placerat. Duis maximus ligula nulla, vel congue urna
commodo id.
The data from Stack Overflow gives us an idea of AI adoption among engineers,
but adoption is only one small part of the overall story. For engineering and
Donec quis augue ac enim consequat ultrices. Nunc dapibus scelerisque
business leaders to judge the potential value of AI coding tools – and make
feugiat. Maecenas vestibulum, justo nec hendrerit efficitur, augue leo
educated investments accordingly – they need to understand impact.
bibendum orci, ut finibus eros arcu vel arcu. Sed vel tellus nec risus varius
ultrices. Sed diam elit, maximus id porta in, ultricies at mauris. Donec at mattis
Jellyfish’s Copilot dashboard gives us a clear look into how one of the most
nisi. Curabitur elit ex, venenatis sed interdum nec, venenatis vitae eros.
popular GenAI tools is reshaping engineering. At a basic level, we can see the
extent to which Copilot helps engineers work more quickly and deliver more
code. We can also see the impact of Copilot on code quality – both in terms
of bugs created and bugs resolved. And we can look one level deeper to make
assumptions on the actual value of Copilot: how much does time saved equate to
actual savings for an organization?

Let’s take a look.

10
T H E S TAT E O F A I C O D I N G A S S I S TA N T S

Steadily Improving Code Quality:

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc congue ex
blandit, interdum diam nec, lobortis arcu. Donec nec elit dui. Nam a turpis
How Varo Bank leo.Uses Copilot
Fusce congue urna vel neque fermentum ornare. Donec sagittis enim
vel neque fringilla, congue bibendum lectus aliquet. Cras mi lectus, posuere
auctor rutrum eu, porttitor in justo. In hac habitasse platea dictumst. Sed
Nicole Nunziata
pharetra Risk
Head of Technology a justo
andvitae cursus.
Director of Engineering, Varo Bank

Nam sollicitudin accumsan massa quis rutrum. Cras aliquam at orci a efficitur.
Varo Bank is the first all-digital nationally-chartered bank in the United States. Our
Vivamus eget dui sit amet quam suscipit pretium. Ut vulputate dui justo, et
company has a small engineering team of around 40 active full-time employees.
bibendum lacus facilisis vel. Integer id tellus ut diam congue vulputate. Sed
As a digital-native organization with a relatively small set of resources, we’re always
ligula purus, imperdiet ac tempus eget, eleifend a magna. Donec maximus
interested in emerging technologies that will help us maximize productivity and
non nunc pulvinar placerat. Duis maximus ligula nulla, vel congue urna
output without dramatically increasing spend.
commodo id.

In early 2024, we started using GenAI coding tools with a limited proof of concept:
Donec quis augue ac enim consequat ultrices. Nunc dapibus scelerisque
we looked at AWS CodeWhisperer, Gitlab Duo and Github Copilot. We started with
feugiat. Maecenas vestibulum, justo nec hendrerit efficitur, augue leo
just 10 engineers, and we decided early on that Copilot offered the best quality of
bibendum orci, ut finibus eros arcu vel arcu. Sed vel tellus nec risus varius
code suggestions to the engineers. Once we knew which tool we were going to
ultrices. Sed diam elit, maximus id porta in, ultricies at mauris. Donec at mattis
use, we rolled it out to all of our FTE engineers.
nisi. Curabitur elit ex, venenatis sed interdum nec, venenatis vitae eros.

We wanted to understand how

our engineers were using Copilot,
so we set up tools to collect both
quantitative and qualitative data.
For quantitative, we started using
Jellyfish’s Copilot Dashboard; for
qualitative, we set up a Copilot user
Slack channel to gather reactions
from our engineers.

Those two tools began to give us a picture of how Copilot was reshaping our
engineering organization. We could see in Jellyfish that our engineers were starting
to use Copilot for code generation. But we also heard in the Slack channel that
they were experimenting with Copilot to help write documentation and conduct
research on different innovative use cases.

11
T H E S TAT E O F A I C O D I N G A S S I S TA N T S

Enabling new coding capabilities

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc congue ex
blandit, interdum diam nec, lobortis arcu. Donec nec elit dui. Nam a turpis
leo.
Shortly after we rolled out Fusce congue
Copilot urna
to our full vel neque fermentum
engineering ornare. Donec
team, we launched an sagittis enim
vel neque
initiative to replatform three of ourfringilla,
products congue bibendum
into React lectus
Native — analiquet. Cras mi lectus, posuere
open source
auctorNative
UI software framework. React rutrumiseu, porttitor
a new in justo.
language forInus,
hac habitasse
which meantplatea
the dictumst. Sed
engineers had to get up pharetra
to speedavery
justoquickly.
vitae cursus.

We encouraged developers Namtosollicitudin

use Copilotaccumsan
to catchmassa quis working
up when rutrum. Cras
withaliquam
React at orci a efficitur.
Vivamus
Native, and the results were eget
strong. duiCopilot
The sit ametDashboard
quam suscipit pretium.
allows us to Ut vulputate dui justo, et
measure
bibendum
the suggestion acceptance rate forlacus
eachfacilisis
languagevel. Integer id tellus
– how often ut diam congue
developers are vulputate. Sed
accepting the code thatligula purus,
Copilot imperdiet
is offering. Theacsuggestion
tempus eget, eleifend
rate a magna.
for React NativeDonec maximus
nonwhich
started out at around 30%, nunc is
pulvinar placerat.
the industry Duis maximus
average. ligula
But over nulla,
time, vel congue
we saw the urna
acceptance rate increasecommodo id.
to 50%. Copilot code quality steadily improved, which
allowed our engineers to deliver quickly against aggressive timelines.
Donec quis augue ac enim consequat ultrices. Nunc dapibus scelerisque
feugiat. Maecenas vestibulum, justo nec hendrerit efficitur, augue leo
bibendum orci, ut finibus eros arcu vel arcu. Sed vel tellus nec risus varius
100%
ultrices. Sed diam elit, maximus id porta in, ultricies at mauris. Donec at mattis
nisi. Curabitur elit ex, venenatis sed interdum nec, venenatis vitae eros.

Suggestion 50%

acceptance rate 50%

for React Native
30%
(industry average)
0%
Suggestion acceptance Suggestion rate
rate start over time

We can’t be sure exactly why the code quality improved at the rate it did. However,
we took an unusual approach by having senior engineers work with Copilot and
act as team leaders for junior engineers. This extra layer of review – having senior
developers review the work coming from junior engineers – put us at ease before
pushing code into production. We feel this team structure provides the additional
context that Copilot needs to improve suggestion quality.

12
T H E S TAT E O F A I C O D I N G A S S I S TA N T S

Boosting productivity and

Lorem developer
ipsum experience
dolor sit amet, consectetur adipiscing elit. Nunc congue ex
blandit, interdum diam nec, lobortis arcu. Donec nec elit dui. Nam a turpis
Of course, everyone on an engineering team wants to deliver code quickly and for
leo. Fusce congue urna vel neque fermentum ornare. Donec sagittis enim
the code to be perfect when it goes into production. But we live in the real world,
vel neque fringilla, congue bibendum lectus aliquet. Cras mi lectus, posuere
and there always has to be a balance between speed and quality. Varo Bank is in a
auctor rutrum eu, porttitor in justo. In hac habitasse platea dictumst. Sed
highly regulated industry, and we have to be very careful not to impact customers
pharetra a justo vitae cursus.
and their bank accounts with software changes.
Nam sollicitudin accumsan massa quis rutrum. Cras aliquam at orci a efficitur.
We have seen an increase in code quality from our Copilot users, but we’ve only had
Vivamus eget dui sit amet quam suscipit pretium. Ut vulputate dui justo, et
the tool for six months. Is that a permanent quality metric? We’ll have to wait and see.
bibendum lacus facilisis vel. Integer id tellus ut diam congue vulputate. Sed
ligula purus, imperdiet ac tempus eget, eleifend a magna. Donec maximus
As we report up the chain to executives about the impact of Copilot, we want to be
non nunc pulvinar placerat. Duis maximus ligula nulla, vel congue urna
realistic about what the tool can offer. We would love to multiply the productivity
commodo id.
of every engineer by 10X, but we would also be happy with 2X – that will help us
get solutions to our customers faster. We’ve counseled our executives not to
Donec quis augue ac enim consequat ultrices. Nunc dapibus scelerisque
expect a huge, 75% productivity increase, but we’re also making clear that it’s
feugiat. Maecenas vestibulum, justo nec hendrerit efficitur, augue leo
having a measurable positive impact overall.
bibendum orci, ut finibus eros arcu vel arcu. Sed vel tellus nec risus varius
ultrices. Sed diam elit, maximus id porta in, ultricies at mauris. Donec at mattis
The other clear indicator of value is qualitative. In our Copilot Slack channel, we
nisi. Curabitur elit ex, venenatis sed interdum nec, venenatis vitae eros.
asked engineers whether they want to keep Copilot in their toolset. The answer
was a resounding “yes”: it adds a great deal of value; we want to keep using it; we
want the company to keep paying for it. As we continue using Copilot in 2025, we
hope to see that qualitative value reflected in quantitative metrics across code
quality and productivity.

13
T H E S TAT E O F A I C O D I N G A S S I S TA N T S

What’s Next Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc congue ex

Should We Be Worried blandit, interdum diam nec, lobortis arcu. Donec nec elit dui. Nam a turpis
leo. Fusce congue urna vel neque fermentum ornare. Donec sagittis enim
About Adoption of AIvel neque fringilla, congue bibendum lectus aliquet. Cras mi lectus, posuere
Coding Tools in 2025?auctor rutrum eu, porttitor in justo. In hac habitasse platea dictumst. Sed
pharetra a justo vitae cursus.

Andrew Lau
Nam sollicitudin accumsan massa quis rutrum. Cras aliquam at orci a efficitur.
CEO & co-founder, Jellyfish
Vivamus eget dui sit amet quam suscipit pretium. Ut vulputate dui justo, et
bibendum lacus facilisis vel. Integer id tellus ut diam congue vulputate. Sed
ligula purus, imperdiet ac tempus eget, eleifend a magna. Donec maximus
non nunc pulvinar placerat. Duis maximus ligula nulla, vel congue urna
commodo id.
As we move into 2025, the conversation around AI coding tools is dominated
by loud voices on both sides of the argument. On one side, you have the true
Donec quis augue ac enim consequat ultrices. Nunc dapibus scelerisque
believers: the AI evangelists who preach we should all be using AI coding tools all
feugiat. Maecenas vestibulum, justo nec hendrerit efficitur, augue leo
day, every day, and anyone who isn’t is going the way of the dinosaur. On the other
bibendum orci, ut finibus eros arcu vel arcu. Sed vel tellus nec risus varius
side, you have the rationalists: those who argue AI tools haven’t demonstrated
ultrices. Sed diam elit, maximus id porta in, ultricies at mauris. Donec at mattis
nearly enough value to justify the investment – yet.
nisi. Curabitur elit ex, venenatis sed interdum nec, venenatis vitae eros.

Who should we follow?

It’s important to remember that innovation isn’t linear, and the best technology
doesn’t always win (right away, anyways). Bill Gates famously said that people
tend to “overestimate the change that will occur in the next two years and
underestimate the change that will occur in the next ten,” meaning significant
change happens gradually, rather than in quick bursts.

When it comes to AI coding tool adoption, if buyers don’t see enough value from
their AI investments, will they fail to fund the last mile before these tools have the
chance to demonstrate their true value? When we look at the market landscape
for 2025, it’s important to remember two important factors:

We’re still in the early innings for this technology.

While we’ve all read plenty of opinions and hot takes about AI over the last two
years, adoption moves on a much slower timeline. Large enterprises are testing
the waters with proof-of-concept projects – in some cases, trialing multiple AI
coding tools at once – but they’ve yet to fully invest. The shape and trajectory of
the market won’t become clear until these major economic players start making
long-term decisions.

14
T H E S TAT E O F A I C O D I N G A S S I S TA N T S

Lorem ipsum
Most organizations still haven’t dolor
figured outsit amet,
how consectetur
to measure theadipiscing
impact elit. Nunc congue ex
of AI coding tools. blandit, interdum diam nec, lobortis arcu. Donec nec elit dui. Nam a turpis
leo.
If we expect adoption of AI Fuscetools
coding congue urna vel neque
to broaden, fermentum
it’s incumbent ornare.
upon Donec sagittis enim
us – the
velstrategies,
industry – to provide better neque fringilla,
best congue bibendum
practices, lectus aliquet.
and blueprints for thatCras mi lectus, posuere
auctor
adoption. We need to convey to rutrum
the endeu, porttitorhow
consumer in justo. In hac
to best habitasse
harness platea dictumst. Sed
the power
pharetra
of AI coding tools rather than a justo
waiting vitae cursus.
for adoption to happen organically. Then, we
need to give these users robust data about that adoption and, more importantly, the
Nam sollicitudin accumsan massa quis rutrum. Cras aliquam at orci a efficitur.
impact and ROI these tools provide. Without access to proper data, adoption will stall.
Vivamus eget dui sit amet quam suscipit pretium. Ut vulputate dui justo, et
bibendum
The opportunity here is bright, but lacus facilisis
we still have avel. Integer
long id tellus
way to ut diam
go. Instead ofcongue
makingvulputate. Sed
ligula purus,
dramatic changes or declaring imperdiet
winners ac tempus
and losers during eget, eleifend
the first a magna.
quarter of the Donec maximus
non nunc
game, we should focus instead onpulvinar
measuringplacerat. Duis maximus
and proving where ligula
thesenulla,
toolsvel
arecongue urna

actually delivering value. commodo id. engineering organizations that use metrics to
In 2025, the
inform their AI strategies will start to separate themselves from the pack – realizing
Donec quis augue ac enim consequat ultrices. Nunc dapibus scelerisque
the benefits and competitive advantages of AI.
feugiat. Maecenas vestibulum, justo nec hendrerit efficitur, augue leo
bibendum orci, ut finibus eros arcu vel arcu. Sed vel tellus nec risus varius
ultrices. Sed diam elit, maximus id porta in, ultricies at mauris. Donec at mattis
nisi. Curabitur elit ex, venenatis sed interdum nec, venenatis vitae eros.

See Jellyfish in Action

GenAI coding tools have the potential to change
engineering teams for the better, but leaders
need the ability to measure impact and verify ROI.
Jellyfish’s Copilot Dashboard allows engineering
leaders to view Copilot adoption rates throughout
their organization, understand how engineers are
using the tool, and measure impact in terms of
both productivity and code quality.

Want to see the Copilot Dashboard in action and

learn how your team can use it to measure and
optimize the industry’s best-known AI coding tool?

Schedule a demo

15
About Jellyfish
Jellyfish, the leading Engineering Management Platform,
enables engineering, finance and business leaders to align
engineering output with strategic business objectives. With
Jellyfish, businesses can measure, improve and communicate
the investment in and effectiveness of their engineering
organizations’ efforts with solutions aimed at improving developer
experience, simplifying software capitalization and understanding
the impact of GenAI coding tools. Clari, Hootsuite, Priceline,
ZoomInfo and PagerDuty use Jellyfish to optimize engineering
resource allocation so their teams can focus on what matters
most to the business. Learn more at jellyfish.co.