1 of 14

Code Red: White

paper

The business
impact of low
code quality
This paper presents data from a large-scale study on how code quality
impacts software companies in terms of time-to-market and product
experience. We conclude with an analysis of the impact and speci c
recommendations towards successful software development.

Target audience About CodeScene

• Business managers CodeScene is the intersection of code and
• Product owners/managers people, empowering companies to build great
• Technical managers software.
• Tech leads
• Development teams CodeScene was born in 2015 when founder
Adam Tornhill published the book “Your Code
as a Crime Scene”. It introduced a new
approach to software analysis which focused
on the evolution of a codebase over time.

CodeScene has become the next generation

of code analysis and is used by global Fo tune
100 companies in a wide variety of domains.
r
fi
 2 of 14

Content

1. Introduction: a software tragedy

Source code lacks visibility 4
Lack of visibility makes it easy to trade sho t-term gains for long-term sustainability 4

2. Where we are: the lack of code quality measurements

Code quality is an abstract concept, inaccessible to the business 5
Why is it so hard to measure the business impact? 5

3. Hard facts: the business impact of code quality

We need to know rather than “know” 6
Development data from 39 companies 6
Measure code quality via the Code Health metric 7
Measure the business impact by Time-In-Development and Defects 7
Results 8
124% faster development in Green Code 8
The unce tainty in Red Code means a feature can take an order of magnitude
longer to implement 8
15 times more defects in Red Code 9

4. Impact: code quality enables successful software

Enable a data-driven approach to software development 10
Increase developer productivity 10
Reduce unce tainty in estimates and commitments 11
Speed + Quality — you can have it all 12

5. Summary: a call to put theory into practice

Key message 13
Stakeholder bene ts 13
Prioritise code health issues via Hotspots 13

6. Fu ther information
The code analysis platform 14
About the author 14
Credits 14

v1.0 March 2022

r
r
r
fi
r
 3 of 14

Key takeaways
1

Everyone in the software industry “knows” that code quality is impo tant, yet we never had any
data or numbers to prove it. Consequently, the impo tance of a healthy codebase is largely
undervalued at the business level.
With this paper we remove the quotation marks so that “knows” becomes knows by attaching
numbers on the impact of unhealthy code. That way, code quality can nally become the
business concern that this study shows that it must be.

• Ef cient software development is a • Our research investigates 39 commercial

competitive advantage that enables codebases from various industries and
companies to maintain a sho t time-to- domains. The nding are peer reviewed,
market with a mature product experience. statistically signi cant, and reproducible.1
All metrics were automated via
• However, research shows that 23-42% of CodeScene.
developer’s time is wasted due to
Technical Debt and bad code. • The results show that code quality has a
dramatic impact on, both, time-to-market
• A key reason that this waste is tolerated is as well as the external quality of the
because code quality lacks visibility to non- product. High quality code has:
tech stakeholders and possible gains in A. 15 times fewer bugs,
code quality are hard to translate into
B. twice the development speed, and
business value.
C. 9 times lower unce tainty in
completion time.
• This paper aims to elevate code quality to
the business level by putting numbers on
the impact of unhealthy code.

Key ndings for healthy code

Implement features Reduce unce tainty in task

15 times fewer defects
twice as fast completion times by an
order of magnitude!

1 A. Tornhill & M. Borg (2022), “Code Red: The business impact of code quality - A Quantitative
Study of 39 Proprietary Production Codebases”. Accepted for publication in Proc. of International Conference on
Technical Debt 2022
fi
fi
r
fi
fi
r
r
r
r
fi
 4 of 14

1. Introduction: a software
tragedy

One of the great tragedies in software This implies that there is an untapped
development is that code quality lacks potential for software projects if the code
visibility. Hence, it becomes far too easy to quality is improved and technical debt paid
trade sho t-term wins like new features for down.
the long-term maintainability of the code
base. Until now, the business impact of code
quality has been vague and frequently
This puts the business at risk: ef cient dismissed as a technical concern. Our
software development is a competitive mission is to change that view: in this paper
advantage that enables companies to we present our ndings from studying 39
maintain a sho t time-to-market without proprietary production codebases with
compromising the quality of their products. respect to lead times for new features and
Adding to that challenge, our industry also the business risks in terms of defects.
faces a global sho tage of software
developers; demand substantially outweighs Our results indicate that improving code
supply. quality could free existing capacity; with 15
times fewer bugs, twice the development
So at the same time that our industry is speed, and 9 times lower unce tainty in
struggling with recruiting enough talent to completion time, the business advantage of
meet ever sho ter product cycles, research code quality is unmistakably clear.
indicates that up to 42% of developers' time
is wasted dealing with technical debt.

“With 15 times fewer bugs, twice the development speed, and 9 times
lower unce tainty in completion time, the business advantage of code
quality is unmistakably clear.”
r
r
r
r
fi
r
r
fi
 5 of 14

23

2. Where we are: the lack of

code quality measurements
Without an industry-wide
standard, code quality has 10% 42%
remained a subjective and
abstract concept that fails to
gain traction at the business Only 10% of business
managers actively
Developers waste 42%
of the work week on
manage technical debt. technical debt.
level.
The costs of Technical Debt and poor code Why is it so hard to measure the
quality are well known.2, 3 In the face of this business costs of low code quality?
trillion dollar problem, it’s surprising that we
still don’t have any business level KPIs around It’s impo tant to point out that the technical
technical debt. The lack of clear and debt costs referenced so far are based on
quanti able bene ts makes it hard to build a surveys and self-repo ted estimates and, to
business case for code quality and, hence, it’s the best of our knowledge, no prior research
unfo tunately much easier to trade sho t-term has measured the relationship between
wins like new features for long-term development speed and code quality.
sustainability.
Pa t of the reason is because existing models
Well-de ned code quality KPIs would allow all for technical debt (e.g. SQALE and SIG TD)
stakeholders to assess the situation, estimate lack a measure of the actual business impact;
the business impact, and prioritise accordingly, the cost of technical debt is not the time it
similar to how nancial KPIs drive businesses would take to x the code — that’s the
today. remediation work -- but rather the additional
development work due to technical quality
We believe that there are two main reasons issues.
behind the absence of technical debt KPIs:
To successfully crack this problem, we need
1. code quality is highly subjective, and to establish a link between a reliable code
quality measure and the impact on the
2. the industry lacks an established business.
relationship between code quality and
business outcomes.

2 Besker, T., Ma tini, A., Bosch, J. (2019) “Software Developer Productivity Loss Due to Technical Debt”

3 https://codescene.com/technical-debt/whitepaper/calculate-business-costs-of-technical-debt.pdf
r
r
fi
fi
r
r
fi
fi
fi
r
r
 6 of 14

3. Hard facts: data on the business impact

4

of code quality
Everyone in the software Development data from 39 companies
industry “knows” that code To measure the relationship between code
quality and the business impact, we collected
quality is impo tant, yet there data from 39 proprietary codebases under
is little data suppo ting that active development. The collected data
represents all development tasks that were
claim. completed over the past 6-12 months,
depending on codebase.
Without quanti able values, we — as an industry
— are unlikely to bridge the existing We included codebases from industry
communication chasm between engineering segments as diverse as retailing,
and business. This is evident in the previously construction, infrastructure, brokerage, data
referenced studies: even though technical debt analysis, and more to ensure a representative
wastes up to 42% of developer’s time, less than sample. We also capture data that generalises
10% of all business managers actually manage across implementation technologies. Hence,
their technical debt. our dataset includes codebases implemented
in 14 different programming languages
But it gets worse. Much worse. A study by (Python, C++, JavaScript, C#, Java, Go, etc.).
Besker et al. repo ts that “none of the
interviewed companies had a clear strategy on All companies gave their consent to
how to track and address the wasted time”4. pa ticipate in the study, and provided us with
This implies that the majority of companies: access to their source code repositories and
Jira product data.
A. don’t know how much time they waste on
poor quality code and technical debt, The measurements are performed through
the CodeScene tool, which automates the
B. don’t know where the main development code quality classi cation as well as
bottlenecks are amongst millions of lines of measuring the cycle time in development and
code, or the ratio of software defects. Let’s look at the
metrics and results.
C. don’t have a strategy to reduce that waste.

The study and numbers presented in this white • 40,000 software modules
paper aim to change the situation by offering
• 14 programming languages
statistically signi cant data on the relationship
• Multiple industry segments
between code quality and business outcome.

4Antonio Ma tini, Terese Besker, and Jan Bosch. 2018. Technical debt tracking: Current state of practice: A survey and
multiple case study in 15 large organizations. Science of Computer Programming 163 (2018), 42–61.
r
r
fi
fi
r
fi
r
r
 7 of 14
Measure
5
Code Quality via the
Code Health metric
Our research uses the Code Health metric
as a proxy for code quality. Code health is
an aggregated metric based on 25+
factors scanned from the source code. 5
The code health factors are known – from
research – to correlate with increased
maintenance costs and an increased risk
for defects.
Based on the code health score, each
source code le is automatically
categorised by the CodeScene tool as:

• Green Code (healthy code with low risk

for maintenance issues),

• Yellow Code (problematic code), and

Visualising code health: each source code le is classi ed based on maintenance costs and risks.

• Red Code (unhealthy code with high

maintenance risks).

Measure the business impact via

Time-In-Development & Defects
The business impact of low code health is From a business perspective, the number of
measured by integrating information from defects impacts the product experience:
product life-cycle tools like Jira. The CodeScene software with a high degree of defects delivers
tool fetches Jira issues, maps them to source a negative product experience, which in turn
code les, and calculates two data sets: impacts customer satisfaction and increases the
risk for customer churn.
1. Number of defects per source code le, and
2. Time-In-Development per le and Jira ticket. A longer Time-In-Development represents waste
that negatively impacts the roadmap execution.
Time-In-Development is the cycle time a Fu ther, if work in low quality code leads to
developer needs to implement the code more unpredictable delivery times, then that will
associated with a Jira backlog item. The cycle lead to lower con dence in commitments and
time is automatically calculated by CodeScene, strain the coordination within the organisation.
and is de ned as the time between when an Unce tainty is the enemy of any software
issue is moved to an “In Progress” state and delivery.
when the last code related to that Jira issue is
committed in version-control.

Code health automatically The Time-In-Development can be calculated

identi es problematic source code automatically: no administrative overhead
for measuring waste & oppo tunities.

5 https://codescene.com/code-health
r
r
fi
fi
fi
fi
fi
fi
fi
r
fi
fi
 8 of 14

124% faster development in Green Code

Our rst research objective investigates the link between
code health and time-to-market. We do that by measuring
the average Time-In-Development for Jira tasks and
correlating those numbers with the code health of the
impacted source code les.

Our results show that implementing a Jira task in Green

Code is 124% times faster than in Red Code for tasks of
similar scope. This means that a feature that takes 2 weeks
to implement could have been delivered in less than one
week if the code had been healthy.

We also note a signi cant impact already at the Yellow

code health level: the average development time for a Jira
issue in Yellow code is 78% longer than in healthy code. Implementing a feature or xing a bug is twice as
expensive in Red Code (relative scale).

The unce tainty in Red Code means a feature

can take an order of magnitude longer to
implement
The additional development time for Red Code matches
the intuition of experienced software developers: there’s
a productivity cost associated with low code quality.
However, to us as a research team, the big surprise was
that Red Code isn’t just more expensive: it also seems to
be more unpredictable than healthy code.

We explored that by measuring the maximum Time-In

Development for each source code le. The results show
that the maximum time to implement a Jira issue in Red
Code is an order of magnitude larger than in healthy
code!

Translated to a business context, an order of magnitude

difference in completion time means very high
unce tainty. As a product owner, high unce tainty makes
it impossible to keep any commitments; be it to Red code: more than 9 times longer average maximum
time leads to unce tainty during development (relative
customers or internal stakeholders. And to a developer,
scale).
unce tainty causes stress, over-time, and missed
deadlines.

In essence, Red Code is not only inherently more expensive:

it also carries a signi cant business risk.
fi
r
r
r
r
fi
fi
fi
fi
fi
r
 9 of 14

15 times more defects in Red Code

In addition to the excess development costs, we also
wanted to explore the external quality perspective:
does Red Code contain more bugs then code of
higher quality?

The results are quite dramatic: Red Code has fteen

times more defects on average than healthy code!
Just like for the maximum Time-In-Development, the
relationship isn’t linear: Red Code is signi cantly more
problematic than Yellow code. That said, even Yellow
code comes with a real cost and has, on average, 4
times as many defects as healthy Green Code.

This nding adds another business dimension,

namely customer satisfaction. A high degree of
Red code: 15 times more defects compared to high-quality
defects also impacts the development team in the code (relative scale).
form of unplanned work. Bug prone code makes it
hard to stay focused on planned tasks, which in turn
causes additional waste via context switches.

All ndings are statistically signi cant

All ndings repo ted in this paper are statistically signi cant. Peer reviewed & accepted for
That means, in layperson terms, that the ndings are the International Conference
unlikely to be just a uke or randomness in the data. on Technical Debt 2022

This is impo tant: when advising on professional software

development, money, jobs, and people are impacted.
Hence, it’s our responsibility as researchers to make sure
our data and resulting recommendations are as accurate Backed by academic research
and reliable as possible; the software profession is still a in collaboration with leading
relatively young eld in need of more facts and fewer software expe ts
opinions.

The formal research publication (A. Tornhill & M. Borg, Proc.

of the International Conference on Technical Debt, 2022)
— the foundation for this white paper — de nes the details
of the data collection, analysis and statistical methods. We
also made the data public so that the results can be
replicated.
fi
fi
fi
r
r
r
fi
fl
fi
fi
fi
fi
fi
fi
 10 of 14

4. Impact: code quality enables successful

6

software
Given that software companies waste up to 42% of
developers’ time on technical debt, it is surprising that as few
as 7.2% of organizations methodically track technical debt6.

Enable a data-driven approach to software

development
One explanation for why this waste is tolerated could
be simply that the impact of technical debt hasn’t
been possible to quantify at the level of the source
code. Consequently, the few companies that do
manage technical debt spend a signi cant amount of
that time on identi cation and prioritization of
potential issues to x. Actual improvements are more
rare, and the outcome unce tain.

The ndings in this paper give us the option to

challenge the status quo and elevate code quality to
the level of a business KPI. More speci cally, knowing
— as a software company — where you have Red,
Yellow, and Green Code enables a data-driven
approach to software development.

Increase developer productivity

Due to the predicted global sho tage of software
developers, companies will not be able to hire as many
developers as might be needed; demand substantially
outweighs supply. The ef ciency loss due to low code quality hasn’t been
possible to assess at code level. The study in this repo t
However, with 15 times fewer bugs and twice the aims to change that by quantifying the costs of Red Code.
development speed, existing capacity is freed to fuel
innovation and product growth. This is a clear and
quanti able business advantage that comes with
healthy code.

6Antonio Ma tini, Terese Besker, and Jan Bosch. 2018. Technical debt tracking: Current state of practice: A survey and
multiple case study in 15 large organizations. Science of Computer Programming 163 (2018), 42–61.
fi
fi
fi
r
fi
fi
r
r
fi
fi
r
 11 of 14

Reduce
7
unce tainty in estimates and commitments
Without visibility into your product’s code prioritising features. Planning a feature that
quality it becomes too easy to ignore any involves Red Code implies that the outcome
warnings from engineering. Over time, the is:
problems will deepen: features that took one A. higher risk,
day to develop a year ago now need 2 weeks, B. more expensive in terms of development
and often lead to unexpected rework. Low time, and
code quality kills innovation and progress. C. a large unce tainty in completion time.

Instead, knowing the health of your code That way, you can make data-driven decisions
reduces risk and aligns expectations. on whether to go ahead with the feature as
Consider a Product Manager (PM) responsible planned or, in case of code health issues,
for the company’s roadmap. If you — as a decide to sta t by refactoring/improving the
potential PM — knows the health of your existing code to reduce the risk.
code, then you’d use that when planning and

Example of a code health visualisation of React.js from Facebook: we immediately see where the risks
area (visualisation via the CodeScene tool).7

7 https://codescene.com/blog/evaluate-code-quality-at-scale/
r
r
r
 12 of 14
Speed
89 + Quality — you can have it all

Software development productivity is a times for deployment, the fewer production

depressing topic: most proposed measures failures. This might sound counter-intuitive at
like velocity, added lines of code, commit rst, but — as noted in this paper — our data
counts, etc. have done more harm than suggests the same relationship for coding:
good. the higher the quality, the quicker your
development.
A prominent exception is the work by the
DevOps Research Assessment (DORA)9 that As such, one goal of our work is to
introduced productivity measures via its Four complement DORA’s delivery metrics with
Key Metrics (FKM) . These metrics were also similar correlations between code quality
popularised through the Accelerate book8, and its business impact. As pointed out in
and are now spreading in the software the previous section, the waste during
industry. development can be signi cant.

The FKM of Accelerate clearly show that

there is no trade-off between speed and
quality. In fact, in order to deliver fast you
also need high quality: the sho ter your cycle

Figure. The scope of this white paper and how it complements the FMK from Accelerate.

“There is no trade-off between speed and quality. In fact, in order to

deliver fast you also need high quality code.”

8N. Forsgren PhD J. Humble and G. Kim. 2018. Accelerate: The science of lean software and devops: Building and
scaling high performing technology organizations. IT Revolution.

9 https://www.devops-research.com/research.html
fi
fi
r
 13 of 14

5. Summary: a call to put

10

theory into practice

Key message Stakeholder bene ts
This research was initiated to make code quality a The Code Health metric used in this research is
business concern by putting numbers on the automated and available via the CodeScene tool.
impact of unhealthy code. It’s an impo tant topic That way, organisations can sta t measuring these
since code quality has been an abstract concept KPIs today.
that fails to get attention at the management level.
Consequently, the software industry is wasting • Business managers: as evident from the ndings
critical developer time on code that’s more in this paper, code quality constrains the
expensive to maintain than it should be. business. Make Code Health a KPI that’s tracked
at the same level as pure business metrics like
Measuring code health enables data-driven ARR, customer churn, EBIT, etc.
decisions around core topics like Technical Debt,
priorities, and roadmap risks. In pa ticular, code • Product owners/managers: manage risk and
quality improvements can come with a business priorities via Code Health views. In pa ticular, be
expectation. That said, the impact of code quality conscious about the inherent unce tainty in Red
goes well-beyond nancial numbers; the inherent Code. Use Code Health measures to balance the
unce tainty in Red Code is likely to cause stress and trade-off between new features vs improving
friction within an organisation. what’s already there.

Finally, we acknowledge the fact that correlation • Development teams: these research ndings
doesn’t imply causation. We plan to conduct future offer a way to communicate improvements to
studies that help uncover the impact of code the product and leadership teams. Many
quality. developers are forced to take on technical debt
when the organisation pushes for wishful
deadlines; use the Code Health views to
There’s more: communicate the risks using a terminology that
Prioritise code health issues via Hotspots means something to the business. Also, use
Code Health trends10 to show the negative
Organisations need to balance sho t- and long-
impact in uenced by accumulated technical
term goals. No matter how much we want, we
debt.
simply cannot x all Red and Yellow code at
once. Instead, we need to prioritise by impact.

CodeScene’s hotspots are a powerful tool for

identifying the most expensive code quality
issues. Read all about it here: https://
codescene.com/blog/prioritize-technical-debt-
by-impact/

10 https://codescene.com/blog/3-code-health-kpis/
r
fl
fi
fi
fi
r
r
r
r
r
fi
r
fi
 14 of 14

6. Fu ther information

The CodeScene tool

CodeScene is a Swedish sta tup founded in 2015. CodeScene
is the intersection of code and people, empowering
companies to build great software. The CodeScene tool
represents a new generation of code analysis and used by
global Fo tune 100 companies in a wide variety of domains.

CodeScene automates all the metrics covered in this paper.

About the author

Adam Tornhill is a programmer who combines degrees in
engineering and psychology. He's the founder of
CodeScene, and the author of Software Design X-Rays, the
best-selling Your Code as a Crime Scene, Lisp for the Web
and Patterns in C.

Credits
A big, big THANKS to Markus Borg from Lund This paper represents the progress towards a
University for co-authoring the original long-term goal of mine: make code quality a
research paper. Invaluable contributions! rst class citizen of business. On this journey,
I learned a lot from the technical debt
Thanks to Aslam Khan, Joseph Fahey, and community as well as from all the people that
Romanela Polutak for reading drafts of this I had the chance to meet and discuss with
paper. Your feedback and comments made over the years. Thanks!
the paper so much better than what I could
have done on my own.
Contact Blog and a ticles
Contact: [email protected] www.codescene.com
Twitter: @codescene www.codescene.com/blog/
LinkedIn: https://www.linkedin.com/
company/codescene

v1.0 March 2022

fi
r
r
r
r