Guide To Penetration Testing

Download as pdf or txt
Download as pdf or txt
You are on page 1of 18

2 0 2 0 U LT I M AT E G U I D E T O

Penetration Testing
Y O U R G U I D E T O C H A N G E S I N T H E I N D U S T R Y,
A N D W H AT ’ S C O M I N G N E X T
TA B L E O F C O N T E N T S

3 Introduction

5 Why Penetration Test?

6 Why Traditional Penetration Tests Aren’t Fit for Purpose

8 Penetration Testing: What are the Options?

10 The 2020 State of Penetration Testing

14 What’s Next for Security Testing?

17 The End of an Era


INTRODUCTION

Pentesting started in the 90’s as adversary simulation. Its job wasn't


to find everything, it was to define the things a malicious attacker
was most likely to do, and actually could do in a given system. As a
result, the practice was absorbed by a variety of compliance initiatives
aimed at assuring regulators, and other stakeholders that a given
organization took seriously ever-evolving active threats in the wild.

In 2006 PCI-DSS mandated pentesting and vulnerability


scanning. In 2007 they published a special interest
group paper that defined the relationship between the
two as "a pentest is a vulnerability scan with manual
confirmation of exploitability." This created an explosion of scanner-
assisted pen tests, which did increasingly less to surface new,
complex threats, at increasingly high margins.

While the practical value of attack simulation itself hasn’t waned, many
security leaders view the current model for resourcing and deploying
pen tests to be a ‘necessary evil’. They know serious vulnerabilities
are often missed during testing — but they also know that penetration
testing addresses an established business need, like attainment of
PCI-DSS, HIPAA, SOC 2, and other compliance initiatives.

2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 3
W H E N R I S K O V E R TA K E S C O M P L I A N C E
Over the last 15 years, cybersecurity has inched closer to the boardroom table. In 2020, it finally has a
seat. Where previously many executives viewed the function purely as a cost center, cybersecurity’s
relatively recent ability to influence the financial decisions of customers, partners, and investors has
greatly elevated its status amongst the C-suite.

What changed? High-profile breaches like those of Target, Equifax, and Marriott made security tangible to
even the least tech-savvy consumer. And at the same time, increasingly large fines from regulators made
it clear that simply achieving compliance with industry frameworks wasn’t enough to keep an organization
safe from cyber attacks.

Pulling on that thread puts traditional penetration testing in the cross-hairs. Where previously these
programs were considered essential for a strong security program, the current method for resourcing and
deploying them has failed to keep up with the evolution of the modern attack surface. After all, how could
1-2 penetration testers accurately mimic the activity of the entire global cybercriminal community in just a
couple of weeks?

THIS REPORT EXAMINES

Why compliance is no longer the #1 reason for security testing,


and what other factors play a role.

Eight major issues with traditional penetration testing, and what


they mean for security teams.

The testing options available to modern organizations, plus


their pros and cons.

What the results of a recent Bugcrowd survey tell us about the


state of penetration testing in 2020.

How the next generation of penetration testing is addressing


the shortcomings of traditional services.

4 | 2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G
W H Y P E N E T R AT I O N T E S T ?

In the past, many organizations used penetration testing primarily as a tool to achieve compliance.

However, as cybersecurity programs have evolved, they have become more risk-based. Most industry
professionals understand compliance is just one of many key security objectives. They also know that
satisfying a compliance framework, while essential, does little to ensure the security of the organization.

In 2020, there are six primary reasons why organizations continue to invest in penetration testing:

P R OT E C T T H E O R G A N I Z AT I O N A N D I T S A S S E T S
Cyber attacks pose a serious — even existential — threat, and any digital asset is a potential target.
Penetration testing is used to identify vulnerabilities in websites, applications, and other digital
systems before they can be exploited by an attacker.

P R OT E C T C U S TO M E R DATA
Customer data is among the most important assets an organization has. Its possession is heavily
regulated. Any breach of customer data is potentially devastating, as it can lead to heavy fines from
industry regulators — not to mention a loss of customer trust. Penetration testing is used to find and
close vulnerabilities that could otherwise be used to gain unauthorized access to customer data.

REDUCE CYBER RISK


Once a vague concept, cyber risk is now a clearly calculable factor. Using tools like the Threat
Category Risk framework , it can be clearly articulated as a dollar value. For organizations with
a mature cybersecurity function, managing cyber risk is the #1 priority in cyber defense, and
penetration testing a critical component.

S AV E T H E O R G A N I Z AT I O N M O N E Y
Cybersecurity is no longer considered a cost center by cyber mature organizations, and for some,
it has a clearly measurable ROI. Security testing expenses can be tied directly to a reduction in the
cost of incident response, remediation, and regulatory fines.

S AT I S F Y S TA K E H O L D E R R E Q U I R E M E N T S
Customers, suppliers, shareholders, and other stakeholders have a huge influence on the decisions
an organization makes. As concepts like supply chain risk have become more widely understood,
key stakeholders have increasingly demanded close attention to cyber risk management.
Penetration testing plays a crucial role in this area.

P R E S E R V E T H E O R G A N I Z AT I O N ’ S I M AG E A N D R E P U TAT I O N
Cyber incidents can fundamentally harm an organization’s ability to operate by undermining
customer trust in its products, services, and brands. A major motivation for investment in penetration
testing is to preserve customer trust by avoiding high-profile incidents.

2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 5
W H Y T R A D I T I O N A L P E N E T R AT I O N T E S T S
AREN’T FIT FOR PURPOSE

In November 2018, a Bugcrowd survey of 200 cybersecurity leaders1 found 56% were dissatisfied with
their current penetration tests. Since then, the traditional penetration testing model has only become less
effective as a tool for promoting security and managing cyber risk.

To be clear, there is a huge distinction between penetration testers — the experts who use their skills to
identify security vulnerabilities — and the model through which they are deployed.

Penetration testers are an incredible resource. Bugcrowd’s ‘Crowd’ of security experts and ethical
hackers includes thousands of penetration testers. If they could, every organization would have dozens of
penetration testers working full-time to identify vulnerabilities in its digital assets. However, this approach is
cost-prohibitive and logistically impossible. There simply aren’t enough penetration testers in the industry.

As important as penetration testers are, however, the traditional penetration testing service model no
longer meets the needs of modern organizations. Here’s why:

S C H E D U L I N G D E L AY S
Organizations are frequently forced to accept long wait times (up to months) for each testing period.
As penetration testing providers seek to reduce time on the ‘bench’ for salaried employees, getting
resources where and when needed is a perpetual challenge.

I N C O M PAT I B L E I N C E N T I V E S
The provider’s need to reduce overhead can result in assignment of those not suited for the
engagement at hand. Unfortunately the only thing protecting customers from the caveat emptor
nature of this model is the provider’s desire to win the renewal next year.

S P E E D O F R E S U LT S
With a standard penetration test, the customer doesn’t receive results until the engagement is
concluded, often 14-24 days after testing begins. This leaves tested assets vulnerable for an
unnecessarily long time.

QUESTIONABLE SKILL FIT


A typical penetration test is carried out by 1-2 testers over a period of two weeks. Regardless of how
experienced the testers are, they can’t be versed in every possible attack technique, and their skill
sets may not be appropriate to the asset being tested. Equally, customers don’t have the option to
select which testers are assigned to their projects.

6 | 2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G
CHECKLIST FOCUSED
Most penetration tests are checklist-based, with minimal time or incentive for testers to use their
initiative or ‘dig deeper’ to find complex vulnerabilities.

P O I N T- I N -T I M E T E ST I N G
Most digital assets are penetration tested a maximum of 1-2 times per year. With modern, agile
development lifecycles, new codebase versions are released much more frequently. While an asset
may be secure immediately following a test, new code releases could leave it vulnerable to attack
until the next scheduled test.

LACK OF INCENTIVE
Traditional penetration testing providers operate a ‘pay for time’ business model, where customers
pay for a certain number of hours, and the assigned tester is only required to finish the methodology
in that time. Number and severity of vulnerabilities surfaced during this time is irrelevant to the
tester’s final pay.

L AC K O F S D LC I N T E G R AT I O N
Traditional penetration tests aren’t constructed in a way that actively integrates security and
development teams. Developers must manually migrate vulnerabilities to their preferred workspace
(e.g., JIRA or ServiceNow) before ‘sifting through’ a long report lacking context, priority, and
guidance on how to safely resolve.

P O O R R E S U LT S
A typical penetration test finds just eight high-value, unknown vulnerabilities on average. These
valid findings are interspersed with false positives and no-risk issues, making them hard to identify
and resolve. Worse, many genuine high-risk vulnerabilities are simply not identified.

Due to poor results, high cost, and time delays, traditional penetration testing services are not a cost
effective security control. Worse, because skill fit for a project is likely sub-optimal and testers aren’t
incentivized to ‘go deep,’ it’s likely that genuine, high-risk vulnerabilities will be missed.

Given this, the traditional penetration testing model is simply ineffective for Cyber Risk Management.

2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 7
P E N E T R AT I O N T E S T I N G : W H AT A R E T H E O P T I O N S ?

While security testing has become synonymous with traditional penetration testing services, there are
actually four primary testing options.

T R A D I T I O N A L P E N E T R AT I O N T E S T I N G
Despite the issues raised in the last section, many organizations still rely on traditional penetration testing
services. The ‘traditional’ model consists of one or two testers working against a set methodology for a
defined period, usually anywhere from three days to two weeks. This format is a mainstay of the security
industry, and at this point, executives and business leaders are pre-sold on the need for it.

PROS CONS
• Established budget line item • Delays to scheduling and results
• A known quantity • Inflexible with questionable skill fit
• Best for targets that require physical • Not optimized to incentivize true
presence to access/test risk reduction

C R O W D S O U R C E D S E C U R I T Y P E N E T R AT I O N T E S T I N G
The crowdsourced security penetration test is a comparatively new method of testing. Crowdsourced
options utilize a large pool of pay-per-project testers that work remotely. Often combined with an
additionally incentivized ‘pay for results’ approach to billing, crowdsourced testing is becoming the go-to
choice for security-conscious organizations.

PROS CONS
• Rapid setup and time to value • Not optimized for highly sensitive or
physical targets too big to ship
• Real-time results and SDLC integration
• ‘Bounty’ approach may not fit buying cycles
• Option to ‘pay for results’ instead of time
• New business case may be required
INTERNAL SECURITY TESTING
While often not feasible for smaller organizations, some enterprises prefer to build and maintain in-house
teams of security testers. This approach allows the organization to set its own testing schedule, and may
reduce barriers in some areas, e.g., provision of credentials.

PROS CONS
• Best for extremely sensitive work (Secret, • Labor-intensive to set up and maintain
NOFORN)
• Impossible to retain all possible testing skills
• Tests can be run as frequently as needed
• Hard to acquire new skills when needed
• Little marginal cost to testing

A MIXED TESTING APPROACH


Some organizations use a combination of traditional, crowdsourced, and internal testing to meet the
specific needs of each project.

PROS CONS
• Includes the best aspects of each method • Includes the worst aspects of each method
• Potential for thorough security coverage • Complex to arrange and maintain
• Testing depth is as-needed for each project • (Potentially) extremely high-cost

2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 9
T H E 2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G

In March 2020, we surveyed 129 cybersecurity engineers, managers, and CISOs to find out how they
conduct their penetration testing. All of our respondents had influence over their organization’s security
testing budget, methodology, and scope. Here’s what we learned:

COMPLIANCE IS NO LONGER THE #1 REASON FOR TESTING.


While 55% of respondents cited compliance as one of their reasons for testing, only 16% test purely for
compliance purposes. Meanwhile, 61% of respondents cited best practice as a reason for testing, and 38%
cited stakeholder requirements.

T R A D I T I O N A L P E N E T R AT I O N T E S T I N G S E R V I C E S A R E S T I L L
# 1 … J U S T.
In the past, traditional penetration testing was a dominant force
in security. However, recently, other approaches have gained
popularity. Our survey shows that in 2020, across all industries and
organization sizes, traditional penetration testing services account
for just 35% of security testing.

Crowdsourced testing has jumped into second place at 25%,


despite being around for a comparatively short time. 24% of
organizations complete most of their testing internally, while 15% use
a mixture of testing approaches.

L A R G E R O R G A N I Z AT I O N S A R E M O V I N G AWAY F R O M
T R A D I T I O N A L P E N E T R AT I O N T E S T I N G S E R V I C E S .
While other options are catching up, traditional penetration testing is the most common testing method
among small organizations with under 1,000 employees. For larger organizations — where increased
budgets open up more options — things are less clear.

Traditional penetration testing and crowdsourced testing are both utilized by just under a third of
organizations with 1-10k employees. Considering that crowdsourced testing is a far more recent option,
this highlights a rapid movement away from traditional penetration testing services.

1 0 | 2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G
At the enterprise level (10K + employees) the percentage of organizations relying on traditional
penetration testing services is barely half the rate we see in small organizations (21% vs. 39%). It’s also
dead equal with the percentage of enterprises using crowdsourced testing. Meanwhile, more than half
(57%) of enterprises rely primarily on internal security testing.

C R OW D S O U R C E D T E ST I N G F I N D S M O R E , H I G H E R -VA LU E
VULNERABILITIES.
When it comes to results, crowdsourced testing is the clear winner. 76% of crowdsourced testers received
at least 10 vulnerabilities per two-week test, compared to 57% of traditional penetration testing services.

The quality of results was also higher. Only a small fraction (13%) of crowdsourced testers received less
than 5% high-value vulnerabilities, while traditional penetration testing services were twice as likely to
deliver a poor result. Meanwhile, crowdsourced testing was 60% more likely than traditional penetration
testing services to deliver a large proportion (26%+) of high-value vulnerabilities.

Internal testing programs performed extremely poorly on both the quality and quantity of results, despite
their popularity with enterprises.

2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 1 1
T R A D I T I O N A L P E N E T R AT I O N T E S T I N G I S FAV O R E D BY
INFREQUENT TESTERS.
66% of organizations that use traditional penetration testing services test very infrequently — once per year
or less. By contrast, over half (52%) of organizations that use crowdsourced testing test at least quarterly.
Organizations that test internally are the most frequent testers, with 60% testing at least quarterly.

C O S T S A R E C O M PA R A B L E , B U T R O I I S N ’ T.

Our respondents placed traditional penetration testing neck-and-neck


with crowdsourced testing on total cost. However, since crowdsourced
delivers more, higher-quality results, it’s a clear winner for ROI.

1 2 | 2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G
While the cost of maintaining an internal testing capability varies, there’s no question that it falls beyond
what most organizations can afford. And, given its poor results, the ROI is questionable at best.

W H AT C A N W E L E A R N F R O M T H I S ?

Larger organizations recognize the issues with traditional


penetration testing services... but haven’t chosen the best
alternative. Internal testing performs poorly on the number and
quality of vulnerabilities found.

Crowdsourced testing finds more and higher-value vulnerabilities


than traditional penetration testing services, internal security
testing, and mixed programs.

Crowdsourced testing offers higher ROI than other methods, as


costs remain comparable while results are consistently better.

2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 1 3
W H AT ’ S N E X T F O R S E C U R I T Y T E S T I N G ?

Traditional penetration tests don’t meet the needs of modern organizations. A different solution is needed.

Crowdsourced testing approaches such as bug bounty programs have addressed many of the
shortcomings of traditional penetration tests. By operating on a pay-for-findings model, these programs
harness the power of the global hacking community to provide on-demand access to the expertise
needed for each engagement.

However, bug bounty programs haven’t fully replaced the need for standardized testing. Compliance is
still a crucial part of security, and most frameworks demand that testing follows a recognized methodology.

T H E N E X T- G E N E R AT I O N P E N E T R AT I O N T E S T
While many organizations share a need to achieve compliance milestones, not all have the same testing
requirements or capacity. Some seek continuous coverage, to match increasingly rapid development
cycles. Others need shorter testing windows throughout the year, as dictated by engineering workflows
or budgetary and procurement cycles. Equally, an organization’s appetite for tester incentivization may be
shaped by its bandwidth to process more vulnerabilities, as well as flexibility in maintaining an elastic pool
of monetary rewards.

To address these varied needs, Bugcrowd has launched the next generation of penetration testing. One
that taps into the diverse expertise of the global hacking community, while providing methodology-based
coverage and essential compliance reporting. And vitally, one where the customer chooses the terms.
C R O W D - P O W E R E D P E N E T R AT I O N T E S T I N G

‘ N E X T- G E N E R AT I O N ’ ‘CLASSIC’
P E N E T R AT I O N T E S T P E N E T R AT I O N T E S T
• Continuous coverage and on-demand • On-demand methodology-driven
methodology-driven testing, with testing over a defined period based
re-testing included on project scope
• Testers incentivized by reward for • Options for re-testing and expedited
valid vulnerabilities reporting
• Options for Premium SLAs and • Cost: Per-day, no incentive pool
Coverage Analysis
• Cost: platform + incentive pool

BOTH
• QSA-ASSESSED COMPLIANCE REPORT: Meet PCI-DSS, NIST 800-53 rev4, ISO 27001 and more
• SET UP IN <72HRS ON AVERAGE: Avoid lengthy scheduling delays and receive results faster
• STREAMING RESULTS: Receive vulnerabilities upon discovery and validation
• SDLC INTEGRATIONS: Push vulnerabilities to the places your developers live like GitHub and
ServiceNow
• REMEDIATION ADVICE: Help development fix quickly with prescriptive instructions based on
vulnerability type
• CROWDMATCHTM: Draw from the largest pool of talent and ensure skills and experience match
project needs.
• ON-DEMAND IN-PLATFORM REPORTING: Monitor vulnerability status and program activity
• FULLY MANAGED: Bugcrowd handles pentester matching, activation, and remuneration, as well
as vulnerability triage and prioritization

2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 1 5
HARNESSING THE CROWD
Harnessing the power of the global hacking community requires structure, process, and deep experience
in human-to-human interaction. All Bugcrowd’s crowd-powered penetration tests utilize the Crowdcontrol™
platform which includes dedicated program management. This combination of technology-enabled
expertise enables us to provide:

Thorough vetting and expert skills-matching of every crowdsourced


penetration tester.

Rapid triage, validation, and risk-ranking of all discovered


vulnerabilities.

Several software Development integrations for faster remediation.

Rapid time to value as results are streamed immediately post-


validation (not at program end).

Full program onboarding, clearly defined SLAs, and dispute resolution.

Full, real-time visibility into team activity, program outcomes, and costs.

Combined, these factors enable crowd-powered penetration tests to identify on average 7X more high-
priority vulnerabilities than traditional penetration tests.

1 6 | 2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G
THE END OF AN ERA

Once the gold standard for cybersecurity, traditional penetration


testing now falls far short of what’s needed by a modern organization.
From cost, to time, to quality of results, these services simply are not
an effective tool for improving security outcomes or managing cyber
risk. With the emergence of alternative methods, a compliance report
alone no longer justifies the opportunity cost of a test that fails to
deliver real results.

Penetration testing providers must evolve to provide both compliance


assurance, as well as deep security insights. Modern, agile
development lifecycles have highlighted the inability of traditional
providers to adapt to this new world in a way that is both functional,
and cost-effective. A new model is required. By leveraging an elastic
network of fully managed premium testing talent, Crowdsourced
security platforms offer organizations a faster path to compliance
without sacrificing the critical insights that help keep products and
customers safe.

In 2020, crowdsourced security testing has already caught up with


traditional penetration testing services in the enterprise market and
is rapidly closing the gap with smaller organizations. In the coming
years, this trend will only continue as more organizations recognize
the shortcomings of traditional penetration testing services and
begin to evaluate their options.

2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 1 7
K E Y TA K E AWAY S

S E C U R I T Y T E S T I N G I S N O L O N G E R P U R E LY F O R C O M P L I A N C E .
Modern organizations have to balance compliance with other needs, including customer and
stakeholder requirements, financial concerns, and cyber risk management.

O R G A N I Z AT I O N S H AV E TA K E N TO M I X I N G M E T H O D S F O R
P E N E T R AT I O N T E S T I N G .
To fill the gaps left by traditional testing services, modern organizations have begun incorporating
other methods where appropriate, for example, crowdsourced testing, internal testing, and hybrid
testing programs.

CROWDSOURCED TESTING DELIVERS MORE AND HIGHER


QUALITY VULNERABILITIES.
Users of crowdsourced security programs report a greater volume of higher-quality vulnerabilities
than traditional penetration testing services provide.

T R A D I T I O N A L P E N E T R AT I O N T E S T I N G S E R V I C E S A R E
L O S I N G P O P U L A R I T Y.
Traditional services are just barely holding the top spot, while organizations are increasingly
incorporating or switching to crowdsourced methods.

C R O W D S O U R C E D P E N E T R AT I O N T E S T I N G I S G A I N I N G
T R AC T I O N W I T H O R G A N I Z AT I O N S O F A L L S I Z E S .
Crowdsourced programs now account for between 20 - 30% of all security testing, depending on
organization size.

You might also like