Guide To Penetration Testing
Guide To Penetration Testing
Guide To Penetration Testing
Penetration Testing
Y O U R G U I D E T O C H A N G E S I N T H E I N D U S T R Y,
A N D W H AT ’ S C O M I N G N E X T
TA B L E O F C O N T E N T S
3 Introduction
While the practical value of attack simulation itself hasn’t waned, many
security leaders view the current model for resourcing and deploying
pen tests to be a ‘necessary evil’. They know serious vulnerabilities
are often missed during testing — but they also know that penetration
testing addresses an established business need, like attainment of
PCI-DSS, HIPAA, SOC 2, and other compliance initiatives.
2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 3
W H E N R I S K O V E R TA K E S C O M P L I A N C E
Over the last 15 years, cybersecurity has inched closer to the boardroom table. In 2020, it finally has a
seat. Where previously many executives viewed the function purely as a cost center, cybersecurity’s
relatively recent ability to influence the financial decisions of customers, partners, and investors has
greatly elevated its status amongst the C-suite.
What changed? High-profile breaches like those of Target, Equifax, and Marriott made security tangible to
even the least tech-savvy consumer. And at the same time, increasingly large fines from regulators made
it clear that simply achieving compliance with industry frameworks wasn’t enough to keep an organization
safe from cyber attacks.
Pulling on that thread puts traditional penetration testing in the cross-hairs. Where previously these
programs were considered essential for a strong security program, the current method for resourcing and
deploying them has failed to keep up with the evolution of the modern attack surface. After all, how could
1-2 penetration testers accurately mimic the activity of the entire global cybercriminal community in just a
couple of weeks?
4 | 2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G
W H Y P E N E T R AT I O N T E S T ?
In the past, many organizations used penetration testing primarily as a tool to achieve compliance.
However, as cybersecurity programs have evolved, they have become more risk-based. Most industry
professionals understand compliance is just one of many key security objectives. They also know that
satisfying a compliance framework, while essential, does little to ensure the security of the organization.
In 2020, there are six primary reasons why organizations continue to invest in penetration testing:
P R OT E C T T H E O R G A N I Z AT I O N A N D I T S A S S E T S
Cyber attacks pose a serious — even existential — threat, and any digital asset is a potential target.
Penetration testing is used to identify vulnerabilities in websites, applications, and other digital
systems before they can be exploited by an attacker.
P R OT E C T C U S TO M E R DATA
Customer data is among the most important assets an organization has. Its possession is heavily
regulated. Any breach of customer data is potentially devastating, as it can lead to heavy fines from
industry regulators — not to mention a loss of customer trust. Penetration testing is used to find and
close vulnerabilities that could otherwise be used to gain unauthorized access to customer data.
S AV E T H E O R G A N I Z AT I O N M O N E Y
Cybersecurity is no longer considered a cost center by cyber mature organizations, and for some,
it has a clearly measurable ROI. Security testing expenses can be tied directly to a reduction in the
cost of incident response, remediation, and regulatory fines.
S AT I S F Y S TA K E H O L D E R R E Q U I R E M E N T S
Customers, suppliers, shareholders, and other stakeholders have a huge influence on the decisions
an organization makes. As concepts like supply chain risk have become more widely understood,
key stakeholders have increasingly demanded close attention to cyber risk management.
Penetration testing plays a crucial role in this area.
P R E S E R V E T H E O R G A N I Z AT I O N ’ S I M AG E A N D R E P U TAT I O N
Cyber incidents can fundamentally harm an organization’s ability to operate by undermining
customer trust in its products, services, and brands. A major motivation for investment in penetration
testing is to preserve customer trust by avoiding high-profile incidents.
2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 5
W H Y T R A D I T I O N A L P E N E T R AT I O N T E S T S
AREN’T FIT FOR PURPOSE
In November 2018, a Bugcrowd survey of 200 cybersecurity leaders1 found 56% were dissatisfied with
their current penetration tests. Since then, the traditional penetration testing model has only become less
effective as a tool for promoting security and managing cyber risk.
To be clear, there is a huge distinction between penetration testers — the experts who use their skills to
identify security vulnerabilities — and the model through which they are deployed.
Penetration testers are an incredible resource. Bugcrowd’s ‘Crowd’ of security experts and ethical
hackers includes thousands of penetration testers. If they could, every organization would have dozens of
penetration testers working full-time to identify vulnerabilities in its digital assets. However, this approach is
cost-prohibitive and logistically impossible. There simply aren’t enough penetration testers in the industry.
As important as penetration testers are, however, the traditional penetration testing service model no
longer meets the needs of modern organizations. Here’s why:
S C H E D U L I N G D E L AY S
Organizations are frequently forced to accept long wait times (up to months) for each testing period.
As penetration testing providers seek to reduce time on the ‘bench’ for salaried employees, getting
resources where and when needed is a perpetual challenge.
I N C O M PAT I B L E I N C E N T I V E S
The provider’s need to reduce overhead can result in assignment of those not suited for the
engagement at hand. Unfortunately the only thing protecting customers from the caveat emptor
nature of this model is the provider’s desire to win the renewal next year.
S P E E D O F R E S U LT S
With a standard penetration test, the customer doesn’t receive results until the engagement is
concluded, often 14-24 days after testing begins. This leaves tested assets vulnerable for an
unnecessarily long time.
6 | 2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G
CHECKLIST FOCUSED
Most penetration tests are checklist-based, with minimal time or incentive for testers to use their
initiative or ‘dig deeper’ to find complex vulnerabilities.
P O I N T- I N -T I M E T E ST I N G
Most digital assets are penetration tested a maximum of 1-2 times per year. With modern, agile
development lifecycles, new codebase versions are released much more frequently. While an asset
may be secure immediately following a test, new code releases could leave it vulnerable to attack
until the next scheduled test.
LACK OF INCENTIVE
Traditional penetration testing providers operate a ‘pay for time’ business model, where customers
pay for a certain number of hours, and the assigned tester is only required to finish the methodology
in that time. Number and severity of vulnerabilities surfaced during this time is irrelevant to the
tester’s final pay.
L AC K O F S D LC I N T E G R AT I O N
Traditional penetration tests aren’t constructed in a way that actively integrates security and
development teams. Developers must manually migrate vulnerabilities to their preferred workspace
(e.g., JIRA or ServiceNow) before ‘sifting through’ a long report lacking context, priority, and
guidance on how to safely resolve.
P O O R R E S U LT S
A typical penetration test finds just eight high-value, unknown vulnerabilities on average. These
valid findings are interspersed with false positives and no-risk issues, making them hard to identify
and resolve. Worse, many genuine high-risk vulnerabilities are simply not identified.
Due to poor results, high cost, and time delays, traditional penetration testing services are not a cost
effective security control. Worse, because skill fit for a project is likely sub-optimal and testers aren’t
incentivized to ‘go deep,’ it’s likely that genuine, high-risk vulnerabilities will be missed.
Given this, the traditional penetration testing model is simply ineffective for Cyber Risk Management.
2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 7
P E N E T R AT I O N T E S T I N G : W H AT A R E T H E O P T I O N S ?
While security testing has become synonymous with traditional penetration testing services, there are
actually four primary testing options.
T R A D I T I O N A L P E N E T R AT I O N T E S T I N G
Despite the issues raised in the last section, many organizations still rely on traditional penetration testing
services. The ‘traditional’ model consists of one or two testers working against a set methodology for a
defined period, usually anywhere from three days to two weeks. This format is a mainstay of the security
industry, and at this point, executives and business leaders are pre-sold on the need for it.
PROS CONS
• Established budget line item • Delays to scheduling and results
• A known quantity • Inflexible with questionable skill fit
• Best for targets that require physical • Not optimized to incentivize true
presence to access/test risk reduction
C R O W D S O U R C E D S E C U R I T Y P E N E T R AT I O N T E S T I N G
The crowdsourced security penetration test is a comparatively new method of testing. Crowdsourced
options utilize a large pool of pay-per-project testers that work remotely. Often combined with an
additionally incentivized ‘pay for results’ approach to billing, crowdsourced testing is becoming the go-to
choice for security-conscious organizations.
PROS CONS
• Rapid setup and time to value • Not optimized for highly sensitive or
physical targets too big to ship
• Real-time results and SDLC integration
• ‘Bounty’ approach may not fit buying cycles
• Option to ‘pay for results’ instead of time
• New business case may be required
INTERNAL SECURITY TESTING
While often not feasible for smaller organizations, some enterprises prefer to build and maintain in-house
teams of security testers. This approach allows the organization to set its own testing schedule, and may
reduce barriers in some areas, e.g., provision of credentials.
PROS CONS
• Best for extremely sensitive work (Secret, • Labor-intensive to set up and maintain
NOFORN)
• Impossible to retain all possible testing skills
• Tests can be run as frequently as needed
• Hard to acquire new skills when needed
• Little marginal cost to testing
PROS CONS
• Includes the best aspects of each method • Includes the worst aspects of each method
• Potential for thorough security coverage • Complex to arrange and maintain
• Testing depth is as-needed for each project • (Potentially) extremely high-cost
2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 9
T H E 2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G
In March 2020, we surveyed 129 cybersecurity engineers, managers, and CISOs to find out how they
conduct their penetration testing. All of our respondents had influence over their organization’s security
testing budget, methodology, and scope. Here’s what we learned:
T R A D I T I O N A L P E N E T R AT I O N T E S T I N G S E R V I C E S A R E S T I L L
# 1 … J U S T.
In the past, traditional penetration testing was a dominant force
in security. However, recently, other approaches have gained
popularity. Our survey shows that in 2020, across all industries and
organization sizes, traditional penetration testing services account
for just 35% of security testing.
L A R G E R O R G A N I Z AT I O N S A R E M O V I N G AWAY F R O M
T R A D I T I O N A L P E N E T R AT I O N T E S T I N G S E R V I C E S .
While other options are catching up, traditional penetration testing is the most common testing method
among small organizations with under 1,000 employees. For larger organizations — where increased
budgets open up more options — things are less clear.
Traditional penetration testing and crowdsourced testing are both utilized by just under a third of
organizations with 1-10k employees. Considering that crowdsourced testing is a far more recent option,
this highlights a rapid movement away from traditional penetration testing services.
1 0 | 2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G
At the enterprise level (10K + employees) the percentage of organizations relying on traditional
penetration testing services is barely half the rate we see in small organizations (21% vs. 39%). It’s also
dead equal with the percentage of enterprises using crowdsourced testing. Meanwhile, more than half
(57%) of enterprises rely primarily on internal security testing.
C R OW D S O U R C E D T E ST I N G F I N D S M O R E , H I G H E R -VA LU E
VULNERABILITIES.
When it comes to results, crowdsourced testing is the clear winner. 76% of crowdsourced testers received
at least 10 vulnerabilities per two-week test, compared to 57% of traditional penetration testing services.
The quality of results was also higher. Only a small fraction (13%) of crowdsourced testers received less
than 5% high-value vulnerabilities, while traditional penetration testing services were twice as likely to
deliver a poor result. Meanwhile, crowdsourced testing was 60% more likely than traditional penetration
testing services to deliver a large proportion (26%+) of high-value vulnerabilities.
Internal testing programs performed extremely poorly on both the quality and quantity of results, despite
their popularity with enterprises.
2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 1 1
T R A D I T I O N A L P E N E T R AT I O N T E S T I N G I S FAV O R E D BY
INFREQUENT TESTERS.
66% of organizations that use traditional penetration testing services test very infrequently — once per year
or less. By contrast, over half (52%) of organizations that use crowdsourced testing test at least quarterly.
Organizations that test internally are the most frequent testers, with 60% testing at least quarterly.
C O S T S A R E C O M PA R A B L E , B U T R O I I S N ’ T.
1 2 | 2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G
While the cost of maintaining an internal testing capability varies, there’s no question that it falls beyond
what most organizations can afford. And, given its poor results, the ROI is questionable at best.
W H AT C A N W E L E A R N F R O M T H I S ?
2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 1 3
W H AT ’ S N E X T F O R S E C U R I T Y T E S T I N G ?
Traditional penetration tests don’t meet the needs of modern organizations. A different solution is needed.
Crowdsourced testing approaches such as bug bounty programs have addressed many of the
shortcomings of traditional penetration tests. By operating on a pay-for-findings model, these programs
harness the power of the global hacking community to provide on-demand access to the expertise
needed for each engagement.
However, bug bounty programs haven’t fully replaced the need for standardized testing. Compliance is
still a crucial part of security, and most frameworks demand that testing follows a recognized methodology.
T H E N E X T- G E N E R AT I O N P E N E T R AT I O N T E S T
While many organizations share a need to achieve compliance milestones, not all have the same testing
requirements or capacity. Some seek continuous coverage, to match increasingly rapid development
cycles. Others need shorter testing windows throughout the year, as dictated by engineering workflows
or budgetary and procurement cycles. Equally, an organization’s appetite for tester incentivization may be
shaped by its bandwidth to process more vulnerabilities, as well as flexibility in maintaining an elastic pool
of monetary rewards.
To address these varied needs, Bugcrowd has launched the next generation of penetration testing. One
that taps into the diverse expertise of the global hacking community, while providing methodology-based
coverage and essential compliance reporting. And vitally, one where the customer chooses the terms.
C R O W D - P O W E R E D P E N E T R AT I O N T E S T I N G
‘ N E X T- G E N E R AT I O N ’ ‘CLASSIC’
P E N E T R AT I O N T E S T P E N E T R AT I O N T E S T
• Continuous coverage and on-demand • On-demand methodology-driven
methodology-driven testing, with testing over a defined period based
re-testing included on project scope
• Testers incentivized by reward for • Options for re-testing and expedited
valid vulnerabilities reporting
• Options for Premium SLAs and • Cost: Per-day, no incentive pool
Coverage Analysis
• Cost: platform + incentive pool
BOTH
• QSA-ASSESSED COMPLIANCE REPORT: Meet PCI-DSS, NIST 800-53 rev4, ISO 27001 and more
• SET UP IN <72HRS ON AVERAGE: Avoid lengthy scheduling delays and receive results faster
• STREAMING RESULTS: Receive vulnerabilities upon discovery and validation
• SDLC INTEGRATIONS: Push vulnerabilities to the places your developers live like GitHub and
ServiceNow
• REMEDIATION ADVICE: Help development fix quickly with prescriptive instructions based on
vulnerability type
• CROWDMATCHTM: Draw from the largest pool of talent and ensure skills and experience match
project needs.
• ON-DEMAND IN-PLATFORM REPORTING: Monitor vulnerability status and program activity
• FULLY MANAGED: Bugcrowd handles pentester matching, activation, and remuneration, as well
as vulnerability triage and prioritization
2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 1 5
HARNESSING THE CROWD
Harnessing the power of the global hacking community requires structure, process, and deep experience
in human-to-human interaction. All Bugcrowd’s crowd-powered penetration tests utilize the Crowdcontrol™
platform which includes dedicated program management. This combination of technology-enabled
expertise enables us to provide:
Full, real-time visibility into team activity, program outcomes, and costs.
Combined, these factors enable crowd-powered penetration tests to identify on average 7X more high-
priority vulnerabilities than traditional penetration tests.
1 6 | 2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G
THE END OF AN ERA
2 0 2 0 S TAT E O F P E N E T R AT I O N T E S T I N G | 1 7
K E Y TA K E AWAY S
S E C U R I T Y T E S T I N G I S N O L O N G E R P U R E LY F O R C O M P L I A N C E .
Modern organizations have to balance compliance with other needs, including customer and
stakeholder requirements, financial concerns, and cyber risk management.
O R G A N I Z AT I O N S H AV E TA K E N TO M I X I N G M E T H O D S F O R
P E N E T R AT I O N T E S T I N G .
To fill the gaps left by traditional testing services, modern organizations have begun incorporating
other methods where appropriate, for example, crowdsourced testing, internal testing, and hybrid
testing programs.
T R A D I T I O N A L P E N E T R AT I O N T E S T I N G S E R V I C E S A R E
L O S I N G P O P U L A R I T Y.
Traditional services are just barely holding the top spot, while organizations are increasingly
incorporating or switching to crowdsourced methods.
C R O W D S O U R C E D P E N E T R AT I O N T E S T I N G I S G A I N I N G
T R AC T I O N W I T H O R G A N I Z AT I O N S O F A L L S I Z E S .
Crowdsourced programs now account for between 20 - 30% of all security testing, depending on
organization size.