Blitzscaling Security - Diary of A Security Engineer
Blitzscaling Security - Diary of A Security Engineer
Shellshock
Rough start
Skeletons in the closet
Tetris game
Trial by fire
Web of attacks
Baby steps
Poking around
First layer
Ad astra
Spilled ink
The cavalry arrives
Now you see me
Git-fucking-ops
Next up
Prologue
When should a startup really invest in information security? Ask
any seasoned consultant and their eyes will gleam at the
opportunity of whipping out slides showing how the cost of fixing a
vulnerability exponentially grows over time. “Of course, you should
strive to build security right from the start,” they claim.
Alright. Allow me to elaborate then. Our startup is composed of
three people. Two programmers who call themselves the CTO and
VP of engineering, and a CEO who is actually closer to a product
manager, accountant and office manager. When she is not dialing
for dollars or thinking about customer experience, she is testing
every line of code feverishly pushed to production by her two
colleagues. The product is a mumble jumble of half-baked features,
all once hailed as the “next big breakthrough”. They don’t know if
their company will survive the next six months. Their runway cash
surely won’t. Product-market fit is still an elusive concept they only
hear about in TED talks. Every hour they don’t spend developing
basic features is another missed opportunity to crack the puzzle of
business growth before the competition. A missed opportunity to
secure a roof over their heads. A missed opportunity to cast away
the ever-looming feeling of failure.
Now, when should this startup invest in their first security
engineer?
Maybe our consultant friend will shrug away any feeling of
uneasiness and stick to their guns. Security is, after all, paramount
to a company’s success. But I want to argue a different stance that
some security experts would dismiss as pure heresy: Forget about
security. First, make it work.
However well-thought-out some online infosec preaching may
appear, the truth is that many accepted security practices will latch
your product to the ground while the competition acquires precious
market share. Can you imagine talking about dynamic code analysis
and network segmentation when the team is really just free-
throwing ideas labelled as features to see what sticks to the wall?
These are hard problems in computer security. The only people who
refer to them as “the basics” never had the exquisite pleasure of
implementing and maintaining them. Of course, it’s easy to write
tight firewall rules that only allow trusted protocols in the network,
but can you deploy them safely without breaking the product or
slowing down the dev team? How will you test new changes in a
safe and controlled way? Should every new feature that makes a
network call entail a request to update firewall rules? What’s the
impact on the feature’s delivery time? Is it really worth it given your
threat profile at this stage?
But I digress. We will get back to these particular problems later
on in this book.
So, how should our typical startup handle its security at such a
young age? I find it easier to make the analogy with performance.
You don’t buy a 3Tb-memory machine to host the alpha version of a
product. An optimist who expects virality in the first 30 days may
get a machine with 4Gb of RAM instead of 2Gb, but that’s about it.
It’s much more sensible—and cheaper—to follow well-defined
metrics and progressively increase the capacity of machines to
match the projected user activity. In the early stages of a startup,
it’s a bad investment to spend days refactoring code to save a few
milliseconds of execution. It would make more sense to just throw
money at the problem: upscale the machine’s capacity with the click
of a button on the cloud console, and then focus on growth.
The same holds true for security. The threat of product failure
dwarfs every other security risk one can imagine. It’s the only risk
that matters. Every vulnerability that does not directly contribute to
this risk is fair game to accept for now. If using generic accounts
speeds up publishing code, so be it. If figuring out AWS permissions
is too darn hard, go with default policies. They’re okay for now.
Upgrading dependencies every week is a waste of time when you
have 0 paying customers. Whatever helps you iterate faster is
probably okay so long as it does not immediately threaten the
livelihood of the product, i.e., publishing the admin panel on the
Internet with test/test credentials is taking it a tad too far.
Reid Hoffmann, founder of LinkedIn, when talking about hyper
growth in the startup world, refers to this early phase as the
“Family” phase: less than 10 employees and a few would-be
customers are trying out the product. Barely any revenue is coming
in. The biggest goal of the Family phase is to reel in those early
adopters. Understand what drives them to use the product, and
quickly morph key characteristics of the product into features most
customers actually want.
A company that succeeds in this phase slowly lifts off the ground
by the sheer force of product-market fit. Investors, once a scarce
breed, suddenly appear at every corner trying to get in on the
action. The company raises its first or second round of funding
around one core idea: scaling. They have the right formula, they’ve
proved it at a tiny scale, now it’s time to expand it to a larger
audience, double down on the core features that fit their customer
profile and hire a hundred people to make it happen. The company
enters what Reid refers to as the “Tribe” phase, and that’s when you
should start really caring about security.
In this book, we will follow the journey of the first security
engineer, Alex, hired by a scale-up to design and implement its first
real security practices. You and I will dive into Alex’s mind and
embody their intellectual experience, from the first encounter with
the company during the interview process all the way to their first
months on the job.
We will follow their day-to-day interactions with other teams, the
difficult but oh so important exercise of prioritizing vulnerabilities,
the hard limits of technology and, most importantly, the
phenomenal clash between common security wisdom and the hard
reality of the field.
Such a position is not usually for entry-level engineers who have
just started to tame their first exploits. It may happen, but it’s not
common, so our protagonist will have a background of a few years
in penetration testing, which will color some of their reflexes and
endeavors. Does that mean that it’s the only path to break into the
startup world and design secure systems? Absolutely not. I had to
settle on an experience trait that I was familiar with and that was in
continuity with my other books, so it made sense to go with this
one.
One of the main goals of this book is to give you a peek at what
it takes to implement practical security in a scale-up, so you can
extrapolate the skills required and work on them. If you have a
different background, great, you’ll get to appreciate the world
through a new lens, which will empower you in your day-to-day life.
If you have no security background, don’t worry, relax and enjoy
the adventure. We’ll take it step by step and explain everything
from first principles with links for further reading. If you have a
thriving career in penetration testing and designing secure systems,
prepare for some controversial statements that will make you wince
and hopefully rethink and challenge how you usually approach
certain issues.
This book will swing between deep technical content and high-
level overviews of both the security and the tech worlds. At first, I
tried to avoid constraining the scenario to specific vendors and their
shortcomings, but the narrative quickly acquired a bland and
generic taste, so far away from the complexities of real-world
issues. Security is easy on paper but very hard in real life.
Therefore, we will dare to name some tech solutions, such as AWS,
Kubernetes and GitLab. We will suffer their limitations and leverage
their capabilities to build a secure environment. That being said, the
principles that we will follow apply to any other technological
environment, from Active Directory to Mainframes. Don’t let the
technological specificity fool you. My goal is to convey a thought
pattern and structure to approach security, not nitpick tools and
vendors.
I am shooting for timeless.
Little talks
“If you wish to make an apple pie from scratch, you
must first invent the universe.”
Carl Sagan