DevOps for the Desperate: A Hands-On Survival Guide

DevOps for the Desperate

A Hands-on Survival Guide

Bradley Smith

DevOps for the Desperate. Copyright © 2022 by Bradley Smith.

All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher.

Second printing

27 26 25 24 23    2 3 4 5 6

ISBN-13: 978-1-7185-0248-2 (print)

ISBN-13: 978-1-7185-0249-9 (ebook)

Publisher: William Pollock

Managing Editor: Jill Franklin

Production Editor: Paula Williamson

Developmental Editor: Jill Franklin

Cover Illustration: Gina Redman

Interior Design: Octopod Studios

Technical Reviewer: Quentin Hartman

Copyeditor: Doug McNair

Compositor: Happenstance Type-O-Rama

Proofreader: Jamie Lauer

Library of Congress Cataloging-in-Publication Data

Names: Smith, Bradley (Software engineer), author.

Title: DevOps for the desperate : a hands-on survival guide / Bradley

   Smith.

Description: San Francisco : No Starch Press, [2022] | Includes

   index. |

Identifiers: LCCN 2021060922 (print) | LCCN 2021060923 (ebook) | ISBN

   9781718502482 (paperback) | ISBN 9781718502499 (ebook)

Subjects: LCSH: Computer software--Development--Management. | Software

   engineering--Management.

Classification: LCC QA76.76.D47 S567 2022 (print) | LCC QA76.76.D47

   (ebook) | DDC 005.1068--dc23/eng/20220111

LC record available at https://lccn.loc.gov/2021060922

LC ebook record available at https://lccn.loc.gov/2021060923

For customer service inquiries, please contact [email protected]. For information on distribution, bulk sales, corporate sales, or translations: [email protected]. For permission to translate this work: [email protected]. To report counterfeit copies or piracy: [email protected].

No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product and company names mentioned herein may be the trademarks of their respective owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark.

The information in this book is distributed on an As Is basis, without warranty. While every precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it.

This book is for all the engineers slogging through on call.

About the Author

Bradley Smith is a director of infrastructure and resides in Denver, Colorado. He has been an engineer for more than 20 years at many startups and businesses, large and small. He has built, trained, and been a member of numerous DevOps, SRE, and software engineering teams. A Boston native, Bradley graduated from the University of Massachusetts Lowell.

About the Technical Reviewer

Quentin Hartman has been living and breathing DevOps since before it had a name. He loves the tech, but more than that, he loves seeing how DevOps practices make software and the lives of people who build it better. Over the course of his nearly 25-year career in technology, Quentin has worked in public education, higher education, nonprofits, and private businesses with anywhere from 3 to 300,000 employees. He has managed telecom systems, datacenters, and public and private clouds. He has acted as a sysadmin, a DBA, a network engineer, an incident responder, and a leader. This broad experience has given him an especially strong foundation in DevOps, which has been his primary focus since 2012. Wherever Quentin is, he puts people before tech and is only really happy when he’s working on a social-impact mission using open source tools. Quentin lives near Denver, Colorado, with his family. He can often be found building things, cooking, and wandering in the woods. He can be reached as qhartman on many platforms, including Mastodon.social, Twitter, and LinkedIn.

Acknowledgments

When writing acknowledgments, you quickly realize how many people make publishing a book possible. This would be a very long section if I thanked everyone who contributed in some way, and since this is not a Nobel Prize acceptance speech, I will try to keep it short and sweet. If I do not mention you below, please know I appreciate your help tremendously.

First, I want to thank everyone at No Starch Press. Without you, this book would not have been possible. The guidance from my editor, Jill Franklin, and technical editors, Kyle Terrien and Quentin Hartman, has been invaluable. Thank you so much for wrangling this idea into a book. I appreciate you all.

We all need help from our friends, and this book has my friends’ fingerprints all over it. Many of you provided feedback, and I thank you all so much. In particular, I want to thank Rishi Malik, Jaden Grossman, and Jeffrey Matthias. You provided support and (more importantly) lent me your precious time. I owe you!

Finally, I want to thank my family. Countless times, I asked you to read a sentence or a paragraph and tell me what you thought of it—even though you had no idea what I was talking about. To my wife, Leilani, you have always encouraged me and made me believe I could do this. Thank you for making time in our lives so I could work on this book. To my daughters, Aiden and Akira, you are my inspiration, and you make me want to be the best person I can be. I love the three of you, always.

Introduction

Every day of their working lives, DevOps engineers immerse themselves in cloud-based trends and technologies. Meanwhile, everyone else in engineering is expected to be familiar with DevOps and keep pace with how it is evolving. The reason is simple: DevOps is an integral part of software development. However, you probably don’t have time to both do your day job and keep tabs on the ever-changing landscape of DevOps—and luckily, you don’t have to. Just gain an understanding of the foundational concepts, terms, and tactics of DevOps, and you’ll go far.

On the other hand, when it comes time to deliver code, you can’t just put your head in the sand and hope someone else will deal with it. Writing configuration files, enforcing observability, and setting up continuous integration/continuous delivery (CI/CD) pipelines have become the norm in software development. You therefore need to be well versed in code and infrastructure.

If you’re a software engineer, developer, or systems administrator, this book will teach you the concepts, commands, and techniques that will give you a solid foundation in DevOps, reliability, and modern application stacks. But be aware that this is an introduction to DevOps, not a definitive guide. I’ve chosen to keep the knowledge fire hose turned down low, and I’ll focus on the following foundational concepts:

Infrastructure as code

Configuration management

Security

Containerization and orchestration

Delivery

Monitoring and alerting

Troubleshooting

Plenty of other great books will take you on a deep dive into the concepts and culture of DevOps. I encourage you to read them and learn more. But if you just want to get started with the basics, DevOps for the Desperate has you covered.

What Is the Current State of DevOps?

Over the past few years, different trends have emerged in DevOps. There is a heavy focus on microservices, container orchestration (Kubernetes), automated code delivery (CI/CD), and observability (detailed logging, tracing, monitoring, and alerting). These topics aren’t new to the DevOps community, but they’re gaining more attention because everyone has swallowed the red pill and gone down the cloud-and-containerization rabbit hole.

Automating and testing the code to customer experience is still one of the most important parts of DevOps, and it will continue to be as late adopters play catch-up. As engineering ecosystems mature, more and more DevOps work is occurring higher up the tech stack. In other words, DevOps engineers are heavily relying on tools and processes so software engineers can self-serve shipping code. Because of this, sharing DevOps practices and techniques with feature teams is paramount to delivering standardized and predictable software.

A few more emerging trends are worth a brief mention here. The first is security. DevSecOps is becoming an essential part of the build process rather than a post-release afterthought. Another trend is the use of machine learning for data-driven decisions like alerting. Machine learning insights can be extremely useful in heuristics and will play a larger role going forward.

Who Should Read This Book?

This book is aimed at helping software engineers feel at home and thrive in a modern application stack. As such, it provides just the right amount of introductory information about DevOps tasks. This is not to say it has nothing to offer established DevOps engineers. On the contrary, it provides plenty of useful information about containerization, monitoring, and troubleshooting. If you are a DevOps engineer or software engineer in a small shop, you can even use this book to help you create your whole application stack, from local development to production.

So, if you’re a software developer looking for knowledge about DevOps, this book is for you. If you’re interested in becoming more of a generalist, this book is for you. And if I’ve paid you money to read this book—well, this book is definitely for you.

How This Book Is Organized

This book is divided into three parts, as follows:

Part I: Infrastructure as Code, Configuration Management, Security, and Administration

Part I introduces the concepts of infrastructure as code (IaC) and configuration management (CM), which are essential for building systems with a repeatable, versioned, and predictable state. We’ll also explore host-based and user-based security.

Chapter 1: Setting Up a Virtual Machine This chapter discusses the concepts of IaC and CM. It then introduces two technologies, Vagrant and Ansible, that you’ll use to create and provision an Ubuntu VM.

Chapter 2: Using Ansible to Manage Passwords, Users, and Groups This chapter looks at how to use CM for user and group creation to restrict file and directory access. It also explains how to use CM to enforce complex passwords.

Chapter 3: Using Ansible to Configure SSH This chapter shows you how to set up public key and two-factor authentication over SSH, thus making it harder for unauthorized users to gain access to your host and sensitive data.

Chapter 4: Controlling User Commands with sudo This chapter shows you how to create a security policy that delegates command access for a specific user and group. Controlling the command access that users and groups have on a host can help you avoid unnecessary exposure to attackers. At a minimum, it prevents you from having a poorly configured OS.

Chapter 5: Automating and Testing a Host-Based Firewall This chapter describes how to create and test a minimal firewall that will block all unwanted access while permitting approved traffic. By limiting port exposure, you can reduce the vulnerabilities your host and application may encounter from the outside.

Part II: Containerization and Deploying Modern Applications

Part II introduces the concepts of containerization, orchestration, and delivery. It also explores some of the components that make up a modern stack.

Chapter 6: Containerizing an Application with Docker This chapter introduces containers and containerization, and it shows how to create a sample containerized application. Having a basic understanding of containers and how to use them for local development and production is key to your ability to work with any modern application stack.

Chapter 7: Orchestrating with Kubernetes This chapter introduces container orchestration and explores how to use technologies like Kubernetes and minikube to deploy an application on a local cluster. It also serves as an example of how to set up a local development environment.

Chapter 8: Deploying Code This chapter discusses the concept of continuous integration and continuous deployment (CI/CD). It also explores some core technologies, like Skaffold, that allow you to create a pipeline on a local Kubernetes cluster. After building an effective CI/CD pipeline, you’ll have a good understanding of how to build, test, and deploy software.

Part III: Observability and Troubleshooting

Finally, Part III introduces the concepts of monitoring, alerting, and troubleshooting. It looks at metric collection and visualization for applications and hosts. It also discusses some common host and application issues, as well as tools you can use to diagnose them.

Chapter 9: Observability This chapter introduces the concept of a monitoring and alerting stack, and it explores the technologies (Prometheus, Alertmanager, and Grafana) that make up this stack. You’ll learn how to detect a system’s state and alert on it when things are out of scope.

Chapter 10: Troubleshooting Hosts The last chapter discusses common issues and errors on a host and some tools you can use to troubleshoot them. Being able to analyze issues on a host will help you in times of crisis and help you understand performance issues in your own code and applications.

What You’ll Need

In order to explore the DevOps concepts in this book, you’ll install some tooling and the free VirtualBox virtualization technology for x86 hardware that allows you to run other operating systems on your local host. Unfortunately, some of the tools needed for these tasks won’t work natively on some OSes and CPUs, such as Windows and Apple Silicon. Using Linux or an Intel-based Mac as the host machine is the most straightforward option. The following list summarizes what you can expect for each OS:

Linux

If you’re on a Linux host, all the examples and sample applications will work out of the box. Since you’ll be installing VirtualBox, you’ll want to be running a desktop version of Linux rather than a headless server.

Intel-based Mac

If you’re running an Intel-based Mac, as with Linux, all the examples and sample applications will work without any modifications. Use the Brew package manager (https://brew.sh) to install software.

Windows

If you’re on a Windows host, installing all the tools and applications in this book can be a challenge. For example, you’ll use Ansible to explore configuration management, but there’s no easy way to install Ansible on Windows. As a workaround, you can use an Ubuntu VM as your starting point. I recommend creating the VM with Hyper-V, since it’s native to Windows. You’ll need Windows 10 or 11 Pro to use Hyper-V. See the Ubuntu Wiki (https://wiki.ubuntu.com/Hyper-V) for instructions on creating an Ubuntu VM on Hyper-V.

You’ll also need to enable nested virtualization since you’ll be installing VirtualBox inside the Hyper-V Ubuntu VM. To enable this feature, enter the following command in an administrative PowerShell terminal:

Set-VMProcessor -VMName VMName -ExposeVirtualizationExtensions $true

You’ll need to run this command when the Ubuntu VM is stopped, or it will fail. Replace VMName with the name of the Ubuntu VM you just created.

After your VM is up and running, you’ll install VirtualBox using the Ubuntu version listed at https://www.virtualbox.org/wiki/Linux_Downloads. After completing that installation, you’ll be able to perform the book’s examples from within the newly created VM.

For older versions of Windows, you can use VirtualBox (yes, VirtualBox within VirtualBox) or VMware (https://www.vmware.com/products/workstation-player.html) to create the Ubuntu VM. Instructions for these options are beyond the scope of this book.

Apple Silicon

If you’re using an Apple Silicon computer as your host machine, VirtualBox is not an option. Apple Silicon’s CPU is based off the ARM architecture, and VirtualBox works only on x86. Instead, you’ll need to use a virtualization technology like Parallels (https://parallels.com), VMware Fusion (https://vmware.com), or Qemu (https://www.qemu.org) to create an ARM-based virtual machine. The first two options are paid software and may provide a better user experience. Qemu is free and open source, and it requires some extra configuration steps. Visit the companion GitHub repository (https://github.com/bradleyd/devops_for_the_desperate/tree/main/apple-silicon/) for detailed instructions on how to set up a suitable lab to follow along on your Apple Silicon Mac.

To get the best experience, your host should have a minimum of 8GB of memory and at least 20GB of free disk space available; your mileage might vary if you have less. This book also makes some basic assumptions about your comfort level with Linux and the command line. You should be familiar with Bash and feel at home editing files.

Downloading and Installing VirtualBox

Download the installer from https://www.virtualbox.org/wiki/Downloads/. Choose the latest version and the correct download for your specific operating system. As mentioned previously, Windows users using Hyper-V will install VirtualBox for Ubuntu Linux. For Intel-based Macs, click the OS hosts link and download the installer. For Linux, you guessed it—click the Linux distributions link to find the download for your distribution. The VirtualBox website has excellent instructions for the different OSes at https://www.virtualbox.org/manual/.

Launch VirtualBox from where you installed it to verify that it works. If everything is okay, you should be greeted with a start screen (see Figure 1).

Figure 1: VirtualBox start screen on macOS (it will look different depending on your host OS)

If you decide to use your OS’s package manager to install VirtualBox, make sure you’ve got the latest version, as older versions might show differences from the examples in this book.

WARNING

If you are running macOS, you’ll need to allow VirtualBox extra permissions when trying to launch the virtual machine. You will be prompted to allow VirtualBox