Mastering Puppet

Table of Contents

Mastering Puppet

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Dealing with Load/Scale

Divide and conquer

Puppet with passenger

Splitting up the workload

Certificate signing

Reporting

Storeconfigs

Catalog compilation

Keeping the code consistent

Rsync

NFS

Clustered filesystem

Git

One more split

One last split or maybe a few more

Conquer by dividing

Creating an rpm

Creating the YUM repository

Summary

2. Organizing Your Nodes and Data

Getting started

Organizing the nodes with ENC

A simple example

Hostname strategy

Modified ENC using hostname strategy

LDAP backend

OpenLDAP configuration

Hiera

Configuring hiera

Using hiera_include

Summary

3. Git and Environments

Environments

Environments and hiera

Multiple hierarchies

Single hierarchy for all environments

Dynamic environments

Git

Why Git?

A simple Git workflow

Git Hooks

Using post-receive to set up environments

Puppet-sync

Playing nice with other developers

Not playing nice with others

Git for everyone

Summary

4. Public Modules

Getting modules

Using GitHub for public modules

Modules from the Forge

Using librarian

Using r10k

Using modules

concat

inifile

firewall

lvm

stdlib

Summary

5. Custom Facts and Modules

Module manifest files

Module files and templates

Naming a module

Creating modules with a Puppet module

Comments in modules

Multiple definitions

Custom facts

Creating custom facts

Creating a custom fact for use in hiera

Summary

6. Custom Types

Parameterized classes

Defined types

Types and providers

Creating a new type

Summary

7. Reporting and Orchestration

Turning on reporting

Syslog

Store

IRC

Foreman

Installing Foreman

Attaching Foreman to Puppet

Using Foreman

Puppet Dashboard

Using passenger with Dashboard

Linking Dashboard to Puppet

Processing reports

mcollective

Installing activemq

Configuring nodes to use activemq

Connecting a client to activemq

Using mcollective

Summary

8. Exported Resources

Configuring puppetdb – using the forge module

Manually installing puppetdb

Installing Puppet and puppetdb

Installing and configuring PostgreSQL

Configuring puppetdb to use PostgreSQL

Configuring Puppet to use puppetdb

Exported resource concepts

Declaring exported resources

Collecting exported resources

Simple example: a host entry

Resource tags

Exported SSH keys

sshkey collection for laptops

Putting it all together

Summary

9. Roles and Profiles

Design pattern

Creating an example CDN role

Creating a sub-CDN role

Dealing with exceptions

Summary

10. Troubleshooting

Connectivity issues

Catalog failures

Full trace of a catalog compile

The classes.txt file

Debugging

Personal and bugfix branches

Echo statements

Scope

Profiling and summarizing

Summary

Index

Mastering Puppet

Mastering Puppet

Copyright © 2014 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: July 2014

Production reference: 1090714

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78398-218-9

www.packtpub.com

Cover image by Gagandeep Sharma (<[email protected]>)

Credits

Author

Thomas Uphill

Reviewers

Ugo Bellavance

C. N. A. Corrêa

Jeroen Hooyberghs

Johan De Wit

Commissioning Editor

Edward Gordon

Acquisition Editor

Meeta Rajani

Content Development Editor

Sharvari Tawde

Technical Editors

Veena Pagare

Anand Singh

Copy Editors

Sarang Chari

Mradula Hegde

Project Coordinator

Danuta Jones

Proofreaders

Faye Coulman

Maria Gould

Indexers

Mariammal Chettiyar

Tejal Soni

Priya Subramani

Graphics

Sheetal Aute

Ronak Dhruv

Production Coordinator

Shantanu Zagade

Cover Work

Shantanu Zagade

About the Author

Thomas Uphill is an RHCA who has been using Puppet since version 0.24. He has been a system administrator for nearly 20 years, more than 10 of which have been with Red Hat Linux and its derivatives. He has presented tutorials on Puppet at LOPSA-East and has spoken at PuppetConf 2013. He enjoys teaching others how to use Puppet to automate as much system administration tasks as possible. When he's not at the Seattle Puppet Meetup, you can find him at http://ramblings.narrabilis.com.

I am very thankful to my friend and colleague Joško Plazonić for introducing me to Puppet and getting me started on this journey. I would like to thank my wife Priya Fernandes for putting up with the long nights and weekends it took to finish this book. Thanks to Nate Tade for his encouragement while I worked on this book, the rest of my team for trying my crazy ideas, and Shawn Foley for a few not-so-crazy ideas. Thanks to Theresa, David, and Ben for their support.

About the Reviewers

Ugo Bellavance has done most of his studies in e-commerce. He started using Linux from RedHat 5.2, got Linux training from Savoir-faire Linux at age 20, and got his RHCE on RHEL 6 in 2011. He's been a consultant in the past, but he's now an employee for a provincial government agency for which he manages the IT infrastructure (servers, workstations, network, security, virtualization, SAN/NAS, and PBX). He's a big fan of open source software and its underlying philosophy. He has worked with Debian, Ubuntu, and SUSE, but what he knows best is RHEL-based distributions. He's known for his contributions to the MailScanner project (he has been a technical reviewer for MailScanner User Guide and Training Manual, Julian Field), but he has also given time to different open source projects such as Mondo Rescue, OTRS, SpamAssassin, pfSense, and a few others. He's been a technical reviewer for Centos 6 Linux Server Cookbook, Jonathan Hobson, Packt Publishing and Puppet 3 Beginner's Guide, John Arundel, Packt Publishing.

I thank my lover, Lysanne, who accepted to allow me some free time slots for this review even with two dynamic children to take care of. The presence of these three human beings in my life is simply invaluable.

I must also thank my friend Sébastien, whose generosity is only matched by his knowledge and kindness. I would never have reached this high in my career if it wasn't for him.

C. N. A. Corrêa (@cnacorrea) is an IT operations manager and consultant. He is also a Puppet enthusiast and an old-school Linux hacker. He has a master's degree in Systems Virtualization and holds the CISSP and RHCE certifications. Backed by a 15-year career on systems administration, Carlos leads IT operations teams for companies in Brazil, Africa, and the USA. He is also a part-time professor for graduate and undergraduate courses in Brazil. Carlos co-authored several research papers on network virtualization and OpenFlow, presented on peer-reviewed IEEE and ACM conferences worldwide.

I thank God for all the opportunities of hard work and all the lovely people I always find on my way. I thank the sweetest of them all, my wife Nanda, for all her loving care and support that pushes me forward. I would also like to thank my parents, Nilton and Zélia, for being such a big inspiration for all the things I do.

Jeroen Hooyberghs has eight years of professional experience in many different Linux environments. Currently, he's employed as an Open Source and Linux Consultant at Open-Future in Belgium. Since the past year, a lot of his time has been going into implementing and maintaining Puppet installations for clients.

I would like to thank my two girls, Eveline and Tess, for understanding that a passion for open source requires evenings and weekends spent on it.

Johan De Wit was an early Linux user, and he still remembers the day he built a 0.9x Linux kernel on his brand new 486 computer that took an entire night. His love for the UNIX operating systems existed before Linux was announced. It is not surprising that he started a career as a UNIX system administrator.

He doesn't remember precisely when he started working with open source software, but since 2009, he is working as an Open Source Consultant at Open-Future, where he got the opportunity to work with Puppet. Right now, Puppet has become Johan's biggest interest. He also loves to teach Puppet as one of the few official Puppet trainers in Belgium.

Johan started the Belgian Puppet User Group a year ago, where he tries to bring some Puppeteers together having great and interesting meetups. When he takes time writing some Puppet-related blogs, he mostly does that at http://puppet-be.github.io/, the BPUG website. Also, from time to time, he tries to spread some hopefully wise Puppet words by presenting talks at Puppet camps across in Europe.

Besides having fun at work, he spends a lot of his free time with his two lovely kids, his two Belgian draft horses, and if time and the weather permits, he likes to (re)build and drive his old-school chopper.

www.PacktPub.com

Support files, eBooks, discount offers, and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print and bookmark content

On demand and accessible via web browser

Free access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.

Preface

Every project changes when you scale it out. Puppet is no different. Working on a small number of nodes with a small team of developers is a completely different task than working with thousands of nodes with a large group of developers.

Mastering Puppet deals with the issues faced with larger deployments, such as scaling and duplicate resource definitions. It will show you how to fit Puppet into your organization and keep everyone working. The concepts presented can be adopted to suit organizations of any size.

What this book covers

Chapter 1, Dealing with Load/Scale, deals with scaling out your Puppet infrastructure to handle a large number of nodes. Using proxying techniques, a sample deployment is presented.

Chapter 2, Organizing Your Nodes and Data, is where we examine different methods of applying modules to nodes. In addition to ENCs (external node classifiers), we use hiera and hiera_include to apply modules to nodes.

Chapter 3, Git and Environments, shows you how to use Git hooks to deploy your code to your Puppet masters and enforce access control for your modules.

Chapter 4, Public Modules, presents several supported modules from the Puppet Forge and has real-world example use cases.

Chapter 5, Custom Facts and Modules, is all about extending facter with custom facts and rolling your own modules to solve problems.

Chapter 6, Custom Types, covers how to implement defined types and create your own custom types where appropriate.

Chapter 7, Reporting and Orchestration, says that without reporting you'll never know when everything is broken. We explore two popular options for reporting, Foreman and Puppet Dashboard. We then configure and use the marionette collective (mcollective or mco) to perform orchestration tasks.

Chapter 8, Exported Resources, is an advanced topic where we have resource definitions on one node applying to another node. We start by configuring puppetdb and more onto real-world exported resources examples with Forge modules.

Chapter 9, Roles and Profiles, is a popular design paradigm used by many large installations. We show how this design can be implemented using all of the knowledge from the previous chapters.

Chapter 10, Troubleshooting, is a necessity. Things will always break, and we will always need to fix them. This chapter shows some common techniques for troubleshooting.

What you need for this book

All the examples in this book were written and tested using an Enterprise Linux 6.5 derived installation such as CentOS 6.5, Scientific Linux 6.5, or Springdale Linux 6.5. Additional repositories used were EPEL (Extra Packages for Enterprise Linux), the Software Collections (SCL) Repository, the Foreman repository, and Puppet Labs repository. The version of Puppet used was the latest 3.4 series at the time of writing.

Who this book is for

This book is for system administrators and Puppeteers writing Puppet code in an enterprise setting. Puppet masters will appreciate the scaling and troubleshooting chapters and Puppet implementers will find useful tips in the customization chapters.

Conventions

In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.

Puppet code words in text, module names, folder names, filenames, dummy URLs, and user input are shown as follows: The file /var/lib/puppet/classes.txt contains a list of the classes applied to the machine.

A block of code is set as follows:

class base {

  file {'one':

    path  => '/tmp/one',

    ensure => 'directory',

  }

  file {two:

    path  => /tmp/one$one,

    ensure => 'file',

  }

}

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

service {'nginx':   require => Package['nginx'],

  ensure  => true,

  enable  => true,

}

Any command-line input or output is written as follows:

$ mco ping worker1.example.com                      time=86.03 ms node2.example.com                        time=96.21 ms node1.example.com                        time=97.64 ms ---- ping statistics ---- 3 replies max: 97.64 min: 86.03 avg: 93.29

New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: Then navigate to the settings section and update the trusted_puppetmaster_hosts setting.

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.

To send us general feedback, simply send an e-mail to <[email protected]>, and mention the book title via the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata is verified, your submission will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support.

Piracy

Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.

Please contact us at <[email protected]> with a link to the suspected pirated material.

We appreciate your help in protecting our authors, and our ability to bring you valuable content.

Questions

You can contact us at <[email protected]> if you are having a problem with any aspect of the book, and we will do our best to address it.

Chapter 1. Dealing with Load/Scale

A large deployment will have a large number of nodes. If you are growing your installation from scratch, you may have started with a single Puppet master running the built-in WEBrick server and moved up to a passenger installation. At a certain point in your deployment, a single Puppet master just won't cut it—the load will become too great. In my experience, this limit was around 600 nodes. Puppet agent runs begin to fail on the nodes, and catalogs fail to compile. There are two ways to deal with this problem: divide and conquer or conquer by dividing.

That is, we can either split up our Puppet master and divide the workload among several machines or we can make each of our nodes apply our code directly using Puppet agent (this is known as a masterless configuration). We'll examine each of these solutions separately.

Divide and