stdout.

no

http://stdout.no/a-modern-puppet-master-from-scratch/

A modern Puppet Master from scratch

The goal is:
Puppet 3 everywhere
Config in shared Git repo
Dynamic environments powered by r10k
Support PuppetDB
Support Hiera

Install Puppet
Add the Puppet Labs repository by installing their handy package:
source /etc/lsb-release
wget https://apt.puppetlabs.com/puppetlabs-release-$DISTRIB_CODENAME.deb
dpkg -i puppetlabs-release-$DISTRIB_CODENAME.deb
rm puppetlabs-release-$DISTRIB_CODENAME.deb
Install Puppet and Puppet Master:
apt-get update
apt-get install puppet puppetmaster

Configure Puppet
Create a folder to hold our environments, and use the puppet group to manage write access to it:
mkdir /etc/puppet/environments
chgrp puppet /etc/puppet/environments
chmod 2775 /etc/puppet/environments
We will never modify the contents of this directory directly, but r10k will use it through a Git hook we set up later
on.
Now make some adjustments to /etc/puppet/puppet.conf. Here is a good starting point:
[main]
environment
confdir
logdir
vardir
ssldir
rundir
factpath
pluginsync

=
=
=
=
=
=
=
=

[agent]
environment
report
show_diff

= production
= true
= true

production
/etc/puppet
/var/log/puppet
/var/lib/puppet
$vardir/ssl
/var/run/puppet
$vardir/lib/facter
true

[master]
environment
= production
environmentpath = $confdir/environments
# Passenger
ssl_client_header
= SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
If you don't have DNS setup so that looking up "puppet" resolves to your server, then you can also work around
that here by adding server = your.server.com to the [main] section.

Configure Hiera
Hiera needs some setup as well. Create the file /etc/puppet/hiera.yaml:
--:hierarchy:
- "nodes/%{::fqdn}"
- "manufacturers/%{::manufacturer}"
- "virtual/%{::virtual}"
- common
:backends:
- yaml
:yaml:
:datadir: "/etc/puppet/environments/%{::environment}/hieradata"
In order to make debugging Hiera lookups (slightly) easier and avoid confusion later, I also like to replace
/etc/hiera.yaml (which Puppet doesn't care about) with a link to /etc/puppet/hiera.yaml:
ln -sf /etc/puppet/hiera.yaml /etc/hiera.yaml

Test Puppet
This would be a good time to restart the Puppet Master service:
/etc/init.d/puppetmaster restart
Test that the Puppet agent works as expected:
puppet agent --test
The output at this point should be along these lines:
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve
information from environment production source(s) puppet://testpm.qix.no/plugins
Info: Caching catalog for testpm.qix.no
Info: Applying configuration version '1384949455'
Info: Creating state file /var/lib/puppet/state/state.yaml
Notice: Finished catalog run in 0.03 seconds
The single error message can be ignored, because we have no config or plugins yet.
Make sure this works before continuing. Problems here are usually DNS related.

Install r10k

The excellent Adrien Thebo has created an equally excellent tool for managing dynamic Puppet environments
and using external modules efficiently, whether you find them on the Puppet Forge or keep them in your own
repositories.
More information can be found on GitHub. Install it like so:
# note: rubygems package not needed on Ubuntu 14.04
apt-get install rubygems
gem install r10k

Configure r10k
The cache directory where r10k stores copies of modules has to be created:
mkdir /var/cache/r10k
chgrp puppet /var/cache/r10k
chmod 2775 /var/cache/r10k
And finally r10k has its own config file. Create /etc/r10k.yaml with the following content:
# location for cached repos
:cachedir: '/var/cache/r10k'
# git repositories containing environments
:sources:
:base:
remote: '/srv/puppet.git'
basedir: '/etc/puppet/environments'
# purge non-existing environments found here
:purgedirs:
- '/etc/puppet/environments'

Install Git
The version of Git that comes with Ubuntu 12.04 unfortunately still suffers from this bug, which sets the wrong
mode (0755) on all new Puppet environments. This breaks sharing of repos among multiple users.
If you're on 12.04 (or you just want the latest Git), add the PPA for Ubuntu Git Maintainers team:
apt-get install python-software-properties
add-apt-repository ppa:git-core/ppa
Install the latest stable Git:
apt-get update
apt-get install git

Create Git repo

Now create a new Git repo that will be the authoritative source of Puppet configuration for your Puppet Master.
All your administrators will pull from and push to this repo, and r10k will pull from it to automatically create (and
destroy) Puppet environments.
Create this new repo under /srv/puppet.git as follows:

git init --bare --shared=group /srv/puppet.git

chgrp -R puppet /srv/puppet.git
cd /srv/puppet.git
git symbolic-ref HEAD refs/heads/production
Note that this repo now has three special properties:
1. It is bare
2. It is shared
3. The master branch is renamed to production

Fix user rights

It's time to start building our Puppet configuration in the Git repo.
You don't want to be working with your Git repo as root on a regular basis, so add your own user to the puppet
group, which we will use to restrict access to the repo:
adduser <myuser> puppet
Log out and back in again for this to take effect.
Double check your effective group membership by running this as your regular user:
id | grep puppet

Create Git hook

Keep working as your regular user account.
Create a file /srv/puppet.git/hooks/post-receive with the following content, which will trigger r10k
whenever someone pushes to the repo:
#!/bin/bash
umask 0002
while read oldrev newrev ref
do
branch=$(echo $ref | cut -d/ -f3)
echo
echo "--> Deploying ${branch}..."
echo
r10k deploy environment $branch -p
# sometimes r10k gets permissions wrong too
find /etc/puppet/environments/$branch/modules -type d -exec chmod 2775 {} \;
2> /dev/null
find /etc/puppet/environments/$branch/modules -type f -exec chmod 664 {} \; 2>
/dev/null
done
Also make sure it is executable:
chmod 0775 /srv/puppet.git/hooks/post-receive

Create first environment

Still as your regular user, go to your home directory and clone the empty repo:
cd
git clone /srv/puppet.git
cd puppet
Create a few needed directories:
mkdir -p hieradata/nodes manifests site
The modules directory will be managed by r10k, so it should not exist in the repo. Local modules (= modules
exclusive to this Puppet Master) will go in the site directory.
Tell Puppet about these directories by creating environment.conf with the following contents:
modulepath = modules:site
Now configure r10k by creating a file called Puppetfile in the root of the repo containing the following:
# Puppet Forge
mod 'puppetlabs/ntp', '3.0.0-rc1'
mod 'puppetlabs/puppetdb', '3.0.0'
mod 'puppetlabs/stdlib', '4.1.0'
mod 'puppetlabs/concat', '1.0.0'
mod 'puppetlabs/inifile', '1.0.0'
mod 'puppetlabs/postgresql', '3.2.0'
mod 'puppetlabs/firewall', '0.4.2'
# A module from your own git server
#mod 'custom',
# :git => 'git://git.mydomain.com/custom.git',
# :ref => '1.0'
The Puppetfile format was first created for use in librarian-puppet by Tim Sharpe, so that is still a good source of
documentation.
Two important points about the Puppetfile:
Unlike the puppet module command, r10k does not (as of 1.1.0) handle module dependencies. You
have to include all dependencies manually.
You can reference Git commits by replacing the tags with Git hashes. For testing you can even track a
branch by setting ref to something like master, but this is probably not a good idea in a production
environment.
We will now make use of two modules by assigning them through Hiera.
All hosts get the ntp module, so create hieradata/common.yaml with the following content:
--classes:
- ntp
ntp::servers:
- 0.pool.ntp.org

- 1.pool.ntp.org
- 2.pool.ntp.org
- 3.pool.ntp.org
This Puppet Master needs the puppetdb module, so create hieradata/nodes/$(hostname -f).yaml and
include the class there, along with the recommended basic configuration:
--classes:
- puppetdb
- puppetdb::master::config
Finally create a very simple manifests/site.pp that includes any classes we define in Hiera:
hiera_include('classes')

Commit and push

Still in the Git repo, it is time to commit and push the first version of the production environment.
Because Git doesn't track empty directories and we still haven't created any local modules, add a dummy file to
the site directory:
touch site/.keep
Now make sure we're on the right branch, add all files, commit and push:
git
git
git
git

checkout -b production
add *
commit -a -m "initital commit"
push -u origin production

The output should look something like this:

Counting objects: 11, done.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (11/11), 867 bytes | 0 bytes/s, done.
Total 11 (delta 0), reused 0 (delta 0)
remote:
remote: --> Deploying production...
remote:
To /srv/puppet.git
* [new branch]
production -> production
Branch production set up to track remote branch production from origin.
Note the bit about --> Deploying production..., which indicates that our Git hook is working.
You can also check that /etc/puppet/environments/production has been created, and that its modules
subdirectory contains the Puppet Forge modules listed in the Puppetfile.

Run Puppet
Back to root again, and run the Puppet agent:
puppet agent --test
You should get several screens of output in all grey and green showing the installation and configuration of the

NTP and PuppetDB services, including a PostgreSQL database for PuppetDB.

Check that the services are up and running:
/etc/init.d/ntp status
/etc/init.d/puppetdb status

Test PuppetDB
Run Puppet one more time to get some data in the database:
puppet agent --test
Now run the following:
puppet node status $(hostname -f)
This should give you output like:
testpm.qix.no
Currently active
Last catalog: 2013-11-20T13:22:05.036Z
Last facts: 2013-11-20T13:22:00.437Z
Bonus tip: Try the following to see everything PuppetDB has stored about your host, in pretty-printed JSON:
puppet node find $(hostname -f) | python -mjson.tool
Your PuppetDB is now operational, and you can use exported resources to do tricks like share ssh host keys
across hosts.

Test Hiera
If you got this far then Hiera is already working, but you may want to test Hiera from the command line during
development.
Assuming you followed my advice about linking /etc/hiera.yaml to /etc/puppet/hiera.yaml then the
following command will list all classes that will be applied to the current host with the production environment:
hiera -a classes ::environment=production ::fqdn=$(hostname -f)
This should return:
["puppetdb", "puppetdb::master::config", "ntp"]

Workflow on Puppet Master

Creating new branches in Git is easy, so the workflow for further development becomes something like this:
# create new branch and make changes
git checkout -b new_feature
vim somefile
git add somefile
git commit -m "best feature ever"
# new branch == new environment
git push --set-upstream origin new_feature

# test (on a test server, right?)

puppet agent --test --noop --environment new_feature
puppet agent --test --environment new_feature
# diff and merge
git checkout production
git diff ..new_feature
git diff --name-only new_feature
git merge new_feature
# put new feature in production
git push
# delete local branch
git branch -d new_feature
# delete remote branch == delete environment
git push origin :new_feature

Workflow for modules

If you're working on a Puppet module that you want to use on several Puppet Masters (but not release publicly),
one way is to put it on a private git server and track the branch you're working on in your Puppetfile:
mod 'my_app',
:git => 'git://git.mydomain.com/my_app.git',
:ref => 'master'
This module will be updated to the latest commit in the master branch every time you push to the repo in
/srv/puppet.git on your Puppet Master. But what if you haven't made any changes to that repo?
In that case call r10k directly. This will update all modules in all environments:
r10k deploy environment -p
Or to update only the testing environment:
r10k deploy environment testing -p
The problem with running r10k like this is that it will probably mess up the permissions below
/etc/puppet/environments, creating trouble in a shared environment. Here's a workaround. Save this
script as /usr/local/bin/deploy and make it executable:
#!/bin/sh
umask 0002
r10k deploy environment $1 -p
find /etc/puppet/environments -mindepth 1 -type d -exec chmod 2775 {} \;
find /etc/puppet/environments -type f -exec chmod 0664 {} \;
Now you can run the following on your Puppet Master when you have updated modules that you're tracking by
branch:
# update all modules in all environments

deploy
# update all modules in testing environment
deploy testing
Once you're done with your module, make sure to tag it on your git server:
git tag -a 1.0 -m "finally no error messages"
git push --tags
...and update your Puppetfile to reference the module by tag rather than branch. You will thank yourself later.