Since 1994: The Original Magazine of the Linux Community

A Look at
the History
behind
systemd
MARCH 2015 | ISSUE 251 | www.linuxjournal.com

SYSTEM
ADMINISTRATION

Build Lightweight Virtual Containers

PLUS
Get a
Fully
Capable
Android
Tablet
for $20

LJ251-March2015.indd 1

Libreboot
for a Free
Software
Laptop

WATCH:

ISSUE
OVERVIEW

Using Puppets
Hiera and
Encrypting
Credentials

2/19/15 9:21 AM

LINUX JOURNAL
ARCHIVE DVD
19942014

NOW AVAILABLE
www.linuxjour nal.com/dvd
LJ251-March2015.indd 2

2/19/15 9:21 AM

Are you
dealing
with
proprietary
storage?
you tired
tieredofof
dealing
with
proprietary
storage?
zStax StorCore

from Silicon

9%24MHDC2SNQ@FD

ZFS Unified Storage

From modest data storage needs to a multi-tiered production storage environment, zStax StorCore

zStax StorCore 64

zStax StorCore 104

The zStax StorCore 64 utilizes the latest in

dual-processor Intel Xeon platforms and fast
SAS SSDs for caching. The zStax StorCore 64
platform is perfect for:

The zStax StorCore 104 is the flagship of the

zStax product line. With its highly available
configurations and scalable architecture, the
zStax StorCore 104 platform is ideal for:

VPDOOPHGLXPRIILFHILOHVHUYHUV
VWUHDPLQJYLGHRKRVWV
VPDOOGDWDDUFKLYHV

EDFNHQGVWRUDJHIRUYLUWXDOL]HGHQYLURQPHQWV
PLVVLRQFULWLFDOGDWDEDVHDSSOLFDWLRQV
DOZD\VDYDLODEOHDFWLYHDUFKLYHV

expert
today:866-352-1173
866-352-1173 - http://www.siliconmechanics.com/zstax
TalkTalk
withwith
an an
expert
today:

LJ251-March2015.indd 3

2/19/15 9:21 AM

CONTENTS

MARCH 2015
ISSUE 251

SYSTEM ADMINISTRATION
FEATURES
58 Using Hiera
with Puppet
Use Hiera to encrypt
sensitive data
in Puppet.
Scott Lackey

68 Managing

Services
in Linux:
Past, Present
and Future
Learn about the
history of init systems
in Linux and
understand how
these systems
evolved over time.
Jonas Gorauskas

82 Infinite

BusyBox
with systemd
Build one Linux
system within
another, using
the latest utilities
within the
systemd suite of
management tools.
Charles Fisher

4 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 4

2/19/15 9:21 AM

COLUMNS
34

22

Dave Taylors
Work the Shell
Lets Play Cards with Acey-Deucey,
Part II

38

Kyle Rankins
Hack and /
Libreboot on an X60, Part I:
the Setup

44

Shawn Powers
The Open-Source Classroom
The Teeny Tiny $20 Tablet

100

Doc Searls EOF

Resurrecting the Armadillo

24

IN EVERY ISSUE
8
10
16
32
54
105

Current_Issue.tar.gz
Letters
UPFRONT
Editors Choice
New Products
Advertisers Index

ON THE COVER
(3VVRH[[OL/PZ[VY`ILOPUKZ`Z[LTKW
)\PSK3PNO[^LPNO[=PY[\HS*VU[HPULYZW
<ZPUN7\WWL[Z/PLYHHUK,UJY`W[PUN*YLKLU[PHSZW
3PIYLIVV[MVYH-YLL:VM[^HYL3HW[VWW
7S\Z!.L[H-\SS`*HWHISL(UKYVPK;HISL[MVYW

44

LINUX JOURNAL (ISSN 1075-3583) is published monthly by Belltown Media, Inc., 2121 Sage Road, Ste. 395, Houston, TX 77056 USA. Subscription rate is $29.50/year. Subscriptions start with the next issue.

WWW.LINUXJOURNAL.COM / MARCH 2015 / 5

LJ251-March2015.indd 5

2/19/15 11:53 AM

Executive Editor
Senior Editor
Associate Editor
Art Director
Products Editor
Editor Emeritus
Technical Editor
Senior Columnist
Security Editor
Hack Editor
Virtual Editor

Jill Franklin
[email protected]
Doc Searls
[email protected]
Shawn Powers
[email protected]
Garrick Antikajian
[email protected]
James Gray
[email protected]
Don Marti
[email protected]
Michael Baxter
[email protected]
Reuven Lerner
[email protected]
Mick Bauer
[email protected]
Kyle Rankin
[email protected]
Bill Childers
[email protected]

Contributing Editors
)BRAHIM (ADDAD s 2OBERT ,OVE s :ACK "ROWN s $AVE 0HILLIPS s -ARCO &IORETTI s ,UDOVIC -ARCOTTE
0AUL "ARRY s 0AUL -C+ENNEY s $AVE 4AYLOR s $IRK %LMENDORF s *USTIN 2YAN s !DAM -ONSEN

President

Carlie Fairchild
[email protected]

Publisher

Mark Irgang
[email protected]

Associate Publisher

John Grogan
[email protected]

Director of Digital Experience

Accountant

Katherine Druckman
[email protected]
Candy Beauchamp
[email protected]

Linux Journal is published by, and is a registered trade name of,

Belltown Media, Inc.
PO Box 980985, Houston, TX 77098 USA
Editorial Advisory Panel
Nick Baronian
Kalyana Krishna Chadalavada
"RIAN #ONNER s +EIR $AVIS
-ICHAEL %AGER s 6ICTOR 'REGORIO
$AVID ! ,ANE s 3TEVE -ARQUEZ
$AVE -C!LLISTER s 4HOMAS 1UINLAN
Chris D. Stark
Advertising
% -!),: [email protected]
URL: www.linuxjournal.com/advertising
0(/.%     EXT 
Subscriptions
% -!),: [email protected]
URL: www.linuxjournal.com/subscribe
MAIL: PO Box 980985, Houston, TX 77098 USA
LINUX is a registered trademark of Linus Torvalds.

LJ251-March2015.indd 6

2/19/15 9:21 AM

LJ251-March2015.indd 7

2/19/15 9:21 AM

Current_Issue.tar.gz

Putting Out
Fires and
Designing FireProof Buildings

SHAWN POWERS

ystem administration is a very

general term. Its our job to
fix problems, repair systems
and remind people to try power
cycling their troubled desktops.
We are also responsible for
creating systems that dont develop
problems, need fewer repairs and
run without being power cycled. In
an ideal world, system administrators
would work themselves out of a
job in short order. Thankfully (or
unfortunately?), thats not how it
goes. We always have problems
to fix, and theres always a better
way to do what were doing. Thus,
system administration is a vibrant

VIDEO:

Shawn Powers runs

through the latest issue.

and ever-changing field. This

month, we learn how to be better
at our jobs, even if the measure of
success is constantly fluctuating.
Dave Taylor starts off this issue with
a continuation of his script-based
card game. Designing games with
Dave is a great way to become better
shell scripters, and so in a very real
sense, we can justify playing games
at work. Kyle Rankin follows Dave
with a nerdier sort of game: trying
to replace the proprietary BIOS on
a ThinkPad with Libreboot. Coreboot
is an open-source BIOS replacement,
and Libreboot goes a step further
by stripping out all the proprietary
code. If you think having a free
BIOS with built-in GRUB sounds
interesting, youll want to check out
Kyles column this month.

8 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 8

2/19/15 9:21 AM

CURRENT_ISSUE.TAR.GZ

My personal contribution to the

System Administration issue is
something I find to be more useful
than I ever expected. Android tablets
are convenient for things like Wi-Fi
sniffing, but they are often unwieldy
to carry around. My solution is to
convert a cheap pre-paid cell phone
into a tiny, pocket-size tablet. If you
already have an Android phone, it
might be redundant, but for me, a
$20 tablet was too hard to pass up. In
my column, I give you all the details.
Puppet is an incredible tool for
managing the system configurations of
multiple nodes. Scott Lackey describes
a great tool we can use to store sitespecific data more efficiently (and
securely). Hiera is a key/value lookup
tool that integrates directly with
Puppet and makes a great tool even
better. If you want to have a clear
separation between your sensitive data
and the Puppet system that uses it,
or if you want to save time by reusing
common data, Hiera is a tool any
Puppet admin will want to check out.
Jonas Gorauskas gives us a history
of systemd. Whether you love the
new initialization system, or think
its a terrible implementation of a
horrible idea, systemd is here to
stayat least for a while. If youve
ever been curious how we got
FROM SIMPLE INIT SCRIPTS TO 3YS6 AND
beyond, youll want to read Jonas

article. Once you understand systemd,

Charles Fisher follows up with a great
tutorial on using the new init system
to create powerful and lightweight
virtual containers utilizing systemd
FOR INITIALIZATION &OR STUBBORN 3YS6
lovers like myself, its great to read
some information on the advantages
systemd might offer.
Doc Searls closes out our issue with
a new look at the 15-year-old Cluetrain
Manifesto. If youre a fan of the Locke,
Levine, Weinberger and Searls project,
youll want to read whats happening
with New Clues today.
If it werent for the modern
technological world we live in, system
administration wouldnt even exist!
Thankfully (or again, unfortunately?),
our world is getting more and more
technological every day. The need
for system administrators and their
tools are more in demand than ever
before, and this issue of Linux Journal
was written to educate, inform and
even entertain those of us in the
digital trenches. We hope you enjoy
this issue as much as we enjoyed
putting it together! Q
Shawn Powers is the Associate Editor for Linux Journal .
Hes also the Gadget Guy for LinuxJournal.com, and he has
an interesting collection of vintage Garfield coffee mugs.
Dont let his silly hairdo fool you, hes a pretty ordinary guy
and can be reached via e-mail at [email protected].
Or, swing by the #linuxjournal IRC channel on Freenode.net.
WWW.LINUXJOURNAL.COM / MARCH 2015 / 9

LJ251-March2015.indd 9

2/19/15 9:21 AM

letters
Im glad youre back in the fold,
welcome home!Shawn Powers
Vagrant Simplified
2EGARDING 3HAWN 0OWERS h6AGRANT
Simplified in the January 2015
ISSUE GREAT ARTICLE ) TRIED 6AGRANT
a few months back, and I couldnt
get the light bulb to turn on. Thus,
I put it aside. Shawns article
supplied the understanding I was
missing. Many thanks.
Tim Parks
Digital Format
An interesting thing happened. I
dropped reading LJ a while back due
to hating to stare into a monitor.
But last week I finally purchased an
Amazon tablet and re-subscribed to LJ
because of the LJ app. Its now easy
TO READ ON A NICE SCREEN %VEN THOUGH
I still enjoy printed magazines, I do
respect the environment and agree
that chopping down green for this
is not good. So good choice on an
environmentally-friendly mag.
Peter K.
Thanks Peter! Paper magazines
have a dear place in my heart as
well, but I cant deny the digital
format has some advantages too.

Thats exactly what I was hoping

for! Im glad it worked, and Im glad
Vagrant is demystified for a few
more people. Thank you for the kind
words.Shawn Powers
Suggestion for Dave Taylor
! WHILE AGO ) WROTE A # PROGRAM
for downloading stock and option
information from Yahoo in Windows.
I remember it took a lot of code
to parse the information, most
particularly the option information.
Since then, I have graduated to Linux,
and I am currently running on Xubuntu.
! FRIEND PIQUED MY INTEREST IN OPTION
trading that caused me to revisit coding
a Linux version of option tracking. I have

10 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 10

2/19/15 9:21 AM

LETTERS

not yet gotten around to coding any GUI

STUFF AND ) WANTED SOMETHING QUICKLY SO
) JUST WROTE A FEW # PROGRAMS USING
Geany to run on the terminal.

0.00

I really didnt want to write hundreds

of lines of code to parse the data and
looked around for some XML parsers.
None looked easy enough for me to
use, but then I looked at the source
page, which I downloaded, and tried
grep , which led me to develop the
following few, or one, line(s) of code
that I thought you might be interested
as a source for some future articles:

Of course, with a slight change to:

wget -O /tmp/_option.html
http://finance.yahoo.com/q/op?s=SPY&date=1429228800

grep 'option_entry\|:volume' /tmp/_option.html | sed -n

's/\r//;s/[^>]*//;s/>//;s/<\/div>//;p' | sed
's/<\/strong>//;s/[^>]*>//;s/<\/a>//;s/%//' > /tmp/_option.txt

10
21
65.14

wget -q -O /tmp/_option.html
http://finance.yahoo.com/q/op?s=SPY&date=1429228800
&& sleep 1 && grep
'option_entry\|:volume' /tmp/_option.html | sed -n
's/\r//;s/[^>]*//;s/>//;s/<\/div>//;p' | sed
's/<\/strong>//;s/[^>]*>//;s/<\/a>//;s/%//'
| grep -A8 SPY141226P00230000

I could say that we can use one line

of code to parse the Yahoo finance
page for the option information!
4HIS IS # IN THAT ) USE SYSTEM TO
execute. Maybe a popen function
might be better as an alternative, but
I didnt think of it at the time.

grep -A8 SPY141226P00230000 _option.txt

These three lines extracted the

following information:
SPY141226P00230000
24.27
22.87
24.09
0.00

In summary, I thought this was pretty

cool, and you may have already done
something similar, but as I said, I
thought it might give you some ideas
for future articles. I enjoy and find
your articles educational, which are
usually one of the first I read after the
titles that catch my eye.
Roger
WWW.LINUXJOURNAL.COM / MARCH 2015 / 11

LJ251-March2015.indd 11

2/19/15 9:21 AM

LETTERS

Dave Taylor replies: Thanks for

your note and code snippet, Roger.
It is rather amazing what you can do
with sed, although when it gets that
complex, you might consider having the
script in a separate file and using the
-f FILE option to sed to retain your
sanity as you debug it. The problem
with all of these crude HTML parsers, of
course, is that if they make the slightest
tweak on the page, your codes broken.
I know; it happens to me all the time.
Response to Letter in the January
2015 Issue Regarding zbackup
Regarding Chris W ills letter in
the January 2015 issue [this letter
is from David Barton, author of
the article Ideal Backups with
ZBACKUPv IN THE .OVEMBER 
issue]: currently I use rsnapshot to
back up the zbackup stores with
hourly, daily, weekly rotations.
Because the zbackup store changes
very slightly each time, it is very
space-effective. Due to the IO load
caused by large numbers of files,
very large numbers of servers may
want to look at options that dont
REQUIRE LINKING ALL THE FILES SUCH AS
rotating thin provisioned snapshots,
"TRFS:&3 SNAPSHOTS OR ROTATING ONTO
removable storage media like tape.
I dont think the snapshots need
to be replicated, since it is a guard

against malicious file corruption

for example, an administrator
inserting random bits into the files.
Also, for readers who are interested in
using zbackup to back up very large
directory structures, there is a pre-release
of software on https://github.com/
davidbartonau/zbackup-tar that
backs up directories about 10x faster
on a non-SSD.
David Barton
Kyle Rankins Dr Hjkl on the
Command Line
Regarding Kyle Rankins article Dr
Hjkl on the Command Line in the
$ECEMBER  ISSUE OF LJ: it seems
that Mr Rankin wants to use vi
keystrokes to manipulate the shell
command line, so why is he explaining
%MACS MODE KEYSTROKES )N THE SHELL
all he needs to type is set -o vi
and use vi mode from then on. Hit
%SC TO ENTER COMMAND MODE THEN
hop to the previous word with b, next
word with w, change the current word
WITH CW AND SO ON %VEN hHJKLv ARE
active, for moving the cursor to the
previous/next letter/shell command.
The mechanism is called GNU
READLINE IT SUPPORTS BOTH %MACS
(default) and vi mode (put set
editing-mode vi in ~/.inputrc),

12 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 12

2/19/15 9:21 AM

and most command-line tools like the

SHELL OR THE -Y31, CLIENT OR THE '$"
debugger will behave accordingly
because theyre using the library.
Mike
Dr Hjkl on the Command Line, II
)N THE $ECEMBER  ISSUE SERIOUSLY
Kyle: set -o vi .
Xaveer
Dr Hjkl on the Command Line, III
)N +YLE 2ANKINS $ECEMBER 
column, he describes being comfortable

LJ251-March2015.indd 13

LETTERS

with the vi command set, as I am too.

He then goes on to describe bash
command-line editing capabilities,
using lots of Ctrl and Alt keys. If youve
ever been sucked into editor wars, Im
sure it occurred to you that those key
SEQUENCES SEEM AWFULLY %MACS LIKE
And, in fact, thats exactly what they are.
Bash starts out with its command-line
EDITING IN %MACS MODE (OWEVER BASH
also has a perfectly functional vi mode
that may seem more familiar to you. Just
do set -o vi to turn on vi mode.

2/19/15 9:21 AM

LETTERS

Now, having said that, I have to

admit that I leave my bash sessions
IN %MACS MODE ALMOST EXCLUSIVELY
The vi mode, like the vi editor, is
modal, and that modality is somewhat
non-intuitive in command-line
editing. However, for a vi fan, its
certainly worth exploring.
Tim Roberts
Kyle Rankin replies: I remember
when I first got really interested
in vi that I changed the command
line to vi mode. I realized pretty
quickly though that I didnt like
having modes on the command
line and switched it back.
In general, I try to keep my
environments set to their defaults,
so you wont find me with custom
bashrc files that set a lot of aliases
or anything like that. Its just too
much of a pain to ship custom
settings like that throughout all
my home and work systems, so
instead, I try to make the most
with the defaults I get.
Digital Format
Happy New Year to you and your
team. I had stop subscribing to this
magazine some time ago and came
back because of the digital format.
Why? Because I am a seaman
who is away for six months at a

time. W ithout the digital format,

its hard to keep reading such a
fine magazine. Seriously, I do not
understand 60% of what is written,
but if one keeps reading, surely
ones knowledge will gradually
improve. Keep up the good work.
KokYY
Awesome! Yes, please keep reading.
Then after a couple months go
back and see if any of the older
stuff makes sense. (Dont worry if
it doesnt all make sense, however;
sometimes the articles make my
head spin too!)Shawn Powers
They Said It
I just wanted to put in a good
word for the They Said It column.
4HE QUOTES ARE NOT ALWAYS
memorable (although they often
are), but they always put me in a
good frame of mind for enjoying
the rest of the issue.
Steven Janke
Thank you Steven. I enjoy looking
for good quotes every month. The
hardest part is making sure I dont
repeat any (unless theyre really
good ones!)Shawn Powers
Kyle Rankins EC2 Security Groups
In the Secure Server Deployments

14 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 14

2/19/15 9:21 AM

in Hostile Territory article in the January

2015 issue, Mr Rankin says: ... its important
to know that Security Groups are assigned
only when an instance is createdyou cant
add or remove Security Groups from an
instance after you create it.
)N A 60# WHICH HAS BEEN %#S DEFAULT FOR A
good while now, this is not true. You easily
can change the Security Group(s) associated
with an instance.
Gx
Kyle Rankin replies: Thanks for the e-mail.
It could be that Im just showing my age in
how long Ive been working with EC2, since
none of my accounts default into VPCs, and
Im still in the wild west of EC2 Classic.
That said, I do think having to think in terms
of the limitations of the classic EC2 Security
Group model helps build more robust security
since you can take less for granted.

WRITE LJ A LETTER
We love hearing from our readers. Please
send us your comments and feedback via
http://www.linuxjournal.com/contact.

PHOTO OF THE MONTH

Remember, send your Linux-related photos to
[email protected]!

At Your Service
SUBSCRIPTIONS: Linux Journal is available
in a variety of digital formats, including PDF,
.epub, .mobi and an on-line digital edition,
as well as apps for iOS and Android devices.
Renewing your subscription, changing your
e-mail address for issue delivery, paying your
invoice, viewing your account details or other
subscription inquiries can be done instantly
on-line: http://www.linuxjournal.com/subs.
E-mail us at [email protected] or reach
us via postal mail at Linux Journal, PO Box
980985, Houston, TX 77098 USA. Please
remember to include your complete name
and address when contacting us.
ACCESSING THE DIGITAL ARCHIVE:
Your monthly download notifications
will have links to the various formats
and to the digital archive. To access the
digital archive at any time, log in at
http://www.linuxjournal.com/digital.
LETTERS TO THE EDITOR: We welcome your
letters and encourage you to submit them
at http://www.linuxjournal.com/contact or
mail them to Linux Journal, PO Box 980985,
Houston, TX 77098 USA. Letters may be
edited for space and clarity.
WRITING FOR US: We always are looking
for contributed articles, tutorials and
real-world stories for the magazine.
An authors guide, a list of topics and
due dates can be found on-line:
http://www.linuxjournal.com/author.
FREE e-NEWSLETTERS: Linux Journal
editors publish newsletters on both
a weekly and monthly basis. Receive
late-breaking news, technical tips and
tricks, an inside look at upcoming issues
and links to in-depth stories featured on
http://www.linuxjournal.com. Subscribe
for free today: http://www.linuxjournal.com/
enewsletters.
ADVERTISING: Linux Journal is a great
resource for readers and advertisers alike.
Request a media kit, view our current
editorial calendar and advertising due dates,
or learn more about other advertising
and marketing opportunities by visiting
us on-line: http://ww.linuxjournal.com/
advertising. Contact us directly for further
information: [email protected] or
+1 713-344-1956 ext. 2.

WWW.LINUXJOURNAL.COM / MARCH 2015 / 15

LJ251-March2015.indd 15

2/19/15 9:21 AM

UPFRONT
NEWS + FUN

diff -u

WHATS NEW IN KERNEL DEVELOPMENT

Nicolas Dichtel and Thierry
Herbelot pointed out that the
directories in the /proc filesystem
used a linked list to identify their
files. But, this would be slow when
/proc directories started having lots
of files, which, for example, might
happen when the system needed
lots of network sockets.
Nicolas and Thierry posted
a patch to change the /proc
implementation to use multiple
linked lists instead of just one.
%ACH SUBDIRECTORY WOULD HAVE ITS
own linked list, keyed to a hash of
the directorys name. According to
their benchmarks, the patch shaved
1/5 of the time needed to chur n
through all the entries of a given
subdirectory.
Stephen Hemminger liked the
speedup, but suggested that there
already were implementations,
like the hlist macro, that might
simplify their hash table code.
Eric W. Biederman also liked
the speedup and kicked himself
for overlooking the /proc issue
when doing other scalability work.

But, he felt that the whole linked

list concept was not the right
APPROACH %SPECIALLY HE FELT THAT
/proc/net/dev/snmp6 was the real
target of Nicolas and Thierrys
patch, and if no one actually
needed the files in that directory
EXCEPT PEOPLE REQUIRING EXTREME
backward compatibility), it would
be even more efficient to do away
with them completely.
This, however, already had come
up in an earlier thread, when
David S. Miller had said that
It potentially breaks tools, its a
non-starter, sorry. So, reworking
the user interface would not be
allowed, which left the linked list
speedup that Nicolas and Thierry
proposed. But, Nicolas said hed
look into an rbtree implementation
instead of a plain linked list,
because rbtrees would potentially
scale better.
Minchan Kim noticed that
putting memory pressure on
qemu-kvm UNDER ,INUX  WOULD
cause a ker nel stack overflow and
crash the system. He dug into the

16 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 16

2/19/15 9:21 AM

code and tried to reduce his own

stack usage, but he wasnt able
to cut back enough to prevent the
crash. And in any case, he said,
trying to reduce everyones stack
usage was not very scalable. He
proposed expanding the ker nel
stack from 8K to 16K, although he
acknowledged that there possibly
were good reasons not to do this
that he wasnt aware of.
Dave Chinner remarked that
8k stacks were never large enough
to fit the Linux IO architecture

UPFRONT

ON X  BUT NOBODY OUTSIDE

filesystem and IO developers
has been willing to accept that
argument as valid, despite
regular stack overruns and
filesystems having to add
workaround after workaround
to prevent stack overruns.
He added, Were basically at
the point where we have to push
EVERY 8&3 OPERATION THAT REQUIRES
block allocation off to another
thread to get enough stack space
for normal operation, and said

LINUX JOURNAL

now available
for the iPad and
iPhone at the
App Store.

linuxjournal.com/ios
For more information about advertising opportunities within Linux Journal iPhone, iPad and
Android apps, contact John Grogan at +1-713-344-1956 x2 or [email protected].

LJ251-March2015.indd 17

2/19/15 9:21 AM

UPFRONT

XFS has always been the stack

usage canary and this issue is
BASICALLY A REPEAT OF THE K STACK
on i386 ker nel debacle.
Borislav Petkov pointed out
that if they increased the ker nel
stack from 8K to 16K, there
undoubtedly would come a time
when 16K wouldnt be enough
either. He wondered if there ever
would be a limit, or if the ker nel
stack ultimately would grow to
one megabyte and beyond.
Steven Rostedt said, If
[Minchans patch] goes in, it
should be a config option,
or perhaps selected by those
filesystems that need it. I hate
to have 16K stacks on a box that
doesnt have that much memory,
but also just uses ext2.
Meanwhile, H. Peter Anvin said,
8K additional per thread is a huge
hit. XFS has indeed always been a
canary, or trouble spot, I suspect
because it originally came from
another ker nel where this was not
an optimization target.
At around this point, Linus
Torvalds remarked that something
like Minchans fix probably would
be necessary at some point,
although the development cycle
was already at -rc7, making it
too late for that particular ker nel

version. Linus also pointed out that

there was plenty of room to reduce
stack usage in the stack trace
Minchan had posted in his original
e-mail. Linus remarked, From a
QUICK GLANCE AT THE FRAME USAGE
some of it seems to be gcc being
rather bad at stack allocation,
but lots of it is just nasty spilling
around the disgusting call-sites
with tons or arguments. A lot
of the stack slots are marked as
%sfp (which is gcc-ese for spill
frame pointer, afaik).
There was a technical discussion
about various ways to reduce stack
usage in general (and some further
consideration of ways in which
GCC might be somewhat to blame),
but with Linus willing to accept
a patch to implement a larger
stack, it seems like something
along the lines of Minchans patch
will soon be part of the ker nel.
At one point, Linus summed up
his position on the issue, saying,
Minchans call trace and this
thread has actually convinced me
that yes, we really do need to make
X  HAVE A K" STACK ;=
The 8kB stack has been somewhat
restrictive and painful for a while,
and Im ok with admitting that it is
just getting too damn painful.
ZACK BROWN

18 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 18

2/19/15 9:21 AM

March 1619, 2015 | Boston, MA

From strategies to essential

technologiesbuild a solid
foundation in software architecture
The OReilly Software Architecture Conference is a new event
designed to provide the in-depth professional training that
software architects and people working on software architecture
need to support the success of their businesses.

Reactive and its variants

Microservices
Continuous Delivery
Integration Architecture
Devops
Scaling
Big Data

Continuous Deployment
Architecture Fundamentals
Business Skills

Save 20%

softwarearchitecturecon.com
@oreillysacon

LJ251-March2015.indd 19

on your ticket
Use code LINUXJ
2/19/15 9:21 AM

UPFRONT

Android Candy:
Bluetooth Auto
Connect
I love my latest Android device (see this issues
Open-Source Classroom column for details), but
for some reason, it wont automatically connect
to my Bluetooth headset. When I turn on my
headset, I want it to connect to my Android
device so I can start using it right away. In order
to make it connect, I have to go into the settings
app, then Bluetooth, and then tap the device to
connect. Thankfully, theres an application that
makes life a lot easier.
Bluetooth Auto Connect is a program that runs in
the background. It doesnt constantly poll for newly
turned on Bluetooth devices, because that would
waste battery power. It has several other ways to
initiate the connection though. My favorite is the
connect when powered on option. Because I
always have to turn the phone on in order to start
my audiobook (or music), its not an inconvenience
to turn the screen on in order to connect Bluetooth.
As soon as the power button is pressed, it connects
to my headset, and by the time I open the media
player application, its ready to rock!
Sometimes its the simplest applications that are
the most useful. Bluetooth Auto Connect is one
of those. Check it out in the Google Play Store
today: https://play.google.com/store/apps/
details?id=org.myklos.btautoconnect.
SHAWN POWERS

They Said It
Do something every
day that you dont
want to do; this is
the golden rule for
acquiring the habit
of doing your duty
without pain.
Mark Twain
Its okay if you mess
up. You should give
yourself a break.
Billy Joel
Let me tell you the
secret that has led
me to my goal. My
strength lies solely
in my tenacity.
Louis Pasteur
If you limit your
choices only to what
seems possible or
reasonable, you
disconnect yourself
from what you truly
want, and all that is
left is a compromise.
Robert Fritz
The highest result
of education is
tolerance.
Helen Keller

20 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 20

2/19/15 9:21 AM

LJ251-March2015.indd 21

2/19/15 9:21 AM

UPFRONT

Non-Linux FOSS:
MenuMeters
It sounds like a back in my day story,
but I really do miss the days when

LAPTOPS HAD ,%$ ACTIVITY LIGHTS FOR HARD

drives and Wi-Fi. Sure, some still have

Menu Bar (screenshot from http://ragingmenace.com)

Customizing MenuMeters
22 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 22

2/19/15 9:21 AM

them, but for the most part, the latest

trend is to have no way of knowing if
your application is pegging the CPU at
100%, or if it just locked up.
The hardware on Apple-branded
LAPTOPS IS AMAZING %VEN IF YOU HATE
the operating system, the solid
aluminum cases are just awesome.
Like most other brands of laptops,
however, they lack any activity lights. A
perfect fix for OS X is the open-source

UPFRONT

MenuMeters application. It puts all sorts

of monitoring ability right in your menu
bar. MenuMeters supports CPU activity,
network activity and even memory
usage. With a wide range of display
options, you can customize MenuMeters
to be as informative or subtle as you like.
MenuMeters is licensed under
the GPL and is available to download
at http://www.ragingmenace.com.
SHAWN POWERS

Tighten Up SSH
SSH is a Swiss Army knife and Hogwarts
magic wand all rolled into one simple
command-line tool. As often as we use it, we
sometimes forget that even our encrypted
friend can be secured more than it is by
default. For a full list of options to turn on
and off, simply type man sshd_config to
read the man page for the configuration file.
As an example, one of the first things
I do is disable root login via SSH. If you
open /etc/ssh/sshd_config as root, search
for a line mentioning PermitRootLogin
and change it to no . If you cant find a
line with that option, just add it to the
end. It will end up looking like:
PermitRootLogin no

Plenty of other security options are

available as well. Disabling the old

SSH version 1 protocol is as simple as
changing (or adding):
Protocol 2, 1

Change it to:
Protocol 2

Then only the far more secure version

 PROTOCOL WILL BE ABLE TO CONNECT %VERY
server situation has different security
needs. Reading through the man page
might reveal some options you never even
considered before. (Note that the sshd
dmon will need to be restarted for the
changes to be applied. Or, if in doubt, just
reboot the computer.) SHAWN POWERS
WWW.LINUXJOURNAL.COM / MARCH 2015 / 23

LJ251-March2015.indd 23

2/19/15 9:21 AM

UPFRONT

Solving ODEs on Linux

Many problems in science and
engineering are modeled through
ORDINARY DIFFERENTIAL EQUATIONS /$%S
http://en.wikipedia.org/wiki/
Ordinary_differential_equation).
!N /$% IS AN EQUATION THAT CONTAINS A
function of one independent variable
and its derivatives. This means that
practically any system that changes
over time can be modeled with an
/$% FROM CELESTIAL MECHANICS TO
chemistry reaction rates to ecology
and population modeling.
"ECAUSE OF THIS UBIQUITY MANY TOOLS
have been developed through the years
TO HELP SOLVE AND ANALYZE /$%S )N
this article, I take a look at one of the
tools available on Linux: Model Builder
(http://model-builder.sourceforge.net).
The project is hosted on SourceForge,
so you always can build it from
source, but most distributions should
have a package available. On Debianbased distros, you can install
it with the command:
sudo apt-get install model-builder

It also installs several Python

modules to support the tasks it
can handle. If you do decide to
build from source, you will need to

handle these dependencies yourself.

Included with the source is a
directory of examples. You can use
them as a starting point and to gain
some ideas of what you can do with
Model Builder. Documentation is a
bit sparse, so you may need to get
your hands a little dirty to take the
most advantage of what is possible
with Model Builder.
To start Model Builder, you either
can click on its menu item in your
desktop environment or run the
command PyMB from a terminal
window. When the main window
pops up, you are presented with
a template where you can define
the problem you are analyzing
(Figure 1). The main pane, titled
$IFFERENTIAL %QUATIONS IS WHERE
you can define the set of ordinary
DIFFERENTIAL EQUATIONS THAT YOU ARE
trying to solve. The general form of
THESE EQUATIONS IS DYDT  FYT 
If your system depends on
different levels of differentiating
the dependent variable, you always
CAN REWRITE IT AS A SYSTEM OF /$%S
When you give Model Builder your
system, you need to write out
only the right-hand side of the
ABOVE EQUATION 4HIS EQUATION CAN

24 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 24

2/19/15 9:21 AM

UPFRONT

Figure 1. When Model Builder starts, you can set several parameters and the equations
you want to analyze.
contain essentially any function or
expression that NumPy understands,
since Model Builder uses Python to
do the heavy lifting.
Because Model Builder is designed
TO HANDLE SYSTEMS OF EQUATIONS
you need to define the y portion as
elements of a list. So the y variable
FOR THE FIRST EQUATION IS LABELED AS
y[0]; the y variable for the second
EQUATION IS LABELED Y;= AND SO ON
These are called the state variables.

The pane to the right of the

EQUATION WINDOW IS WHERE YOU CAN
place any parameters that you need,
one per line. They can be used in
THE EQUATION WINDOW WHERE THEY
are labeled as p[0], p[1] and so on.
If you want to use time in either the
PARAMETERS OR EQUATIONS THAT YOU
have defined, you just need to use
the t variable.
Because Python is used in the
back end, you even can use lambda
WWW.LINUXJOURNAL.COM / MARCH 2015 / 25

LJ251-March2015.indd 25

2/19/15 9:21 AM

UPFRONT

functions to define more complex

structures. You may want to take
a look at the documentation
available on the NumPy site to
see what options are available
(http://www.numpy.org).
Below these two panes is where
you define the rest of the options
for your problem. In the Initial
values box, you can enter the initial

values for each state variable at

THE TIME T 4HEY NEED TO BE
separated with a space and put in
THE ORDER OF THE EQUATIONS GIVEN
IN THE EQUATION PANE
Below the Initial values, you can
enter the start time, the end time
and the time step to use in the
solution. The critical time steps
box is usually left empty, so lets

Figure 2. Once you finish defining the problem and run the integration, a result window
pops up with a graph of the integration.
26 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 26

2/19/15 9:21 AM

UPFRONT

Figure 3. You always can get a typeset display of your equations to verify what they
should look like.
leave it alone here. The first
step box is the size of the first
step. Usually, you should leave
this as 0 to allow for automatic
determination. The minimum
and maximum step size boxes set
these variables that are used in
the variable step size algorithm.
Typically, you should leave these
as 0 as well to allow for automatic

determination. The full output

check box will print out more useful
information about the integration in
the results spreadsheet.
Once everything is entered, all
you need to do is click the Start
icon, and the integration will be
calculated. If this is a system that
you will want to work with over
time, you can click on the menu
WWW.LINUXJOURNAL.COM / MARCH 2015 / 27

LJ251-March2015.indd 27

2/19/15 9:21 AM

UPFRONT

Figure 4. You can pull up all of the results of your integration and do further analysis.
item FileASave to save the model
to a file. This file format is an
XML file, so you could edit it with
a text editor if you want. When
you are ready to do more work
with it, you can load it by clicking
on FileAOpen.
Once the calculations are done,
which may be fast for simple
problems, a results window will pop
up (Figure 2). matplotlib handles

this graph window, so you can

manipulate it just like any other
matplotlib window. This includes
panning, zooming or changing the
plot window. You also can save the
resulting plot as an image file in
one of several different formats.
Going back to the main window,
lets look at some other available
tools. Clicking on the Show
EQUATIONS ICON POPS UP A WINDOW

28 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 28

2/19/15 9:21 AM

UPFRONT

Figure 5. You can generate a power spectrum of any column of your results.
WHERE YOU CAN SEE THE EQUATIONS
typeset (Figure 3). Beside this icon
is the Results icon. Clicking on
that pops up a spreadsheet of all
of the results from your integration
&IGURE   4HE COLUMNS OF DATA
include the time, the value of
y[0] and the step sizes, among
other things. You can select a
couple columns by holding down
the Ctrl key and clicking on the

column headers. Then, click on

the plot button to plot them in a
new window. You can get a power
spectrum for any one column by
selecting one of interest and clicking
on the Spectrum icon. This pops up
two new windows, the first a power
spectrum of the column (Figure 5)
and the second a spectrogram of the
column (Figure 6).
The last tool available is a wavelet
WWW.LINUXJOURNAL.COM / MARCH 2015 / 29

LJ251-March2015.indd 29

2/19/15 9:21 AM

UPFRONT

Figure 6. You also can generate a spectrogram of your results.

transform. When you select a
column, you can apply a continuous
wavelet transform to the data.
When you are done with Model
Builder, you can save this data into
A COMMA SEPARATED VALUES #36 FILE
from the spreadsheet window. Then,
you can import it into other tools,
like R, to do even further analysis.
Now that you have seen the
options available in Model Builder,

hopefully you will consider it

WHEN LOOKING AT /$% PROBLEMS
It provides a pretty simple interface
to the tools available in Python to
SOLVE /$%S NUMERICALLY !LTHOUGH
other more powerful tools are
available, Model Builder fits into
THE NICHE OF EXPERIMENTING QUICKLY
WITH DIFFERENT EQUATIONS AND PLAYING
with ideas.
JOEY BERNARD

30 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 30

2/19/15 9:21 AM

'15HPCLinux-LinuxJnlMarchad.qxp_Layout 1 2/10/15 3:34 PM Page 1

12th Annual

C
F
Re onf ree
w
ua gi ere
w
lif str nc
w.
G
fla o ied ati e
gg O
E on
m nli nd
gm ne U
t.c - sers
Roosevelt Hotel, NYC
.
om
Madison Ave and 45th St, next to Grand Central Station
/li
nu
x

2015 HPC FOR WALL STREET

fo

rQ

CLOUD TECHNOLOGY

April 6, 2015 (Monday)

Plan to Attend:

2015 HPC for Wall Street will deliver

top-notch content and connections.
Low-cost conference at $295. save $100.
Full program, including lunch.
Dave Weber
Ken Barnes
Financial Services SVP Corp Dev, Options
Free Conference Registration for quali- Global
Segment Leader, Lenovo Information Technology
fied end users. Register online as end
user.

Bernard S Donefer
Associate Director,
Baruch College

Mike Blalock
Global Sales Director,
Intel

Cloud Technology,Big Data, Low Latency, Networks, Data Centers, APIs, Scalability, cost savings for the global financial markets.
Leading Wall Street IT directors and vendor technology experts
will speak on the program.

Paul Jameson
Managing Director,
Global Fin Services,
Cisco Systems

Dave Malik
Senior Director,
Advanced Services,
Cisco Systems

Dino Vitale
Dir, Morgan Stanley
Quality Assurance &
Production Mgmt

Harvey Stein
Head of Credit Risk
Modeling,
Bloomberg

Fadi Gebara
Sr Manager,
IBM Research

Terry Keene
CEO,
iSys

Rob Krugman
VP Digital Strategy,
Broadridge Fin Sols

Lee Fisher
VP Marketing, Redline
Trading Solutions

Jeremy Eder
Perf Engineering,
Red Hat

Matt Smith
Sol Architect,
Red Hat

David B. Weiss
Sr Analyst,
Aite

Rick Aiere
Architect Specialty,
AIG

Jeffrey Scheel
Senior Technical Staff,
IBM Linux Tech Center

Ed Turkel
Mgr WW HPC Mkting,
Hewlett-Packard

Charles Milo
Enterprise Technical
Specialist, Intel

Davor Frank
Sr Solutions Architect,
Solarflare

Phil Albinus
Editor, Traders Magazine, SourceMedia

Speakers will cover 2015 Cloud, HPC and the latest programs
to increase speed, put-through, and reduce costs.
Full conference program includes industry luncheon, general
sessions, drill down sessions, exhibits, post show receptions.
Dont have time for the full Conference? Attend the free
Show. Register in advance at: www.flaggmgmt.com/linux
2015 Sponsors

Shagun Bali
Analyst,
TABB Group

Show Hours: Mon, April 6

Conference Hours: Mon, April 6

8:00 - 4:00
8:30 - 4:50

Visit: www.flaggmgmt.com/linux

LJ251-March2015.indd 31

Show & Conference:

Flagg Management Inc
353 Lexington Avenue,
New York 10016
(212) 286 0333
fax: (212) 286 0086
[email protected]

2/19/15 9:22 AM

EDITORS' CHOICE

NmapNot Just
for Evil!
If SSH is the Swiss Army knife of the
system administration world, Nmap
is a box of dynamite. Its really
easy to misuse dynamite and blow
your foot off, but its also a very
powerful tool that can do jobs that
are impossible without it.
When most people think of
Nmap, they think of scanning
servers, looking for open ports
to attack. Through the years,
however, that same ability is
incredibly useful when youre in
charge of the server or computer
IN QUESTION 7HETHER YOURE TRYING
to figure out what kind of server
is using a specific IP address in
your network or trying to lock
down a new NAS device, scanning
networks is incredibly useful.
Figure 1 shows a network scan
OF MY 1.!0 .!3 4HE ONLY THING )
use the unit for is NFS and SMB file
sharing, but as you can tell, it has
a ton of ports wide open. W ithout
Nmap, it would be difficult to figure
out what the machine was running.
Another incredibly useful way
to use Nmap is to scan a network.

EDITORS
CHOICE

You dont even have to have root

access for that, and its as simple
as specifying the network block you
want to scan. For example, typing:
nmap 192.168.1.0/24

WILL SCAN THE ENTIRE RANGE OF 

possible IP addresses on my local
network and let me know which
are pingable, along with which
ports are open. If youve just
plugged in a new piece of
hardware, but dont know what
IP address it grabbed via DHCP,
Nmap is priceless. For example,
the above command revealed this
on my network:
Nmap scan report for
TIVO-8480001903CCDDB.brainofshawn.com (192.168.1.220)
Host is up (0.0083s latency).
Not shown: 995 filtered ports
PORT

STATE

SERVICE

80/tcp

open

http

443/tcp

open

https

2190/tcp open

tivoconnect

2191/tcp open

tvbus

9080/tcp closed glrpc

32 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 32

2/19/15 9:22 AM

Figure 1. Network Scan

This not only tells me the address

of my new T ivo unit, but it also
shows me what ports it has open.
Thanks to its reliability, usability
and borderline black hat abilities,

.MAP GETS THIS MONTHS %DITORS

Choice award. Its not a new
program, but if youre a Linux
user, you should be using it!
SHAWN POWERS
WWW.LINUXJOURNAL.COM / MARCH 2015 / 33

LJ251-March2015.indd 33

2/19/15 9:22 AM

COLUMNS

WORK THE SHELL

Lets Play
Cards with
Acey-Deucey,
Part II

DAVE TAYLOR

Dave adds the necessary code to turn a demo into a playable

game, complete with some rule variants.
In my last article, I started
developing a simple card game
called Acey-Deucey, in which you
deal two cards face up, then bet
on whether the next card is going
to be between those two in rank
value. In other words, if a 5 of
diamonds and a jack of spades
were flipped up, the bet would be
whether the next card was going
to be between a 6 and a 10.
I also dug into the math too, if
you missed it, because this is a great
game for understanding odds and
probability. Remember, any given card
has a 1 in 52 chance of appearing,
and because two cards already have
been exposed, that means any given
card actually has 1:50 odds.

For the example above, there are

four 6s, four 7s, 8s, 9s and 10s,
MEANING THAT THERE ARE 
  
  OR A  CHANCE THAT
the next card flipped up will indeed
be between the two exposed cards.
Make that 5 of diamonds an ace of
diamonds, and the odds get crazy
good: 80%. Id take those odds!
The math will factor into the
script because you actually can
have the game suggest what to
do based on the odds. The greater
the spread, the better the odds
easy enough.
I ended my last article with the
game being able to shuffle and deal
three cards: two exposed and one
hidden. Running the program with

34 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 34

2/19/15 9:22 AM

COLUMNS

WORK THE SHELL

More important, it also means that the game

can identify situations where theres no point
in betting, like when a 7 of diamonds and 8 of
clubs are dealt out.
just that code results in this:
$ sh acey-deucey.sh
I've dealt:
Ace of Hearts

helpful, particularly knowing that the

dealCards function ensures that the
two cards displayed are in order of
increasing rank, which means that this
is a darn helpful addition:

Queen of Diamonds
$

splitValue=$(( $rank2 - $rank1 ))

Theres not much to do yet,

because theres no game logic, so
lets add some.

More important, it also means

that the game can identify situations
where theres no point in betting,
like when a 7 of diamonds and 8
of clubs are dealt out. There are no
cards that can be between them.
This is added with a simple test:

Turning the Code into a

Playable Game
To start, lets initialize and deal out
the cards. With the highly mnemonic
function names already assigned, its
QUITE READABLE

if [ $splitValue -le 1 ] ; then

echo "No point in betting when you can't win!"
continue

initializeDeck

fi

shuffleDeck
dealCards
echo "Do you think the next card will be between? (y/n/q) "
read answer

This is good for a start, but as I

mentioned earlier with the math
discussion, it can be a bit more

The third card already has

been dealt within the function
dealCards , its rank calculated (as
$rank3 ) and its display name set (as
$cardname3 ). So, the test to see if the
new card is or isnt between the two
existing ranks is the next section of the
WWW.LINUXJOURNAL.COM / MARCH 2015 / 35

LJ251-March2015.indd 35

2/19/15 9:22 AM

COLUMNS

WORK THE SHELL

So you can pick three cards randomly out of the

deck, you can calculate their ranks and display
names, and you can prompt the user to guess
whether the next card will or wont be between
the two, then test to see if they were right.
CODE REQUIRED AND IT TOO IS EASY

won=$(( $won + 1 ))
else

if [ $rank3 -gt $rank1 -a $rank3 -lt $rank2 ] ; then # winner!

winner=1

echo "Bad betting strategy. You lose."

fi

else
winner=0
fi

So you can pick three cards randomly

out of the deck, you can calculate their
ranks and display names, and you can
prompt the user to guess whether the
next card will or wont be between the
two, then test to see if they were right.
Whats left? Scoring. And, thats
done with the $won variable, which is
incremented in a conditional statement
that appears immediately after the test to
see if the third card is a $winner or not:
if [ $winner -eq 1 -a "$answer" = "y" ] ; then
echo "You bet that it would be between the two and it is.
You WIN!"
won=$(( $won + 1 ))
elif [ $winner -eq 0 -a "$answer" = "n" ] ; then

Youll notice that in this

implementation of Acey-Deucey, Im
allowing the player to win if he or she
bet the card wont be between the
two, and it turns out that it isnt. This
is probably too generous, because all
you need to do is pick the more likely
scenario, which is to say any situation
where the spread is six cards or less (like
at the very beginning of this article).
3TILL ITS NOT 6EGAS OR !TLANTIC #ITY ITS
just a shell script, right? So Ill be nice.
If youd rather not offer that option,
simply change the message in the first
elif conditional code block and skip
incrementing the $won variable.
All thats left to do is to wrap the
entire code block in a big loop thatll run
FOREVER AND USE THAT STANDARD TECHNIQUE
of shell script programmers worldwide:

echo "You bet that it would not be between the two and
it isn't. You WIN!"

while [ /bin/true ] ; do

36 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 36

2/19/15 9:22 AM

COLUMNS

WORK THE SHELL

You probably wondered why /bin/true

existed in Linux, didnt you? So thats
the first line of the main code block,
and lets increment the $games
variable in the last line of the block:

The spread is 3. Do you think the next card will

be between them? (y/n/q) n
I picked: 9 of Hearts
You bet that it would not be between the two
and it isn't. You WIN!
I've dealt:

games=$(( games + 1 ))

Ace of Hearts
7 of Spades

But theres one more fragment

needed, and thats the test to see if
the user guessed that the third card
would or would not be between the
two displayed cards, or if the user
QUIT THE GAME )N THE LATTER SITUATION
its time to display some stats. Thats
easy enough, and it turns out that
you can just leave $answer alone
for the yes/no answer:

The spread is 6. Do you think the next card will

be between them? (y/n/q) y
I picked: 3 of Spades
You bet that it would be between the two
and it is. You WIN!
I've dealt:
7 of Spades
10 of Spades
The spread is 3. Do you think the next card will
be between them? (y/n/q) q
You played 2 games and won 2 times.

if [ "$answer" = "q" ] ; then

echo "You played $games games and won $won times."

exit 0
fi

)N FACT YOULL NEVER QUIT THE GAME

by falling out of the while loop, but
that makes sense since the conditional
test of /bin/true is, well, always true.
Stitch all these fragments together
and you have a game, by George!

! PERFECT SCORE .ICE ,AS 6EGAS

here I come! Q
Dave Taylor has been hacking shell scripts for more than 30
yearsreally. Hes the author of the popular Wicked Cool

Shell Scripts (and just completed a 10th anniversary revision

to the book, coming very soon from OReilly and NoStarch
Press). He can be found on Twitter as @DaveTaylor and more
generally at his tech site http://www.AskDaveTaylor.com.

$ sh acey-deucey.sh
I've dealt:
6 of Hearts
9 of Clubs

Send comments or feedback via

http://www.linuxjournal.com/contact
or to [email protected].
WWW.LINUXJOURNAL.COM / MARCH 2015 / 37

LJ251-March2015.indd 37

2/19/15 9:22 AM

COLUMNS

HACK AND /

Libreboot on
an X60, Part I:
the Setup

KYLE RANKIN

Find out what Libreboot is and why you should dust off that old
ThinkPad and give it a fresh BIOS.
Recently I wrote a review
for the Linux Journal Web site
on the Purism Librem 15 laptop
(http://www.linuxjournal.com/
content/purism-librem-15-review).
The goal of this laptop is to provide a
piece of modern hardware that can run
100% free software not just for the OS,
but also all device drivers and firmware
up to and including the BIOS. At the
time Im writing this, the last major
sticking point along those lines for the
PROJECT IS THE )NTEL -ANAGEMENT %NGINE
a proprietary piece of firmware that is
REQUIRED TO BOOT UP MODERN SYSTEMS )N
that review, I wrote the following:
It turns out its rather difficult to
have a fully free software laptop.
%VEN IF YOU CAN PICK HARDWARE THAT
can use free software drivers, theres
still that pesky BIOS. While coreboot

and libreboot are great free software

BIOS implementations, to get it on
MANY LAPTOPS REQUIRES HARDWARE
BIOS chip flashing with pomona
clipsthe kind of thing I wasnt
ready to brick a laptop to try. Like
other privacy advocates, I turned
to the old ThinkPad X60 laptop
series. While its old, underpowered
and has a low-res screen by todays
standards, the keyboard is great and
more important, you could flash its
BIOS with coreboot or libreboot from
within Linux itselfno hardware
HACKING REQUIRED 3O THATS WHAT ) DID

Although the Purism 15 laptop

seems to be a viable choice for
those who want a free software
laptop, at the time of this writing,
the crowdfunding campaign is still in
process, and even after it completes,

38 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 38

2/19/15 9:22 AM

COLUMNS

HACK AND /

Ive been able to find used ThinkPad X60 laptops

on auction sites as cheap as $30, so if you are
willing to live with some of the limitaions of
hardware that old, it is an inexpensive route to
a decent machine that runs only free software.
it will take some time until they ship.
Plus, a new laptop like that doesnt
come cheap, and many people who
may want a laptop that runs 100%
FREE SOFTWARE MAY NOT HAVE 
to spend on it. Ive been able to
find used ThinkPad X60 laptops on
auction sites as cheap as $30, so if
you are willing to live with some of
the limitations of hardware that old,
it is an inexpensive route to a decent
machine that runs only free software.
The first time I attempted to flash
an X60 with coreboot, it was one of
the more difficult things Id done with
Linux to the point that I wasnt ever
planning on writing it up in Linux
Journal. More recently, I tried again,
only this time with Libreboota
coreboot BIOS distribution that
has all of the proprietary software
removed. The process was greatly
simplified and automated to the point
where I feel relatively comfortable
recommending others try it (with a
few caveats Ill explain later).
In my next couple articles, Im

going to walk through the journey

that brought me to the X60 running
Libreboot that Im using to type this
column. In this first part, I discuss the
setup, including what Libreboot is,
what hardware it currently supports
and some of the risks around flashing
your BIOS. If I havent scared you
off by the end of this article, in
future articles, Ill cover how to
download Libreboot and verify its
integrity, how to flash the BIOS itself
in detail with sample script output
and how to modify the default GRUB
bootloader. If you cant wait until
next month, a lot of my process
is based on the excellent guide
provided at https://github.com/
bibanon/Coreboot-ThinkPads/wiki/
ThinkPad-X60.
Free as in BIOS
To understand Libreboot, it helps to
understand coreboot first. Coreboot
is an open-source BIOS replacement.
With coreboot, you can replace a
proprietary BIOS with open-source
WWW.LINUXJOURNAL.COM / MARCH 2015 / 39

LJ251-March2015.indd 39

2/19/15 9:22 AM

COLUMNS

HACK AND /

Libreboot is a custom distribution of coreboot that

removes all proprietary software from the BIOS.
software on supported hardware
with a minimal amount of proprietary
firmware included to support things
like video hardware in the BIOS or the
)NTEL -ANAGEMENT %NGINE ON NEWER
hardware. Coreboot doesnt currently
support all hardware out there,
although the list continues to grow,
and you might be surprised to know
that Chromebooks ship with coreboot
by default. To install coreboot on
much of the supported hardware, you
must use external hardware including
a connector like an 8-pin Pomona
clip to reflash the BIOS chip. Thats
pretty intense for a lot of people, but
fortunately, some hardware including
the X60, X60s, X60 tablet and T60
can be flashed completely in software.
When I first attempted to flash an
X60 with coreboot a few months ago,
the process involved disassembling the
laptop to inspect the underside of the
motherboard with a magnifying glass
so I could determine which of two BIOS
chip types I had. I used that information
to hand-patch the flashrom software
with custom code and compiled a
special version just to unlock my BIOS.
Then I downloaded, configured and
compiled a custom coreboot BIOS

image for my laptop and went through

a two-phase flash. In the end, I got it
working; however, I needed to strip
out and include the proprietary video
firmware from my proprietary BIOS
to get any video at boot timeuseful
when you want to select between hard
drive and USB boot.
Libreboot is a custom distribution of
coreboot that removes all proprietary
software from the BIOS. Instead of
proprietary BIOS boot selector, for
instance, Libreboot boots straight into
its own GRUB menu that you can use
to load your own underlying OS. In
addition, Libreboot has automated a
lot of the difficult processes around
installing coreboot and provides
custom scripts and pre-build ROMs for
its officially supported hardware.
But, why would you want a free
software BIOS? For those who
fully support the Free Software
Foundation and the principles of
free software, you dont need any
further justification. Although I have
traditionally taken a more pragmatic
approach to the free vs. open-source
software debate, Ive recently been
more motivated to seek out free
software whenever I can find it as I

40 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 40

2/19/15 9:22 AM

COLUMNS

HACK AND /

explain in my Librem 15 review:

In the past, I didnt care all that
much if I had to use a binary blob
to get a wireless card or video card
working as long as it worked, and I
definitely never cared that my BIOS
was proprietary software.
Then the Snowden leaks happened.
The sheer depth and breadth of
the loss of privacy motivated me to
step up my game in terms of overall
security and focus on privacy. In the
past it would seem rather paranoid
to think that there might be some
sort of NSA-sanctioned spyware in a
binary blob, firmware, or the BIOS.
After the Snowden leaks and the
SUBSEQUENT DISCLOSURES ABOUT THE
ANT catalog, these things stopped
seeming so far-fetched. I found
myself leaning more toward the
Stallman camp. One of the only
ways to be truly sure that you dont
have a backdoor on your system is
to be able to see the source code
for all of it from the browser plugins
to the kernel drivers all the way to
the BIOS.

Supported Hardware
Due to the fact that Libreboot avoids
any proprietary firmware in the BIOS,
its hardware support is somewhat

limited. Among other reasons, this

is due to the fact that modern Intel
HARDWARE REQUIRES THE PROPRIETARY )NTEL
-ANAGEMENT %NGINE FIRMWARE EVEN TO
boot. Although you may be able to get
Libreboot to work on other hardware,
at this point, only a few laptops are
listed on its hardware compatibility
list (http://libreboot.org/docs/hcl/
index.html#supported_list) as
officially supported:
Q Lenovo ThinkPad X60/X60s
Q Lenovo ThinkPad X60 Tablet
Q Lenovo ThinkPad T60
Q Apple MacBook1,1
Q Apple MacBook2,1

You may find one major thing in

common with all the laptops on this
list: they are old. In most cases, we
are talking about 32-bit Intel Core
$UO PROCESSORS OR  BIT #ORE  $UOS
in some cases (and the T60s CPU
CAN BE REPLACED WITH A  BIT #05
apparently). That said, the X60 is a
decent piece of hardware with a solid
keyboard and decent battery life,
even if the CPU is slow and the screen
resolution is low by todays standards.
%VEN ON THIS LIST OF SUPPORTED
WWW.LINUXJOURNAL.COM / MARCH 2015 / 41

LJ251-March2015.indd 41

2/19/15 9:22 AM

COLUMNS

HACK AND /

hardware there are some exceptions.

Although all X60s are supported,
only T60s that use Intel GPUs are
supported, and those with ATI GPUs
are not. The Libreboot hardware
compatibility page has more
information to help you figure out
whats supported and what isnt. The
page also lists recommended Wi-Fi
chipsets that are known to work well
with Libreboot and Linux in general,
AS THEY DONT REQUIRE ANY PROPRIETARY
binary blobs to function.
Risky Business
If it doesnt already go without saying,
reflashing the BIOS on your laptop with
custom software is risky! Although Ive
had success so far flashing a couple
different X60s, I did temporarily brick
one laptop when I got fancy and
tried an initial flash with one of my
own custom ROMs instead of one
provided by Libreboot. For the most
part, the process is straightforward
and automated, but as youll see in
my follow-up article that describes
each step, many of the automated
scripts call other software that output
some pretty scary warnings and
errors during the process that you are
supposed to ignore.
There are two primary ways you can
brick your laptop during the process.
First, you could have a bad flash during

the initial bootstrapping flash phase. If

that happens but you were using one
of the Libreboot-supplied ROMs, all
you should have to do is shut off the
machine, unplug the CMOS battery for
a few seconds, reconnect it and power
on your machine to get back to the
original BIOS.
If you flash during the initial
bootstrapping phase with a custom
ROM like I tried one time, lose power
during the process, attempt this on
incompatible hardware or otherwise
encounter a worst-case scenario,
you could end up with a completely
unbootable machine. Because you
cant boot back to your OS, you
cant attempt to reflash, so you are
stuck with a bricked laptop unless
you buy hardware that can flash your
BIOS chip, such as a BusPirate or a
Raspberry Pi running custom software.
That said, if you have that hardware,
wire it properly and you remembered
to back up your original BIOS first,
you should be able to restore your
laptop to normal.
Although so far Ive been successful
when Ive stuck strictly to the
directions, there is still a possibility
you will brick your laptop, so if
you are particularly attached to
your laptop and cant risk it being
OUT OF SERVICE WHILE YOU ACQUIRE
hardware flashing tools, you may

42 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 42

2/19/15 9:22 AM

COLUMNS

HACK AND /

want to reconsider going down this

road. Again, you can get used X60s
relatively cheap on-line if you shop
around, so if you are concerned,
you may want to try this first with
a sacrificial machine.

AND REQUIRES A NUMBER OF UNUSUAL

steps, most of the hard work already
has been done for you, and in the end
youll have a trusted machine without
any proprietary firmware. Q
Kyle Rankin is a Sr. Systems Administrator in the San Francisco

Conclusion
Well, if I havent scared you off yet, I
hope you check out my next column
in this series where I jump right into
step-by-step instructions on how to
flash an X60 with Libreboot. Although
THE PROCESS ISNT QUITE AS SIMPLE AS
updating a traditional proprietary BIOS

Bay Area and the author of a number of books, including The

Official Ubuntu Server Book, Knoppix Hacks and Ubuntu Hacks.

He is currently the president of the North Bay Linux Users Group.

Send comments or feedback via

http://www.linuxjournal.com/contact
or to [email protected].

LINUX JOURNAL
on your
Android device
Download the app now on
the Google Play Store

www.linuxjournal.com/android
For more information about advertising opportunities within Linux Journal iPhone, iPad and
Android apps, contact John Grogan at +1-713-344-1956 x2 or [email protected].

LJ251-March2015.indd 43

2/19/15 9:22 AM

COLUMNS

THE OPEN-SOURCE CLASSROOM

The Teeny
Tiny $20 Tablet

SHAWN POWERS

Whats better than a pocket-sized Android tablet? One for $20.

For reasons other than which do
you like better, my cell phone is an
Apple iPhone. Mainly its because the
rest of my family members use Apple
products, and I want to be able to fit
into their environment. With three
teenage daughters, its nice to run
Find my iPhone and see why theyre
running late. That leaves me with two
problems. First, theres the ridicule
and teasing from my geeky friends.
(You know who you are!) The second
problem is that I really love Android
apps for much of what I do on a
day-to-day basis. My Nexus 7 is too
unwieldy to carry around all the time,
so I really need a tiny little Android
tablet I can keep in my pocket. If the
roles were flipped, I could just buy
an iPod Touch and be done with it.
It turns out things arent so simple
in the Android world.
I was able to find the Samsung
Galaxy Player in several sizes, but
not only did they cost hundreds of
dollars, they also were discontinued

before I ever could order one. It seems

my demographic is tiny enough that
it cant support a line of devices.
Thankfully, my demographic is also
pretty nerdy, so with a little research
and hard work, I got a better solution
altogetherfor $20.
My Prepaid Non-Phone
The short version is that I bought a
prepaid Android phone and never
activated it. That version of the story
leaves out some really important
and really cool details, however.
My end result is a pocket-sized
Android device that I can use for
listening to audiobooks via Bluetooth
headset, make and receive calls,
play games, and sorta use for a GPS
device while driving. The best part is
that my uber-micro-tablet really did
cost me only $20.
If youre lazy, you can just buy
a prepaid phone off the shelf and
never activate it. Most (but not
all) will allow you to cancel the

44 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 44

2/19/15 9:22 AM

COLUMNS

THE OPEN-SOURCE CLASSROOM

If you do some research and dont mind a little

hard work, however, you can get a cheap Android
device that does everything you want without any
nag screens or limitations.
activation screen and use the device
without cell service. If you do some
research and dont mind a little hard
work, however, you can get a cheap
Android device that does everything
you want without any nag screens
or limitations. I describe my process
here, and if it sounds like something
interesting, you can do the same.
My Requirements
I wanted my new anti-iPod to be
every bit as useful as the Samsung
Galaxy Player would have been.
Heres what I expected:
Q ! SMALL BUT NICE QUALITY SCREEN

I didnt want a cheap plastic

screen that would haze over
with tiny scratches. Preferably,
I wanted Gorilla Glass.
Q Wi-Fi: this seems obvious, but

with cheap prepaid phones, you

never can tell. Its always safest
to make sure!
Q At least a dual-core CPU: I didnt

want a powerhouse, but I wanted

to be able to do things with the
device. I wanted at least 2GB
of RAM as well, but I ended up
settling for 1GB.
Q MicroSD expansion slot: this

is vitally important, because

prepaid phones generally come
with absurdly small amounts of
internal storage.
Q Bluetooth: the main purpose of

this device will be to listen to

audiobooks. For that, it needs
to work with my knockoff-brand
version of the Logitech HB-730.
Q Must be rootable: this is as

important as the MicroSD slot.

7ITH THE ADVENT OF !NDROID 
you need to have a rooted device
in order for applications to be able
to write to the SD card. I personally
think its about the dumbest
feature a new version of Android
could offer, but at least with root
access, it can be fixed.
WWW.LINUXJOURNAL.COM / MARCH 2015 / 45

LJ251-March2015.indd 45

2/19/15 9:22 AM

COLUMNS

THE OPEN-SOURCE CLASSROOM

Q Must be affordable. I already have

a phone (the iPhone), so I have to

be able to convince my wife that
its not wasteful to buy a prepaid
phone I never plan to activate.
(APPY WIFE  HAPPY LIFE

great battery life along with really

great cameras. You currently can
pick up this device for around $60,
and for the hardware you get,
Im guessing Boost is losing some
money on every sale.

My New Non-Phone
I spent a very, very long time
researching what phone to purchase.
Since what I was proposing goes
against everything the prepaid
vendors stand for, its not like I could
check their Web sites to see if the
phones were rootable or if theyd
work without activation. I considered
several models:

3. LG Realm from Boost Mobile: the

Realm is what I ended up buying
(Figure 1). The specs are a step
DOWN FROM THE 6OLT BUT ) WAS ABLE
to get the device for $19.99 from
"EST "UY AT THE END OF  IN THE
last chance to get Black Friday
Sale Prices sale. That sale probably
still is running; it seems thats how
Black Friday sales work nowadays.

 -OTOROLA -OTO ' FROM 6ERIZON

and Boost Mobile: the Moto G is
a pretty decent-looking phone,
and it has a beautiful screen.
Unfortunately, although it has
8GB of onboard storage, it lacks
a MicroSD expansion slot. Its also
around $80, which is reasonable
considering how nice of a device
it is, but without that SD slot, its
more than I was willing to pay.

Phone models change all the time.

Rather than make decisions based
on my findings from a few months
ago, I urge you to look for the latest
and greatest (or cheapest!) prepaid
options out there, and make sure
THEY MEET YOUR LIST OF REQUIREMENTS
I cant stress enough how important
it is for the phone to be rootable
though, so do at least that much
research before buying one.

 ,' 6OLT FROM "OOST -OBILE THIS

phone is probably what Id buy
if I were going to buy another
device right now. It checks all
the boxes above, and it has really

The Rooting
Sometimes the hardest part of the
PROCESS IS TO GET OUT OF THE h!#4)6!4%
-% ./7v SCREEN 7ITH ENOUGH
button pressing, I was able to put the

46 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 46

2/19/15 9:22 AM

COLUMNS

THE OPEN-SOURCE CLASSROOM

Figure 1. Oddly,
a replacement
battery for this
phone costs more
than the phone
itself. At $19.99,
you cant go
wrong!

activation screen in the background.

%VERY TIME THE PHONE BOOTED HOWEVER
it had the same annoying screen trying
to force me to activate. Therefore, the
very first thing I recommend doing is
rooting the phone.
Usually, thats as simple as visiting

http://towelroot.com from the

phones browser and installing the
tr.apk file. As long as your phone is
supported, its literally 23 clicks, and
your phone is rooted. Then install
SuperSU from the Google Play store,
and your phone is ready to hack.
WWW.LINUXJOURNAL.COM / MARCH 2015 / 47

LJ251-March2015.indd 47

2/19/15 9:22 AM

COLUMNS

THE OPEN-SOURCE CLASSROOM

Its important to note that rooting

a phone is not the same as installing
a third-party ROM. Although its dead
simple to root most phones, installing
something like Cyanogenmod is far
more difficult, and often its not
possible even if the phone is rooted.
Thankfully, once the phone is rooted,
the existing ROM can be made to
function a little nicer. Getting rid of
the nag screens is the first obstacle
in that journey.
Stop the Nags!
Once your phone is rooted, its time
to start looking for the applications
that are doing all the nagging.
Unfortunately, this will take some
googling, some guessing and a
little bit of luck. The process is itself
pretty straightforward:
1. Download a root-enabled file
manager app like Root Browser
or something similar.
2. Figure out what app(s) are
responsible for the activation nag
screens. Basically, google your
phones model along with disable
activation screen or something
like that.
3. Rename the apk files to add
.bak at the end. In my case,

I had to rename /system/app/

LGDMSClient.apk to /system/app/
LGDMSClient.apk.bak.
 2EBOOT THE PHONE AND SEE IF IT
worked. If it did, celebrate. If not,
do some more googling, or just
educated guessing, and try again.
There is some danger here, but as
long as youre not deleting files,
just renaming them, most bad
guesses can be reversed.
Other Anti-iPod Tweaks
Once your phone is working, and
youre able to reboot it without the
frustrating nag screens, head over to
the settings app. Here is where you
can disable all cellular data radios.
Since youre not going to activate the
phone with cell service, it will save
some serious battery power if you
disable the radios entirely. Depending
on your model, you may have to
DISABLE ',4% AND ' SEPARATELY
The one frustration I have is that
try as I might, Ive not been able to
remove the cellular radio icon from
the top of the phone (Figure 2).
There are some apps in the Google
Play store to remove icons, but they
remove the W i-Fi icon too, and that
doesnt help me at all. Oh well,
its a small price to pay. Plus, its a
great way for me to keep track of

48 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 48

2/19/15 9:22 AM

COLUMNS

THE OPEN-SOURCE CLASSROOM

Figure 2. Although
the cellular radio
icon bothers
me, its the only
frustration I
havent been able
to eliminate!

Sprint coverage in my area. If it

ever gets strong enough, Ill
probably invest in a Karma router
(http://www.yourkarma.com),

which would make my new minitablet even more useful on the road.
One last recommendation I have
is to download one of the SD Fix
WWW.LINUXJOURNAL.COM / MARCH 2015 / 49

LJ251-March2015.indd 49

2/19/15 9:22 AM

COLUMNS

THE OPEN-SOURCE CLASSROOM

apps from the Google Play store.

7ITH THE ADVENT OF !NDROID  THE
SD card isnt writable by apps like
FolderSync, and as such, it makes
managing audiobooks or MP3 files
really difficult. With a rooted phone,
its another two-click solution to make
your SD card functional again. I still
cant believe Google crippled Android
 LIKE THAT 4HANKFULLY ROOT ACCESS
and Linux can save the day.
Heres My Number, Call Me Maybe
If the cell radio icon bothers me, just
imagine how much it bothers me to
have a phone with microphone and
speaker, but no phone service. I know,
I said I wasnt looking for a phone,
but I have OCD, so that unused
HARDWARE REALLY ANNOYS ME %NTER 3)0
Some phones come with a firmware
that supports Android SIP calling out
of the box. Most prepaid phones,
however, disable that feature because
they want you to use their service.
It makes sense. Thankfully, you can
download a third-party app called
CSipSimple and add complete Wi-Fibased SIP calling to your phone. It
even integrates with the native
dialer application, so you use it
like a regular phone! Im still using
'OOGLE 6OICE SERVICE THROUGH
http://www.simonics.com, but because
that ability was supposed to stop almost

a year ago, I wouldnt count on it

working forever. Any SIP provider will
work with CSipSimple, however, so
EVEN IF THE FREE 'OOGLE 6OICE OPTION
through Simonics stops working, you
can get the prepaid phone working
without paying the cellular provider.
GPS!
Its hard to buy an Android device
that doesnt come with GPS built in.
Although the lack of cellular radio
means you cant do real-time map
downloading on the road (unless you
have mobile Wi-Fi or a hotspot on
your actual cell phone), its simple to
use your new mini-tablet as a GPS
device with a little bit of planning.
Google Maps allows you to download
map data for specific areas locally to
the device. This doesnt work great
for long trips, because grabbing all
those map sections is tedious, but
for short trips to unknown locations
it works well.
There are also several off-line
GPS apps available in the Google
Play store, and although most cost
money, theyre cheaper than buying
a standalone GPS unit at the store.
If Im being completely honest, I still
use a Garmin GPS for long trips, but
thats probably because Im old and
dont always trust technology to
work as expected.

50 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 50

2/19/15 9:22 AM

LJ251-March2015.indd 51

2/19/15 9:22 AM

COLUMNS

THE OPEN-SOURCE CLASSROOM

What Else?
For me, having a tiny Android
device that lets me sync audiobooks
with FolderSync and play with
the Listen app is all I could want
and more. That doesnt mean I
couldnt think of more things to
do with a cheap Android device,
however. I actually bought another
LG Realm when they were on sale,
and I have lots of plans for it
things like:
Q A Plex player for watching movies.
Q A really cheap IP camera (birdcam!)

using the IPWebcam app.

Q Music player for the bathroom

counter, connected to speakers.

Q Tiny gaming device for bored

children who visit.

Q Cheap Skype/Hangout device to

give someone I want to keep in

contact with.

Really, Id do anything you can do

with any other Android device, but
without the guilt of spending a ton of
money. Having a $20 Android device
capable of doing so many things really
makes it easy to come up with fun
projects. Its also nice to be able to
mount one in the car and just leave it
there. With FolderSync, it will download
any new media when youre parked in
the garage, and youll never have to
take your phone out of your pocket!
)M REALLY QUITE HAPPY 3AMSUNG
discontinued its Galaxy Player devices.
I worry that if they were available, I
might have purchased one. Im so much
happier with my $20 tablet than I would
have been with a $300 media player. If
you have a need for another Android
device in your life, but dont want to
spend a fortune, I urge you to check out
the prepaid phone options out there.
Its surprising what $20 will buy! Q
Shawn Powers is the Associate Editor for Linux Journal.
Hes also the Gadget Guy for LinuxJournal.com, and he has an
interesting collection of vintage Garfield coffee mugs. Dont let
his silly hairdo fool you, hes a pretty ordinary guy and can be

Q Surprisingly affordable alarm clock

for my nightstand.

reached via e-mail at [email protected]. Or, swing by

the #linuxjournal IRC channel on Freenode.net.

Q Universal XBMC/Kodi remote for all

our televisions.
Q Wi-Fi testing tool (WiFi Analyzer).

Send comments or feedback via

http://www.linuxjournal.com/contact
or to [email protected].

52 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 52

2/19/15 9:22 AM

LJ251-March2015.indd 53

2/19/15 9:22 AM

NEW PRODUCTS

Gumstix Inc.s Geppetto

Gumstix Inc. is so proud of the embedded
systems it designed with its home-grown
Geppetto design tool that it wants the
wider world to enjoy similar benefits.
Gumstix calls the new Geppetto 2.0 the
most advanced version of the companys
on-line build-to-order tool for designing
custom-embedded Linux systems. This new
iteration of Geppetto introduces Tux-approved recommended mappings for buses,
ensuring optimal compatibility between customer-created hardware and standard
Linux images. In addition, version 2.0 offers an expanded module selection, improved
dimensioning, faster UI and video tutorials. As part of the Geppetto announcement,
'UMSTIX ALSO ANNOUNCED THE 'EPPETTO DESIGNED !ERO#ORE4-  -ICRO !ERIAL 6EHICLE
#ONTROL "OARD AND THE 'EPPETTO DESIGNED 0EPPER $6) $ SINGLE BOARD COMPUTER
http://www.gumstix.com

Investintech.coms Able2Extract PDF

Converter
)TS NOT A STRETCH TO CALL )NVESTINTECHCOMS !BLE%XTRACT  0$& #ONVERTER
THE h3WISS ARMY KNIFEv OF 0$& CONVERTERS .OT ONLY IS !BLE%XTRACT ABLE
to convert PDFs to a wide range of formats accurately, but it also features
THE UNIQUE ABILITY TO WORK ACROSS ,INUX 5BUNTU AND &EDORA  -AC /3 8
AND 7INDOWS PLATFORMS )NVESTINTECHCOM NOTES THE ABILITY OF !BLE%XTRACT
to maintain intact all aspectsimages, colors, formatting and fonts
regardless of file format. Supported formats include converting PDF to
/PEN/FFICEORG -3 /FFICE !UTO#!$ %XCEL AND COMMONLY USED IMAGE
formats. The upgrade version 9 adds secure PDF creation, improved custom
0$& TO %XCEL CONVERSION AND AN IMPROVED '5) AND OVERALL USER EXPERIENCE
http://www.investintech.com

54 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 54

2/19/15 9:22 AM

NEW PRODUCTS

Linutop XS
Until the era of the Linutop computer, the word
minuscule has not been a common descriptor for
a full-fledged PC. That word nevertheless hits the
NAIL SQUARELY ON THE HEAD TO DESCRIBE THE NEW
Linutop XS, a truly tiny Linux computer designed
to reduce TCO from shipping to deployment,
operation and maintenance. As Linutops smallest and most energy-efficient computer
to date, the Linutop XS weighs a mere 3.3 ounces (92 g), measures about the size
OF A TYPICAL PLAYING CARD AND OPERATES ON ONLY  6OLTS AND  7ATTS ,INUTOP SAYS
that the Linutop XS comes loaded with Debian Weezy and ready-to-use software,
including Libre Office and Linutop Kiosk, making it an ideal system for a wide range
of applications in business, government, education and the home.
http://www.linutop.com

JetBrains Upsource
The idea for JetBrains new team
collaboration tool for developers,
called Upsource, originally came
from the intention to make a
TOTALLY DIFFERENT TOOL )NTELLI* )$%!
available from both the desktop
and the Web. The final result is Upsource 1.0, a new Web-based team collaboration
tool that helps developers read, browse and review code maintained in Git, Mercurial,
Subversion and/or Perforce repositories. Both a repository browser and a code-review
tool, Upsource 1.0 provides instant read access to code developed throughout an
ORGANIZATION AND HELPS IMPROVE CODE QUALITY BY ENABLING EASY CODE REVIEW *ET"RAINS
ADDS THAT THANKS TO PLATFORM SHARING WITH THE )NTELLI* )$%! )$% FOR *AVA *AVA TEAMS
enjoy an additional advantage. Upsource boasts in-depth knowledge of Java code and
is able to execute server-side static code analysis on Java projects, as well as provide
code-aware navigation and smart search for code usages.
http://www.jetbrains.com/upsource
WWW.LINUXJOURNAL.COM / MARCH 2015 / 55

LJ251-March2015.indd 55

2/19/15 9:22 AM

NEW PRODUCTS

Corsair Flash Voyager Slider Series

X1 and X2
4HE NEW &LASH 6OYAGER 3LIDER 8 AND 8 FAMILIES OF 53"  &LASH
drives expand Corsairs already formidable arsenal of memory
products. Combining the speed of USB 3.0 with the functionality of
A CAP LESS 53" DRIVE THE &LASH 6OYAGER 3LIDER 8 AND 3LIDER 8 SERIES
share a sleek, glossy design that allows the USB cap to slide back
conveniently into the drive housing, says Corsair. The company added that the Slider
8 IS AVAILABLE IN '" '" '" '" AND '" CAPACITIES AND THANKS TO ITS
USB 3.0 interface, is able to reach read speeds of up to 130MB/s. Meanwhile, Slider
X2 knocks the performance up a level with read speeds of 200MB/s in capacities of
'" '" '" AND '" "OTH #ORSAIR DRIVE FAMILIES ARE COMPATIBLE WITH ,INUX
Windows and Mac OS X, and they also are fully USB 2.0-backward compatible.
http://www.corsair.com

Regina O. Obe and Leo S. Hsus

PostGIS in Action, 2nd ed.
(Manning Publications Co.)
Hybrid GIS and Linux geeks know that the open-source
0OST')3 GIVES SUPPORT FOR GEOGRAPHIC OBJECTS TO 0OSTGRE31,
allowing the relational database to serve as the back end
for ArcGIS, GRASS GIS and other geospatial programs. The
new 2nd edition of PostGIS in Action from Regina O. Obe
AND ,EO 3 (SU TEACHES READERS OF ALL LEVELS TO WRITE SPATIAL QUERIES THAT SOLVE
real-world problems. Obe and Hsu start by getting readers feet wet with a
background in vector-, raster- and topology-based GIS, followed by a tutorial
IN ANALYZING VIEWING AND MAPPING DATA 2EADERS LEARN HOW TO OPTIMIZE QUERIES
for maximum speed, simplify geometries for greater efficiency, analyze rasters,
vectorize rasters, better manage data utilizing topologies and create custom
FUNCTIONS 4HE BOOK COVERS 0OST')3  AND  0OSTGRE31,   AND 
features and shows how to integrate PostGIS with other GIS tools.
http://manning.com
56 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 56

2/19/15 9:22 AM

NEW PRODUCTS

Mahesh Venkitachalams Python

Playground (No Starch Press)
Putting the subtitle Geeky Weekend Projects for the Curious Programmer onto
a book is a sure way to charm ones way onto these geek-friendly Linux Journal
pages. The main title of said book is Python Playground A NEW BOOK FROM -AHESH 6ENKITACHALAM
and irreverent publisher No Starch Press. No Starch describes the book as a collection of fun
programming projects that will inspire you to new heights as a Pythonista. Readers will learn
to use Python for all kinds of playful purposesfor example, to manipulate images, build
simulations and interact with hardware using Arduino and Raspberry Pi. As readers work through
each project, they power up their programming skills and learn how to leverage external libraries
for specialized tasks, how to break problems into smaller, solvable pieces and how to translate
an algorithm into code. The fun projects include an autostereogram generator, an ASCII art
maker, a Conways Game of Life simulator, a ray casting volume renderer and an Arduino rig.

http://www.nostarch.com

Deciso OPNsense Firewall

$ECISO "6 IS A .ETHERLANDS BASED MANUFACTURER OF NETWORKING
EQUIPMENT THAT DEVELOPED AND RECENTLY RELEASED /0.SENSE A NEW
open-source firewall that reportedly combines the best of opensource and closed-source firewalls. Deciso adds that OPNsense
brings the rich feature set of commercial offerings with the
benefits of open and verifiable sources, combined with a simple, two-clause BSD license. The latter
permits companies to create a branded firewall based on OPNsense, extend its features, or even
create a fork and build upon the same codebase. Key features of OPNsense include load balancing,
high availability and captive portal. The easy-to-use Bootstrap-based GUI makes configuring and
managing the firewall a comfortable task for administrators. The kicker, boasts Decisio, is that all
sources and build tools are freely available without special clauses and without licensing costs. The
company also puts a great deal of value on the community surrounding OPNsense, which it says will
give users, developers and businesses a friendly, stable and transparent environment.

http://www.opnsense.org and http://www.deciso.com

Please send information about releases of Linux-related products to [email protected] or
New Products c/o Linux Journal, PO Box 980985, Houston, TX 77098. Submissions are edited for length and content.

WWW.LINUXJOURNAL.COM / MARCH 2015 / 57

LJ251-March2015.indd 57

2/19/15 9:22 AM

FEATURE Using Hiera with Puppet

USING
HIERA
WITH
PUPPET
A GUIDE TO USING HIERA WITH PUPPET,
SEPARATING CODE FROM DATA
AND ENCRYPTING PASSWORDS
AND CERTIFICATES.
SCOTT LACKEY

58 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 58

2/19/15 9:22 AM

ith Hiera, you can

externalize your systems
configuration data and
easily understand how those values
are assigned to your servers. With
that data separated from your Puppet
code, you then can encrypt sensitive
values, such as passwords and keys.
Separating code and data can be
tricky. In the case of configuration
management, there is significant value
in being able to design a hierarchy of
dataespecially one with the ability
to cascade through classifications
of servers and assign one or several
options. This is the primary value that
Hiera providesthe ability to separate
the code for how to configure the
/etc/ntp.conf from the values that
define what ntp servers should each
node use. In the most concise sense,
Hiera lets you separate the how
from the what.
The idea behind separating code
and data is more than just having a
cleaner Puppet environment; it allows
engineers to create more re-usable
Puppet modules. It also puts your
variables in one place so that they too
can be re-used, without importing
manifests across modules. Hieras use
cases include managing packages
and versions or using it as a Node
Classifier. One of the most compelling
use cases for Hiera is for encrypting

credentials and other sensitive data,

which I talk about later in this article.
Puppet node data originally was
managed through node inheritance,
which is no longer supported, and
SUBSEQUENTLY THROUGH USING A PARAMSPP
module subclass. Before Hiera, it was
necessary to modify the params.pp
module class locally within the
MODULE WHICH FREQUENTLY DAMAGED
the re-usability of the module.
params.pp still is used in modules
today, but as of Puppet version 3,
Hiera is not only the default, but also
the first place checked for variable
values. When a variable is defined
in both Hiera and a module, Hiera
takes precedence by default. As youll
see, its easy to use a module with
params.pp and store some or all of
the variable data in Hiera, making it
easy to migrate incrementally.
To get started using Hiera with your
existing Puppet 3 implementation, you
wont have to make any significant
changes or code migrations. You need
only a hierarchy file for Hiera and a
yaml file with a key/value pair. Here is
an example of a Hiera hierarchy:
hiera.yaml:

:backends:
- yaml
:yaml:
WWW.LINUXJOURNAL.COM / MARCH 2015 / 59

LJ251-March2015.indd 59

2/19/15 9:22 AM

FEATURE Using Hiera with Puppet

:datadir: /etc/puppet/hieradata
:hierarchy:
- "node/%{::fqdn}"
- "environment/%{::env}/main"
- "environment/%{::env}/%{calling_module}"
- defaults

And a yaml file:

/etc/puppet/hieradata/environment/prod/main.yaml:
--$nginx::credentials::basic_auth: 'password'

Hiera can have multiple back ends,

but for now, lets start with yaml,
WHICH IS THE DEFAULT AND REQUIRES NO
additional software. The :datadir:
is just the path to where the hierarchy
search path should begin, and is
usually a place within your Puppet
configuration. The :hierarchy:
section is where the core algorithm of
how Hiera does its key/value lookups
is defined. The :hierarchy: is
something that will grow and change
over time, and it may become much
more complex than this example.
Within each of the paths defined
in the :hierarchy: , you can
reference any Puppet variable, even
$operatingsystem and $ipaddress ,
if set. Using the %{variable} syntax
will pull the value.
This example is actually a special
hierarchical design that I use and

recommend, which employs a

fact assigned to all nodes called
@env from within facter. This @env
value can be set on the hosts either
BASED ON &1$. OR TAGS IN %# OR
elsewhere, but the important thing
is that this is the separation of one
large main.yaml file into directories
named prod, dev and so on, and,
therefore, the initial separation of
Hiera values into categories.
The second component of this
specific example is a special Hiera
variable called %{calling_module} .
4HIS VARIABLE IS UNIQUE AND RESERVED
for Hiera to indicate that the yaml
filename to search will be the same as
the Puppet module that is performing
the Hiera lookup. Therefore, the
way this hierarchy will behave when
looking for a variable in Puppet is like:
$nginx::credentials::basic_auth

First, Hiera knows that its looking

in /etc/puppet/hieradata/node for a file
NAMED HOSTNAMEDOMAINTLDYAML
and for a value for
nginx::credentials::basic_auth .
If either the file or the variable
isnt there, the next step is to
look in /etc/puppet/hieradata/
ENVIRONMENTPROD\STAGE\DEV
main.yaml, which is a great way
to have one yaml file with most

60 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 60

2/19/15 9:22 AM

of your Hiera values. If you have a

lot of values for the nginx example
and you want to separate them for
manageability, you simply can move
them to the /etc/puppet/hieradata/
ENVIRONMENTPROD\STAGE\DEV
nginx.yaml file. Finally, as a default,
Hiera will check for the value in
defaults.yaml at the top of the
hieradata directory.
Your Puppet manifest for this lookup
should look something like this:

Hiera will override it. This kind of

Hiera lookup is called Automatic
Parameter Lookup and is one of
several ways to pull data from Hiera,
but its by far the most common in
practice. You also can specify a Hiera
lookup with:
modules/nginx/manifests/credentials.pp

class nginx::credentials (
basic_auth = hiera('nginx::credentials::basic_auth'),

modules/nginx/manifests/credentials.pp

class nginx::credentials (
basic_auth = 'some_default',
){}

This class, when included, will

pull the value from Hiera and can
be used whenever included in your
manifests. The value set here of
some_default is just a placeholder;
Hiera will override anything set in a
parameterized class. In fact, if you
have a class you are thinking about
converting to pull data from Hiera,
just start by moving one variable
from the class definition in {} to a
parameterized section in (), and
Puppet will perform a Hiera lookup
on that variable. You even can leave
the existing definition intact, because

){}

These will both default to a priority

lookup method in the Hiera data files.
This means that Hiera will return the
value of the first match and stop
looking further. This is usually the
only behavior you want, and its a
reasonable default. There are two
lookup methods worth mentioning:
hiera_array and hiera_hash .
hiera_array will find all of the
matching values in the files of the
hierarchy and combine them in an
array. In the example hierarchy, this
would enable you to look up all values
for a single key for both the node
and the environmentfor example,
adding an additional DNS search
path for one hosts /etc/resolv.conf.
To use a hiera_array lookup, you
must define the lookup type explicitly
WWW.LINUXJOURNAL.COM / MARCH 2015 / 61

LJ251-March2015.indd 61

2/19/15 9:22 AM

FEATURE Using Hiera with Puppet

(instead of relying on Automatic

Parameter Lookup):

* default
:backends:
* psql

modules/nginx/manifests/credentials.pp

:psql:
:connection:
:dbname: hiera
:host: localhost

class nginx::credentials (

:user: root

basic_auth = hiera_array('nginx::credentials::basic_auth'),

:password: password

){}

A hiera_hash lookup works in the

same way, only it gathers all matching
values into a single hash and returns
that hash. This is often useful for an
advanced create_resources variable
import as well as many other uses in
an advanced Puppet environment.
Perhaps Hieras most powerful
feature is the ability to pull data
from a variety of back-end storage
technologies. Hiera back ends are
too numerous to list, but they include
JSON, Redis, MongoDB and even HTTP
to create a URL-driven Puppet value
API. Lets take a look at two useful
back ends: Postgres and hiera-eyaml.
4O START WITH THE PSQL BACK END YOU
NEED TO INSTALL THE HIERA PSQL GEM ON
your Puppet master (or each node if
youre using masterless Puppet runs
with Puppet apply), with a simple
hiera.yaml file of:
:hierarchy:
* 'environment/%{env}'

You can do lookups on a local

Postgres installation with a single
database called hiera with a single
table called config with three
COLUMNS 0ATH +EY AND 6ALUE
path

'environment/prod'

key

'nginx::credentials::basic_auth'

value

'password'

This is extremely useful if you want

to expose your Hiera data to custom
in-house applications outside Puppet,
or if you want to create a DevOps
Web console or reports.
Storing credentials in Puppet
modules is a bad idea. If you store
credentials in Puppet and your
manifests on an external code
repository, youre not only unable to
share those manifests with developers
with less-secure access, but youre
obviously exposing vital security
data outside the organization, and
possibly in violation of various types

62 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 62

2/19/15 9:22 AM

of compliance. So how do you encrypt

sensitive data in Puppet while keeping
your manifests relevant and sharable?
The answer is with hiera-eyaml.
Tom Poulton created hiera-eyaml
to allow engineers to do just that:
encrypt only the sensitive string of
data inside the actual file rather than
encrypting the entire file, which also
can be done with hiera-gpg (a very
useful encryption gem but not covered
in this article).
To get started, install the hiera-eyaml
gem, and generate a keypair on the
Puppet master:
$ eyaml createkeys

Then move the keys to a secure

location, like /etc/puppet/secure/keys.
Your hiera.yaml configuration should
look something like this:
hiera.yaml:
--:backends:
- eyaml
- yaml
:yaml:
:datadir: /etc/puppet/hieradata
:eyaml:
:datadir: /etc/puppet/hieradata
:extension: 'yaml'

# <- so all files can be named .yaml

:pkcs7_private_key: /path/to/private_key.pkcs7.pem
:pkcs7_public_key: /path/to/public_key.pkcs7.pem

:hierarchy:
- "node/%{::fqdn}"
- "environment/%{::env}/main"
- "environment/%{::env}/%{calling_module}"
* defaults

To encrypt values, you need only the

public key, so distribute it to anyone
who needs to create encrypted values:
$ eyaml encrypt -s 'password'

This will generate an encrypted

block that you can add as the value in
any yaml file:
main.yaml:
nginx::credentials::user: slackey

#cleartext example value

nginx::credentials::basic_auth : >

#encrypted example value

ENC[PKCS7,Y22exl+OvjDe+drmik2XEeD3VQtl1uZJXFFF2Nn
/HjZFXwcXRtTlzewJLc+/gox2IfByQRhsI/AgogRfYQKocZg
IZGeunzwhqfmEtGiqpvJJQ5wVRdzJVpTnANBA5qxeA==]

%DITING ENCRYPTED VALUES IN PLACE

is one of the coolest features of the
hiera-eyaml back end. eyaml edit
opens a copy of the eyaml file in your
editor of choice and automatically
decrypts all of the values in the file.
Here you can modify the values just
as though they were plain text. When
you exit the editor by saving the file,
it automatically encrypts all of the
modified values and saves the new
file in place. You can see that the
WWW.LINUXJOURNAL.COM / MARCH 2015 / 63

LJ251-March2015.indd 63

2/19/15 9:22 AM

FEATURE Using Hiera with Puppet

unencrypted plain text is marked

to allow the eyaml tool to identify
each encrypted block, along with the
encryption method that originally
was used. This is used to make sure
that the block is encrypted again only
if the clear text value has changed
and is encrypted using the original
encryption mechanism:
nginx::credentials::user: user1
nginx::credentials::basic_auth : DEC(1)::PKCS7[very secret password]!

Blocks and strings of encrypted text

can get rather onerous once you have
more than a hundred entries or so.
Because these yaml files are meant to
be modified by humans directly, you
want them to be easy to navigate. In
my experience, it makes sense to keep
your encrypted values in a separate
file, such as a secure.yaml, with a
hierarchy path of:
:hierarchy:
- "node/%{::fqdn}"
- "environment/%{::env}/secure"
- "environment/%{::env}/main"

encrypted files in a separate repository,

perhaps in a different Git repository.
Only the private keys need to be
protected on the Puppet master. I
also recommend having separate
keys for each environment, as this
can give more granular control over
who can decrypt different datafiles in
Hiera, as well as even greater security
separation. One way to do this is to
name the keys with the possible values
for the @env fact, and include that in
the path of the hierarchy. Youll need
to encrypt values with the correct key,
and this naming convention makes it
easy to tell which one is correct:
:pkcs7_private_key: /path/to/private_key.pkcs7.pem-%{::env}
:pkcs7_public_key: /path/to/public_key.pkcs7.pem-%{::env}

When using Hiera values within

Puppet templates, either encrypted or
not, you must be careful to pull them
into the class that contains the templates
instead of calling the values from
within the template across classesfor
example, in the template mytest.erb in a
module called mymodule:

- "environment/%{::env}/%{calling_module}"
mytest.erb:

This isnt necessary, as each value

is encrypted individually and can be
distributed safely to other teams. It
may work well for your environment,
however, because you can store the

...
username: user1
passwd: <%= scope.lookupvar('nginx::credentials::basic_auth') %>
#don't do this
...

64 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 64

2/19/15 9:22 AM

LJ251-March2015.indd 65

2/19/15 9:22 AM

FEATURE Using Hiera with Puppet

Puppet may not have loaded a value into

mytest.erb:

nginx::credentials::basic_auth

...

yet because of the order of

operations. Also, if you are using the
%calling_module Hiera variable, the
calling module in this case would be
mymodule, and not nginx, so it would
not find the value in the nginx.yaml
file, as one might expect.
To avoid these and other issues,
its best to import the values into the
mymodule class and assign local values:

username: user1

mymodule.pp:

Scott Lackey is a 17-year engineering veteran and Sr. DevOps

class mymodule {

Engineer for Salesforce.com. Hes passionate about helping

passwd: <%= @basic_auth %>

Youre now ready to start introducing

encrypted Hiera values gradually into
your Puppet environment. Maybe after
you separate data from your Puppet
code, you can contribute some of your
modules to the PuppetForge for others
to use! Q

include nginx::credentials

companies migrate to the cloud and mentoring prospective

$basic_auth = "${nginx::credentials::basic_auth}"

DevOps engineers. He lives in Los Angeles with his dachshund

file { '/etc/credentials/boto_cloudwatch.cfg':

Zelda. Reach him at [email protected].

content =>

template ("mymodule/mytest.erb"),

And then reference the local value

from the template:

Send comments or feedback via

http://www.linuxjournal.com/contact
or to [email protected].

Resources
DocsHiera 1 Overview: https://docs.puppetlabs.com/hiera/1
First Look: Installing and Using Hiera:
http://puppetlabs.com/blog/first-look-installing-and-using-hiera
TomPoulton/hiera-eyaml: https://github.com/TomPoulton/hiera-eyaml
dalen/hiera-psql: https://github.com/dalen/hiera-psql
Encrypting sensitive data in Puppet: http://www.theguardian.com/info/developer-blog/
2014/feb/14/encrypting-sensitive-data-in-puppet

66 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 66

2/19/15 9:22 AM

Interested in Site Reliability Engineering?

SREcon is back!
SREcon15

SREcon15EUROPE

www.usenix.org/srecon15

www.usenix.org/srecon15europe

MARCH 1617, 2015

SANTA CLARA, CALIFORNIA, USA

MAY 1415, 2015

DUBLIN, IRELAND

Following 2014s inaugural sold-out conference, SREcon has expanded

to two venues for 2015.
If you already work in an SRE environmentor want to learn how its
being used by many of the largest companies todaytake advantage of
this rare opportunity to meet with other engineers and discuss tricks of
the trade.
Register today at www.usenix.org

LJ251-March2015.indd 67
srecon15_lj.indd 1

2/19/15 9:23 AM
2/17/15 11:03 AM

FEATURE Initializing and Managing Services in Linux: Past, Present and Future

INITIALIZING
AND MANAGING
SERVICES
IN LINUX:
PAST,
PRESENT
AND
FUTURE
systemd is the new init system
used by many of the top Linux distributions,
but do you know the history behind it
and how we got here?
Learn about the history of init systems
in Linux and their UNIX legacy.
Gain a better perspective about how Linux
manages services and other support processes.
Jonas Gorauskas
68 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 68

2/19/15 9:23 AM

ne of the most crucial pieces

of any UNIX-like operating
system is the init dmon
process. In Linux, this process is
started by the kernel, and its the first
userspace process to spawn and the
last one to die during shutdown.
During the history of UNIX and
Linux, many init systems have gained
popularity and then faded away. In
this article, I focus on the history
of the init system as it relates to
Linux, and I talk about the role of
init in a modern Linux system. I also
relate some of the history of the
3YSTEM 6 )NIT 3YS6 SCHEME WHICH
was the de facto standard for many
Linux distributions for a long time.
Then I cover a couple more modern
approaches to system initialization,
such as Upstart and systemd. Finally,
I pay some attention to how things
work in systemd, as this seems to be
the popular choice at the moment for
several of the largest distributions.
The Role of Init
Init is short for initializer, and its
both a startup manager and a session
manager for Linux and other UNIXes.
Its a startup manager, because it
plays a crucial role in the startup of
Linux. Its the process that creates or
initializes userspace and, ultimately,
all userspace processes. It also may

be considered a session manager,

because it takes care of many aspects
of userspace and its processes once
the system is up and running.
The call to start this process is, in
fact, hard-coded in the Linux kernel.
Download the latest kernel sources
and look for a function called
kernel_init in the file init/main.c.
Among the files that the Linux kernel
will try to execute is /sbin/init. If Linux
cannot find one of these processes, it
throws a kernel panic and halts.
The kernel gives the init process an
ID of 1 or PID 1. All other userspace
processes are forked from init, and
therefore, PID 1 claims ancestral rights
to all other userspace processes. PID 1
also automatically will become the
direct parent process of any userspace
process that is orphaned.
A Little Bit of History
Now that I have set the stage for the
article and given you a very basic
understanding of what init is and
does, Id like to digress into a little bit
of UNIX history.
There has been a lot of diversity in
the initialization schemes for UNIXlike operating systems over time. Two
of the most important init schemes
that had a historical impact on how
different Linux distributions do things
ARE THE RC SCHEME USED IN THE  "3$
WWW.LINUXJOURNAL.COM / MARCH 2015 / 69

LJ251-March2015.indd 69

2/19/15 9:23 AM

FEATURE Initializing and Managing Services in Linux: Past, Present and Future

A Linux distribution implementing a SysV

scheme can be in one of many distinct
states in which a predetermined number
of processes may be running.
AND THE 3YS6 SCHEME USED IN 3UN/3
and Solaris.
4HE  "3$ INIT SYSTEM IS PRETTY
simple and monolithic. When booting,
the kernel runs /sbin/init, which would
spawn a shell to run the /etc/rc script.
The /etc/rc script contained commands
to check the integrity of hard
drives and mount them, start other
processes, and start the networking
subsystem. This scheme was contained
completely within a few scripts:
namely /etc/rc, /etc/rc.local and
/etc/netstart. This scheme also had
no specific shutdown procedure. Init
WOULD RECEIVE A 3)'4%2- SIGNAL AND
SEND A 3)'(50 ANDOR A 3)'4%2- TO
its children, and after all processes
exited, it would drop to single-user
mode and shut down.
Today, the systems that have
inherited the rc initialization scheme
are Free-BSD, Net-BSD and the
Slackware Linux distribution. These
modern systems have improved
QUITE A BIT ON THE ORIGINAL  "3$
scheme and are much more modular

than the original.

Most other Linux distributions
have, historically, been adepts of
THE 3YS6 SCHEME WHICH ORIGINALLY
was implemented in AT&T UNIX and
derivative systems like Solaris.
System V Init
A Linux distribution implementing
A 3YS6 SCHEME CAN BE IN ONE OF
many distinct states in which a
predetermined number of processes
may be running. These states are
called runlevels and to get to a certain
runlevel means that the system is in a
certain operational stage.
The meaning for each runlevel may
vary based on your distribution of
Linux. For example, there are a few
distributions (such as Ubuntu) that
use runlevel 2 to mean multi-user
graphical mode with networking
enabled. Others (like Fedora) use
runlevel 5 to mean the same thing.
)N A 3YS6 ,INUX MACHINE THE KERNEL
runs /sbin/init as usual, which in turn
will load parameters and execute

70 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 70

2/19/15 9:23 AM

directives defined in /etc/inittab.

This file defines the default runlevel
for the whole system, describes
what happens when Ctrl-Alt-Del is
pressed, loads keymap files, defines
which terminals to spawn gettys for,
spawns terminal login processes, runs
the /etc/init.d/rcS script, and it also
influences the order of execution of
other runlevel scripts.
The /etc/init.d/rcS script will put
the system in a single-user mode in
order to finish probing hardware,
mount disks, set hostname, set up
networking and so on. Take a look
at /etc/rcS.d/ in a Debian 7 system
for all the gory details. Next,
/sbin/init will switch itself to
the default runlevel to start all
the system services. The default
runlevel value is defined in the
initdefault line of /etc/inittab.
This actually translates into a call
to the /etc/init.d/rc script with the
parameter of 2 for the runlevel value.
The rc script will then execute all of the
+
FOR +ILL AND 3
FOR 3TART SCRIPTS
in the /etc/rc2.d/ directory. These are
actually links to the real scripts in
/etc/init.d/. The names of the links
follow the format S##<service-name>
or K##<service-name> , where the
## token is the two-digit number used
to determine the order in which the
script should run. Order is alphabetical,

and Kill scripts execute before Start

scripts. The last thing to happen is
to run the /etc/rc.local script, which
is where you can add custom system
commands that you want to execute
at startup.
! SYSTEM THAT USES THE 3YS6 SCHEME
usually comes with the service
program used to manage the services
while the system is running. You can
check on the status of a service, or all
services, and start or stop a service,
respectively, using the service utility:
Q $ service <service> status
Q $ service status -all
Q # service <service> start|stop

To manage the assignment of

services to a particular runlevel, you
can use a tool called sysv-rc-conf ,
which manages the setup of all links
in the respective rc directories. You
also can switch the runlevel of the
system at any time when you use the
command telinit as a privileged
user. For example, telinit 6 will
REBOOT A 3YS6 SYSTEM
4HE 3YS6 SCHEME STILL IS IN USE
today in Debian 7 (Wheezy) systems.
However, the Debian developers will
be changing the init system in version
8 to systemd. I cover systemd in more
WWW.LINUXJOURNAL.COM / MARCH 2015 / 71

LJ251-March2015.indd 71

2/19/15 9:23 AM

FEATURE Initializing and Managing Services in Linux: Past, Present and Future

The SysV scheme has been great,

but it started to show its age around
the time when Linux on the desktop
gained a little more momentum.
detail below, but first, lets look at
why we need a new init system.
The Problem with System V Init
4HE 3YS6 SCHEME HAS BEEN GREAT
but it started to show its age around
the time when Linux on the desktop
gained a little more momentum.
7HEN THE 3YS6 SCHEME ORIGINALLY
was designed, computers where
NOTHING LIKE THEY ARE TODAY 3YS6
was not designed to handle certain
things well:

shutting down services prior to

shutdown. As a result, the design
was strictly synchronous, blocking
future tasks until the current one
had completed.
This left the system unable to
handle various events that were not
related to the startup or shutdown
of the system. Things that we
take for granted today were really
cumbersome to handle elegantly
DURING THE HEYDAY OF 3YS6 INIT
Q There was no real process

Q USB devices.
Q %XTERNAL STORAGE VOLUMES
Q Bluetooth devices.
Q The cloud.

4HE 3YS6 SCHEME WAS DESIGNED

for a world that was static and slow
moving. This init scheme originally
was responsible only for bringing
the system into a normal running
state after power on or gracefully

supervisionfor example, dmons

were not automatically restarted
when they crashed.
Q There was no real dependency

checking. The order of script

naming determined the order in
which they were loaded.
Q The addition or removal of USB

drives and other portable storage/

network devices while the machine
was running was cumbersome and
OFTENTIMES REQUIRED A REBOOT

72 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 72

2/19/15 9:23 AM

Q There were no facilities to

discover and scan for new storage

devices without locking the
system, especially when a disk
might not even power on until it
was scanned.
Q There were no facilities to load

firmware for a device, which may

have needed to occur after it was
detected but before it was usable.
Inevitably, around the 2005/2006
time frame, several alternative
efforts tried to fix all the issues with
THE 3YS6 SCHEME "UT THE EFFORT THAT
looked most promising during that
time was the Upstart init project
sponsored by Canonical.
Upstart
To be sure, Upstart init doesnt share
ANY CODE WITH THE 3YS6 INIT SCHEME
but its rather a superset of it,
providing a good degree of backwardcompatibility. The main departure
FROM THE TRADITIONAL 3YS6 WAY OF DOING
things is that Upstart implements an
event-driven model that allows it to
respond to milestones asynchronously
as they are reached. Upstart also
implements the concept of jobs,
which are described by the files under
ETCINIT
CONF AND WHOSE PURPOSE
is to execute a script section that

spawns a process. As such, system

initialization can be expressed as a
consecutive set of spawn process X
when event Y occurs rules.
*UST LIKE IN THE 3YS6 SCHEME
the Linux kernel gives control
to Upstart when it executes the
Upstart implementation of /sbin/init.
At this point, things may work a
little differently depending on your
distribution of Linux. For Red Hat
%NTERPRISE ,INUX 2(%,  USERS
youll still find a file at /etc/inittab,
but the sole function of this file is
to set the default runlevel for the
system. If your distribution is one of
the Ubuntu derivatives, /etc/inittab
doesnt even exist anymore, and the
default runlevel is set in a file called
/etc/init/rc-sysinit.conf instead.
The Upstart version of /sbin/init will
emit a single event called startup,
which triggers the rest of the system
initialization. There are a few jobs
that specify the startup event as their
start condition, the most notable of
which is mountall , which mounts all
filesystems. The mountall job then
triggers various other events related
to disk and filesystem initialization.
These events, in turn, trigger the
udev kernel device manager to start,
and it emits the event that starts the
networking subsystem.
This is when one of the most critical
WWW.LINUXJOURNAL.COM / MARCH 2015 / 73

LJ251-March2015.indd 73

2/19/15 9:23 AM

FEATURE Initializing and Managing Services in Linux: Past, Present and Future

jobs is triggered by Upstart. This job is

called rc-sysinit , which has a start
dependency on the filesystem and
network-up events. The role of this
job is to bring the system to its default
runlevel. It executes the command
telinit <runlevel> to achieve this.
The telinit command then emits
the runlevel event, which causes many
other jobs to start. This includes the
/etc/init/rc.conf job, which implements
A COMPATIBILITY LAYER FOR THE 3YS6 INIT
scheme. It executes /etc/init.d/rc
<runlevel> and determines if a
/etc/rc#.d/ directory exists for the current
runlevel, executing all scripts in it.
In Upstart-based systems, such
AS 5BUNTU AND 2(%,  YOU CAN
use the tools sysv-rc-conf or
chkconfig , respectively, to manage
the runlevel of different services.
You also can manage jobs via the
initctl utility. You can list all jobs
and their respective start and stop
events with the command initctl
show-config . You also can check
on job status, list available jobs and
start/stop jobs with the following
commands, respectively:
Q $ initctl status <job>
Q $ initctl list
Q # initctl start|stop <job>

The Upstart scheme has been used

in popular distributions of Linux, such
AS &EDORA FROM VERSIONS  UP TO 
THE 2(%,  SERIES AND 5BUNTU SINCE
version 6.10 to present. But for all
the flexibility that Upstart init brings
to Linux, it still falls short in a few
fundamental ways:
Q It ignores the system state between

events. For instance, a system has

a power cord plugged in, then the
system runs on AC power for a
while, and then the user unplugs the
power cord. Upstart focuses on each
event above as a single discrete and
unrelated unit, instead of tracking
the chain of events as a whole.
Q The event-driven nature of the

system turns the dependency chain

on its head. Instead of doing the
absolute minimum amount of
work needed to get the system to
a working state, when an event
is triggered, it executes all jobs
that could possibly follow it. For
example, just because networking
has started, it doesnt mean that
NFS also should start. As a matter
of fact, the opposite is the correct
order of things: when a user
REQUESTS ACCESS TO AN .&3 SHARE
the system should validate that
networking is also up and running.

74 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 74

2/19/15 9:23 AM

The main design goals of this init scheme

are, according to Lennart Poettering,
lead developer of systemd, to start less,
and to start more in parallel.
Q The dependency chain is still

present. Although many more things

happen in parallel in Upstart, the
user has to port the original script
SEQUENCE FROM 3YS6 INIT TO A SET
of event trigger action rules in the

CONF FILES IN ETCINIT &URTHERMORE

because of the spanning tree
structure of the event system, it is
a real nightmare to figure out why
something happened and what
event triggered it.
There is another init scheme
whose purpose is to address the
issues listed above.
systemd
systemd is the latest milestone on the
road to init system nirvana. The main
design goals of this init scheme are,
according to Lennart Poettering, lead
developer of systemd, to start less,
and to start more in parallel. What
that means is that you execute only
that which is absolutely necessary to
get the system to a running state, and

you run as much as possible at the

same time.
To accomplish these goals, systemd
aims to act against two major trouble
spots of previous init schemes: the
shell and parallelism. The main
executable for systemd, /lib/systemd/
systemd, performs all calls that
originally were present in scripts, thus
eliminating the need to spawn a shell
environment. What about the call to
/sbin/init thats hard-coded in the Linux
kernel? Its still there in the form of a
symbolic link to /lib/systemd/systemd.
To address parallelism, you need to
remove the dependency chain between
the various services or at least make it
a secondary concern. If you look at the
problem at its most fundamental level,
the dependency between the various
services boils down to one thing: having
a socket available for the processes
to communicate among themselves.
systemd creates all sockets first and
then spawns all processes in parallel.
For example, services that need to write
to the system log need to wait for the
WWW.LINUXJOURNAL.COM / MARCH 2015 / 75

LJ251-March2015.indd 75

2/19/15 9:23 AM

FEATURE Initializing and Managing Services in Linux: Past, Present and Future

/dev/log socket to become available,

but as soon as it is available, these
services can start. Therefore, if systemd
creates the socket /dev/log first, then
thats one less dependency that blocks
OTHER SERVICES %VEN IF THERE IS NOTHING
to receive messages at the other end of
the socket, this strategy still works. The
kernel itself will manage a buffer for
the socket, and as soon as the receiving
service starts, it will flush the buffer
and handle all the messages. The ideas
above are not new or revolutionary.
They have been tried before in projects
like the xinetd superserver and the
launchd init scheme used in OS X.
systemd does introduce the new
concepts of units and targets. A target
is analogous to a runlevel in previous
schemes and is composed of several
units. systemd will execute units to
reach a target. The instructions for each
unit reside in the /lib/systemd/system/
directory. These files use a declarative
format that looks like a Windows INI
file. The most common type of these
units is the service unit, which is used
to start a service. The sshd.service file
from Arch Linux looks like this:

[Service]
ExecStart=/usr/bin/sshd -D
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always
[Install]
WantedBy=multi-user.target

This format is really simple and

really portable across several different
distributions. There are other types
of unit files that describe a system,
and they are socket, device, mount,
automount, swap, target, path, timer,
snapshot, slice and scope. Going into
all of them in detail is beyond the
scope of this article; however, I want to
mention one thing: target is a special
type of unit file that glues the other
types together into a cohesive whole.
For example, here are the contents of
basic.target from Arch Linux:
[Unit]
Description=Basic System
Documentation=man:systemd.special(7)
Requires=sysinit.target
Wants=sockets.target timers.target paths.target

[Unit]
Description=OpenSSH Daemon
Wants=sshdgenkeys.service

slices.target
After=sysinit.target sockets.target timers.target
paths.target slices.target

After=sshdgenkeys.service

JobTimeoutSec=15min

After=network.target

JobTimeoutAction=poweroff-force

76 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 76

2/19/15 9:23 AM

You can follow the chain of

dependencies if you look at what
BASICTARGET REQUIRES AND WANTS
Those are actual unit files in the
same /lib/systemd/system/ directory.
The Requires and Wants directives
above are how systemd defines
the dependency chain among the
units. The Requires directive
DENOTES A HARD REQUIREMENT
and Wants denotes an optional
REQUIREMENT !LSO KEEP IN MIND THAT
Requires and Wants dont imply
order. If the After directive isnt
specified, systemd will start the
units in parallel.
Timer units are also really
interesting. They are unit files
that contain a [Timer] section
and define how the TimeDateD
subsystem of systemd will activate
a future event. In these timer units,
you can create two types of timers:
one that will activate after a time
period based on a variable starting
point, such as the systems boot,
and another that activates at fixed
intervals like a cron job. As a matter
of fact, timer units are an alternative
to cron jobs.
One last thing to mention about
systemd unit files is that they
provide the means to describe easily
what to do when a service crashes.
You can do that by using the

directives Restart or RestartSec

in your unit files. This feature allows
systemd to take the role of process
supervisor as well.
systemd refers to the init
dmon executable itself, namely
/lib/systemd/systemd, but it also
refers to the set of utilities and
programs used to manage the system
and services. Chief among these
utilities is the systemctl program
thats used to manage services.
You can use it to enable, start and
disable services, find the status of a
given service and also list all loaded
units. For example:
Q # systemctl enable sshd
Q # systemctl start sshd
Q # systemctl stop sshd
Q # systemctl status sshd
Q # systemctl list-units

Some Linux distributions, like

2(%,  AND #ENT/3  PROVIDE A
compatibility layer that translates
3YS6 AND 5PSTART COMMANDS INTO
systemd commands. If you issue
the command service sshd
status in CentOS 7, you will get
the following output:
WWW.LINUXJOURNAL.COM / MARCH 2015 / 77

LJ251-March2015.indd 77

2/19/15 9:23 AM

FEATURE Initializing and Managing Services in Linux: Past, Present and Future

Redirecting to /bin/systemctl status

sshd.service

sshd.service - OpenSSH server daemon

Q Display log since last boot:

# journalctl -b

Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled)

Active: active (running) since Mon 2014-12-08 02:01:53 PST;
12h ago

Q Display errors from last boot:

# journalctl -b -p err

Process: 915 ExecStartPre=/usr/sbin/sshd-keygen (code=exited,

status=0/SUCCESS)
Main PID: 937 (sshd)
CGroup: /system.slice/sshd.service

I urge you to look at the

documentation of the different schemes
presented here to learn more.

...937 /usr/sbin/sshd -D

Notice that first line of console

output above and how it indicates
THAT THE 3YS6 STYLE COMMAND WAS
redirected to the systemd-style of
command. This allows the user to
ease into the systemd way of doing
things while still allowing the user to
leverage the previous skill set.
Another really important program
in the systemd toolbox is the
journalctl utility. It allows you
to view and manage the systemd
logging subsystem called journald.
systemds logfile is a binary file and
using journalctl really simplifies
the user experience. Here are some
interesting examples:
Q Display full log: # journalctl --all
Q Tail the log: # journalctl -f
Q Filter log by executable:
# journalctl /lib/systemd/systemd

Controversies
From my vantage point, the future is
not 100% certain when it comes to init
schemes for Linux. The clear leader, as
) WRITE THIS IN LATE  IS SYSTEMD !
lot of distributions are adopting it; the
LATEST ONES ARE 2(%,  AND $EBIAN 
However, the adoption of systemd
has been controversial, and these
distributions have received a lot of
strong feedback from their respective
communities. Of note is the Debian
technical committee debate that
occurred in the Debian mailing list and
a complaint by Linus Torvalds himself
in the Linux kernel mailing list.
systemd is not just an init scheme.
It unifies everything that is related to
starting and managing system services
into a centralized and monolithic
whole: user login, cron jobs, network
services, virtual TTY management
and so on. The use of shell scripts to
control system startup has the benefit
of providing flexibility, and a lot of

78 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 78

2/19/15 9:23 AM

Where every interaction matters.

break down
your innovation barriers
power your business to its full potential
When youre presented with new opportunities, you want to focus on turning
them into successes, not whether your IT solution can support them.

Peer 1 Hosting powers your business with our wholly owned FastFiber NetworkTM,
solutions that are secure, scalable, and customized for your business.
Unsurpassed performance and reliability help build your business foundation to
be rock-solid, ready for high growth, and deliver the fast user experience your
customers expect.

Want more on cloud?

Call: 844.855.6655 | go.peer1.com/linux | Vew Cloud Webinar:

Public and Private Cloud

LJ251-March2015.indd 79

Managed Hosting

Dedicated Hosting

Colocation

2/19/15 9:23 AM

FEATURE Initializing and Managing Services in Linux: Past, Present and Future

members of the community want to

be able to choose their favorite init
scheme. This has spawned some forks
of systemd and even a faction of the
Linux community that is for completely
boycotting systemd. Check out the
site http://boycottsystemd.org.

influenced user choice in this space

over time. I hope this article will foster
further discussion, and your feedback
is highly encouraged. Q
Jonas Gorauskas is technically a software developer by trade
but also a generalist with background in operations. In the past
he has been one or more of the following: programmer, technical

Conclusion
The userspace initialization and
management of Linux systems has
had a rich and diverse history. I hope
that this article has given you a new
perspective for how we got to where
we are today with systemd becoming
the new standard. I have covered all
the pros and cons of the different
schemes and how those factors have

support analyst, technical writer, systems designer, database

administrator, amateur cook and professional curmudgeon.
Jonas is currently working at Intuit in Reno, Nevada, as part of
the Application Operations Engineering team helping them with
operations, deployment, DevOps or anything else they can think of.

Send comments or feedback via

http://www.linuxjournal.com/contact
or to [email protected].

Resources
The source code of various Linux distributions, including:
Q Debian 7 and 8
Q CentOS 6.5 and 7
Q Slackware 14
Q Fedora 20
Q Ubuntu 12.4 and 14.4
Q Arch Linux
The Web site of Lennart Poettering: http://0pointer.net/blog
The systemd Documentation: http://freedesktop.org/wiki/Software/systemd
Upstart Documentation: http://upstart.ubuntu.com/cookbook

80 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 80

2/19/15 9:23 AM

NEW!

Linux Journal
eBook Series
GEEK GUIDES
Slow Down to Speed Up:

Continuous Quality Assurance in a DevOps Environment

FREE

Down
loa
NOW d
!

By Bill Childers
DevOps is one of the newest and largest movements in Information
Technology in the past few years. The name DevOps is a portmanteau
of Development and Operations and is meant to denote a fusion of
these two functions in a company. Whether or not your business actually
does combine the two functions, the lessons and tools learned from the
DevOps movement and attitude can be applied throughout the entire
Information Technology space. This eBook focuses on one of the key
attributes of the DevOps movement: Quality Assurance. At any point,
you should be able to release your product, code or configurationso
long as you continue keeping your deliverables in a deployable state. This is done by slowing
down to include a Quality Assurance step at each point in your workflow. The sooner you catch
an error or trouble condition and fix it, the faster you can get back on track. This will lower the
amount of rework required and keep your teams momentum going in a forward direction,
enabling your group to move on to new projects and challenges.

Build a Private Cloud for Less Than $10,000!

By Mike Diehl
This eBook presents a compelling argument as to why you should
consider re-architecting your enterprise toward a private cloud. It
outlines some of the design considerations that you need to be
aware of before implementing your own private cloud, and it
describes using the DevCloud installer in order to install OpenStack
on an Ubuntu 14 server. Finally, this eBook will familiarize you with
the features and day-to-day operations of an OpenStack-based
private cloud architecture, all for less than $10K!

DOWNLOAD NOW AT: http://linuxjournal.com/geekguides

LJ251-March2015.indd 81

2/19/15 9:23 AM

FEATURE Infinite BusyBox with systemd

Infinite
BusyBox
with
systemd
Lightweight virtual containers
with PID 1.
Charles Fisher

82 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 82

2/19/15 9:23 AM

n this article, I demonstrate

a method to build one Linux
system within another using
the latest utilities within the
systemd suite of management
tools. The guest OS container
design focuses upon BusyBox
and Dropbear for the userspace
system utilities, but I also work
through methods for running more
general application software so the
containers are actually useful.
This tutorial was developed on
Oracle Linux 7, and it likely will run
unchanged on its common brethren
(Red Hat, CentOS, Scientific Linux),
and from here forward, I refer to this
PLATFORM SIMPLY AS 6 3LIGHT CHANGES
may be necessary on other systemd
PLATFORMS SUCH AS 353% $EBIAN OR
5BUNTU  /RACLES 6 RUNS ONLY ON THE
X? PLATFORM SO THATS THIS ARTICLES
primary focus.
Required Utilities
Red Hat saw fit to remove the
long-included BusyBox binary from
ITS 6 DISTRIBUTION BUT THIS EASILY
is remedied by downloading the
latest binary directly from the
projects Web site. Since the /home
filesystem gets a large amount of
space by default when installing
6 LETS PUT IT THERE FOR NOW 2UN
the commands below as root until

indicated otherwise:
cd /home
wget http://busybox.net/downloads/binaries/latest/busybox-x86_64

You also can get a binary copy of

the Dropbear SSH server and client
from this location:
wget http://landley.net/aboriginal/downloads/
binaries/extras/dropbearmulti-x86_64

For this article, I used the

following versions:
Q BusyBox v1.21.1.
Q $ROPBEAR 33( MULTI PURPOSE V

These are static binaries that do not

link against shared objectsnothing
ELSE IS REQUIRED TO RUN THEM AND THEY
are ideal for building a new UNIX-ish
ENVIRONMENT QUICKLY
Build a chroot
The chroot system call and the
associated shell utility allow an
arbitrary subdirectory somewhere
on the system to be declared as
the root for all child processes.
The commands below populate the
chroot jail, then lock you in. Note
that the call to chroot needs your
CHANGE TO THE 3(%,, ENVIRONMENT
WWW.LINUXJOURNAL.COM / MARCH 2015 / 83

LJ251-March2015.indd 83

2/19/15 9:23 AM

FEATURE Infinite BusyBox with systemd

BusyBox changes its behavior depending upon

how it is calledit bundles a whole system of
utility programs into one convenient package.

variable below, as you dont have

bash inside the jail (and its likely the
DEFAULT VALUE OF 3(%,, 
export SHELL=/bin/sh
mkdir /home/nifty
mkdir /home/nifty/bin
cd /home/nifty/bin
cp /home/busybox-x86_64 /home/dropbearmulti-x86_64 .
chmod 755 busybox-x86_64 dropbearmulti-x86_64
./busybox-x86_64 --list | awk '{print "ln -s
busybox-x86_64 " $0}' | sh
chroot /home/nifty
export PATH=/bin
ls -l
###(try some commands)
exit

Take some time to explore your

shell environment after you launch
your chroot above before you
exit. Notice that you have a /bin
directory, which is populated by
soft links that resolve to the
BusyBox binary. BusyBox changes
its behavior depending upon how
it is calledit bundles a whole

system of utility programs into one

convenient package.
Try a few additional UNIX
commands that you may know. Some
that work are vi , uname , uptime
and (of course) the shell that you are
working inside. Commands that dont
work include ps , top and netstat .
4HEY FAIL BECAUSE THEY REQUIRE THE
/proc directory (which is dynamically
provided by the Linux kernel)it has
not been mounted within the jail.
Note that few native utilities will
run in the chroot without moving
many dependent libraries (objects).
You might try copying bash or gawk
into the jail, but you wont be able to
run them (yet). In this regard, BusyBox
is ideal, as it depends upon nothing.
Build a Minimal UNIX System
and Launch It
The systemd suite includes the
eponymous program that runs
as PID 1 on Linux. Among many
other utilities, it also includes the
nspawn program that is used to

84 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 84

2/19/15 9:23 AM

launch containers. Containers that

are created by nspawn fix most of
the problems with chroot jails. They
provide /proc, /dev, /run and otherwise
EQUIP THE CHILD ENVIRONMENT WITH A
more capable runtime.
Next, you are going to configure
a getty to run on the console of the
container that you can use to log in.
Being sure that you have exited your
chroot from the previous step, run the
following commands as root:
mkdir /home/nifty/etc
mkdir /home/nifty/root
echo 'NAME="nifty busybox container"' >
/home/nifty/etc/os-release
cd /home/nifty

that appear in the child container

also appear on the host system, but
different PIDs will be assigned between
the parent and child.
Note that youll also receive the
message: The kernel auditing
subsystem is known to be
incompatible with containers. Please
make sure to turn off auditing with
AUDIT ON THE KERNEL COMMAND
line before using systemd-nspawn.
Sleeping for 5s... The audit
settings dont seem to impact the
BusyBox container login, but you
can adjust your kernel command
line in your grub configuration
(at least to silence the warning and
stop the delay).

ln -s bin sbin
ln -s bin usr/bin
echo 'root::0:0:root:/root:/bin/sh' >
/home/nifty/etc/passwd
echo 'console::respawn:/bin/getty 38400 /dev/console' >
/home/nifty/etc/inittab
tar cf - /usr/share/zoneinfo | (cd /home/nifty; tar xvpf -)
systemd-nspawn -bD /home/nifty

After you have executed the nspawn

above, you will be presented with a
nifty login prompt. Log in as root
(there is no passwordyet), and try a
few more commands. You immediately
will notice that ps and top work, and
there is now a /proc.
You also will notice that the processes

Running Dropbear SSH in

Your Container
Its best if you configure a nonroot user of your system and forbid
network root logins. The reasoning
will become clear when I address
container security.
Run all of these commands as root
within the container:
cd /bin
ln -s dropbearmulti-x86_64 dropbear
ln -s dropbearmulti-x86_64 ssh
ln -s dropbearmulti-x86_64 scp
ln -s dropbearmulti-x86_64 dropbearkey
ln -s dropbearmulti-x86_64 dropbearconvert
WWW.LINUXJOURNAL.COM / MARCH 2015 / 85

LJ251-March2015.indd 85

2/19/15 9:23 AM

FEATURE Infinite BusyBox with systemd

Above, you have established the

names that you need to call Dropbear,
both the main client and server,
and the sundry key generation and
management utilities.
You then generate the host keys that
will be used by this container, placing
them in a new directory /home/nifty/
etc/dropbear (as viewed by the host):
mkdir /etc/dropbear
dropbearkey -t rsa -f /etc/dropbear/dropbear_rsa_host_key
dropbearkey -t dss -f /etc/dropbear/dropbear_dss_host_key
dropbearkey -t ecdsa -f /etc/dropbear/dropbear_ecdsa_host_key

6ARIOUS DIRECTORIES ARE THEN CREATED

that you will need shortly:

echo root:::::::: > /etc/shadow

chmod 600 /etc/shadow
echo root:x:0: > /etc/group
passwd -a x root

Note that the BusyBox passwd

call used here generated an MD5
hashthere is a $1$ prefix in the
second field of /etc/shadow for root.
Additional hashing algorithms are
available from this version of the
passwd utility (the options -a s will
generate a $5$ SHA256 hash, and
-a sha512 will generate a $6$ hash).
However, Dropbear seems to be able
to work only with $1$ hashes for now.
Finally, add a new user to the
system, and then halt the container:

mkdir -p /var/log/lastlog
mkdir /home

adduser -h /home/luser -D luser

mkdir /var/run

passwd -a x luser

mkdir /tmp
mkdir /var/tmp

halt

chmod 01777 /tmp /var/tmp

You then create the inittab, which will

launch syslogd and Dropbear once at
startup (in addition to the existing getty
that is respawned whenever it dies):
echo ::sysinit:/bin/syslogd >> /etc/inittab
echo '::sysinit:/bin/dropbear -w -p 2200' >> /etc/inittab

Next, you add a shadow file and

create a password for root:

You should see container shutdown

messages that are similar to a system halt.
When you next start your container,
it will listen on socket 2200 for
connections. If you want remote
hosts to be able to connect to your
container from anywhere on the
network, run this command as root on
the host to open a firewall port:
iptables -I INPUT -p tcp --dport 2200 --syn -j ACCEPT

86 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 86

2/19/15 9:23 AM

The port will be open only until

you reboot. If youd like the open
port to persist across reboots, use the
firewall-config command from
within the X Window System (set the
port on the second tab in the GUI).
In any case, run the container with
the previous nspawn syntax, then try
to connect from another shell within
the parent host OS with the following:

 BIT LIBRARIES AVAILABLE WITH AN

argument to nspawn that establishes
a bind mount:

ssh -l luser -p 2200 localhost

9OU THEN WILL FIND THAT MANY  BIT

binaries that you copy in from the
host will run (running /bin/gawk -V
RETURNS h'.5 !WK vAN ENTIRE
Oracle 12c instance is confirmed to
run this way). The read-only library
bind mount also has the benefit of
receiving security patches immediately
when they appear on the host.
There is a significant security
problem with this, however. The root
user in the container has the power to

You should be able to log in to the

luser account under a BusyBox shell.
Executing Programs with
Runtime Dependencies
If you copy various system programs
from /bin or /usr/bin into your
container, you immediately will notice
that they dont work. They are missing
shared objects that they need to run.
If you had previously copied the
gawk binary in from the host:
cp /bin/gawk /home/nifty/bin/

you would find that attempts to

execute it fail with gawk: not found
errors (on the host, there usually will
be explicit complaints about missing
shared objects, which are not seen in
the container).
You easily can make most of the

systemd-nspawn -bD /home/nifty --bind-ro=/usr/lib64

Then, from within the container, run:

cd /
ln -s usr/lib64 lib64

mount -o remount,rw /usr/lib64

and, thus, gain write access to your

host library directories. In general,
you cannot give root to a container
user that you dont know and trust
among other problems, these mounts
can be abused.
You also might be tempted to
mount the /usr/lib directory in the
same manner. The difficulty you
will find is that the systemd binary
will be found under that directory
WWW.LINUXJOURNAL.COM / MARCH 2015 / 87

LJ251-March2015.indd 87

2/19/15 9:23 AM

FEATURE Infinite BusyBox with systemd

tree, and nspawn will try to execute

it in preference to BusyBox init.
%NABLING  BIT RUNTIME SUPPORT
likely will involve more directory
and mounting gymnastics than was
REQUIRED FOR USRLIB
And now, Im going off on a tangent.
systemd Service Files
You will need to call on the host PID
1 (systemd) directly to launch your
container in an automated manner,
potentially at boot. To do this, you
need to create a service file.
Because there is a dearth of clear
discussion on moving inittab and
service functions into systemd, Ill
cover all the basic uses before creating
a service file for the container.
Start by configuring a telnet server.
The telnet protocol is not secure, as
it transmits passwords in clear text.
Dont practice these examples on a
production server or with sensitive
information or accounts.
Classical telnetd is launched by
the inetd superserver, both of which
are implemented by BusyBox. Lets
configure inetd for telnet on port
12323. Run the following as root
on the host:
echo '12323 stream tcp nowait root
/home/nifty/bin/telnetd telnetd -i -l
/home/nifty/bin/login' >> /etc/inetd.conf

After the configuring above,

if you manually launch the inetd
contained in BusyBox, you will be
able to telnet to port 12323. Note
THAT THE 6 PLATFORM DOES NOT
include a telnet client by default,
so you either can install it with yum
or use the BusyBox client (which
the example below will do). Unless
you open up port 12323 on your
firewall, you will have to telnet
to localhost.
Make sure any inetd that you
started is shut down before
proceeding to create an inetd
service file below:
echo '[Unit]
Description=busybox inetd
#After=network-online.target
Wants=network-online.target
[Service]
#ExecStartPre=
#ExecStopPost=
#Environment=GZIP=-9
#OPTION 1
ExecStart=/home/nifty/bin/inetd -f
Type=simple
KillMode=process
#OPTION 2
#ExecStart=/home/nifty/bin/inetd
#Type=forking

88 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 88

2/19/15 9:23 AM

#Restart=always

#User=root

Kernel 3.10.0-123.9.3.el7.x86_64 on an x86_64

#Group=root

localhost.localdomain login: jdoe

Password:

[Install]
WantedBy=multi-user.target' >
/etc/systemd/system/inetd.service

Checking the status again, you see

information about the connection and
the session activity:

systemctl start inetd.service

[root@localhost ~]# systemctl status inetd.service

After starting the inet service

above, you can check the status
of the dmon:

inetd.service - busybox inetd

Loaded: loaded (/etc/systemd/system/inetd.service; disabled)
Active: active (running) since Sun 2014-11-16 12:34:04 CST;
7min ago

[root@localhost ~]# systemctl status inetd.service

inetd.service - busybox inetd

Main PID: 3927 (inetd)

CGroup: /system.slice/inetd.service

Loaded: loaded (/etc/systemd/system/inetd.service; disabled)

3927 /home/nifty/bin/inetd -f

Active: active (running) since Sun 2014-11-16 12:21:29 CST;

4076 telnetd -i -l /home/nifty/bin/login

28s ago

4077 -bash

Main PID: 3375 (inetd)

CGroup: /system.slice/inetd.service
3375 /home/nifty/bin/inetd -f

Nov 16 12:21:29 localhost.localdomain systemd[1]: Started

busybox inetd.
Try opening a telnet session from a different console:

/home/nifty/bin/telnet localhost 12323

You should be presented with a

login prompt:
Entering character mode
Escape character is '^]'.

You can learn more about

systemd service files with the man
5 systemd.service command.
There is an important point to make
hereyou have started inetd with the
-f Run in foreground option. This
is not how inetd normally is started
this option is commonly used for
debugging activity. However, if you
were starting inetd with a classical
inittab entry, -f would be useful in
conjunction with respawn. Without
-f , inetd immediately will fork into
the background; attempting to
WWW.LINUXJOURNAL.COM / MARCH 2015 / 89

LJ251-March2015.indd 89

2/19/15 9:23 AM

FEATURE Infinite BusyBox with systemd

respawn forking dmons will launch

them repeatedly. With -f , you can
configure init to relaunch inetd should
it die.
Another important point is stopping
the service. With a foreground
dmon and the KillMode=process
setting in the service file, the child
telnetd services are not killed when
the service is stopped. This is not
the normal, default behavior for
a systemd service, where all the
children will be killed.
To see this mass kill behavior,
comment out the OPTION 1 settings
in the service file (/etc/systemd/
system/inetd.service), and enable
the default settings in OPTION 2 .
Then execute:
systemctl stop inetd.service
systemctl daemon-reload
systemctl start inetd.service

Launch another telnet session, then

stop the service. When you do, your
telnet sessions will all be cut with
Connection closed by foreign host.
In short, the default behavior of
systemd is to kill all the children of a
service when a parent dies.
The KillMode=process setting can
be used with the forking version of
inetd, but the -f Run in foreground
in the first option is more specific and,

thus, safer.
You can learn more about the
KillMode option with the man
5 systemd.kill command.
Note also that the systemctl
status output included the word
disabled. This indicates that
the service will not be started at
boot. Pass the enable keyword to
systemctl for the service to set it
to launch at boot (the disable
keyword will undo this).
Make some note of the
commented options above. You
may set environment variables for
your service (here suggesting a
COMPRESSION QUALITY  SPECIFY A NON ROOT
user/group and commands to be
executed before the service starts or
after it is halted. These capabilities
are beyond the direct features
offered by the classical inittab.
Of course, systemd is capable of
spawning telnet servers directly,
allowing you to dispense with inetd
altogether. Run the following as root
on the host to configure systemd for
BusyBox telnetd:
systemctl stop inetd.service

echo '[Unit]
Description=mytelnet

[Socket]

90 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 90

2/19/15 9:23 AM

ListenStream=12323
Accept=yes

[Install]

to the telnet server indicates

that systemd should not pay
attention to any stats return codes
from the process.

WantedBy=sockets.target' >
/etc/systemd/system/mytelnet.socket

echo '[Unit]
Description=mytelnet

Q In the client telnet sessions, the

command cat /proc/self/
cgroup will return detailed

connection information for the

IP addresses involved.

[Service]
ExecStart=-/home/nifty/bin/telnetd telnetd -i -l
/home/nifty/bin/login
StandardInput=socket' >
/etc/systemd/system/[email protected]

systemctl start mytelnet.socket

At this point, I have returned from

my long-winded tangent, so now lets
build a service file for the container.
Run the following as root on the host:
echo '[Unit]
Description=nifty container

Some notes about inetd-style services:

[Service]

Q The socket is started, rather than

the service, when inetd services are

launched. Similarly, they are enabled
to set them to launch at boot.
Q The @ character in the service file

indicates this is an instantiated

service. They are used when a
number of similar services are
launched with a single service file
(getty being the prime example
they also work well for Oracle
database instances).
Q The - prefix above in the path

ExecStart=/usr/bin/systemd-nspawn -bD /home/nifty

KillMode=process' > /etc/systemd/system/nifty.service

Be sure that you have shut down

any other instances of the nifty
container. You optionally can
disable the console getty by
commenting/removing the first line
of /home/nifty/etc/inittab. Then use PID 1
to launch your container directly:
systemctl start nifty.service

If you check the status of the

service, you will see the same level of
WWW.LINUXJOURNAL.COM / MARCH 2015 / 91

LJ251-March2015.indd 91

2/19/15 9:23 AM

FEATURE Infinite BusyBox with systemd

information that you previously saw

on the console:

[root@localhost ~]# size /home/busybox-x86_64

text

data

bss

dec

hex

942326

29772

19440

991538

f2132

filename
/home/busybox-x86_64

[root@localhost ~]# systemctl status nifty.service

nifty.service - nifty container
Loaded: loaded (/etc/systemd/system/nifty.service; static)
Active: active (running) since Sun 2014-11-16 14:06:21 CST;
31s ago
Main PID: 5881 (systemd-nspawn)
CGroup: /system.slice/nifty.service
5881 /usr/bin/systemd-nspawn -bD /home/nifty

If you want to conserve the memory

used by BusyBox, one way would be
to create a common /cbin that you
attach to all containers as a read-only
bind mount (as you did previously
WITH LIB  AND RESET ALL THE LINKS IN
/bin to the new location. The root user
could do this:

Nov 16 14:06:21 localhost.localdomain systemd[1]: Starting

nifty container...

systemctl stop nifty.service

Nov 16 14:06:21 localhost.localdomain systemd[1]: Started

nifty container.
Nov 16 14:06:26 localhost.localdomain systemd-nspawn[5881]:

mkdir /home/cbin
mv /home/nifty/bin/busybox-x86_64 /home/cbin

Spawning namespace container on /home/nifty

mv /home/nifty/bin/dropbearmulti-x86_64 /home/cbin

(console is /dev/pts/4).

cd /

Nov 16 14:06:26 localhost.localdomain systemd-nspawn[5881]:

Init process in the container running as PID 5883.

ln -s home/cbin cbin
cd /home/nifty/bin
for x in *; do if [ -h "$x" ]; then rm -f "$x"; fi; done

Memory and Disk Consumption

BusyBox is a big program, and if you
are running several containers that
each have their own copy, you will
waste both memory and disk space.
It is possible to share the text
segment of the BusyBox memory
usage between all running programs,
but only if they are running on the
same inode, from the same filesystem.
The text segment is the read-only,
compiled code of a program, and you
can see the size like this:

/cbin/busybox-x86_64 --list | awk '{print "ln -s

/cbin/busybox-x86_64 " $0}' | sh
ln -s /cbin/dropbearmulti-x86_64 dropbear
ln -s /cbin/dropbearmulti-x86_64 ssh
ln -s /cbin/dropbearmulti-x86_64 scp
ln -s /cbin/dropbearmulti-x86_64 dropbearkey
ln -s /cbin/dropbearmulti-x86_64 dropbearconvert

You also could arrange to bindmount the zoneinfo directory,

saving a little more disk space in
the container (and giving the
container patches for time zone

92 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 92

2/19/15 9:23 AM

It might interesting to launch tens, hundreds,

or even thousands of containers at once.

data in the bargain):

cd /home/nifty/usr/share
rm -rf zoneinfo

Then the service file is modified to

bind /cbin and /usr/share/zoneinfo
(note the altered syntax for sharing
/cbin below, when the paths differ
between host and container):
echo '[Unit]
Description=nifty container

[Service]
ExecStart=/usr/bin/systemd-nspawn -bD /home/nifty
--bind-ro=/home/cbin:/cbin --bind-ro=/usr/share/zoneinfo
KillMode=process' > /etc/systemd/system/nifty.service

systemctl daemon-reload

systemctl start nifty.service

Now any container using the

BusyBox binary from /cbin will share
the same inode. All versions of the
BusyBox utilities running in those
containers will share the same text
segment in memory.

Infinite BusyBox
It might interesting to launch tens,
hundreds, or even thousands of
containers at once. You could launch
the clones by making copies of the
/home/nifty directory, then adjusting
the systemd service file. To simplify,
you will place your new containers
in /home/nifty1, /home/nifty2,
/home/nifty3 ... using integer suffixes
on the directories to differentiate them.
Please make sure that you have
disabled kernel auditing to remove
the five-second delay when launching
containers. At the very least, press
e at the grub menu at boot time,
and add the audit=0 to your kernel
command line for a one-time boot.
Im going to return to the subject of
systemd instantiated services that I
touched upon with the telnetd service
FILE THAT REPLACED INETD 4HIS TECHNIQUE
will allow you to use one service
file to launch all of your containers.
Such a service has an @ character
in the filename that is used to refer
to a particular, differentiated instance
of a service, and it allows the use
of the %i placeholder within the
WWW.LINUXJOURNAL.COM / MARCH 2015 / 93

LJ251-March2015.indd 93

2/19/15 9:23 AM

FEATURE Infinite BusyBox with systemd

service file for variable expansion.

Run the following on the host as
root to place your service file for
instantiated containers:

make a thousand of them:

cd /home
for x in $(seq 1 999)
do

echo '[Unit]

mkdir "nifty${x}"

Description=nifty container # %i

(cd nifty; tar cf - .) | (cd "nifty${x}"; tar xpf -)

sed "s/2200/$((x+2200))/" < nifty/etc/inittab >

[Service]
ExecStart=/usr/bin/systemd-nspawn -bD /home/nifty%i
--bind-ro=/home/cbin:/cbin --bind-ro=/usr/share/zoneinfo

nifty${x}/etc/inittab
systemctl start nifty@${x}.service
done

KillMode=process' > /etc/systemd/system/[email protected]

The %i above first adjusts the

description, then adjusts the launch
directory for the nspawn . The content
that will replace the %i is specified on
the systemctl command line.
To test this, make a directory
called /home/niftyslick. The service
file doesnt limit you to numeric
suffixes. You will adjust the SSH
port after the copy. Run this as root
on the host:

As you can see below, this test

launches all containers:
$ ssh -l luser -p 3199 localhost
The authenticity of host '[localhost]:3199 ([::1]:3199)'
can't be established.
ECDSA key fingerprint is 07:26:15:75:7d:15:56:d2:ab:9e:
14:8a:ac:1b:32:8c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:3199' (ECDSA)
to the list of known hosts.
luser@localhost's password:
~ $ sh --help

cd /home

BusyBox v1.21.1 (2013-07-08 11:34:59 CDT) multi-call binary.

mkdir niftyslick
(cd nifty; tar cf - .) | (cd niftyslick; tar xpf -)
sed "s/2200/2100/" < nifty/etc/inittab > niftyslick/etc/inittab

systemctl start [email protected]

Bearing this pattern in mind, lets

create a script to produce these
CONTAINERS IN MASSIVE QUANTITIES ,ETS

Usage: sh [-/+OPTIONS] [-/+o OPT]... [-c 'SCRIPT'

[ARG0 [ARGS]] / FILE [ARGS]]

Unix shell interpreter

~ $ cat /proc/self/cgroup
10:hugetlb:/
9:perf_event:/

94 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 94

2/19/15 9:23 AM

8:blkio:/

2882 /bin/dropbear -w -p 2210

7:net_cls:/
6:freezer:/

Nov 18 23:01:21 localhost.localdomain systemd[1]:

5:devices:/

Starting Container nifty10.

4:memory:/

Nov 18 23:01:21 localhost.localdomain systemd[1]:

3:cpuacct,cpu:/

Started Container nifty10.

2:cpuset:/
1:name=systemd:/machine.slice/machine-nifty999.scope

The output of systemctl will list each

of your containers:
# systemctl
...
machine-nifty1.scope

loaded active running

Container nifty1

machine-nifty10.scope

loaded active running

Container nifty10

machine-nifty100.scope

loaded active running

Container nifty100

machine-nifty101.scope

loaded active running

Container nifty101

machine-nifty102.scope

loaded active running

Container nifty102

The raw number of containers that

you can launch with this approach
is more directly impacted by kernel
limits than general disk and memory
resources. Launching the containers
above used no swap on a small system
with 2GB of RAM.
After you have investigated a few
of the containers and their listening
ports, the easiest and cleanest way to
get all of your containers shut down is
likely a reboot.

...

More detail is available with

systemctl status :
machine-nifty10.scope - Container nifty10
Loaded: loaded (/run/systemd/system/machine-nifty10.scope;
static)
Drop-In: /run/systemd/system/machine-nifty10.scope.d
90-Description.conf, 90-Slice.conf,
90-TimeoutStopUSec.conf
Active: active (running) since Tue 2014-11-18 23:01:21 CST;
11min ago
CGroup: /machine.slice/machine-nifty10.scope
2871 init
2880 /bin/syslogd

Container Security
A number of concerns are raised with
these features:
1) Since BusyBox and Dropbear
were not installed with the RPM host
package tools, updates to them will
have to be loaded manually. It will
be important to check from time to
time if new versions are available
and if any security flaws have been
discovered. If it is necessary to load
new versions, the binaries should
be copied to all containers that are
potentially used, which should then
be restarted (especially if a security
WWW.LINUXJOURNAL.COM / MARCH 2015 / 95

LJ251-March2015.indd 95

2/19/15 9:23 AM

FEATURE Infinite BusyBox with systemd

The crux is that untrusted users cannot have

the container root, any more than you would
give them full system root.

issue is involved).
2) Control of the root user in the
container cannot be passed to an
individual that you do not trust.
For a particular example, if the
LIBCBINZONEINFO BIND MOUNTS
above are used, the container root
user can issue the command:
mount -o remount,rw /usr/lib64

at which point the container root

will have full write privileges on your
 BIT LIBRARIES CONTAINER BIN OR ZONEINFO
The systemd-nspawn man page goes
even further, with the warning:
Note that even though these
security precautions are taken
systemd-nspawn is not suitable
for secure container setups. Many
of the security features may be
circumvented and are hence
primarily useful to avoid accidental
changes to the host system from
the container. The intended use
of this program is debugging

and testing as well as building

of packages, distributions and
software involved with boot and
systems management.

The crux is that untrusted users

cannot have the container root, any
more than you would give them full
system root. The container root will
have the CAP_SYS_ADMIN privilege,
which allows full control of the
system. If you want to isolate
non-root users further, the container
environment does limit non-root
users visibility into host activities, as
they cannot see the full process table.
3) Note that the BusyBox su and
passwd utilities above do not work
when installed in the manner outlined
here. They lack the appropriate
filesystem permissions. To fix this,
chmod u+s busybox-x86_64

could be executed, but this is also

distasteful from a security perspective.
Removing the links and copying the
BusyBox binary to su and passwd
before applying the setuid privilege

96 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 96

2/19/15 9:23 AM

might be better, but only slightly. It

would be best if su was unavailable
and another mechanism was found for
password changes.
 4HE -w argument to the Dropbear
SSH server above prevents root logins
from the network. It is somewhat
distasteful, from a security perspective,
to relax this limitation. The net effect
is that root is locked out of active use
in the container when -w is forced,
and su/passwd do not have setuid. If
it is at all possible to live with such an
arrangement for your container, try to
do so, as the security is much improved.
systemd Controversy
There is a high degree of hostility
toward systemd from users of
Linux. This hostility is divided into
two main complaints:
Q The classic inittab from UNIX

3YSTEM 6 SHOULD NOT BE CHANGED

because it is well understood.

6 INIT IS NOT ABLE TO NSPAWN AND HAS

far less control over processes running
on a system. The features delivered by
systemd surely justify the inconvenience
of change in many situations.
Toward the second point, much
thought was placed into the adoption
of the architecture of systemd
by skilled designers from diverse
organizations. Those most critical
of the new environment should
acknowledge the technical success
of systemd as it is adopted by the
majority of the Linux community.
In any case, the next decade will
see popular Linux server distributions
EQUIPPED WITH SYSTEMD AND COMPETENT
administrators will not have the option
of ignoring it. It is unfortunate that the
introduction of systemd did not include
more guidance for the user community,
but the new features are compelling
and should not be overlooked. Q
Charles Fisher has an electrical engineering degree from
the University of Iowa and works as a systems and database

Q Increasing features are bundled

into systemd that bring

dangerous complexity to a
critical system process.
Toward the first point, nostalgia for
legacy systems is not always misguided,
but it cannot be allowed to hinder
progress unreasonably. A classic System

administrator for a Fortune 500 mining and manufacturing

corporation. He has previously published both journal articles
and technical manuals on Linux for UnixWorld and other
McGraw-Hill publications.

Send comments or feedback via

http://www.linuxjournal.com/contact
or to [email protected].
WWW.LINUXJOURNAL.COM / MARCH 2015 / 97

LJ251-March2015.indd 97

2/19/15 9:23 AM

KNOWLEDGE HUB
WEBCASTS
Learn the 5 Critical Success Factors to Accelerate
IT Service Delivery in a Cloud-Enabled Data Center
Today's organizations face an unparalleled rate of change. Cloud-enabled data centers are increasingly seen as a way to accelerate
IT service delivery and increase utilization of resources while reducing operating expenses. Building a cloud starts with virtualizing
your IT environment, but an end-to-end cloud orchestration solution is key to optimizing the cloud to drive real productivity gains.

> http://lnxjr.nl/IBM5factors

Modernizing SAP Environments with Minimum

Riska Path to Big Data
Sponsor: SAP | Topic: Big Data
)S THE DATA EXPLOSION IN TODAYS WORLD A LIABILITY OR A COMPETITIVE ADVANTAGE FOR YOUR BUSINESS %XPLOITING MASSIVE AMOUNTS
of data to make sound business decisions is a business imperative for success and a high priority for many firms. With rapid
advances in x86 processing power and storage, enterprise application and database workloads are increasingly being moved
from UNIX to Linux as part of IT modernization efforts. Modernizing application environments has numerous TCO and ROI
benefits but the transformation needs to be managed carefully and performed with minimal downtime. Join this webinar to
HEAR FROM TOP )$# ANALYST 2ICHARD 6ILLARS ABOUT THE PATH YOU CAN START TAKING NOW TO ENABLE YOUR ORGANIZATION TO GET THE
benefits of turning data into actionable insights with exciting x86 technology.
> http://lnxjr.nl/modsap

WHITE PAPERS
White Paper: JBoss Enterprise Application
Platform for OpenShift Enterprise
Sponsor: DLT Solutions
2ED (ATS *"OSS %NTERPRISE !PPLICATION 0LATFORM FOR /PEN3HIFT %NTERPRISE OFFERING PROVIDES )4 ORGANIZATIONS WITH A SIMPLE AND
STRAIGHTFORWARD WAY TO DEPLOY AND MANAGE *AVA APPLICATIONS 4HIS OPTIONAL /PEN3HIFT %NTERPRISE COMPONENT FURTHER EXTENDS
THE DEVELOPER AND MANAGEABILITY BENEFITS INHERENT IN *"OSS %NTERPRISE !PPLICATION 0LATFORM FOR ON PREMISE CLOUD ENVIRONMENTS
5NLIKE OTHER MULTI PRODUCT OFFERINGS THIS IS NOT A BUNDLING OF TWO SEPARATE PRODUCTS *"OSS %NTERPRISE -IDDLEWARE HAS BEEN
HOSTED ON THE /PEN3HIFT PUBLIC OFFERING FOR MORE THAN  MONTHS !ND MANY CAPABILITIES AND FEATURES OF *"OSS %NTERPRISE
Application Platform 6 and JBoss Developer Studio 5 (which is also included in this offering) are based upon that experience.
This real-world understanding of how application servers operate and function in cloud environments is now available in this
SINGLE ON PREMISE OFFERING *"OSS %NTERPRISE !PPLICATION 0LATFORM FOR /PEN3HIFT %NTERPRISE FOR ENTERPRISES LOOKING FOR CLOUD
benefits within their own datacenters.
> http://lnxjr.nl/jbossapp

98 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 98

2/19/15 9:23 AM

KNOWLEDGE HUB
WHITE PAPERS
Linux Management with Red Hat Satellite:
Measuring Business Impact and ROI
Sponsor: Red Hat | Topic: Linux Management
Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT
organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility
workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows
IN IMPORTANCE IN TERMS OF VALUE TO THE BUSINESS MANAGING ,INUX ENVIRONMENTS TO HIGH STANDARDS OF SERVICE QUALITY
AVAILABILITY SECURITY AND PERFORMANCE BECOMES AN ESSENTIAL REQUIREMENT FOR BUSINESS SUCCESS
> http://lnxjr.nl/RHS-ROI

Standardized Operating Environments

for IT Efficiency
Sponsor: Red Hat
4HE 2ED (AT 3TANDARD /PERATING %NVIRONMENT 3/% HELPS YOU DEFINE DEPLOY AND MAINTAIN 2ED (AT %NTERPRISE ,INUX
AND THIRD PARTY APPLICATIONS AS AN 3/% 4HE 3/% IS FULLY ALIGNED WITH YOUR REQUIREMENTS AS AN EFFECTIVE AND MANAGED
process, and fully integrated with your IT environment and processes.
Benefits of an SOE:
3/% IS A SPECIFICATION FOR A TESTED STANDARD SELECTION OF COMPUTER HARDWARE SOFTWARE AND THEIR CONFIGURATION FOR USE
ON COMPUTERS WITHIN AN ORGANIZATION 4HE MODULAR NATURE OF THE 2ED (AT 3/% LETS YOU SELECT THE MOST APPROPRIATE
solutions to address your business' IT needs.
SOE leads to:
s $RAMATICALLY REDUCED DEPLOYMENT TIME
s 3OFTWARE DEPLOYED AND CONFIGURED IN A STANDARDIZED MANNER
s 3IMPLIFIED MAINTENANCE DUE TO STANDARDIZATION
s )NCREASED STABILITY AND REDUCED SUPPORT AND MANAGEMENT COSTS
s 4HERE ARE MANY BENEFITS TO HAVING AN 3/% WITHIN LARGER ENVIRONMENTS SUCH AS
s ,ESS TOTAL COST OF OWNERSHIP 4#/ FOR THE )4 ENVIRONMENT
s -ORE EFFECTIVE SUPPORT
s &ASTER DEPLOYMENT TIMES
s 3TANDARDIZATION
> http://lnxjr.nl/RH-SOE

WWW.LINUXJOURNAL.COM / MARCH 2015 / 99

LJ251-March2015.indd 99

2/19/15 9:23 AM

EOF
Resurrecting
the Armadillo

DOC SEARLS

Fifteen years after giving the world a pile of hopefully helpful

memes, Cluetrain rides again.
1999 was a crazy year for
business on the Internet,
and for Linux. It was when Red
Hat went public, with a record
VALUATION AND 6! ,INUX FOLLOWED
with a bigger one. Both were cases
in point of the dot-com boom, a
speculative bubble inflated by huge
expectations of what the Internet
would mean for business.
In April of that year, Chris
Locke, Rick Levine, David
Weinberger and I put up a Web site
called The Cluetrain Manifesto
(http://cluetrain.com), attempting
to make clear that the Internet was
for everybody and everything, and
not just for companies looking to
exit into wealth on Wall Street.
Our bulls-eye was marketing,
which spoke about users in terms
that were barely human. We are
not seats or eyeballs or end users or

consumers, Cluetrain said. We are

human beings and our reach exceeds
your grasp. Deal with it.
We addressed Cluetrain to People
OF %ARTHv BUILT IT AROUND  THESES
(because that worked for Martin
Luther) and called it a manifesto
(because that worked for Karl Marx).
The Cluetrain name came from an
OLD 3ILICON 6ALLEY PUT DOWN h4HE
clue train stopped there four times a
day and they never took delivery.
)T WAS A HIT 6OLUNTEERS TRANSLATED
it into 13 languages. The Wall Street
Journal covered it on the front page
of its Marketplace section. Book
offers came in. We accepted one
and wrote the book version of The
Cluetrain Manifesto that summer. It
came out in January 2000 and was
a business bestseller, even though it
could also be read for free on-line at
the Cluetrain Web site. It went on

100 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 100

2/19/15 9:23 AM

EOF

New Clues was tuned for our timeone in which

Internet usage has been migrating from wired to
cellular connections, from the Web to apps, and
from the Nets wide open spaces to the closed
and proprietary walled gardens of Facebook,
Twitter, Apple, Google and other feudal lords.
to be published in nine languages,
and still sells at a nice clip, 15
years later. Same goes for a 10th
anniversary edition that came
out in 2010.
Today the word cluetrain, which
didnt exist before 1999, appears in
thousands of books and is tweeted
many times per day. One-liners
from its list of thesesMarkets
are conversations, Hyperlinks
SUBVERT HIERARCHYvARE QUOTED
endlessly. Not bad for an project that
had no promotion, no budget, no
conferences, no bumper stickers,
no t-shirts and no interest in
becoming a business or an
institution. It was just a bunch of
ideas people could put to use.
The biggest irony of Cluetrains
success is that most books that
cite it are marketing books, and
most tweets about it seem to be
by marketing people. Is marketing
better because of it? To some

degree, I suppose. But I dont care.

Nor do I care that Cluetrain is often
credited with having something
to do with social media. What I
care about is that Cluetrain hasnt
yet succeeded at its main mission:
to make clear that the Internet is
something more than the pipes
we get it from, the content we
find there, and the companies and
governments that would have us
think they run the thing.
3O ON !UGUST   WHEN
somebody pointed me to yet another
Cluetrain story that failed to grok
what it was really about, I wrote
this to the other three guys: I feel
an urge to publish something that
says The Cluetrain Manifesto was
not about clearing the way for social
media. David Weinberger wrote
back, Anyone ready for a new
manifesto? A few minutes later,
he shared the first draft of one: a
collection of theses, similar to the
WWW.LINUXJOURNAL.COM / MARCH 2015 / 101

LJ251-March2015.indd 101

2/19/15 9:23 AM

EOF

original in style and length. Chris

Locke followed with a wordless
image of a woman gleefully shooting
thumbs-up images out of a machine
gun. The rest of the back-and-forth
was between David and me. (Chris
and Rick stayed busy with other
things.) The result was New Clues,
which went up on January 8, 2015,
at the original Cluetrain site:
http://cluetrain.com/newclues.
New Clues was tuned for our
timeone in which Internet usage
has been migrating from wired
to cellular connections, from the
Web to apps, and from the Nets
wide open spaces to the closed
and proprietary walled gardens of
Facebook, Twitter, Apple, Google
and other feudal lords. An organby-organ body snatch of the Internet
is already well underway, it says in
the preamble. Make no mistake:
with a stroke of a pen, a covert
handshake, or by allowing memes to
drown out the cries of the afflicted
we can lose the Internet we love.
One hundred and twenty one
numbered clues follow, under
thematic subheads:

Q The Internet is not content.

Q The Net is not a medium.
Q How did we let conversation get

weaponized anyway?
Q Marketing still makes it harder to talk.
Q Kumbiyah sounds surprisingly good

in an echo chamber.
Q The Gitmo of the Net.
Q Gravitys great until it sucks us all

into a black hole.

Q Privacy in an age of spies.
Q Privacy in an age of weasels.
Q A pocket full of homilies.
Q Being together: the cause of and

solution to every problem.

We wanted to make New Clues, and
every piece of it, as useful, mixable
and remixable as possible. So:
Q %VERY SUBHEAD AND EVERY CLUE HAS A

Q The Internet is us, connected.

Q The Internet is nothing and has

no purpose.

link of its own.

Q The text is released to the public

domain with a Creative Commons

102 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 102

2/19/15 9:23 AM

EOF

We wanted to make New Clues, and every piece of

it, as useful, mixable and remixable as possible.

:ERO  5NIVERSAL ## 

Public Domain Dedication. In
other words, no copyright at
all. When asked for permission
to republish New Clues, David
replies, You dont have our
permission. Go ahead!

the great Steven Levy):

https://medium.com/backchannel/
internet-under-fire-gets-newmanifests-207a922b459e.
Q A version by Kevin Marks that

accepts fragmentations and

webmentions.

Q The whole thing is on GitHub,

with machine-readable versions

(JSON, XML and OPML, so far). The
GitHub folks also have offered to
set up a way for us to maintain a
single data file (YAML) that will
automatically create all the other
versions we need.
The results so far (Im writing this a
week after it went up) include:
Q A listicle that Dave W iner hacked

together on his own software,

which he improved in the course
of posting it. (Some people like
the listicle version better than
the one-page text one. Try it
out: http://scripting.com.)
Q An artful posting on Backchannel

at Medium (by invitation from

Q A site by John Johnston that

randomly generates one clue per

click: http://johnjohnston.info/
oddsandends/givemeaclue.
Q Thousands of tweets and re-tweets

(hashtags: #cluetrain #newclues).

Q 6OLUNTEER TRANSLATIONS INTO 'ERMAN

Italian, Italian (yes, there are two

different ones), Catalan and French
(see Resources).
Q Lots of great blog posts, such

as this one by JP Rangaswami:

http://confusedofcalcutta.com/
2015/01/11/new-clues-calling-oneveryone-to-be-dutiful-individuals.
Q A Gillmor Gang devoted to

to the subject, with David and

WWW.LINUXJOURNAL.COM / MARCH 2015 / 103

LJ251-March2015.indd 103

2/19/15 9:23 AM

EOF

Now the question is, Will it work? Or will it be,

like so much else that gets published on the Web,
snow on the water?
myself: http://techcrunch.com/2015/
01/10/gillmor-gang-kind-of-clue.
Q A discussion page on Facebook:

https://www.facebook.com/
login.php?next=https%3A%2F%2F
www.facebook.com%2Fgroups%
2Fnewclues%2F.
Our only common design element
between Cluetrain and New Clues
is an armadillo. On Cluetrains index
page is the image shown in Figure 1
of a flattened armadillo in the
middle of a road, painted over with
yellow divider lines. The provenance
of the photo is unknown to us.
Chris Locke found it somewhere,
and nobody has ever stepped
forward to claim it.
The one for New Clues is shown in
Figure 2. It was posted by e. res on
Flickr and made useful by a Creative
Commons Attribution 2.0 Generic
(CC BY 2.0) license. (The shot was
taken at Alki Beach in Seattle, the
town where Linux Journal was born.)
For New Clues, we cropped the shot
and made it black and white. For his

Figure 1. Armadillo from Cluetrains

Index Page
listicle version, Dave Winer kept the
color but darkened it.
Among the few criticisms of New
Clues is that its not so disruptive
as Cluetrain was. For the last few
YEARS 3ILICON 6ALLEY HAS BEEN SO GAGA
over disruption that it even has a
conference named after it. The term

104 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 104

2/19/15 9:23 AM

comes from Clayton Christensens

work on disruptive innovation, which
is defined as a process by which a
product or service takes root initially
in simple applications at the bottom
of a market and then relentlessly
moves up market, eventually displacing
established competitors. This can
apply to ideas as well as technologies.
Cases in point: free software and open
source. Both of which, of course,
informed Cluetrain and New Clues.
.OW THE QUESTION IS Will it work?
Or will it be, like so much else that
gets published on the Web, snow on
the water? Cant say, so soon after
its published. But the two publishing
dates, a decade and a half apart, came
at very different times on the Web, and
we did our best to leverage both.
In 1999, the Web was a collection
of almost physical places. You put up
or built Web sites on domains with
locations that people visited and
browsed. Search engines might take
days or weeks to index a page. But
then, after blogs came along, with
syndication through RSS, what my son
Allen in 2003 described as the Live
Web began to emerge. Technorati
and other search engines for live stuff
appeared. My October 2005 column
in Linux Journal was titled The World
Live Web. In it I said the Live Web was
about time and people, rather than

Advertiser
Index
Thank you as always for supporting our
advertisers by buying their products!

ADVERTISER

URL

PAGE #

Drupalize.me

http://drupalize.me

%MBEDDED ,INUX
Conference

HTTPEVENTSLINUXFOUNDATIONORG
events/embedded-linux-conference



%MPEROR,INUX

HTTPWWWEMPERORLINUXCOM



HPC Wallstreet

http://www.flaggmgmt.com/linux

31

Libre Planet 2015

https://libreplanet.org/2015/

21

LinuxFest Northwest

http://linuxfestnorthwest.org/2015

65

Netgate

http://www.netgate.com

/g2EILLY 3OFTWARE
Architecture Conference

HTTPOREILLY#BB+)



Peer 1 Hosting

http://go.peer1.com/linux

79

Silicon Mechanics

http://www.siliconmechanics.com

32%CON

HTTPSWWWUSENIXORGCONFERENCE 
srecon15

6AULT

HTTPEVENTSLINUXFOUNDATIONORG
events/vault

108



ATTENTION ADVERTISERS
The Linux Journal brands following has
grown to a monthly readership nearly
one million strong. Encompassing the
magazine, Web site, newsletters and
much more, Linux Journal offers the
ideal content environment to help you
reach your marketing objectives. For
more information, please visit
http://www.linuxjournal.com/advertising.

WWW.LINUXJOURNAL.COM / MARCH 2015 / 105

LJ251-March2015.indd 105

2/19/15 9:23 AM

EOF

Figure 2. New Clues Armadillo

sites and content. This was a year
before Twitter and Facebook took off,
and search engines time-to-index was
reduced nearly to zero. Sites today are
geysers of content, and the Live Web
is a giant short-attention-span theater.
What shows there is hyper-social and
temporary in the extreme, made more
for sharing than for saving.
This is why New Clues is a collection
of stuff for sharing, presented atop an
old chunk of bedrock, which is what
Cluetrain has become. If we had only
put it on Medium, or on Facebook,
it would have come and gone in a

week. As buzz, thats pretty much

what it did. But as durable food for
re-thinking what the Net is, and how
we might lose it, maybe it will have
lasting effects. Sure hope so. Q
Doc Searls is Senior Editor of Linux Journal. He is also a
fellow with the Berkman Center for Internet and Society at
Harvard University and the Center for Information Technology
and Society at UC Santa Barbara.

Send comments or feedback via

http://www.linuxjournal.com/contact
or to [email protected].

106 / MARCH 2015 / WWW.LINUXJOURNAL.COM

LJ251-March2015.indd 106

2/19/15 9:23 AM

EOF

Resources
Dot-Com Bubble: http://en.wikipedia.org/wiki/Dot-com_bubble
Chris Locke: http://rageboy.com
Rick Levine: https://twitter.com/ricklevine
David Weinberger: http://www.hyperorg.com/blogger
The Cluetrain Manifesto: http://cluetrain.com
The entire original text of The Cluetrain Manifesto: http://cluetrain.com/book
What The Cluetrain Manifesto Teaches Us On Social Media...11 Years Later:
http://visionarymarketing.com/en/blog/2010/02/what-the-cluetrain-manifesto-teaches-uson-social-media-11-years-later
The Cluetrain Legacy and Social Media: http://www.chrisg.com/cluetrain-social-media
Invasion of the Body Snatchers: http://en.wikipedia.org/wiki/Invasion_of_the_Body_Snatchers
CC0 1.0 Universal (CC0 1.0) Public Domain Dedication:
http://creativecommons.org/publicdomain/zero/1.0
e. res on Flickr: https://www.flickr.com/photos/iamtheloop
Creative Commons Attribution 2.0 Generic License: https://creativecommons.org/licenses/by/2.0
Backchannel: New Clues:
https://medium.com/backchannel/internet-under-fire-gets-new-manifests-207a922b459e
Steven Levy: http://www.stevenlevy.com
John Johnston: http://johnjohnston.info
German Translation: http://conceptbakery.de/blog/2015/01/11/new-clues-deutscheuebersetzung-die-neuen-thesen-von-den-verfassern-des-cluetrain-manifest
Italian Translation 1:
https://medium.com/bee-free-the-social-bee/cluetrain-15-anni-dopo-9d6b4def4d57
Italian Translation 2: https://medium.com/@nuovetesi/nuove-tesi-4a1def360351
Catalan Translation: https://ca.wikisource.org/wiki/New_clues
New Clues: Calling on everyone to be Dutiful Individuals by JP Rangaswami:
http://confusedofcalcutta.com/2015/01/11/new-clues-calling-on-everyone-to-be-dutiful-individuals
Gillmor Gang: Kind of Clue: http://techcrunch.com/2015/01/10/gillmor-gang-kind-of-clue
Facebook Discussion Page: https://www.facebook.com/login.
php?next=https%3A%2F%2Ffanyv88.com%3A443%2Fhttps%2Fwww.facebook.com%2Fgroups%2Fnewclues%2F
Disrupt Conference: http://techcrunch.com/event-type/disrupt
Clayton Christensen: http://www.claytonchristensen.com
Disruptive Innovation: http://www.claytonchristensen.com/key-concepts
Snow on the Water: http://blogs.law.harvard.edu/doc/2014/08/03/snow-on-the-water
The World Live Web by Doc Searls in the October 2005 issue of LJ:
http://www.linuxjournal.com/article/8549

WWW.LINUXJOURNAL.COM / MARCH 2015 / 107

LJ251-March2015.indd 107

2/19/15 9:23 AM

Instant Access to Premium

Online Drupal Training
Instant access to hundreds of hours of Drupal
training with new videos added every week!
Learn from industry experts with real world
H[SHULHQFHEXLOGLQJKLJKSUROHVLWHV
Learn on the go wherever you are with apps
for iOS, Android & Roku
We also offer group accounts. Give your
whole team access at a discounted rate!

Learn about our latest video releases and

RIIHUVUVWE\IROORZLQJXVRQ)DFHERRNDQG
7ZLWWHU#GUXSDOL]HPH
Go to http://drupalize.me and
get Drupalized today!

LJ251-March2015.indd 108

2/19/15 9:23 AM