Continuous Integration using Jenkins

Creating More Robust Code,

Developing More Rapidly,
Improving Testing,
Deploying More Reliably,
Gaining Confidence,
and
Saving Money

2012-05-14
Chris Shenton
@shentonfreude
[email protected]
http://koansys.com/tech/jenkins/

We're a small consulting company with Federal, non-profit and startup clients.
We're UNIX guys who do python, pyramid, mongodb, plone, etc.
 What’s Continuous
Integration?
Continuous Integration is a software development
practice where members of a team integrate their
work frequently, usually each person integrates at
least daily - leading to multiple integrations per day.
Each integration is verified by an automated build
(including test) to detect integration errors as
quickly as possible. Many teams find that this
approach leads to significantly reduced integration
problems and allows a team to develop cohesive
software more rapidly.
-- Martin Fowler

Nowadays, developers tend to commit code checkins much more frequently, so se can checkout and build many times
throughout the day.
 Continuous Integration:
Some Free Systems

• Cruise Control
• BuildBot
• Hudson/Jenkins

Cruise Control and Jenkins are Java, BuildBot is Python: they all can build projects written in any language. I find Jenkins to
be the easiest to setup, and have the most active developer community
 Hudson replaced by Jenkins

Dismissed by Forked by
Oracle Community
Written by Kosuke Kawaguchi while at Sun
Since the split in November:
* 4x increase in commits and fixed tickets
* Over 13,000 downloads a week
* 1500 users on mailing list, 4000 followers on twitter
 How Developers Code
while ( not done ) {
write some code;
test by poking at it tediously;
fix the bugs you see;
}
deploy;
cross your fingers nobody finds new bugs;
return;
Isn’t There Enough
World Suffering??
 How to Improve It?
• What if we could hire a new developer?
• Make them test all day long
• Every time we make a code change
• Track and report progress
• Give metrics, graphs on improvements
• Report failures quickly
 That New Guy is Jenkins

• Doesn’t get bored

• Doesn’t draw a salary
• Tells you about problems it finds
• Doesn’t make fun of you or your bugs
• http://jenkins-ci.org/
• https://github.com/jenkinsci/jenkins
Being non-judgmental is surprisingly important for developers
Basic Continuous Integration
• Check the code repo for changes every
few minutes
• Build or compile the code
• the build must be automate-able
• Run your tests: unit, regression, etc
• of course you have to have tests to run!
• Alert if problems
• Gather metrics if appropriate
 Jenkins Benefits
• Never gets bored doing builds and tests
• Catches problems fast: rapid feedback
• Alerts developers while code is fresh in
their minds
• Prevents bugs from propagating
downstream
• Cheaper to fix bugs earlier, before QA
or Deployment
Importance of Rapid Feedback, as Rebecca Parsons said
 Expensive,
Hard to Install
 Expensive,
Hard to Install
NOT!
 Expensive,
Hard to Install
NOT!
• Free download (9 seconds)
• $ curl -L -O http://mirrors.jenkins-ci.org/war/latest/jenkins.war

• Run it (18 seconds until ready)

• $ java -jar jenkins.war [ --httpPort=4321 ]

• Configure though the web

• http://localhost:4321/

• Profit!
For production use, youʼd put this behind Apache or Tomcat or something,
and youʼd want to have access controls on it,
or use Jenkinsʼ built-in user manager
 Distributed Builds

• Master + slave nodes

• Start Master-only, add slaves later
• Slaves launched by ssh, WMI+DCOM,
custom script, Java Web Start (JNLP)
• Slaves can be different platforms
• Allows build/testing on different OSes

Can use JNLP on Slaves inside a firewall to connect to a publicly-visible Master.

The JNLP mechanism can be done via Slave-side web browser or even headless.
We're setting up an OpenStack cloud so Jenkins can spin up different VMs and build/test on them.
 Alerts by Mail, RSS
Alerts and Commands by IM
Chris Shenton
!status
Jenkins ContinuousIntegration
status of all projects:
pyrajenkins: last build: 6 (2 days 12 hr ago): SUCCESS: http://localhost:8080/job/pyrajenkins/6/
webcompare: last build: 17 (2 days 20 hr ago): FAILURE: http://localhost:8080/job/webcompare/17/

Chris Shenton
!health
Jenkins ContinuousIntegration
health of all projects:
pyrajenkins: Health [Build stability: 1 out of the last 5 builds failed.(80%), Cobertura Coverage: 72%
(78/108) Lines(90%), Test Result: 0 tests failing out of a total of 7 tests.(100%): http://localhost:8080/
job/pyrajenkins/6/
webcompare: Health [Build stability: 2 out of the last 5 builds failed.(60%): http://localhost:8080/job/
webcompare/17/

Jenkins ContinuousIntegration
Project pyrajenkins build #7: FAILURE in 24 sec: http://localhost:8080/job/pyrajenkins/7/
Chris Shenton: Add support for pylint so Jenkins can yell at me

Jenkins ContinuousIntegration
Yippie, build fixed!
Project pyrajenkins build #9: FIXED in 5.5 sec: http://localhost:8080/job/pyrajenkins/9/

I find the IM integration to be flakey sometimes, the first command works, subsequent ones get no response; might be
a bug (see the tracker), might be network/firewall, or could be me.
 Helpful Plugins
• Subversion: code repo checkout
• Git: code repo checkout, build all branches
• Cobertura: test coverage reporter
• Maven: many plugins available
• Various style violations checkers
• Continuous Integration Game: high scores
• Chuck Norris: motivational help :-)
 Over 300 Plugins
• Authentication: LDAP, AD, OpenID, SQL, ...
• Trac, GitHub, Jira, Redmine: trackers
• Selenium, Sauce, Windmill: functional,
click testing

• Libvert, VirtualBox, VMware: test in VM

• Xvnc: UI testing
• Android emulation

This large ecosystem is a sign of a healthy developer community

 Fun With
Extreme Feedback

• Ambient Orbs
• Lava Lamps
• Code Smells

Or Growl or Twitter or Sound files or ...

 Immediate Benefits

• Easy to put projects under Jenkins

• Find build problems, quicker to fix
• Find test breakage, quicker to fix
• Reports on increasing test coverage
• Motivation to increase test coverage

At a recent sprint, we had a bunch of coders all working on one project. If any of them committed code that broke the
build, Jenkins alerted us immediately, the responsible developer was notified, and they could fix the problem before
other developers were affected. This was a big help, and avoided a lot of frustration.
 Critical Prerequisites:
Automated Build

• No human intervention allowed

• ant, maven
• setup.py, pip, buildout
• (every language has one)
 Critical Prerequisites:
Code that Tests Your Code
• Faster than clicking around your app
• Can run without firing up the whole
stack (web framework, security, ...)
• unit tests: junit, nose, ...
• functional tests: selenium, abott, ...
• Run locally before checking in your code:
“don’t break the build”

Philip von Weitershausen: Untested code is broken code.

Chris McDonough: 100% test coverage is the LEAST you can do.
 Continuous Deployment

• Relies on extensive test suite

• Deploy to production when all pass
• Phased deployment to subset of servers
• Monitor for problems
• Mechanism to roll-back
• You’ve got to ask yourself one question:
Do I feel lucky? Well, do ya, punk?
Rebecca Parsons: "Delivery is where you get the value"
 Simple Unit Tests
Can Save Hours Debugging
class TestUrlManglers(unittest.TestCase):

def test_normalize_url(self):

from webcompare import Walker

w = Walker("fakeurl", "fakeurl”, ignoreres=['\\?.*', '#.*', '/RSS.*', '<bound.*'])

self.assertEquals(w._normalize_url("http://example.com/something"),
"http://example.com/something")

self.assertEquals(w._normalize_url("http://example.com/something/RSS"),
"http://example.com/something")

self.assertEquals(w._normalize_url("http://example.com?querystring"),
"http://example.com")

self.assertEquals(w._normalize_url("http://example.com#fragment"),
"http://example.com")
 Recap
• Jenkins watches the code repo for commits
• Checks out code
• Builds it with your build ‘scripts’
• Runs your tests: unit, performance, UI, ...
• Reports problems by email, IM, etc
• Tracks and graphs metrics and trends
• Can build artifacts, deploy, etc
 Take Away

• (Visual) Feedback encourages better

code

• Non-judgmental, friendly competition

• So easy to setup, no excuse not to! :-)
These Books are Helpful
• They’re not specifically about Jenkins, SVN,
Java, Python, Ruby, .net, ...
• They cover real programmer hands-on
practice with code repos, tests, continuous
integration

• Pragmatic Project Automation: How to

Build, Deploy, and Monitor Java
Applications (Mike Clark)

• Ship It! A Practical Guide to Successful

Software Projects (Richardson, Gwaltney)
[excellent for SW project managers]
 Danger!
Live Demo!
 Demo: Create Project (1) search
Build after other projects are built

Poll SCM

Schedule
Build after other projects are built */10 * * * *

Project name search

pyrajenkins Poll SCM

Description Schedule Build periodically

*/10 * * * *
Demonstration pyramid app for Jenkins talk
Project name pyrajenkins Build

Description Execute shell

Build periodically
Demonstration pyramid app for Jenkins talk
Build Command echo SETUP
Discard Old Builds Execute shell
cd pyracms
/usr/local/python/2.7/bin/virtualenv --no-site-
Days to keep builds 7 Command packages --distribute .
echo SETUP
Discard Old Builds cd pyracms
echo BUILD
if not empty, build records are only kept up to this number of /usr/local/python/2.7/bin/virtualenv --no-site-
days bin/python setup.py develop
Days to keep builds 7
packages --distribute .
Max # of builds to keep See the list of available environment variables
10
if not empty, build records are only kept up to this number of echo BUILD
Delete
days if not empty, only up to this number of build records are kept bin/python setup.py develop
Max # of builds to keep See the list of available
Execute shell environment variables
10 Advanced...
Delete
if not empty, only up to this number of build records are kept Command pyracms/bin/nosetests --with-coverage --with-xunit
This build is parameterized Execute shell --where=pyracms --cover-package=pyracms
(trend) Advanced...
pyracms/bin/coverage xml --
Disable Build (No new builds will be executed until the project is re- Command pyracms/bin/nosetests --with-coverage --with-xunit
include="pyracms/pyracms/*"
27:25 PM This build is parameterized
trend) enabled.) --where=pyracms --cover-package=pyracms
36:47 PM pyracms/bin/coverage xml --
Disable Build concurrent
Execute (No new builds will ifbenecessary
builds executed until
(beta)the project is re- include="pyracms/pyracms/*"
25 PM See the list of available environment variables
00:53 PM enabled.)
47 PM Advanced Project builds
Options Delete
30:36 PM Execute concurrent if necessary (beta)
See the list of available environment variables
53 PM Advanced... Execute shell
00:36 PM Advanced Project Options Delete
36 PM
Source Code Management Command cd pyracms
10:36 PM Advanced... Execute shell
36 PM bin/pylint --rcfile .pylintrc -f parseable pyracms
None Command cd pyracms > ../pylint.txt || echo "Return Success"
13:27
36 PM PM
Source Code Management
bin/pylint --rcfile .pylintrc -f parseable pyracms
57:59 NoneCVS
27 PM PM > ../pylint.txt || echo "Return Success"
:43:53 CVSGit
59 PM PM See the list of available environment variables
Repositories
Git URL of repository file:///Users/chris/Projects/presentations/pyrajenkins
or failures
3:53 PM Delete
See the list of available environment variables
Repositories URL of repository file:///Users/chris/Projects/presentations/pyrajenkins
ailures Delete
Advanced... Add build step
Advanced... Add build step
Delete Repository Post-build Actions
Delete Repository Post-build Actions
Publish JUnit test result report
Add
Publish JUnit test result report
Add Test report XMLs
Branches to build **/nosetests.xml
Branch Specifier (blank for default): ** Test report XMLs
Branches to build **/nosetests.xml
Branch Specifier (blank for default): ** Fileset 'includes' setting that specifies the generated raw XML
reportthat
Fileset 'includes' setting files, such asthe
specifies 'myproject/target/test-reports/*.xml'.
generated raw XML
Delete Basedir of the fileset is the workspace root.
report files, such as 'myproject/target/test-reports/*.xml'.
Delete Basedir of the fileset is the workspace root.
Branch Retain long standard output/error
Branch Retain long standard output/error
Add Publish Javadoc
Add Publish Javadoc
Build other projects
Advanced... Build other projects
Advanced...
Archive the artifacts
Repository browser (Auto) Archive the artifacts
Repository browser (Auto)
Aggregate downstream test results
Aggregate downstream test results

Subversion Recordoffingerprints
Record fingerprints files to trackofusage
files to track usage
Subversion

Build Triggers Activate

Activate Chuck Norris Chuck Norris
Build Triggers

BuildBuild
afterafter
otherother projects
projects are built
are built Publish
Publish Cobertura Cobertura
Coverage Coverage Report
Report

1. Give it a name, a code repo to watch; for Git it can build all branches that change.
2. Poll every 10 minutes, then 3 text boxes with (here) Unix commands to run the build and tests.
It can be tricky to get the tests to run where you want them, and output where Jenkins expects it.
You likely will want to turn these commands into scripts once you get them debugged through the web.
Report
Demo: Create Project (2)
Publish Cobertura Coverage Report

Cobertura xml report pattern **/coverage.xml Source encoding default

*/coverage.xml Source encoding default
This is a file name pattern that can be used to locate the
cobertura xml report files (for example with Maven2 use Git Publisher
is is a file name pattern that can be used to locate the **/target/site/cobertura/coverage.xml). The path is
bertura xml report files (for example with Maven2 use relative to the module root unless you have configured your Git Publisher
E-mail Notification
/target/site/cobertura/coverage.xml). The path is SCM with multiple modules, in which case it is relative to the
ative to the module root unless you have configured your workspace root. Note that the module root is SCM-specific, E-mail Notification
M with multiple modules, in which case it is relative to the Recipients [email protected]
and may not be the same as the workspace root.
rkspace root. Note that the module root is SCM-specific, Cobertura must be configured to generate XML reports for Recipients [email protected]
d may not be the same as the workspace root. this plugin to function. Whitespace-separated list of recipient add
bertura must be configured to generate XML reports for reference build parameters like $PARAM. E-m
Consider only stable builds when a build fails, becomes unstable or ret
s plugin to function. Whitespace-separated list of recipient addresses. May
reference build parameters like $PARAM. E-mail will be sent
Include only stable builds, i.e. exclude unstable and failed Send e-mail for every unstab
when a build fails, becomes unstable or returns to stable.
ones.
clude only stable builds,
Sourcei.e. exclude unstable and failed
Encoding ASCII
Send e-mail for every unstable build Send separate e-mails to indi
es.
broke the build
ASCII source.encoding.description Send separate e-mails to individuals who
Coverage Metric Targets Jabber Notification
broke the build
urce.encoding.description
Conditionals 70 0 0
Targets [email protected], chris.shento
Jabber Notification
Conditionals 70 0 0 Lines Delete 80 0 0
Targets Whitespace separated list of accounts to s
[email protected], [email protected]
(like '[email protected]') -- for gr
Lines Delete 80 0 0 Methods Delete 80 0 0 a '*' (e.g. '*[email protected]
Whitespace separated list of accounts to send notifications
note that JIDstothat contain '@conference.'
Add (like '[email protected]') -- for group chats, prepend
recognized as group chats (no need to pre
Methods Delete 80 0 0 a '*' (e.g. '*[email protected]') --
Configure health reporting thresholds. note that JIDs that contain '@conference.' are automatically
Add For the row, leave blank to use the default value (i.e. 80). recognized as group chats (no need to prepend '*')
For the and rows, leave blank to use the default Save
nfigure health reporting thresholds. Advanced...
values (i.e. 0).
r the row, leave blank to use the default value (i.e. 80).
r the and rows,Report Violations
leave blank to use the default Save
Page generated: May 13, 2011 10:24:30 PM
ues (i.e. 0).
XML filename pattern

Page generated: May 13, 2011 10:24:30 PM Jenkins ver. 1.411

checkstyle 10 999 999
XML filename pattern
codenarc 10 999 999
10 999 999
cpd 10 999 999
10 999 999
findbugs 10 999 999
10 999 999
fxcop 10 999 999
10 999 999
gendarme 10 999 999
10 999 999 jcreport 10 999 999

10 999 999 jslint 10 999 999

10 999 999 pmd 10 999 999

10 999 999 pylint 10 999 999pylint.txt

10 999 999 simian 10 999 999

3. The Cobertura code coverage

stylecop
10 999 999pylint.txt 10 999 999
plugin needs to know the data file to display, same with the Violations plugin for Python
styles Per file limit
10 999 999
4. Notify by email and IM 100
10 999 999
Source Path<br />Pattern
Demo: Status Overview
Jenkins » pyrajenkins
search

ENABLE AUTO REFRESH

Back to Dashboard
Project pyrajenkins
Status
Demonstration pyramid app for Jenkins talk
edit description
Changes
Disable Project
Workspace Test Result Trend
Build Now

Delete Project

Configure

Coverage Report

Violations

Git Polling Log

Build History (trend)

(just show failures) enlarge
#9 May 13, 2011 9:27:25 PM Chuck Norris doesn't need garbage collection because he doesn't call .Dispose(), he
#8 May 13, 2011 7:36:47 PM calls .DropKick().
Code Coverage
#7 May 13, 2011 6:00:53 PM
Classes 100% Conditionals 100% Files 100% Lines 72% Packages
#6 May 10, 2011 9:30:36 PM 100%

#5 May 10, 2011 6:00:36 PM

#4 May 10, 2011 5:10:36 PM

#3 May 10, 2011 3:13:27 PM

#2 May 10, 2011 2:57:59 PM

#1 May 10, 2011 12:43:53 PM

RSS for all RSS for failures

pylint 40
Demo: Status: Failed Test search

Jenkins » pyrajenkins » #5 » Test Results » pyracms.tests » PageEditTests » test_save_url ENABLE AUTO REFRESH

Back to Project
Regression
Status
pyracms.tests.PageEditTests.test_save_url (from nosetests)
Changes Failing for the past 1 build (Since #5 )
Took 0 ms.
Console Output add description

Edit Build Information

Error Message
History
DummyResource instance has no attribute 'title'
Polling Log
Stacktrace
Git Build Data
Traceback (most recent call last):
Test Result File "/usr/local/python/2.7/lib/python2.7/unittest/case.py", line 318, in
run
testMethod()
Previous Build File
"/Users/chris/.hudson/jobs/pyrajenkins/workspace/pyracms/pyracms/tests.py",
Next Build line 48, in test_save_url
info = self._make_one()
File
"/Users/chris/.hudson/jobs/pyrajenkins/workspace/pyracms/pyracms/tests.py",
line 41, in _make_one
return page_edit(context, request)
File
"/Users/chris/.hudson/jobs/pyrajenkins/workspace/pyracms/pyracms/views.py",
line 46, in page_edit
'crumbs': _crumbs(context, request),
File
"/Users/chris/.hudson/jobs/pyrajenkins/workspace/pyracms/pyracms/views.py",
line 20, in _crumbs
crumbs.append({'title': resource.title, 'url': url})
AttributeError: DummyResource instance has no attribute 'title'
 Demo: Status: Coverage (1)

Coberturaʼs designed for Java, so only the Lines applies to my Python code.
The dip at checkin #4 was due to added code without corresponding tests;
I then wrote a test committed in #6 to bring it back up.
Note the Red bars indicating missing coverage.
 Demo: Status: Coverage (2)

The graph shows coverage over time for this file.

The code highlighted in Green has test that exercises it,
but the Red needs new tests to provide coverage.
HAWT! This is a big motivator.
 Demo: Adding a Test
Last build (#10), 2 min 14 sec ag
search Last stable build (#10), 2 min 14

Tests and Coverage Successful builds

Last successful build (#10), 2 mi
ENABLE AUTO REFRESH Last failed build (#8), 3 hr 54 min
Last unsuccessful build (#8), 3 h
ect pyrajenkins
improved in Build #10
ration pyramid app for Jenkins talk
edit description
make Chuck smile
Disable Project
Test Result Trend

(just show failures) enlarge

All arrays Chuck Norris declares are of infinite size, because Chuck Norris knows no bounds.
Code Coverage
Classes 100% Conditionals 100% Files 100% Lines 83% Packages 100%

pylint 49 (+9)
fin
 Questions?

http://www.flickr.com/photos/drewdlecam/3501280430/in/photostream/