Basic deployment pipeline structure

Deployment pipeline practices.

1. Build the Binary only once.

2. Deploy the same way, to every environment.

3. Smoke test the deployments.

4. Deploy into a copy of production.

5. All the changes should propagate through the pipeline instantly.

6. In case, if any part of the pipeline fails, stop the line.

 BUILD AND DEPLOYMENT SCRIPTING
Build Tools
In a task-oriented tool, each task will know whether or not it has already been
executed as part of the build process, even though the init task is involved twice, it
would really be executed once.

In a product-oriented tool, the compiled source and compile test will result in a
single file that contains all the compiled code known as source.so and tests.so. The
run test generates file called testreports.zip.

A product-oriented build system ensures order of execution. It invokes Run test after
compiled source and compiled test with Timestamps generated in file for each task.

In compiling C/C++, the build ensures that the source code files are compiled only if it is
changed from the last build, known as Incremental Build.

In languages that run virtual machine, the compiler just creates the Bytecode and the Virtual
Machines Just In Time (JIT) compiler does the optimization at the runtime.
 BUILD AND DEPLOYMENT SCRIPTING

The building and testing of simple projects - (IDE).

Large projects need script for build, test and packaging the application.

Automated continuous integration environment run scripts or command to create binaries.

For example,

•Rails project runs the default Rake task.

•.NET projects use MsBuild, Java projects use Ant, Maven Buildr, and Gradle.
 BUILD AND DEPLOYMENT SCRIPTING

Build Tools
Build tool has two important essential features, that include
all those things that the build tool does, and
all those things the build tool depends.

The build tools can be

Task-oriented –all those things that the build tool does in the build process.
includes 'Ant' and 'NAnt' and 'MsBuild'.

Product-oriented -describe all the activities in terms of product they generate such as executables.
 Example: Make build tool performance depends on certain prerequisites to be executed exactly once.

The compiled source generates source.so and compile test will result in tests.so. The run test generates file
called testreports.zip.
 The build system ensures to invoke Run test after compiled source and test with Timestamps for each task.
The various builds tool includes
1. Make Build Tool
2. Ant Build Tool
3. Nant Build Tool
4. NAnt and MSBuild Build Tools
5. Maven Build Tool
5. Rake Build Tool
6. Buildr Build Tool
7. Psake Build Tool
Make
 Powerful product-oriented Build tool, capable of tracking dependencies within a Build and Components
with change.

Drawbacks
1.With complex applications and a large number of dependencies between the components increases the complexity of
rules build inside the Make becomes much harder to Debug. The teams solve this complexity by
creating a make-file for each directory and keeping at top and recursively runs the make-file in each of the
subdirectories. This leads to spread of make-files that include build information and processes of the make process.

2.make-files are susceptible to very hard to find class of Bugs. For example, sometimes in the command script, if White
spaces are included in the place of Tab unknowingly, the script will not work.

3.Make-files relies on shell to do any function as a result the make-files are specific to an Operating System (OS).

Due to these problems, the Make built tool is rarely used as in commercial applications.
C++ developers, use SCons in preference to Make, it . This can build files written in Python, and makes it powerful.
Ant
Apache
By the development of cross-platform applications in Java, caused Make tool with many limitations. Therefore, Java community
combined Make with XML to build Apache Build tool, that is fully cross platform tool that has a set of tasks written in Java to
perform common operations such as compilation and file system manipulation. This enabled easy extension of any new tasks
written in Java and made this a de-facto build tool for Java projects.

Ant is a task-oriented tool whose runtime components are written in Java. But scripts use an external DSL written in XML. This
combination gives powerful cross-platform capabilities, making it extremely flexible and powerful, but has several shortcomings.
Disadvantages:
•Ant requires build scripts for performing compile, create .Jar files, run Test etc are written in declarative XML language
are difficult to model and write. The declarative language syntax creates confusion to the developers, which generated
build statistics difficult to understand.

Nant, MSBuild, and the .NET framework

Microsoft introduced Nant, MSBuild, and the .NET framework that has many features in common with Java language and
its environment. Uses the same syntax as used by Ant with some differences.
 NAnt was built as a descendant of Ant but is more tightly integrated into visual studio solutions and projects.
 NAnt modified with some minor variations and this model is called MSBuild.
 NAnt and MSBuild has many limitations suffered by Ant.
Maven
• Maven has removed several disadvantages with Ant files.
• Maven performs almost any build, deploy, test and release task in a single command without writing many lines.
• Supports automated management of Java libraries and dependencies between the project in Java projects.
• Supports complex and rigid partitioning scheme in the software, that can decompose complex solutions into smaller
components.

Disadvantages
project that are not confirming to Mavens structure and life cycle, it is extremely difficult to work with this and forces
development team to structure their projects according to Maven's requirement dictating inflexibility.
Maven also uses DSL written in XML, makes it difficult to code when needed to be extended. Maven solves this by the use
of several plugins.
The default Maven core is very small with a self-updating configuration. To make it functional, it downloads it's own plugin
from the internet. Every time when maven is run, it tries to update itself and as a result of the upgrade or downgrade of
plugins can result in unpredictable build failure.
Long time is required to restructure a build, If the versions of the dependencies used by Maven is not effectively tracked,
can may end up in Dependency problems and breakages due to change in the version.
Rake
 Rake is product-oriented tool as well as a task-oriented tool.
 Rake is a general purpose, build and scripting tool for Ruby language.
 It uses an internal domain-specific language DSL in Ruby.
 Rake scripting uses plain ruby script and uses rubies API to carry out the task.
 It can create powerful platform-independent build-files with good understanding of dependencies.
 It has used general-purpose language which is available for normal development and maintaining the build
scripts. Along with this, in case of occurrence of bug in execution, the Rake build script has a stack structure to
help the developer understand what went wrong.
 Ruby provides classes open for extension and can add more methods to Rake classes from within the build script
for debugging purpose.

Disadvantages
•Needs good runtime on the platform
•Needs interaction with Ruby gems for the execution.
Buildr
 Buildr is built on top of Rake. Everything that can be done in Rake can also be done in Buildr.
 Faster than Maven.
 This is simpler and has the power of Rake.
 This is a new generation build tool that include builder, gradle, and Gantt.
 Extremely simple to customize tasks, and create new one.
 Better dependency management and easy completion of multi-project build.
 Buildr or Gradle are preferred replacements for Ant or Maven with new java projects.

Psake
 Basic of Psake is an internal DSL written in power shell.
 supports task-oriented dependency network.
 This is meant for Windows users.
Principles and Practices of Build and Deployment Scripting

 Create a script for each stage in the deployment pipeline.

 Use of appropriate Technology to deploy the application.

 Use the same scripts to deploy to every environment used.

 Use of operating systems packaging tools.

 Ensure the deployment process is idempotent.

 Evolve the deployment system incrementally.

Commit Stage
 Begins with the mark of a project as committed to the VC system.
 The commit-If successful, a collection of binary artifacts and deployable
assemblies are used in the subsequent test and release stages.
 Creating binaries to be deployed into any environment.
 Ensures to minimize the time spent on code level integration, performs
analysis and ensures quality of the code and speed the delivery to begin
deployment.
 Creating other artifacts useful later in the deployment line.
 When a change is checked-in into the main-line or trunk in the VC, the CI
server detects the change, checks out the source code, and performs a series
of tasks.
 The Tasks are automated by build script that is run by the CI server.
 The binaries and other reports are stored in the central artifact repository
Commit stage principles and practices.
Primary goal of deployment pipeline:
 Eliminate any build which is not fit to make into production line.
 Create deployable artifacts, or it should fail fast and notify the team regarding the reason for failure.

Principles and practices to make effective commit stage includes:

 Provide fast and useful feedback.

 What should break the commit stage?

 Do the commit stage carefully.

 Give developers ownership.

 Use a build master for every large teams.

Results of Commit stage
 The commit stage in a deployment pipeline has both inputs and outputs.
 The inputs include source code and the outputs are binaries, the various
reports and the code test result.
 If a test fails and reports from the analysis of the code base will help to
establish the quality of code base.
 The binaries generated in the commit stage is reused throughout the
pipeline.

Artifact repository
•The modern continuous integration servers provide an artifact repository to store reports, binaries to be reused in
the later stages of the pipeline.
•Does not store any failed stage in deployment pipeline.
•Allows to track back any version/release of software and to connect with any instance of the deployment pipeline
and the revisions in the version control system.
•Provides a web interface for the team to access the reports and binaries.
•Dedicated artifacts repository such as Nexus and other Maven style repository managers can handle binaries and
for storing reports.
Artifact repository process
1)A team member in the delivery team commits a change.
2)The continuation continues integration server runs the commit stage.
3)On successful completion, the binaries as well as any reports and
metadata are saved to the artifact repository.
4)The continuation in continuous integration server, the deployment
pipeline retrieves the binaries created by the commit stage and deploys to
a production like test and environment.
5)The continuous integration server runs the ‘acceptance test’ and reuse
the binaries created by the commit stage.
6)On the successful completion of the acceptance test, the release
candidate is marked as having passed the ‘acceptance test’.
7)The testers obtain the list of all binaries which have passed acceptance
test and the press of a button runs automated process to deploy them into
manual testing environment.
Artifact repository process (Contn..)
8)Next, Testers perform ‘manual testing’.
9)On successful completion of manual Testing, testers update the
status of the release candidate to indicate. It has ‘passed manual
testing’.
10)Next, the continuous integration server retrieves, the latest
candidate that has passed acceptance, testing or manual, testing
depending upon the stage in the pipeline configuration from the
artifact repository and deploys. The application to the production test
environment.
11)Next, the capacity tests are run against the release can rate.
12)On successful completion of the capacity test, the status of the
candidate is updated to “capacity tested “.
13)This is repeated for as many stages in the pipeline.
14)Once the code has passed all the relevant stages, it is ready for
release. It can be released by anybody with appropriate
authorization.
15)And the conclusion of the release process, the RC is marked as
“released”.
Commit Tests Suite, Principles and Practices.
 There are some important principles and practices that govern the design of a commit tests suite.

 In an automation test suite pyramid is shown below,

 Unit test forms the vast majority of the test. Should be fast to execute. The application fails to build if the test
used is not sufficiently fast.
 Covers 80% of the code base, and gives a good confidence that the application will pass.
 The unit test starts with unit test cases and should complete in seconds.
 In comparison to unit test the acceptance test are only few in number, which is subdivided into service test and
UI tests.
 Service test and the UI test take longer time to execute because they run against the full running system.
 These levels ensure that the application is working and delivering the expected business value.
There are several design strategies which is required for designing commit test.
These are used to achieve the goal to minimize the scope of any given test and keep it focused on testing one
aspect of this system.

They include:
1. Avoid the user interface
2. Use Dependency injection.
3. Avoid the Database.
4. Avoid a Synchrony in Unit Test.
5. Use Test Doubles.
6. Minimize State in test.
7. Faking time
8. Brute force
Avoid the user interface
Since the user interface is the most obvious place where bugs occur. The natural tendency is to over focus with test
efforts fetching more code. Therefore, the best recommendation with commit test is to avoid the test via UI. Because,
the UI involves a lot of components or levels of software under it and this may be problematic and takes effort and
time to get all the paces, ready for the test before, executing the test itself.

Dependency injection method

Also known as inversion of control describes how the relationship between the objects should be established from
outside the application.

Avoid the database

If the automated testing code is written to test the interaction with some layer of the code. It stores the results in
data base, and produces very slow run. But this is difficult when several tests need to be executed in succession and
creates a complexity of infrastructure setup making the whole testing approach complex to establish and manage.
Here the database can be avoided from the code base by good layering and separation of concerns from the code and
result in the development of better code. The developer must be able to separate the code under test from its
storage.
Avoid Asynchrony in Unit test
Asynchronous behaviour of the code within the scope of a single test will make the test process very difficult.
Therefore, the simplest approach is to avoid the centrally by splitting the test by running one test run up to the
point of its boundary and then a separate test starts. For example, if the system is posting a message and then is
acting on the message, the row message can be wrapped in a separate technology with an interface of the user.
The call of the message, can be confirmed, whether it is as per the expectation. In one test case, this can be done
by implementing it in a separate step or messaging interface. After this is second test can be added which verifies
the behaviour of the message handler.

Using Test Doubles

The unit test is focused on a small closed related, number of code components which are typically a single class or
few closely related classes. This shows good and encapsulation. Several problems occur while testing in the
middle of network of relationships and may require lengthy setup. Among all the surrounding classes, this can be
solved by faking the interactions with the class dependent. The normal solution used this stubbing the code of
such dependencies. Stubbing is a replacement of a part of a system with a simulated version that provides quicker
responses. Stubbs is widely used for large scale components and subsystems. There are several mocking or
stubbing tools such available Mokito Rhino, EasyMock, JMock, NMock and Mocha.
Minimizing State in Test: The unit test asserts the behaviour of the system. It is difficult for effective test design with
the creation of different state around the test. The problem is that some input values are required to any component
of the system to get some results back. The test is written by organizing the relevant data structure, so that the inputs
can be submitted in the correct form and results can be compared with the expected output. But without proper care,
the system and associated tests becomes more and more complex and it is easy to fall into trap of building. Elaborate
hard to understand and hard to maintain data structure. In order to support the test. In order to avoid and minimize
the dependency on the state of the test, always it is sensible to maintain a constant focus on the complexity of the
environment which is to be constructed in order to run the test.

Faking time: Time is a serious problem in automated testing process. Sometimes the test requires 500 milliseconds
before processing with the next step. Sometimes it needs to do something different as the time shifts to February 29
of a leap year. All of these cases are very tricky and hard to deal with and can be potentially disastrous while. Doing
the unit testing status if it is considered with the real system clock, the best strategy for any time-based behaviour is
to abstract the need for time information into a separate class. That is under the developer’s control. The dependency
injection is done to inject the wrapper of the system level time behaviour. Here the time stopped or moved affects the
behaviour of our clock class or the abstraction used which is fully under control.

Brute Force: Developers always wish for a fast commit cycle. The speed of the commit cycle should have the ability to
identify the most common errors, which are likely to be introduced. The optimization process is involved and works
solely through trial and error. In some cases, it is better to accept slower commit stage than to spend too much time,
optimizing the test for speed or reducing the proportion of bugs got. Normally the aim is to keep the commit stage
under 10 minutes. Using Build Grid can make the commit run faster.