SOFTWARE ENGINEERING

UNIT IV

TESTING IN AGILE DEVOPS ENVIRONMENTS

The combined adoption of Agile and DevOps enables organizations to cope with the
increasing complexity of managing customer requirements and requests. It fosters the emergence
of a more collaborative and Agile framework to replace the waterfall models applied to software
development flow and the separation of development teams from operations. This study aims to
explore the benefits of the combined adoption of both models. A qualitative methodology is
adopted by including twelve case studies from international software engineering companies.
Thematic analysis is employed in identifying the benefits of the combined adoption of both
paradigms. The findings reveal the existence of twelve benefits, highlighting the automation of
processes, improved communication between teams, and reduction in time to market through
process integration and shorter software delivery cycles. Although they address different goals
and challenges, the Agile and DevOps paradigms when properly combined and aligned can offer
relevant benefits to organizations. The novelty of this study lies in the systematization of the
benefits of the combined adoption of Agile and DevOps considering multiple perspectives of the
software engineering business environment.
The software development process can be viewed as a set of tasks required to produce high-
quality software. The literature shows that the quality of the software produced reflects the way
the process was carried out. Although several software development processes exist, generic
activities common to all of them stand out, as Sommerville highlights, such as software
specification (e.g., requirements definition, software constraints), software development (e.g.,
software design and implementation), software validation (e.g., software must be validated to
ensure that the implemented functionality conforms to what was specified), and software
evolution (e.g., software evolves as per customer need). The software development process
provides an interaction between users and software engineers, between users and technology, and
between system engineers and technology. In this sense, software development is an interactive
learning process, and the result is an embodiment of knowledge gathered, transformed, and
organized as the process is conducted.
A software development methodology includes a set of activities that assist in the
production of software. These activities result in a product that demonstrates how the
development process was conducted. Agile methodologies arise from the need to overcome the
difficulties and disadvantages of applying traditional methodologies in project management and
implementation. The Agile methodology assumes short periods of time between each delivery to
ensure early and continuous delivery of software susceptible to evaluation. In it is also
recognized that software
implementation according to this paradigm is interactive and incremental, enabling early
confirmation of whether or not the delivered artifact meets the needs and making the respective
corrections with low risk and cost.
The main social and human factors involved in the adoption of Agile methodologies are the
impact on organizational culture, namely by the collaborative culture imposed on developers and
the implications of being embedded in an Agile team. Constant feedback to all team participants
on the activities being carried out, and the commitment to the team’s goals, are highlighted in
Junker as key elements for a well-functioning Agile team. Feedback and collective awareness are
essential as opposed to individualism and lack of communication. This view is also confirmed by
Sweetman and Conboy when they highlight that feedback loops are the essence of the empirical
and complex processes found in software engineering that require continuous adaptation based
on learning obtained daily. Furthermore, complex projects are very unpredictable and therefore
need a process that incorporates unpredictability.
Inspired by the success of Agile methods, the DevOps (Development and Operations)
movement emerged that aims to take this line of reasoning to a higher level. This movement
comes to break the traditional culture where there was almost no interaction between teams and,
as highlighted by Luz et al., the goal is to create a culture of collaboration between development
and operations teams that allows increasing the flow of completed work. In summary, it is
intended to increase the frequency of deployments while increasing the stability and robustness
of the production environment. Beyond a cultural change, the DevOps movement also focuses on
the practices of automating the various activities required to deliver quality code into production,
such as creating environments for testing, automating testing, configuring infrastructure, data
migration, auditing, and security, among others.

Fig. Agile Automation with DevOps

In the literature, we can essentially find studies on DevOps that explore ways to align
development teams with operations, the benefits that this methodology can bring to
organizations,
and the challenges that are posed. However, there is a research gap in the characterization of the
simultaneous adoption of Agile and DevOps in organizations. In this dimension, the number of
available scientific studies is limited and they mostly present individual views of their
implementation, which does not allow for a sufficiently comprehensive characterization of the
benefits of their combined adoption. We acknowledge the study conducted by Hemon, which
characterizes the different phases of Agile to DevOps transition (e.g., Agile, ongoing integration,
and constant delivery), while Melgar et explore the benefits of the combined SCRUM-DevOps
approach in terms of increasing speed during the deployment process and increasing the quality
of software processes. In both studies, there is just one empirical case study, which makes it
difficult to generalize the findings. In this sense, this study seeks to bridge this research gap and
presents an analysis of the benefits that can be found by the combined approach of DevOps and
Agile considering a comprehensive set of twelve case studies that are representative of practices
of simultaneous adoption of both methodologies. This approach supported by multiple case
studies avoids the individual limits of each company’s vision and reduces the risk of bias, and
allows comparing, grouping, and systematizing the main benefits of the combined adoption of
both methodologies.
The rest of this manuscript is organized as follows: Initially, a theoretical contextualization
of the DevOps model is performed and the similarities between DevOps and Agile are explored.
Next, the methodology and associated methods adopted in the study are described. This is
followed by the presentation of the results and discussion of their relevance to the perception of
the benefits of the combined adoption of DevOps and Agile. Finally, the conclusions are
enumerated. It is also in this last phase of the manuscript that the limitations of the study are
addressed and some suggestions for future work are provided.
 Fig. Testing in Agile and DevOps Environments

DevOps Concepts and Model

Paul Hammond and John Allspaw presented the talk “10+ Deploys Per Day: Dev and Ops
Cooperation at Flickr”. They explained how the developers’ teams (Dev) and operations teams
(Ops) could contribute to more agile and scalable software development. Tight integration
between Dev and Ops to safely achieve several software deployments (more than 10) in a single
day was a disruptive idea regarding software development and its evolution. Later, Patrick
Debois coined the term DevOps (Development and Operations) and created the DevOps Day
event. Although the DevOps movement has been discussed for more than a decade it still lacks a
unique formal definition. For Wiedemann et al, the lack of a unique definition could be
intentional to allow each team to choose the definition that better suits its needs. Nevertheless,
several authors proposed definitions such as the one by Leite et“DevOps is a collaborative and
multidisciplinary effort within an organization to automate continuous delivery of new software
versions while guaranteeing their correctness and reliability”—and others view it as a
combination of values, principles, methods, practices, and tools . Some other common
definitions can be found in.
One of the key points in the execution of a project is the approach used to manage it. The
traditional approach based on the waterfall model looks at the project in a linear way with
several events, while in the iterative approach the development of software is undertaken through
successive progress. Therefore, it is common that the system is presented still incomplete or with
some deficient parts. The objective is that the refinement of the product happens in stages until
the desired result is achieved. The software development process does not end with the release of
the code, but only when it closes the feedback loop between those who write the code and those
who
use it. DevOps aims to remove the barriers between traditionally independent teams:
development and operations. Under the DevOps approach, these teams should work together
across the entire software life cycle, from development and testing through deployment to
operations. More than only a technical subject, DevOps deals with the organizational and human
issues that arise in the software life cycle. It promotes a culture of collaboration, integration, and
communication between teams to reduce the disconnect between them while assuring the
delivery of software in an agile, safe, and stable way. According to Rajapakse et al., the DevOps
movement is currently a widely adopted software development approach having as a major
benefit the ability to deploy releases more frequently and at a higher rate.
Some related concepts used in conjunction with DevOps are Continuous Integration,
Continuous Delivery, and Continuous Deployment. As noted in, these concepts are considered
key practices within DevOps, but are not always clearly used, as stressed by Stahl al. Continuous
Integration is a development practice where developers frequently integrate the code they
produce, that has successfully passed testing, to the project under development. Those
integrations occur typically once a day. Continuous Delivery is a development practice where the
software is kept in a reliable deployable state at any time. Potentially, after every change, the
software can be released. This leads to several release candidates that are evaluated. The
deployment to production is made manually by a team member, with the appropriate authority,
who decides when and which candidate should be released. Apart from Continuous Delivery, in
Continuous Deployment, release candidates resulting from software changes are automatically
deployed to production without the intervention of any team member.
Regarding how organizations could adopt DevOps and measure its success, the CALMS
framework is considered a foundational model for DevOps. CALMS is an acronym for Culture-
Automation-Lean-Measurement-Sharing. CALMS was created by Jez Humble, co-author of The
DevOps Handbook. The acronym highlights the five core elements of CALMS.

Culture
Cultural change focusing on collaboration and integration between developers’ team and
operations’ team.

Automation
A high level of automation to achieve continuous delivery running each code change
through automated tests.
Lean
The application of lean principles to increase efficiency and reduce complexity.
Measurement
Keeping key performance indicators for measuring performance and identifying where
improvements can be achieved
Sharing
Knowledge and best practice should be shared in the organization and across organizational
boundaries.
 Security issues concerning DevOps led to the spread of another term: DevSecOps. It adds
“Security” to “Development” and “Operations”, which were already part of the DevOps term.
According to Rajapakse et al, security is often treated as a non-functional requirement, handled
at a later stage of the software development life cycle. Under DevSecOps, security should be
built into every part of the DevOps life cycle. The purpose of the DevSecOps philosophy is to
align the speed of code delivery with building secure code, merged into one streamlined process.
The application of DevOps still must deal with some problems and concerns that can limit
its use (e.g., resistance to change, organizational vision, legacy systems). Misuse of the term,
lack of clear guidelines and, as already mentioned, the lack of a clear definition creates some
ambiguity about how to use DevOps principles. Those principles presuppose that, before
DevOps, development teams and operations teams were working independently with almost no
knowledge about each other’s work. This lack of knowledge across teams is not, in general, as
deep as DevOps assume. The whole software development process is improved with better
collaboration between teams, as DevOps advocates, but it does not mean that DevOps teams did
not previously cooperate. Another concern around DevOps is that its adoption rate is still low.
There is a close relation between DevOps and Agile methods in software development.
According to Leite et al., DevOps is an evolution of the Agile movement since software
development under Agile favors small release iterations with customer reviews.

Fig. DevOps Concepts and Model

Similarities and Differences between DevOps and Agile
In the context of Software Engineering, as discussed in this paper, DevOps can be
understood as a behavioral evolution of Agile development, which was gradually conceived
through practical experiences of implementation in software development. However, it is
important to point out that
the Agile method has its focus directed specifically to software development, while DevOps
aims to involve the software development area in the implementation and operation of the
software developed or still under development, which shows us that DevOps processes are being
implemented within the Agile processes.
Currently, in the professional community of Information and Communication Technologies
(ICT) there is a growing consensus, in practice, that DevOps can be understood as “DevOps =
Agile + Lean + IT service management (ITSM)”. In its method and processes, DevOps adopts
characteristics of frameworks related to the technical area of Agile software development
together with ICT management processes. Complementarily, other relevant methodologies (e.g.,
Extreme Programming, Dynamic Systems Development Method, Kanban, SCRUM) have
approaches intrinsically related to the Agile philosophy. SCRUM is very well-known and
intensely used in the Agile method, and is generally enhanced by the Kanban tool, for managing
the workflow of software development, but which also fits very well into the DevOps
development process itself the problems or gaps in the Agile method that are solved by adopting
the DevOps method.

Testing Strategies in Agile

Test Automation in Agile

Implement Parallel Testing

In agile methodology, continuous releases in multiple iterations could cause the test
automation teams to miss the quality standards. By following parallel testing with the
development cycles, it helps to identify the bugs, fix the issues and test them effectively. This
will be an efficient way to increase productivity and avoid last-minute surprises.

Automation Tool
It is essential to select and invest in the right automation tool by validating the tool
capabilities. It is necessary to ensure whether the selected automation tool meets the expected
features or criteria for successful test automation under an agile environment.

Maintain smaller test cases

It is important to run test cases that are small. This will eliminate the unnecessary data
which might not add any difference with or without the addition of the data to the test case.
In addition to this, smaller test cases help to make changes easily according to the regression
testing requirements.

Monitor the Development Environments

It is crucial to continuously monitor the developers and development environment to
understand the functionality. There are a number of phases in a software development process –
it comprises of a network, back-end to front-end system architecture, etc. Hence, it is important
to understand the functionality and their association with product development. This will help
to focus on the product quality rather than finding the bugs through different data.

Implement Quality Tests

Developing robust tests is an important factor to make test automation successful in agile.
This will help the developers to create quality scripts that can easily integrate with the regression
testing. The test scripts should be flexible to fulfil the regression testing requirements by meeting
the criterion of accuracy, portability, integrity, portability, performance, etc.