Performance Testing in Agile Process
1 like•2,760 views
This white paper discusses the critical importance of integrating performance testing in agile software development to enhance application quality and expedite market readiness. It highlights the shortcomings of traditional development methods, particularly their late-stage approach to performance testing, which can lead to significant delays and costs. The document outlines steps for embedding performance testing throughout the agile process to ensure continuous feedback and early detection of issues, ultimately driving better business outcomes.
Performance Testing in Agile Process
- 1. Advantageous of Implementing Performance Testing in Agile Process White Paper idexcel
- 2. Background In many web or cloud applications, performance testing is critical part of application testing since it affects business revenue, credibility, and customer satisfaction. Conventional software development models are known to pushing the performance testing to the very end of project, with the expectations that, only minor tweaks and tune up are required to meet the performance requirements from the business, however any major performance bottlenecks found during this phase were major factors for delay in Go to Market. With more and more companies are adapting the agile software development process which believes in performance testing should never be an afterthought but it should tightly integrate from initial planning to production analysis of software development lifecycle. This white paper explains how any company can integrate performance testing into agile process, and key barriers for agile performance testing when team decides to adopt agile performance testing. idexcel 2 Page Performance Testing in Agile Process
- 3. Performance Testing in Traditional development Agile Development Practice Performance engineering is an important software Business Impact: Performance issues detected late may not only end in small tuning, but may force business to renegotiate contracts Conducting performance testing in agile, iterative way may increase the efficiency of perfor- mance testing. development discipline that ensures that applications are architected, designed, built and tested for performance. Conventional development models like waterfall models are known to pushing the performance testing to the very end of the project, because of many reasons like last minute changes, bugs that prevent test execution, infrastructure limitations, or lack of dedicated testing resources. This flawed approach is having many disadvantages like Slow: It assumes that system is completely ready to execute performance test cases; these practice results pushing the performance testing at the end of the project lifecycle. Any performance bottlenecks found during this phase will postpone GTM date. Late Changes: As late changes are always expensive to fix, any performance bottlenecks found during this phase are expensive to fix. 3 Page idexcel Performance Testing in Agile Process Performance in all teams and all roles are missing: It is mindset and comfort zone in traditional development teams that, performance testing is carried out by special- ized team at the end of development lifecycle. However most of the teams fail to realize that performance engineering must be architected, designed, and tested. This mindset forces to miss the performance testing at unit level, integration and functional level which leaks the performance issues to last gate of development cycle Analyzing the results: Running all scripts and/or large tests, gives minimal information, about the system behavior. Analyzing will become complicated and time consuming when we run all scenarios and scripts all together. Business Impact: Performance issues detected late may not only end in small tuning, but may force business to renegotiate contracts Conducting performance testing in agile, iterative way may increase the efficiency of performance testing. Growing need for quicker and adaptive solutions to tech problems is pushing firms to adopt the agile methodolo- gy. Today more and more companies are addressing different technology issues by adopting this iterative approach to software development and releasing high quality software faster and more efficiently. Organization sees agile software development as a faster way to create products that would generate sales. Agile methodology promotes self-organization, flexibility to midcourse action, better alignment between the team, accountabili- ty, open collaboration throughout the lifecycle of the project. One of the main goals of agile is to efficiently manage change in both external and software functionality and accelerate the development of quality software. Agile puts development, QA, and support staff on the same page and all stakeholders, including those on the business side need to be tightly involved.
- 4. 4 Page idexcel them to react early to any detected performance or scalability issue. Performance Test at Every Level: In agile, team may not get the complete features which are needed for perfor- mance test at the end of every sprint. This forces teams to conduct performance testing on portion of the code or component which is ready. In agile Start performance testing at component level of function and if all the components work fine individually fine then hypotheti- cally they also work well together. This knowledge will help the team to analyze and debug performance bottlenecks when team test on the system level later on. Team need to look at the performance testing in agile as testing of different performance layers that can be approached in cadence as the sprints progress. Develop and Execute Test Cases: As stated early, agile may not generate workable product at the end of each sprint. We may not be able to use the standard load testing tools as not all components would be present which standard load testing tool may require. So, in most cases there is an inherent challenge in simulating load and test for performance. Hence tools usage may vary between the phases. For example a stub or a harness might be used in the initial phase along with JunitPerf, and extra effort for this need to be taken into consideration during project planning phase. Once created, development and execution of test cases would follow, with the help of profiling tool; most of the performance hotspots can be identified. At later stages of the project, the traditional perfor- mance scripting and execution will replace the harness that was created in the earlier stage. So there will be a big rework which again would be a challenge consider- ing the sprint timelines. So these tasks should be consid- ered in backlog and sprint should be planned according- ly. Integrating Performance Testing in Agile The concept of performance testing in agile is different from conventional model. Performance testing is an integral part of agile process. The goal is to test perfor- mance early and often in the development effort, and to test functionality and performance in the same sprint. To remain efficient in an agile environment, team need to change the way it used to managing performance testing. One of the big challenge team face when integrating performance testing is that agile movement shifts the inter team process of software engineering to one that is focused on software development, to the exclusion of QA and the exclusion of operations Below section describe the x step approach for integrating performance testing into agile process in any organiza- tion Definition of done: In agile, scrum team along with the product owners and scrum master, defines a definition of done to claim a product backlog to be completed. Usually it includes incremental product architecture design, feature design, development, unit testing, integration testing, functional testing, and documenta- tion. It is highly recommended to list the performance test requirement in the product backlog and include performance as a definition of done for a sprint at com- ponent, integration and system level. Add Performance Test to Build Process: As continuous improvements takes center stage in agile, so adding performance tests into continuous integration process is one important step to continuous performance engineering. This entire process starts when developer checks in code and continues through the build and test process. I n addition to the functional testing results, performance test results are analyzed and recorded for each build. this integration of CI performance testing with the agile makes developers to collect feedback on changed com- ponents in continuous feedback loop, which enables Performance Testing in Agile Process
- 5. 5 Page idexcel Decide on Test environment: In Agile, although perfor- mance testing starts early in development lifecycle, there are certain performance testing aspect that are hard to do until later in the delivery cycle like stress and capacity testing. While development and test environ- ment performance is not typically as big concern as production, but can be used for performance smoke testing, and once all components are ready, team can move towards staging and production environment with appropriate tools with valid test data Although performance testing at an early stage with iterative approach looks to be very good idea, while integrating performance testing in agile practice, team may face many challenges. Unstable Builds: As agile process is adaptive, designed to accommodate, and encourage inevitable changes. This may lead to changes in system. But Performance testing at system level is painful process. It is difficult to get the application deployed and running the perfor- mance testing, teams may get frustrated to test multiple builds with changed requirements in same iteration. It is responsibility of management to make these specialist teams to understand the importance of iterative system testing and motivate them in agile practice Test Data Preparation: Because of the nature of agile environment, sprints are short which shrinks testing time in every sprint. This forces team to decide on test data for performance testing, early in development lifecycle preferably in planning stage this gives enough time for performance testing during the sprint. Team need to decide on whether to use the production data for testing or dummy data, does any regulatory aspects to use the production data. The earlier test data prepa- ration takes place, the more time team will have for testing in short agile sprints. So performance testers should work with stake holders in the planning stage to prepare the test data. Plan carefully to get the test data early in the life cycle. Performance Testing in Agile Process Key Barriers of Agile Performance Testing High skilled Performance team: Any tester who could record, generate the script, test and provides the perfor- mance results using standard commercial tools may not be suitable for performance testing in agile. The agile methodology provides lot of challenges from perfor- mance standpoint. Examples are, conducting perfor- mance testing early in lifecycle at component level and performance testing of incomplete components. This makes it important to performance test only a portion or a component of an application. Hence it is imperative that team members should have the confidence to get their hands dirty and learn the technology and program- ming skills to develop stubbing which simulates parts of an application that are either not written or not avail- able. If an application uses outside third party integra- tion then performance engineers need to develop stubs which simulates 3rd party integration, as they may not get the third party production server for performance testing at every sprint.
- 6. 6 Page idexcel Stake holders buy in: Traditional QA departments segre- gate performance test specialists to single group who will perform the work for different development teams. Contrary, performance testing in agile demands perfor- mance testers early in the lifecycle of the project, having an additional performance testing team from the early phase of the project directly impacts the budget and timelines of the project. Adding to it, advantages of performance testing are more of an intangible kind. Thus it becomes more difficult to get a stake holder buy-in for budget and team for performance testing early in lifecycle of agile Project Management Vision: Although performance testing is critical part of application testing as it affects business revenue, credibility and customer satisfaction, many project managers are still unaware of perfor- mance test process and challenges performance team faces. Within agile framework, this would cause more conflicts between the project managers and perfor- mance teams. Hence it is important to discuss and nego- tiate a plan with project managers, performance team and business sponsors, and come up with a plan which will for “approach of performance testing in agile”, “ From which sprint to start the performance testing”, “Transition of performance test tools as the application development progress”, etc. Look at the other side of the coin: Although perfor- mance testing at an early stage with iterative approach looks to be very good idea, there may be limited possi- bility of presence of bottleneck in code base if we have a proper reviews and architecture in place. The more performance bottlenecks that may be present, would be more on infrastructure side which are easier to fix and with almost no functional/regression testing. Hence before adopting early testing which consumes huge effort and cost, have tradeoff between, talent in the team, nature of application, business impact, business and nature of contract and decide whether you can address the performance challenges later in the devel- opment life cycle or in successive releases. Conclusion: Performance engineering is an important software development discipline that ensures that applications are architected, designed and built and tested for performance. With more and more companies are adapting the agile software development process which believes in performance testing should never be an afterthought but it should tightly integrate from initial planning to production analysis of software develop- ment lifecycle, it is important to integrate performance engineering to agile development process. Unlike performance testing in conventional development process, team has to start performance testing early in development lifecycle which has a positive outcome in code tuning, earlier rejection, performance bug which appears in production. Performance Testing in Agile Process
- 7. 7 Page idexcel In this competitive IT days, organizations are becoming more and more conscious about the performance of their Enterprise Solutions with in short time span along with development like not spending separate time line for it, till developers complete their coding for application and making it functionally stable. There are several competi- tive applications with better performance are coming with high quality software in less time before one finish out their project functionally ready , As the IT industry matures and the technology evolves, so does the awareness about expectations from an Enterprise Application. Focusing just on the design / implementation and Zero-functional-de- fect solutions then checking the performance of applica- tion are things of the past. With increasing maturity in technology and IT staff, the best solution is driving every- thing in to agile process. As per this now days most of the ITs moving towards agile, the challenge is how they imple- ment performance testing in agile model. Performance Testing: Performance testing of an applica- tion / system is basically the process of understanding how the application and its operating environment behave at various user loads. In general, it is performed by simulating virtual users to determine / validate the scalability, availability, robustness, and hardware & software resource utilization of the application thus ideally paving the way for effective capacity planning of the system for future usage. IT is one of the single biggest catalysts to significant changes in architecture, code, hardware and environ- ments. One can utilize this to benefit by exploring the performance issues. Performance testing that usually takes place at the end of the lifecycle in a waterfall model. Performance testing in Agile? - The (in) famous sentence. What is it? What does it mean? Why and how to imple- ment and drive performance testing in to agile? In this White paper, we'll touch upon Performance Project Phases traditional waterfall approach to do performance testing, what an agile performance testing advantages of imple- menting performance testing in to agile model. Performance Testing in Agile Process The Performance project Phases: 1. Sizing and Capacity Planning • System usage scenarios • Load during normal usage and peak hours • Performance requirements • Right-sizing systems to meet needs without wasting money and creating unnecessary complexity. 2. Performance Engineering • Proactively engineering for performance • Technical design with performance in mind • Tools Selection • Test environment & data preparation • Test script design and preparation
- 8. 8 Page idexcel Main Performance Types Waterfall development methodology is a sequential process in which development is seen as flowing steadily downwards (like a waterfall) through the phases of requirements analysis Most of the cases performance testing phase will follow traditional water fall approach. After development of finished, application functionally tested and verified and System is ready then the performance scripting starts for requested scenarios. Preparing test runs with required combinations and settings then gathering the results and compare with the requirements will happen. If require- ments not met team alerts the developer to get back for PERFORMANCE TESTING LOAD TESTING STRESS TESTING CAPACITY TESTING Performance Testing in Agile Process 3. Performance Testing • Tests execution • Result analysis • Recommendations • Fix problems • Monitoring and Troubleshooting • Understanding systems and solving problems in produc tion Main types of performance tests to cover repeatedly in Performance Phases are: Load testing: Part of performance testing and it applies a “normal” amount of load, reflective of real-world num- bers, Shows bottlenecks with different user levels and any immediate problems when typical load is applied. Stress testing: Performance testing beyond normal or “peak” amounts of load and possibly for extended periods of time. Shows the how application will behave under stress, problems if any. Capacity Testing: Performance test to determine how many users and/or transactions a given system will support and still meet performance goals. All of this testing is essential to truly understand the systems performance characteristics Traditional water fall performance approach and constraints: Requirements analysis Design Develop Deploy Test Maintain Waterfall SDLC moduleTime taken to statt test
- 9. 9 Page Agile SDLC Module: idexcel Performance Testing in Agile Process work to meet the requirements. Though this methodology has been proven success full approach there are some constraints and problems mentioned below which will be costlier when considering the time in to factor. • No functional system until the very end and everything must be ready to startup performance testing. • It assumes that the system is completely ready • Developers may not be available by the time the perfor mance testing starts • Late changes as per the requirements if not met across the board will be the costlier. • Performance scripting development may took long time to debug modify as per the functionality. • Analyzing will become complicated and time consuming when we run all scenarios and scripts all together. • Lot of repeated set of runs needed to build any Perfor mance Model for future but after one successful run. ROI: Return On Investment What is exactly Agile Methodology and how it works? The word agile here doesn't refer to any specific develop- ment process or methodology, the meaning of word Agile is “able to move quickly and easily “. Agile software development is a group of software devel- opment methods based on iterative and incremental envelopment, where requirements and solutions evolve through collaboration between self-organizing, cross-func- tional teams. Agile methodology promotes close teamwork, appropriate reviews, more iterations and self-organization. This approach enhances rapid delivery of high quality software and enables a business practice that aligns software devel- opment with customer needs and company goals. Application stakeholders, programmers and testers must make performance a primary consideration in all iterations of agile development process. Challenges in Implementing Agile Performance testing and successful approach: DESIGN DEVELOP TEST REQ, ANALYSIS
- 10. idexcel Performance Testing in Agile Process Performance Testing in Agile Process • How to cover all performance project phases? (Sizing, Capacity Planning, Performance Engineering, and Perfor mance Testing) • Do we need to Waite till the developers give code feature by feature? • If any application feature changes do we need to re test the one that has changed? • How we go ahead without scalability and performance requirements? • Below three steps approach will give complete solutions for above challenges Approach: As already explained how agile works above let us see how we approach and face the challenges in application performance testing in agile methodology. The main reason of point and its advantage of doing the performance testing in agile process are to save time factor we have to start concentrate about performance testing early life cycle of SDLC .thus have to determine the performance potentials in main business scenarios as early as possible during the project development. Test the available, analyze and fix performance bottlenecks at the code level. The three Step approach for implement- ing agile performance testing is described below • Test environment setup • Tools selection • Perfomance requirements • Test data preparation • Load tests • Stress tests • Capasity testing • Resillience testing Iter.0 Benchmark Test Focus Test SDLC + STLC SDLC + STLC Focus Test Focus Test Focus Test Benchmark Test Benchmark Test Iter.1 Iter.2 Iter.3 .... Iter.x-1 Iter.x .................................................... 10 Page
- 11. idexcel Performance Testing in Agile Process Performance Testing in Agile -Tree Steps approach: First step: Test and find the potentials at unit code level as it will be developed. Execute the tests to fix optimize bottlenecks at unit/Code level. This is performed when the scalability and performance requirements are specified at method level. Unit performance tuning proceeds along with the devel- opment of the application and unit testing of the method to tune is completed. Application Load Turning Product Released Yes Hardware/ Software Optimization Integration/ Functional Testing Hardware/ Software Optimization Perfomance Bug Found? Perfomance Bug Found? No Yes No Application Load Turning Component Level Turning Component Level Perfomance Testing UAT Testing Component Level Perfomance Testing Application Load Turning Yes Hardware/ Software Optimization Perfomance Bug Found? No Component Level Turning Yes Code Optimization Perfomance Bug Found? No Unit Level Turning Unit Testing Unit Level Perfomance Testing Integration/ Functional Testing Second Step: The Components level testing (testing components): Executing tests to isolate and fix bottlenecks in application components level. Third Step: Application flow level testing: Testing the critical application flows for user experience under normal and maximum high loads. 11 Page
- 12. Performance Testing in Agile Process idexcel Performance at Unit Code Level Goal: • Identifying the methods. • Scripts are prepared to invoke individual method. • Methods are tested under normal load for response time while measuring CPU and memory utilization. • If method does not show acceptable level of perfor mance, it is fine-tuned and test is re-executed • Tests are focused on discovering poorly performing methods. • This is performed when the scalability and perfor mance requirements are specified at method level. When to do? Unit performance tuning proceeds along with the development of the application and unit testing of the method to tune is completed. How to do? • Identifying the methods. • Scripts are prepared to invoke individual method. • Methods are tested under normal load for response time while measuring CPU and memory utilization. • If method does not show acceptable level of perfor mance, it is fine-tuned and test is re-executed. Component Level Tuning Goal: • Here the tests are targeted towards finding bottle necks in application tasks and activities. • The major objective is to evaluate the system behavior when it is pushed beyond its breaking point When to do? The components are deployed on their respective application server. How to do? The components can be identified based on following characteristics: • High business criticality and usage. • Can be deployed and invoked independently. • Deployed at application or middle tier. • Scalability and response time requirement of each component is captured. • Scripts are prepared to invoke individual component. • Tests are executed for each component separately • With respect to two major areas of bottlenecks, the following two tests are executed: 1. Concurrency Test: Component is tested for simultane- ous requests under incremental user load, until failure is noticed. 2. Test for Throughput: Component is tested for high number of hits per second with limited number of users. With the help of above results, the component is tuned by optimizing hardware/software configuration and test is re-executed. 12 Page
- 13. idexcel Performance Testing in Agile Process Application Load Tuning Goal: • Here, critical application flows are tested for user experience under normal and maximum loads. • The major objective is to capture the performance metrics and verify whether the performance objectives are met. When to do? Acceptance Testing is completed and application is stable. How to do? • The critical application flows are identified here. • Scripts are prepared for the flows identified. • Application is tested for normal load and maximum load. Server and client metrics are captured. • As per the results, the application is tuned by optimiz ing hardware/software configuration and test is re-ex ecuted. Performance testing like On Demand The model is the functional equivalent of outsourcing to an internal team, but this is where most companies that are trying to integrate their performance testing into agile processes start. This is because the performance testing was actually an On Demand model before the agile transi- tion. Some of the steps to be handled for this model are mentioned below: • You need to make use of On Demand services periodi cally. • The performance targets, goals and objectives should be a standard module of each user story. • Developers should take the responsibility for unit-lev el, component level, integration-level testing and tuning. • A full time team member has to take overall responsi bility for managing performance related tasks. On Retainer: Goal: On Retainer model can be used as an interim step between the On Demand and Full Immersion models. Some organi- zations may not have the required testers, performance testing environments and testing tools to support the Full Immersion model. Here, each development project is assigned a specific performance tester. That performance tester is also assigned to two or more development projects. Though this model provides more performance testing expertise to individual project and more domain knowledge to the performance tester, the tester may fall short of becoming a fully integrated contributor to the team. As a result, the tester works independently, but provides guidance period- ically. By following the same steps as listed in the above model and with the help of two additional steps as mentioned below, this model can provide more value. • The performance tester should be available to the team at times that is critical to the performance evolu tion of the project. • This model may require more performance testers or tools than On Demand model. 13 Page
- 14. Full Immersion idexcel Performance Testing in Agile Process If the performance is really important to the value of the product and reputation of the company then this model should be the goal of every team in an agile process. Here, full time team members are specialized in both delivering and testing performance and have responsibility for managing and coordinating performance related activities throughout the lifecycle of product development. Risk Mitigation Benefits Agile Performance Testing has positive outcome in risk mitigation with respect to: Code Tuning: Unit level tuning is done early - so no need for code optimization at later stage on testing lifecycle. Application Rejection: Reduces risk of the application being rejected because of issues such as memory leaks, hard coded values in code and database locking. Earlier Detection: Reduces the effort and duration for performance tuning and retesting. Release Dates: Iterative testing leads to constant awareness of the application performance and therefore more confidence in meeting release dates. Performance Bugs: Less number of performance bugs in subsequent phases. Performance Testing Efforts: Performance script can be re-used thus saving 60% of the efforts of performance cycle. Conclusion Hence, Performance testing is one of the single biggest catalysts to significant changes in architecture, code, hardware and environments. You can utilize this to your benefit by exploring the performance issues. Performance testing in an agile project environment allows you to manage the testing in a highly flexible way. Iterative testing approach leads to better code that is optimized for performance. In particular, this approach allows you to revisit the project vision and reprioritize tasks based on the value they add to the performance test at a given point in time. 14 Page
- 15. idexcel Performance Testing in Agile Process About the Author Venkat Prasad works as QA Lead in the Mobility Division of Idexdcel. He has around 7years of experience in Automation and manual testing of mobile applications. He is previously worked with IBM India Pvt Ltd before joining to Idexcel technologies Pvt Ltd. He worked with IBM India for almost 6 years as QA Automaion tool consultant for mobile applica- tions includes Billing systems, Sales applications, Commissions applications. About Idexcel Idexcel is an innovative provider of IT Products & Services focused on emerging technologies. We help world leading companies build efficiencies and stronger businesses. With more than 15 years into existence Idexcel’s main focus is client satisfaction and technology innovation. Our industry expertise and a global, collaborative workforce forms the backbone of our services. We offer high degree of skills in Enterprise Applications, Cloud Services, Data-warehousing, Big Data, Analytic, QA & Testing Services, IT consulting and Staffing. Idexcel product line includes: NDS, ERP, and Cync - A revolutionary credit monitoring application for the manufacturing and financial management. For more information log on to www.idexcel.com. Global Head quarters 459 Herndon Parkway Suite 11 Herndon, VA 20170 Tel: 703-230-2600 Fax: 703-467-0218 Email: [email protected] India Operations “Crystal Plaza” 9, 10 ,11 Bhuvanappa Layout, Hosur Road Bengaluru – 560 029 Karnataka Tel: +91-80-2550 8830 Email: [email protected] © Copyright, Idexcel. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Idexcel. The information contained herein is subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.