Why you shouldn't use test cycles with DevOps & Continuous Delivery

Many development teams and companies are focusing on improving how frequently they deploy and deliver value to customers; it can be key to a company’s financial success. Companies that do this are adopting techniques such as Continuous Delivery, DevOps, Continuous testing, and Microservices. When the concepts from test case management tools and traditional testing techniques are applied with these modern software delivery techniques, it can place a brake on the increased deployment frequency that is the key reason for adopting them.

Test Cycles is one such concept that can have a profound impact on delivery frequency, but its popular with test management tools, even ones claiming to be Agile. This impact is caused by two different issues:

1.      When the Test Cycle is executed: You can choose to execute the cycle after every task or user story, or group together a batch of tasks or stories that are ‘ready for testing’.

2.      The size of the Test Cycle: A test cycle is a group of test cases that need to be executed, typically this is large number as 100% coverage is aimed for.

Test Cycles encourage a large batch of testing work and anyone who works with Continuous Delivery and DevOps knows large batch sizes are kryptonite to their success. The size of the batch and the possible delayed execution slow down the overall time from idea to functionality being used by customers or users.

The large size of a test cycle, the large number of test cases within it, and consequential time it takes to execute this large number of test cases encourages teams to delay the execution of the cycle so a group of tasks or stories can be ‘tested’ together. This is compounded by the type of testing activity within the cycle usually being on one or two types and have the aim to getting close to 100% coverage.

It may seem more efficient to batch up task or stories for a long activity like executing test cycle, but it delays feedback (positive and negative information like bugs) to the developers, product managers and the rest of the team. Developers can move onto other tasks or stories and context switching between them is mentally expensive and time consuming. If any negative information is discovered during the test cycle the developer will have to stop their new task and come back to the old causing wasted time due to the context switch.

Teams successfully doing Continuous Delivery and DevOps typically change the compositions of their testing activities they execute and spread the execution throughout the Software Development Life Cycle (SDLC) to avoid a large testing phases. Rather than focusing on one or two techniques these teams choose from a broad range of testing techniques that support each other so 100% coverage never has to be achieved, as the outcome is greater than the sum of the parts. Also, these techniques are executed at different stages of the SDLC so testing is never a bottleneck blocking a deployment or a ‘crunch’ focused on testing has to be performed. This redistribution of testing activities is often termed shift-left, shift-right, and Continuous Testing.

Conclusion

Large batch sizes have been proven time and again to slow down frequent deployments, and by their very nature test cycles are large batch sizes. Managers and test case management tools need to leave the comfort zone of using Test Cycles to avoid putting the frequent deployments (multiple times per day) that the business desires from DevOps and Continuous Delivery at risk of failing.

If you can’t avoid the use of Test Cycles, I strongly recommend you deliberately limit the size of the cycle to be as small as possible and execute them for each task or story. It may seem counterinitiative, but it does increase throughput.