Automated scenarios is enough test coverage

Anti-patterns in BDD

Working with BDD can be a great way to increase collaboration and conversation among team members. It can help teams with their shared understanding and keep teams on track with their delivery. However, succeeding with BDD requires patience, teamwork and feedback — both on what works and what doesn’t work for your team.

Working as a team can be fraught with anti-patterns, whether they’re caused by biases, losing sight of common goals, or relying too heavily on tooling. In this series, we are looking at some of these anti-patterns, uncovering what they look like, how they are formed, and how they can be avoided.

In today’s article, we’ll dig deeper into the anti-pattern of considering that automated scenarios provide enough test coverage.

Automated scenarios and test coverage

Imagine you’re in a team that is regularly meeting, discussing and collaborating around requirements and from those conversations examples are being captured. The team is then automating those captured examples, which are then run regularly as regression tests to determine whether a build is suitable for release.

Whilst this is done regularly, when it comes to other testing activities very little else is carried out, and testers are trained and encouraged to focus on automating and maintaining the examples. This means the automated examples are the only testing activity that is being carried out.

Why is it an anti-pattern to equate automated scenarios to sufficient test coverage?

When we test our products, we test them to learn about how they work and use that information to:

• Challenge our assumptions

• Help teams make an informed decision about what next to do with their products.

We determine what we want to learn — which also means what we want to test — by discussing and identifying risks that might affect our product. The more risks we test for, the more we know about our product. The value in automating our examples (specifically up front before production code is implemented) is to mitigate the specific risk of a team not delivering what the business wants.

Automated examples guide development towards a common shared goal agreed up front with the business in collaborative sessions. But what about other risks? It’s wrong to assume that our automated examples tell us everything about our products. Other testing activities are required.

What’s the solution?

It’s important to remember that BDD is NOT a testing strategy, including when you automate examples captured in mature collaborative sessions. BDD can, however, facilitate testing especially when it comes to identifying risks that we want to learn about.

Make sure your teams identify risks during collaborative sessions by encouraging your testers to use their critical and lateral thinking skills, as well as the team’s domain knowledge of the product. The risks that are identified can help inform different types of testing activities that can be executed through automation or otherwise. This will ensure that your team is better informed when it comes to delivering their product.