When shouldn't you use Cucumber?

Cucumber is a great tool for creating living documentation within behaviour driven development (BDD) using test automation techniques. However, there are situations where it’s not ideal or cost-effective to use Cucumber. This could be in certain areas of a project’s testing strategy, or the whole project itself.

The regex tax

There is an overhead when using Cucumber. Cucumber works by mapping individual steps - a line within the scenario - to methods of code, and sequentially executing the matching methods when executing the scenarios. The mapping is done using regular expressions, which need to be created and maintained by the developers or testers.

Compared to writing test automation directly in code where you don’t have to write regular expressions or edit the steps in the feature files, in Cucumber you do, and this difference is the ‘regex tax’. While this tax can be small, the overall costs creep up over the many months or years that development is running. 

When the business is actively involved with BDD, living documentation and specification by example, then this overhead is worth paying since the human-readable format enables the business involvement, and the return on investment for avoiding unnecessary rework outweighs the overhead.

Taxing test coverage

Within BDD, scenarios are only created for key examples of behaviour and are not used to document every single edge case as tests. However, when teams look at or consider their test coverage, they often get concerned about the low coverage they have achieved using Cucumber. 

The typical response to the poor test coverage is to write more scenarios in their existing BDD Cucumber framework to increase the test coverage. This knee-jerk reaction is caused by the investment already made into their Cucumber framework, but the regex tax is overlooked within the equation. A team going down this route will quickly start paying the overhead of making the tests human-friendly, without getting any benefits for it.

Mixing tools and code

Adding an additional automation tool to avoid the tax when expanding test coverage doesn’t have to mean you throw away your investment in Cucumber. All the programming languages that Cucumber supports also have unit testing tools; just because they are called ‘unit testing tools’ doesn’t mean they can’t be used for other types of testing.

The foundation of good BDD Cucumber automation, or any test automation, is the abstractions, patterns, and libraries that you use for the code interacting with the application under test, for example page object pattern, test data builder pattern, and writing utility methods. This is where most of a project’s investment in Cucumber has been directed, and you can simply reuse these abstractions as a shared code base with a unit testing tool running in the integration, or as acceptance test phases to create new automated tests.

Conclusion

Cucumber is a powerful tool but its use isn’t always the most suitable one to use, particularly when you are pursuing high test coverage or a broad test automation strategy. Teams would do better to select the most appropriate tool for each individual job and should remember that more than one automation tool can be used within each testing stage.