Session-based test management

This post on session-based test management follows on from part 1 and part 2 in the series.

Once you have identified risks that you would like to test for, by creating a series of test charters, you are ready to begin an exploratory testing session. Exploratory testing is an exciting and challenging way of testing that reaps many rewards when done well.

So how do you go about running an exploratory testing session?

What is exploratory testing?

When it comes to exploratory testing, one of the best resources out there that defines and teaches others about exploratory testing is Elisabeth Hendrickson’s book Explore It!, in which she says exploratory testing is:

“...simultaneously learning about the system while designing and executing tests, using feedback from the last test to inform the next.”

So what does that mean exactly?

During an exploratory testing session a typical test would look like this:

So a test consists of our:

Knowing some information about our product
Coming up with a question about our product, based on what we know so far and designing a test based on that question
Executing the test we designed
Analysing the outcome of that test to learn new information about a product

In exploratory testing, we follow this pattern many times over until we’ve exhausted all our test ideas or have reached the time limit of our session (we can always run more sessions if required).

So you can imagine an exploratory testing session would look something like this:

The benefit of this process is that it allows a tester to explore the product freely and learn as much as possible based on their knowledge, skill and experience.

Your knowledge, skill and experience of testing are important factors when it comes to exploratory testing because you are ultimately responsible for what you test during a session. You have the freedom to explore whatever you like but also need to ensure what you are learning is useful for the rest of the team. Therefore, you need to strike a balance when exploring, and test charters can help you with this balance.

As you are testing, you can ask yourself if the tests you are executing are uncovering information related to the test charter or if you are going ‘off charter’. If you are going ‘off charter’ then perhaps you need to get back to exploring more relevant areas of the product. Or if you feel what you are testing is important, ensure that you run future sessions around the original charter to fill the gaps.

Building tests during exploratory testing

Having the freedom to generate any type of test during a session can be daunting. How do you test beyond confirming requirements? What if you’re testing the wrong thing and what if you are missing important tests? It takes skill to ensure you end positively answering these questions, so keep practicing at running sessions. The more you do, the better you will get at running awesome exploratory testing sessions.

Fortunately, to get you started there are some useful tools for exploratory testers that you may want to try out:

Heuristics - Heuristics are a fallible method for solving a problem or making a decision. We can use different heuristics to help trigger new ideas, but we have to use the right heuristic for the right situation. In testing there are lots of different heuristics out there (Elisabeth Hendrickson collated a series of them in a very useful cheat sheet) but here is an example of one:

During my testing, I find that my product contains records that are stored in a database and rendered on a web page. To come up with test ideas I use the CRUD heuristic which helps me come up with ideas to test for the Creation, Reading, Updating and Deleting of those records.

Remember, a heuristic approach may not necessarily be optimal. For example, the CRUD heuristic is useful when working with data. However, it wouldn’t be useful when thinking of tests around accessibility.

Focusing and defocusing - When you start an exploratory testing session you’ll likely start by taking in lots of general information. This is known as de-focusing where you’re exploring to learn broad, general details about the product. You can then take one of these details that you’ve learnt and focus on it. That means trying as many test ideas as you can think of to learn about a specific part of the product and if you identify other areas of testing, they get noted and put on hold. Once you have exhausted all the test ideas you’ve come up with, you can then de-focus again to search for the next detail you want to test around.
Organising your learning - Before you jump straight into deep-dive testing of a specific feature, it’s good to take time to absorb as much as possible before getting into the extreme edge-case testing. For example, starting with happy path journeys across a feature can give you enough context to start building up your test ideas. Next, you’ll want to explore the rules and boundaries around the feature to understand its limits. Once you understand a feature’s key workings and its limits you can then go to town with your testing!

Discovering bugs during an exploratory testing session

As you explore, your knowledge of what you are testing will increase, but how do you know what is a bug and what isn’t? When exploring and looking for bugs, functional requirements and specifications are limited in how much they can be used to judge what is acceptable and what is not. However, there are other sources we can use during testing known as ‘oracles’.

Oracles are fallible sources of information that can be used to judge information against, to determine if something is a bug or not. Functional requirements and specifications are one type of oracle also known as a ‘claim’. Michael Bolton has written extensively about the different oracles you can use during an exploratory testing session and has come up with the mnemonic FEW HICCUPPS to help remember them. Here are a few examples of them:

Comparable products - Products are not built in a vacuum and it’s likely there are similar products created by competitors. We can judge our product against popular features or standardised functionality in similar products to determine if we think something is a bug.
User’s desires - What a user wants in a product is sometimes lost in requirements. For example, you may have a requirement around an event occurring when a button is pressed. That requirement might be met, but what if it looks ugly or takes time to load? We can consider a user’s desires or expectations to judge whether those might be bugs or not.
History - As the product grows, it changes - both positively and negatively. If we see something that used to work but no longer does, then we can use the history oracle as means to justify it as a bug.

Creating useful test notes

The goal of an exploratory testing session is to learn as much as possible and then share that information with others. This will involve you recounting your testing story for that session an hour, a day or a week after its execution, so you will need reliable test notes. Your note-taking approach will influence how you share your information and debrief others. So if you are struggling to debrief what you’ve learnt based on your notes now, then what chance do you have of sharing that information two weeks later?

There are many different ways to take notes and you should experiment with different approaches such as mind maps, video, screenshots, the Cornell method and sketchnotes. Good test notes strike the balance between being detailed enough to help you tell your testing story and but not so detailed that they consume your exploratory testing. Some of the points to ensure are recorded in your debrief are:

What you did and didn’t test - Give an overview of all the different test ideas that you came up with, and perhaps any test ideas you recorded that you simply ran out of time for. You may also include whether you began testing something outside of the confines of your exploratory test session that was worth investigating (also know as going ‘off charter’). This might highlight new sessions that need to be arranged.
What you learned - For testers, it’s common for us to share negative things such as bugs with others, but we need to remember that our goal is to share information to allow others to make informed decisions. This means we should share everything we’ve learnt both good and bad so our stakeholders end up with a more complete picture.
What issues you faced - Issues such as test environments being unavailable, time-consuming creation of data sets or constant meeting interruptions can all have a negative effect on your testing and slow you down. You will want to discuss these to see how they can be remedied for future exploratory testing sessions.
What bugs you raised - You will also want to ensure that the bugs you have raised contain quality information that allows developers to quickly debug and fix any issues. This means not just sharing what bugs you’ve raised but also how you’ve raised them.

When to stop testing?

There are two ways in which you can finish an exploratory testing session successfully.

The first is if you run out of time. Exploratory testing is a mentally draining task and the more you test, the more you can’t help but focus on certain details and ignore others. Timeboxing an exploratory testing session isn’t just used to track when the session should be run; taking a break and running another session around the same charter once you’re refreshed can help you identify new details that might have been missed, and generate new test ideas.

The second is if you run out of test ideas. If you feel there are simply no more test ideas that you can come up and you’ve explored everything you feel needs to be explored for your test charter, then it’s time to stop. You may feel frustrated that you are missing testing opportunities, even though you have no more test ideas. However, this is where debriefs can help. A good conversation with another team member can help you identify new test ideas or increase your confidence in your test coverage.

Exploratory testing in summary

To become skilled at exploratory testing you need time and practice. Each time you run a session you will learn more about your skills, new testing techniques and heuristics you can use. Just remember these key points to keep yourself on track and you will be a skilled exploratory tester in no time:

Exploratory testing is about continuous learning and designing/executing tests
This gives the tester the freedom to explore and test much more than via other approaches
To ensure that there is a loose structure we test around a test charter to learn enough information to make choices related to that charter
We can use different test techniques such as heuristics to help us generate new and interesting tests
We also can use oracles to help us interpret our information and determine if there are bugs to raise
Whilst we execute our session it’s important that we have robust test notes