Charting a Course

My experiences in testing with an exploratory mindset and methodology have been positive.

Finding important, messy, and hard-to-nail-down bugs is easier than by using the old blindly-run-test-cases-anyone-at-all-could-do-because-little-thinking-goes-on variety.  I personally thrive in being able to use my experiences, curiousity, a variety of test techniques, and brain power to test software (see more in my recent post Be An Explorer!).

Moving into using session-based exploratory testing to help manage testing across a team and provide metrics to management was easy for me.  It was straightforward to write charters that were not so specific as to be test cases, but also not so generic that they were not helpful.  I was able to quickly write a charter that identified the mission and goal of a test session with appropriate depth and breadth.

In working with many dozens of testers over the years, I discovered that creating appropriate charters was not easy for many of them.

Many ‘charters’ I saw were a test-case in disguise, and in some cases, a really bad one that you could see right through to the meat-and-bones of it.  I wondered if these testers did not have the right mindset to be able to understand what made a charter different from a test case.  Or maybe they weren’t given the coaching they needed to be able to write a good charter.  Or maybe something else entirely.

Other ‘charters’ I came across were akin to a movie title or headline, like ‘User Phone Home’, ‘Save The File’, or ‘When Files Go Bad’.  Sure, these ‘charters’ could be appropriate in some situation, but they leave something to be desired in communicating the mission of a test session.

What should a Charter look like?

James Bach says that Charters:

Are intended to communicate the mission of a test session clearly and succinctly to testers who have already been trained in the expectations, vocabulary, techniques and tools used by the organization. Remember, in ET we make maximum use of skill, rather than attempting to represent every action in written form.

Elaborating and combining with my own experiences, the basic questions a charter is likely to address are:

• Where do you want to go?
• What do you want to learn about?
• What should you use to get there?

I recently worked with a group of testers in a workshop to practice different testing skills, paired testing, and in writing test charters and session notes.  Day-to-day, many of the testers were writing test cases instead of charters, which was in opposition to their work environment.  So one key objective for the workshop was to help them learn how to write appropriate charters that clearly communicated the mission of a test session.

Participants finished with a brainstorming session to identify what constitutes a good charter – results are included below.  The applicability of some attributes will depend on the skill and experience level of the tester, so consider your context and what fits for you.

Attributes of a Great Test Session Charter:

• Defines objective, a clear purpose
• Identifies the area of focus
• Identifies general scope of the test coverage intended (features, test approaches, depth of testing)
• May provide guidance on what to look for (if someone else will test the session, and is inexperienced)
• Uses language that encourages exploring!
• It is clear, concise, accurate, and to the point
• It is generic and non-precise in the domain of the focus area (NOT a specific test case – it wouldn’t be exploratory testing then!)
• May associate application under test with another similar application (as an oracle – this is helpful if application very new, and there is little information available about how it should work).
• Contains NO mention of expected results
• Does NOT direct the testing that should occur, or identify what the tester should or should not find
• Generalized, leave open to interpretation to who is running it (don’t assume who will test it)
• Provide guidance for things to look out for
• Bullets not numbered lists (if you are going to use them)
• Consider though that lists may be limiting for people (not mind expanding)
• Examples can be helpful if generic enough, helps guide and think of other ideas
• Be careful of being too generic!  Don’t want to necessarily create a charter that can be applied to anything
• All depends on context – whether people are experienced or not, whether you have information or not, etc..

What do you think?

Is it helpful to apply these types of attributes to help people create better test charters?  Are there any that you would add, or some you would take off the list?  Or, do you think that the writing of charters themselves is limiting and unhelpful?

I would love to hear from you about this!