Checklists

A regular correspondent (hello, thanks for triggering this) asked me about checklists and testing. I had a half-written blog entry on some checklist rules of thumb, and shared it with him. Here it is:

Personal

• Checklists I make are typically more useful than checklists I get. I expect this is more closely-related to ownership than quality.
• I tend to do items at the top first, so ordering may become important even if it isn't meaningful.
• I can't take in more than around 20 items on a list without giving it some structure. To help make a list comprehensible, I use outliners, item codes etc, and group items into categories.

General

• Laundry-type lists are different from shopping-type lists. Laundry lists tend to be set up as general resources, tend to be long, many items don't apply to current circumstances and so one picks a few, can be inspiring. Shopping lists are often made for a specific need and discarded later, one tries to cover everything on the list. I can use each type to enhance the other, but it's generally a bad thing if someone else uses one of my laundry lists as a shopping list and vice versa. Testing Example: "All my charters" is a laundry-type list. "All my charters for today" is a shopping-type list.
• The value given to a laundry-type list can come from an assumption that it is exhaustive, and that its items are mutually exclusive. Few lists are either – there are often missing items, and existing items overlap. Testing example: Lists of non-functional testing qualities. The real value in this laundry-type list is often that it inspires a shopping-type list.
• When categorising, it's important not to use categories as proxy items. Even if the list is exhaustive, many items can be re-categorised – so doing none of, or all of, a category can be more arbitrary than it seems. Testing example: charters grouped in a hierarchy.

My correspondent indicated I might be interested in a not-so-recent New Yorker article by Atul Gawande about checklists in medicine. Surfacing briefly into the rolling boil of tester blogs, it turns out that the article has triggered a gentle meme-ripple through the industry.

Gawande's article describes shopping-type lists, to aid memory and set out minimum requirements. It describes how those lists were used by medical professionals to help them do their jobs more reliably. The checklists covered the simple stuff, and called for no judgement or skill in their use (of course, plenty of skill is needed in their construction, and in following whatever comes next to the little tickety box). The results were impressive – but just as impressive was that the medics actually used the simple things.

It seems that the people involved were responsible for making their own lists (perhaps collectively rather than individually) and also for finding out if the lists were working. They were supported at multiple levels – nurses checked that the checklist was in use and the boxes ticked off, executives made sure that necessary supplies were available for the tasks on the list – so that there were few reasons not to actively use the checklist.

I'll add another note to my 'General' list above:

• When you're doing a difficult job under pressure, checklists help you concentrate on the job, by allowing you to expend less attention on avoiding mistakes. Testing Example: a list of possible input means (type, mouse, paste, drag in, cause default, undo delete, refresh).

The trick in this counter-intuitive heuristic is the difference between concentrate on, and do. A checklist can let your mind work more freely because the standard stuff isn't going to be forgotten. Indeed, I make shopping lists to go shopping with so I can multitask (for which read listen to The Goons on my iPod).

The article doesn't deal with two important ideas;

• Change; when+why do items come off a checklist (important for shopping-type lists).
• Use; what kinds of situation are most amenable to lists.

Aside from recommending regular reviews, I have nothing to say here about changing checklists.
Checklists generally help in situations which are:

• well-known
• busy (in the sense of being dense with stimulus)

It's clear that the massive confirmatory unit tests (and 'acceptance' tests) that characterise agile development can be seen as shopping-type lists, and are all the more powerful for it. The subject is well known (and the tests describe that knowledge) and the environment busy (in the sense of very many tests being run in quick succession). As a list, it helps exactly because it allows one to expend less attention on mistakes. The great (though arguable) strengths of massive confirmatory tests are, however, a special case.

Software development is certainly busy, but much of it is not all that well known. From a test point of view, often we're looking for the unexpected. From an engineering point of view, it's often hard to know what is reliably effective, sufficient and harm-free. Partly as a consequence of this, we tend to start out with laundry-type lists rather than shopping-type lists.

To pick out a few busy+well-known areas specific to testing, one might look at test environment setup, and the list of information on the top of a session form. Both these have a kinship with pre-flight checklists, and if you're not already checklisting in these situations, I expect you would find it valuable.

However helpful checklists are, they are frequently resisted as 'dumbing-down' a skilled task. As one who has resisted in the past, and who is likely resist in the future – I feel this is an entirely fair criticism. Perhaps the best bet is to take an approach similar to that taken by Peter Pronovost, the article's protagonist:

1. get whoever is making the {mistakes you're trying to avoid} to put together their own mnemonic/minimum standard list


2. get them to measure themselves with and without the list in use


3. provide strategic support to help ensure that there's no practical reason why something on the list can't be done, tactical support to help ensure that the list is actually used and used honestly.