Checklists

A regular correspondent (hello, thanks for triggering this) asked me about checklists and testing. I had a half-written blog entry on some checklist rules of thumb, and shared it with him. Here it is:

Personal

• Checklists I make are typically more useful than checklists I get. I expect this is more closely-related to ownership than quality.
• I tend to do items at the top first, so ordering may become important even if it isn't meaningful.
• I can't take in more than around 20 items on a list without giving it some structure. To help make a list comprehensible, I use outliners, item codes etc, and group items into categories.

General

• Laundry-type lists are different from shopping-type lists. Laundry lists tend to be set up as general resources, tend to be long, many items don't apply to current circumstances and so one picks a few, can be inspiring. Shopping lists are often made for a specific need and discarded later, one tries to cover everything on the list. I can use each type to enhance the other, but it's generally a bad thing if someone else uses one of my laundry lists as a shopping list and vice versa. Testing Example: "All my charters" is a laundry-type list. "All my charters for today" is a shopping-type list.
• The value given to a laundry-type list can come from an assumption that it is exhaustive, and that its items are mutually exclusive. Few lists are either – there are often missing items, and existing items overlap. Testing example: Lists of non-functional testing qualities. The real value in this laundry-type list is often that it inspires a shopping-type list.
• When categorising, it's important not to use categories as proxy items. Even if the list is exhaustive, many items can be re-categorised – so doing none of, or all of, a category can be more arbitrary than it seems. Testing example: charters grouped in a hierarchy.

My correspondent indicated I might be interested in a not-so-recent New Yorker article by Atul Gawande about checklists in medicine. Surfacing briefly into the rolling boil of tester blogs, it turns out that the article has triggered a gentle meme-ripple through the industry.

Gawande's article describes shopping-type lists, to aid memory and set out minimum requirements. It describes how those lists were used by medical professionals to help them do their jobs more reliably. The checklists covered the simple stuff, and called for no judgement or skill in their use (of course, plenty of skill is needed in their construction, and in following whatever comes next to the little tickety box). The results were impressive – but just as impressive was that the medics actually used the simple things.

It seems that the people involved were responsible for making their own lists (perhaps collectively rather than individually) and also for finding out if the lists were working. They were supported at multiple levels – nurses checked that the checklist was in use and the boxes ticked off, executives made sure that necessary supplies were available for the tasks on the list – so that there were few reasons not to actively use the checklist.

I'll add another note to my 'General' list above:

• When you're doing a difficult job under pressure, checklists help you concentrate on the job, by allowing you to expend less attention on avoiding mistakes. Testing Example: a list of possible input means (type, mouse, paste, drag in, cause default, undo delete, refresh).

The trick in this counter-intuitive heuristic is the difference between concentrate on, and do. A checklist can let your mind work more freely because the standard stuff isn't going to be forgotten. Indeed, I make shopping lists to go shopping with so I can multitask (for which read listen to The Goons on my iPod).

The article doesn't deal with two important ideas;

• Change; when+why do items come off a checklist (important for shopping-type lists).
• Use; what kinds of situation are most amenable to lists.

Aside from recommending regular reviews, I have nothing to say here about changing checklists.
Checklists generally help in situations which are:

• well-known
• busy (in the sense of being dense with stimulus)

It's clear that the massive confirmatory unit tests (and 'acceptance' tests) that characterise agile development can be seen as shopping-type lists, and are all the more powerful for it. The subject is well known (and the tests describe that knowledge) and the environment busy (in the sense of very many tests being run in quick succession). As a list, it helps exactly because it allows one to expend less attention on mistakes. The great (though arguable) strengths of massive confirmatory tests are, however, a special case.

Software development is certainly busy, but much of it is not all that well known. From a test point of view, often we're looking for the unexpected. From an engineering point of view, it's often hard to know what is reliably effective, sufficient and harm-free. Partly as a consequence of this, we tend to start out with laundry-type lists rather than shopping-type lists.

To pick out a few busy+well-known areas specific to testing, one might look at test environment setup, and the list of information on the top of a session form. Both these have a kinship with pre-flight checklists, and if you're not already checklisting in these situations, I expect you would find it valuable.

However helpful checklists are, they are frequently resisted as 'dumbing-down' a skilled task. As one who has resisted in the past, and who is likely resist in the future – I feel this is an entirely fair criticism. Perhaps the best bet is to take an approach similar to that taken by Peter Pronovost, the article's protagonist:

1. get whoever is making the {mistakes you're trying to avoid} to put together their own mnemonic/minimum standard list


2. get them to measure themselves with and without the list in use


3. provide strategic support to help ensure that there's no practical reason why something on the list can't be done, tactical support to help ensure that the list is actually used and used honestly.

Applied Improvisation

I just got back from the "First in a series of Applied Improvisation Network (AIN) London Events", and thought I'd share some impressions.

The AIN describes itself as "Spreading the Transforming Power of Improvisation". At this particular event, AIN founder Paul Jackson was going to "use improvisation activity to introduce a business theme ... [using] complexity/emergence as the business theme example". I decided to go along because it sounded fun. Re-reading this, I wonder if perhaps my taste for fun has become a little over-sophisticated. Never mind.

The evening was pretty much as described. Just under a dozen people turned up to the usual oddly-shaped room in a re-purposed building. We talked, interacted as individuals and as a group in a set of well-structured exercises, and pushed off for a pint/meal. All was fine and dandy, and I'll go again. I found the exercises interesting, and may adapt* some for my own consultancy and groups - LEWT people, expect to gather in self-selecting groups sometime soon.

However, in this evening's exercises, there was a frustrating focus on game over content. I was reminded of peer events I have attended which degenerate (and I mean degenerate) into good teachers swapping their favourite lessons. Enjoyable and informative, but I took much more away about facilitation exercises and ways to get people to engage in improv than about the structures and ideas of improvisation. Emergent behaviours were discussed, but as personal lessons emerging from an exercise, rather than as properties emerging from a system. Business was discussed, but in terms of getting business people up and interacting in workshops, not in terms of translating improvisational skills into their working environments.

Reacting to this, I revisited some of the ideas about improvisation that I was playing with over the summer, and present them here, tidied and decorated for your amusement.

I'm interested in the improvisation involved in exploring a city, making an extempore speech, singing harmony to an unfamiliar tune. We improvise when we cook a meal with whatever is in the fridge, when we need to get a USB key from behind that hotel radiator, when someone falls off a ladder in front of us, when we get lost - especially when we get lost. To be expert is to be able to improvise with confidence.

It's clear that my interest in improvisation is, in particular, the improvisation we do as individuals under pressure from external circumstances. Perhaps I'm just not a team player; after all, my sports are/were skiing, swimming and fencing. Ask the Choir if they agree before you jump directly to any conclusions...

Improvisation as AIN addresses it is useful, interesting, but seems (on the strength of a single meeting and a swift half) to be biased towards shortform group improvisation under circumstances imposed by the group. This is more complex in at least two ways, and a wonderful field of study - but the interests I list above would be poorly served if this was where improvisation stopped. Conversations indicated that, perhaps, improv was the only improvisation the group could discuss with engagement. I think there's more, and I look forward to interacting with the group and its approaches.

One-line summary: improvisation ≠ improv. Who knew!?


* at the workshop, no ownership was claimed, and no attribution given. One could use the viral meme (pace GPL) and apply the same rules when passing it on, or apply one's own standards if more stringent. I choose to apply my own standards - these exercises were facilitated, and may have been devised, by Paul Z Jackson. However, if it is the practice within this industry to change and neither claim nor attribute, I many yet adjust those standards to fit the context. For those interested in improv exercises, http://improvencyclopedia.org/ is a resource with more than enough (500+) to tickle your fancy.

Decisions and responsibility

There's plenty of mileage in group decisions, and in the wisdom of crowds. I presume that, as technology enables the convening of groups, we'll see more decisions made collectively. I hope - and believe - that in general those decisions will be better.

However, making a decision is not the whole picture. There is a degree of responsibility that goes with a decision, and I'm coming to the conclusion that a group decision is worth nothing if the individuals in the group are not prepared to take individual responsibility for their decision.

One-line summary: watch out for decisions made by groups whose members are disengaged

Agile2008 (and Glasto 2008)

Under the inspired guidance of Rachel Davies, Agile2008 modelled itself on Glastonbury Festival. I found myself performing at both, which came as a bit of a surprise.

Aside: I'll leave the muck and general debauchery of Glasto to your imaginations - suffice it to say that I was entirely sober, stayed at my Mum's, and was on-stage with my wife*. With 25 stages and 150000 revellers spread across a Somerset valley, Glastonbury's scale is as staggering now as it was when I first went in 1987 (no wife, no sleep, not sober). It turns out I'm no less impressionable at 40 than I was at 19. I loved it.

With 1500 people, Agile2008 was a couple of orders of magnitude smaller than Glastonbury. Like Glastonbury, it was as fascinating as it was overwhelming. Attendance was about as big as a relatively-technical hotel conference gets, but the truly staggering element was the 25 concurrent tracks. Being a power-law thing, this of course did not mean 60 people in each session, but hundreds in a few, and a small handful of patient listeners in most others. It meant that nobody saw more than a tiny fraction of the material on offer. However, the keynotes** were attended by a vast majority of participants, and served to align the subjects of conversation. With this, and the attention given to breakout areas, triggers for discussion, and informal entertainment/events there was a clear feeling of community. 

Performing or not (unless you're headlining), these events are at least as much about going and being with the crowd as they are about seeing the stars. You're as likely to love an act you stumble upon as an act you've waited years to see. I'm perversely proud of wandering away from the mighty (and very favourite) Massive Attack at the height (depth?) of their Other Stage thunder and into the elderly groove of the wonderful (and utterly new to me) Ethiopiques. I'm happy to have voted with my feet in Ron Jeffries / Chet Hendrickson's surprisingly artificial Natural Laws of Software Development and just as happy to have made the temporary acquaintance of that embittered sage, Brian Foote. I crossed paths with Toby Mayer half-a-dozen times, each time coming away with insight and inspiration.

The most interesting element for me was the degree to which the (formal) practices of Agility were not only reinvented by each team, but to a significant extent rejected. Two talks highlighted this particularly well:

Bob Martin's keynote gave the most visceral example, as he asked everyone in the room to put their hands up if they were involved with an agile project, then read a list of common practices and asked people to put their hands down as he listed practices which they did *not* do. By the time he had got about five items down his list, 1500 hands in the air had reduced to just one group, and a couple of dozen isolated hands across the hall. His next point; 'keep your hands up if all your tests are automated' took out the group (oddly enough, a gang of testers from Menlo Innovations) and only very few individuals remained in the game.

Scott Ambler's talk (which packed the 'Questioning Agile' stage/room) put real numbers on this phenomenon with a survey from Dr Dobbs in February. You can read his conclusions ("Agile in Practice: What Is Actually Going On Out There?") on his site, but better yet see a video of the talk (from about 5 feet from where I was squashed in) on InfoQ. You might be interested to know that he's made his data available for analysis. I've been looking through it from a testing point of view for a financial client, and his conclusion seems supported: "The easier practices, particularly those around project management, seem to have a higher adoption rate than the more difficult practices (such as TDD) ... For all the talk about TDD that we see in the agile community, it’s nowhere near as popular as doing a bit of up-front modeling, which we rarely hear anything positive about.". Indeed, I'd be tempted to say that the numbers indicate that practices related to testing are typically among the less-likely to be used. None the less, 80% of respondents felt they had better quality and happier customers. 

For those of you who are interested, my own talk (also on the Questioning Agile stage) went well (rather better than the Guardian Stage at Glasto), captured a good audience and generated some fruitful discussions. I took my slightly-jetlaggy part in the pre-conference "functional test tools" workshop (a physical extension of the ongoing discussion on yahoo group aa-ftt) which was worthwhile, but not terribly conclusive.

An excellent event - great for new perspectives, for new people, and for fun. I'd certainly go to Glastonbury again - and with any luck, Agile20xx.

~ o ~

* she leads, and I sing in, the London Bulgarian Choir. The lovely British Sea Power lent us part of their acoustic spot in the Guardian tent, and the girls sang with them for songs on the John Peel stage and the Left Field stage.

** James Surowiecki on diversity/wisdom of crowds, Alan Cooper on engineering user experience and iterative/incremental methods, Bob Martin on a 5th line to the Agile Manifesto ("we value craftsmanship over crap" - although I think there are efforts to make this more boardroom-friendly)