Exploratory Testing - collected stuff

tl;dr: lots about Exploratory Testing

A colleague asked me if I had written articles or blogs about Exploratory Testing. Why yes, I have. It looks like it's time to share them more prominently.

Exploratory Testing is a crucial element that is often poorly integrated, so when I write about testing, I tend to make reference to Exploratory Testing. However, I don't particularly think of ET as the only interesting game in town, so when I write about it, I hope I put it in a relatively rational, consistent and practical testing landscape. Hence this feels like the first time I have consciously made an inventory of the stuff I've written about ET.

Here goes:

Of my long-form ET stuff, the paper that gets the most citations is 'Adventures in Session-Based Testing', which is about managing ET. I wrote it around ten years ago with the very brilliant Niel van Eeden. It won 'Best Paper' at EuroSTAR and STARWest. I updated it, so it occasionally gets called 'Further Adventures...'

The one I'm most fond of is 'Four Exercises for Teaching Exploratory Testing',* but although it went to the Workshop on Teaching Software Testing 5 back in 2006 and should have been part of the online materials, it vanished instead into online limbo. Astonishingly, when I searched just now, it's finally there - but orphaned from the rest of the site. If you can find a path to it from http://www.testingeducation.org/ I'll give you a hug.

You might already recognise the Black Box machines, which get plenty of attention, and are the single biggest cause of random strangers saying hello at conferences. Occasionally someone on a train (typically to or from Paddington) will say "aren't you... didn't you...", which is odd, but nice**. I know of a dozen or so people and organisations who use them for recruitment purposes – so let's call them my attempt at a balance to the useless five years I spent trying to nudge the ISEB exam marginally closer to fit for purpose.

Agile people tend to have come across Elisabeth Hendrickson's excellent Test Heuristics Cheat Sheet, which has my name on it. I contributed when she and I put our Exploratory Testing classes together for a (very enjoyable) 2-hander that we ran in London and in California.

There's plenty of other papers available at http://www.workroom-productions.com/papers.html, and many have related material. A Positive View of Negative Testing has more on techniques, Things Testers Miss is on bug stories, Testing in an Agile Environment covers my experiences of the fit (and friction) of testers - often using ET - on agile projects. In my most recent long-form paper, The Irrational Tester, I appropriated some fashionable ideas from behavioural economics, and I hope not only gave them testing context, but enabled more substantial exploration by drilling back from the pop science to the original research. That one got another 'Best Paper', this time from STAREast.

On that same 'papers' page, you'll find a short series of more conversational short-form things under the heading 'Exploratory Testing Notes'. They go with my Getting a Grip on Exploratory Testing workshop - they're not carefully-checked whitepapers, but nor are they short sharp blog postings. There's yet more that goes with the course, but it changes pretty much every time I do the thing.

As for blog postings, well, you're here - but Blogger is, ironically, a rotten thing for searching.
This is an entry on tools for ET http://workroomprds.blogspot.com/2008/06/tools-for-exploratory-testing.html ,
and this for assurance http://workroomprds.blogspot.com/2011/06/how-to-assure-exploratory-testing.html ,
and two together that may be of interest http://workroomprds.blogspot.com/2008_11_01_archive.html .

Reading this, it's obvious (to me) that lots of the other ET-related stuff I've written over the years has slipped away. That's the problem with the internet - ephemera are eternal, but useful stuff gets drowned. I'll fish some out and post them over the next few weeks.

Finally - I run a workshop from time to time called Getting a Grip on Exploratory Testing. It's all hands-on, and is limited to 12 people. I'm running a public class in Oxford on 25-27 January. Lots of friendly testers have twittered about it, and some (who have been on the thing) have recommended it. You'll need to look now, and book quickly, to get to the early-bird discount by the end of the week.



* Two things to note: firstly, the exercises describes are software, and available to all. Contact me and I'll send you them - plenty of people do, and they're used all over the world. Secondly, they're deep enough exercises to still be part of my workshop. By all means have a play, but if you're thinking of coming on my workshop, be aware that novelty is important to exploration and you won't get as much from the workshop. That said, I always have alternatives available if someone turns out to be familiar with the exercises.
** Sometimes they just want me to sing a song, which is still odd, kind-of-nice, and tends to mean they're a Bulgarian. Once, someone on a train was both a tester and a Bulgarian. We had lots to talk about. I should call him and arrange lunch.

Bias: Illusions and corruption

tl;dr we're all nuts

Two recent articles tickled my interest in bias.

In The Observer (How cognitive illusions blind us to reason, an extract from Thinking, Fast and Slow) Daniel Kahneman reminds us that cognitive illusions are stubborn, particularly when one is exercising hard-won, high-level skills. He illustrates this using stock traders, saying that their skill in evaluating the business prospects of a firm is serious work that requires extensive training. "Unfortunately, [this skill] is not sufficient for successful stock trading, where the key question is whether the information about the firm is already incorporated in the price of its stock."

In The Economist, All power tends to corrupt is subheaded "But power without status corrupts absolutely". It describes an experiment in which subjects were asked to select tasks for a colleague to perform. Some of the tasks were demeaning. Before they made their selection, they were given a job they might respect or look down on (the descriptions read a little like fun vs dull testing roles) and a sense of whether they or their colleague had more influence. Those put into the position of having influence but no respect chose significantly more demeaning tasks for their colleague.

Some testers tell me that they do a skilled and difficult job, but don't get much respect. I believe them – and I've found that the articles above have helped me understand my own behaviour a little more clearly.

Byalo Rade - new track from the London Bulgarian Choir

I don't usually do this, but apparently you're interested...

Here's a track from the London Bulgarian Choir's new album, Goro Le Goro. The album will be released on November 26th, with a big gig and party in London. I'll be wearing my furry hat, but you'll need to buy a ticket.




White Rada is sweeping her yard, her slender figure swaying, her arms like pale wings. As she sings, pearls flow from her mouth. ‘Beautiful Rada, my daughter, don’t leave your yard, don’t lift your eyes, don’t give your flower away.'

Something for the Weekend? 005 - Patterns and evolution

Catherine Young sees plenty in clouds.
(which reminds me: Q: what is the sky? A: all of the above from @gimboland/@posh_somme )

I've had a huge print of this sequence over my desk for years. Every time I actually raise my eyes and look at it, I think of change, excellence, strangeness, practice, talent, choice and stopping. Or something. Hope you've had a lovely weekend. Picasso's Bull suite.

A playful exercise for testers

tl;dr; here's part of a workshop

This exercise, like many exercises used with testers*, encourages people to discover rules** and build models. I last used it in public at Tony Bruce's excellent London Tester Gathering, and continue to use it at corporate clients. However, unlike most of my stuff, it doesn't involve software. Over the years, I've built up a small collection of bits of landscape. My assembly of palm-sized stones generally adorns one of my monitors in the studio, but occasionally has a second life in workshops. The exercise in which I use them is fun and seems valuable, so I thought I'd spread it about.

----

In a nod of respect to Johanna Rothman***, I call this exercise 'Bring Me a Rock****'.

Here are my instructions:

Set our your stones somewhere where everyone can see and get at them.
Pair up the workshop participants. If you have an odd number of people, pair up with the oddest yourself...

• Both of you: write down something to identify a rock. Keep it secret.
• Become ASK and FETCH.
  • ASK says "Bring me a rock"
  • FETCH brings a rock.
  • ASK accepts, or rejects the rock, based on the secret they wrote down earlier.
  • This continues until ASK accepts a rock
• When ASK accepts a rock, FETCH proposes a model
  • If FETCH is wrong, swap roles.
  • If FETCH is right, ASK writes down new criteria, and FETCH brings another rock
• carry on...

Meta-instructions:

• find a new pair if bored
• observe how you are modelling your partner's models

----

The exercise is fun, but gets a lot more valuable when the participants talk about it afterwards. My approach is to:

• discuss the different approaches to test and discovery
• pay special attention to the way that patterns are set up and broken
• apply learnings about discovery and patterns to testing
• avoid too much time spent talking about the different rocks

There's a little fiddle in the instructions that I'd like to draw your attention to. I'm sure you've noticed that the ASK role has a higher status than FETCH. With this in mind, it seems odd that FETCH should stay as FETCH if they correctly predict ASK's model. A playground sense of fairness means that the roles should swap on a 'win'. Do play it this way if you want to. However, I find that swapping on 'lose' introduces interesting, more subversive*****, behaviour. Say hello to the Imp of the Perverse.

If you run an exercise like this, please feel free to change stuff. Also, I would love to hear how the exercise worked for your group. Finally, I'd be grateful if you'd give participants a link to this post.


* I've commonly heard of 'mastermind' being used, but some notable testers use more mutable approaches with lots of rule exploration and discovery. Michael Bolton told me of his new coin meta-game the other day, James Bach and other use cards and dice. If you're a tester interested in games, you'll certainly need to watch Dale Emery and Elisabeth Hendrickson – and try to get along to Elisabeth's Agilistry Studio one of these days.
** There's a whole class of games whose purpose is the discovery of rules - here's Wikipedia's living list of Games with Concealed Rules. You might also want to have a look at Nomic. Nomic's rules aren't concealed, but their evolution is the game. There's a blog post about this around here somewhere...
*** Johanna Rothman's seminal article should be read by all: http://www.jrothman.com/Newsletter/Bringmearock.html
**** Some are smooth, some rough, some round, some flat, some solid, some holey. They're all sorts of different colours and sizes. One is incised with the work 'luck'. One is (was) a plum. Take your pick.
***** ie subverting the game, not each other. I find this approach rewards just-about-guessable reasons for picking a rock, so ultimately this helps people in the pair co-operate in in the learning, rather than allowing one party to stay in the high-status position just by being wilfully obscure. I'm proud of this wrinkle. You might think it's dumb. Your choice.

Something for the Weekend? 004 - Games with unknown rules

tl;dr - some games don't have fixed or known rules. Go here or here.

A long time ago, in a wooden house on the shoulder of a snowcovered mountain, I found myself sitting in a circle of people I didn't know at all. We were idly tossing a ball about. The ball was a complex thing with lights and buttons and fresh batteries – and as it moved from person to person, we shared arbitrary rules that were immediately forgotten. We noticed we had the attention of a couple sitting outside the group. They were, it emerged, players in the UK's Go team, and they were simply fascinated.

We continued to throw the ball. Occasionally, another rule would turn up. One of the Go players objected that our rules were inconsistent. Rising to the bait, we gently corrected him by explaining a hitherto unappreciated subtlety to the ruleset. More Go players were drawn towards the circle. As the Go players got down to organising and double-checking, our new rules explored variables beyond buttons and lights; the pause between throws, how hard the chuck, where the target was looking, whose friend they were, whether their name started with a vowel, whether a previous in-game action had temporarily changed their name...

This was, of course, nerd-sniping. But, more interestingly, it was an unexpected kind of game; the kind of game where the rules change. The Go players, on a team trip and consequently pretty much only in contact with other Go players, were more-than-usually locked into a pattern where rules were constant. Believing that all games have fixed rules is an easy habit to fall into.

Nevertheless, games with variable rules aren't unusual – I've spent happy hours playing bar chess, word disassociation, and parroting Mornington Crescent. If you play any games with a five year old, you'll know that rules are (a) important and (b) made up on the spot.

It's tempting to see the world as a game. Some people who explain the world use games correspondingly; as metaphors for the real world. The trouble is that games with fixed, agreed and finite rules are not always a great model. It's all very well to count heads and tails, but we forget that sometimes the coin falls in our tea and the resulting dousing trashes the laptop on which we're attempting to keep score.

The rules of our world are generally local, temporary and inconsistent. Very few rules are universal, very few activities not involving time, energy and accountancy are zero-sum. For me, the joy and value of maths, physics, music, cookery and coding is the discovery of rules; these deeper, emergent, unexpected truths. It's not about playing by the rules, but playing with. So, if you're going to play a game as a metaphor for life, you might consider using a game where the rules are on the unreasonable side of realistic.

All of which is a long and late introduction to: Something for the Weekend? 004 - Games with unknown rules
List of games with mutable rules
List of games with concealed rules

Enjoy.

James Bach moves very fast...

Something for the Weekend? 003

This turned up in Donella Meadows' excellent Thinking in Systems: A Primer*.
I've not seen it before, and thought you'd enjoy it.

A system is a big black box
Of which we can’t unlock the locks,
And all we can find out about
Is what goes in and what comes out.

by Kenneth Boulding

...there's more, of course. But I'm not convinced that posting the whole thing is a fair (ab)use of copyright. You'll find it on less fussy people's sites. Go fish.

* That's an Amazon affiliate link... trying it out.

7+3 = 11

tl;dr – a boring trivial bug is causing me to procrastinate by writing about it
tool – a spreadsheet to help you choose input data to spot the particular pathology

I love moo cards*.

Here's a snap of a recent bill from moo. Spot the bug.

100 cards £9.17 ... Shipping £2.50 ... VAT £2.33 ... Total £14.01 ... new blog post: priceless

The problem is a known pathology**. It's not uncommon to find that basket calculations are sometimes off by a penny; the calculations are done with precision, and those precise numbers are fiddled to fit with our quantum of currency - the penny. The error fits the fiddle.

In this case, the total including delivery and VAT looks as if it should be precisely £14.004. Expecting this to be £14.00, one might be tempted to speculate that the total has been rounded up my mistake, but two things make me not so sure.

Ⴀ) I generally see problems related to truncations (which always go down; £14.004 -> £14.00) and normal rounding (£14.005-> £14.01, but £14.004-> £14.00).
Ⴁ) thinking about it, I had a 10% discount on the normal price as a sop for the knock on effects of a previous bug. Discounts add another layer of complexity.

Let's work the numbers:

• £10.19 is the normal price.
• After the 10% discount, that would be £9.171, not £9.17.
• Add £2.50 delivery to arrive at £11.671.
• 20% VAT on is £2.3342.
• The precise total is £14.0052 - which will be rounded up to £14.01.
• The VAT component is £2.3342 - which will be rounded down to £2.33.

That seems more plausible.

Is this a rare combination of numbers? I built a spreadsheet to explore, and it is not; 300 prices between 1p and £10 show this behaviour.

All this is in the context of a 10% discount, 20% tax and £2.50 delivery. But my spreadsheet is a model, so I can change the conditions. Playing with it gives me the following empirical understandings:

• you don't see this problem without a discount;
• within reasonable ranges, picking alternative discounts doesn't change the incidence much;
• within reasonable ranges, changing the tax doesn't change the incidence much - I've seen it go down to 200;
• the range of incidence seems to be 200-300 for 'reasonable' ranges of tax and discount
• the delivery charge doesn't matter if it's to 2dp (and my model is inaccurate with 3dp)

Constraining myself to a basket with one item, I expect that I can sit down and demonstrate mathematically to my own satisfaction that in order to see a total that rounds up (ie £14.005), and an associated tax that rounds down(ie £2.3342), you need a price with a third decimal place - ie a normal price that has already been adjusted in some way. But that efficiency, while attractive, is a procrastination too far. For now, I'm happy with the general rule of thumb; you only see this problem when at least one thing in your basket can have a price that includes fractions of a penny - but if the potential is there, you'll see if for 20-30% of your possible prices.

Coders: One solution is do all calculations off-screen to full precision, but produce the totals on the bill from the numbers that actually go on the bill. Another is to round your total to 2dp before calculating tax. Of course this can mean having two containers for very similar information.

So far, so fun. For testers.

Frankly, I don't mind paying the extra penny. My problem is what the penny does to my paperwork.

I'm doing my VAT accounts, where I separate the £2.33 from the rest of the total. Moo's fluff on their bill means that stuff that should add up to zero, doesn't. I'll have to fudge the penny, which means introducing a special case. I'll have to be careful, because special cases are where I make accounting mistakes. That's a pain. I hope that you (or Moo) can use that description to advocate a fix for similar bugs.

And I hope that you go out there to find them. Here's the link to that spreadsheet again. I'll use it to generate data to help me reveal this issue***. You may use it and abuse as you wish. Please attribute me if you use it in public. It's got a second page that shows incidence, and a third with instructions, license and known bugs.

*  Those of you who have had a business card from me are charmed by them, too. Moo's custom postcards will lend excellent grooviness to a game I have in mind. I want to make special stickers for a bunch of post-it related activities. Moo have always responded swiftly and sweetly to problems, and  to top it all, they're local.

** I know this pathology, and I look for it when I test. Indeed, I've got an exercise based on something very similar in one of my classes. Some people question the veracity of that exercise; surely no-one really has obvious errors like this any more. Ha.

*** I tried a google docs version, but it runs like a three legged dog on Safari and Firefox. I was so discouraged I didn't bother with Chrome...

Something for the Weekend? 002

Briefly – I'm back in the studio for the weekend* – I'm fascinated by the way that technology enables interactive art. Exploration, discovery and emergent properties are desirable, even crucial qualities of the work.

Here are the sites of two people whose work I find especially interesting:

Brendan Dawes – and you should also try MagneticNorth
Robert Hodgin – who has more at his blog, Flight404

I'm thinking of getting to know Processing, before trying to get to grips with Cinder. Any of you got experience to share?

Enjoy the weekend.



* teaboy, mainly

Broken by design

tl;dr – I can't print or save a filled-in form generated by  HMRC software

A pet peeve, involving the Taxman and Adobe.

The Taxmen need a form. They'd like it online, and generously supply free software to help me get the numbers in the boxes. I use their software. It produces a form as a .pdf.

It's a dynamically-filled form*, so if I use one of my usual pdf readers, the boxes are devoid of numbers. Only with Adobe's reader can I see my numbers in the boxes. Adobe's reader is desperately slow and buggy, and I need to explicitly allow it to trust this locally-made form in order to see anything meaningful – but that's not my peeve. My peeve starts when I get to a point where the form is useful, and I'm shown a neat purple message:

        You cannot save data typed into this form. Please print your completed form if you would like a copy for your records.

Well, I would. Note that I've not typed any data into the form; it's been generated for me by the Taxman's tool. I go to print the form. I tend to print to .pdf, as I'm swamped with archived paper as it is, and a .pdf is both searchable and findable. A dialog appears, jauntily sporting the following:

        Saving a PDF file when printing is not supported. Instead, choose File > Save.

I consider printing it to paper**, scanning it in, OCR-ing the thing and calling it quits. Just in case, I try File > Save. No one will be surprised to know that I'm told:

        Data typed into this form will not be saved. Adobe Reader can only save a blank copy of this form.

A blank form? I'm sure that's what the taxman intended. The observant will notice that, as happens so often, following the instructions will put me into a self-defeating infinite loop. I've met this before, and that's my peeve.

It's big guns time. I pull out Acrobat 6 Professional. We're into software-that-costs-money territory here, and indeed have plunged straight into that unhappy valley of software-that-I-need-once-in-a-blue-moon-but-buggers-up-my-machine-to-such-an-extent-that-I-wince. Acrobat Professional is, for those of you unacquainted with Adobe's upgrade paths, ongoingly expensive. It also plays nasty with the other children in the sandpit, and doesn't do anything (except this) that I need.

A minute or two later, after it has managed to load, trashed the screen redraw, bunged the CPU to 100% and asked me to upgrade (not on your nelly, you eight-year-old, tired, hack, although I admit I have considered it), I try printing again.

        Saving a PDF file when printing is not supported. Instead, choose Save from the File menu.

There's that 'not supported' message again. Acrobat aside, I've not yet met an application that can print, but that can't aim it at a pdf. Perhaps I should set up a .pdf printer - but choosing not to address the bristles on that yak for the moment, I choose Save from the File menu, and - astonishingly - I can.


I suspect that by not supported, Adobe actually means restricted to the paid-for version. I suspect (suspicious tester that I am) that Adobe have done this on purpose. The taxman has chosen to provide me with a tool that throws my data away, unless I pay Adobe for the joy of keeping it. I wonder  whether the Taxman intended, condoned, or just didn't notice this behaviour.


Post scriptum***: As it happens, the tool turns out to be a dead end. The unprintable form is for my records only. Once I'm done slapping the desk, I fill in the online form in seconds and I'm done.



* For the initiated, this means that the .pdf (empty, pretty) is accompanied by a .fdf (just the numbers).
** Portable Document Format? My arse. Portable when folded up and shoved in a briefcase.
*** As distinct from PostScript. Print joke. Ah ha ha ha, bonk****.
**** Man laughing his head off.

Something for the Weekend? 001 (zero-padded in hope)

Wil Shipley writes code; I use tools he has had a hand in* at least weekly, more so when I’m onsite**. He seems to be an auteur, involved in all stages of translating ideas into code into cash. He also writes words – copiously, but no longer regularly as far as his blog is concerned. A few years ago, he wrote up a narrative describing his thought processes and discoveries as he worked through a rotten bug. It’s called The Greatest Bug of All, and is packed with meaty goodness.

For those of you who are more visual, those who are interested in variation and multiples, or those fascinated by the anonymous human touch, here is Stephen Wragg’s collection of walking men. Note the specification.

Enjoy.




* typically OmniOutliner and OmniGraffle, although Shipley has moved on since to focus on Delicious Library, which I don’t use so actively.
** If I’m allowed to use my own kit…