Workroom Productions

Comedy bug, and upcoming events

2012-01-31T11:19:00.000Z

So I saw this, and I thought you should, too: Just an Ordinary Day in Skyrim . Particularly funny bug at 20s or so in.

While I'm here*, the Exploratory Testing workshops in Oxford went very well, thankyou. Those of you in the know will already know that I'm organising a 2-day workshop on exploratory testing techniques in Amsterdam on 12 and 13 April. Contact me if you want to be kept up-to-date (and to get the earliest-bird discount). Details to hit the web soon. There's been a bit of a hiatus in the 'lots of ways to manage ET' series, but the rest are in the works. Expect to see one at most this week, none at all next week. So much for writing a new post daily; turns out day-to-day stuff gets in the way.

If I was around, I'd be setting off to one of the following low-cost local events;
Paul Gerrard's UK Test Manager Forum is next week, in London. 2 days (tutes, talks and dinner) for just under £300 inc VAT. Maybe, on reflection, that's not a particularly astonishingly low-cost event. The TMF's track format has a keen eye for conversation rather than lecture, which is all to the good, and so it's hard to pick out a particular recommendation. In the tutorials, look out for Dave Evans' hands-on workshop 'Specification By Example, in Practice'.

Chris Ambler is organising TestFest in Brighton on Feb 22. Tickets are £60. As he describes it, it will have not one, but two parallel interactive elements demonstrating test tools and approaches. Twin TestLabs anyone? The event has a clear focus on Brighton (and the UK's) gaming industry, and is as far as I'm aware the first time that a testing conference has paid the large and creative community much attention beyond a couple of track sessions. There's lots to discover from game testing (see bug above), and it looks great. Anyone with an interest in where we've come from will want to know that Geoff Quentin** will be in evidence.

++Missed this in the initial posting... ++
TCL is running a Zappers event in London, also on February 22nd. Zappers is a great opportunity to meet and talk and test. The event is free, and TCL don't simply lay on food and beer, they also set up a software/system target or two, and award prizes for finding bugs. TCL organise regular Zappers events all over the world, and huge kudos to them for doing so.

++Missed this too... ++

Tony Bruce and Nathan Bain set up the Agile Testers meetup a few years ago, and Tony's persistence has made it a monthly must-do. The next is on Feb 27th. Breaking the mould, (the meetup is generally in a pub, and is more about talking to each other than being spoken to) it's at SkillsMatter in London, and Chris Guest from Microsoft will be talking about Powershell.

++And another...++
The Agile Tester's Meetup is back to its usual format a couple of days later. From 5:30 on Wednesday 29 Feb (leap evening, so perhaps there will be a tale to tell about bugs of the day from iPhone owners) at the Shooting Star opposite Liverpool Street station. Free entry, good chat. Sign up on LinkedIn or (my preference) Meetup.

The SIGiST is doing its thing on 7th March for £132. Thankfully, this time you've got Lee Copeland, Julian Harty and Lloyd Roden, all of whom are worth your time. As a special treat, the usually-fascinating Allan Kelly gets a 15 minute slot too.

Ministry of Testing (the ever-present STC in ninja disguise) are running TestBash in Cambridge on March 23rd. It's a day long, and costs £99-£150 depending on how fast you move. Among the excellent selection of speakers, Steve Green and Alan Richardson are two of the best practicing and practical testers I've worked*** with. Not only this, but both have a keen focus on the exploratory end of the spectrum, and both have novel and well-thought through things to say.

* I'm not going to be nerd-sniped into diagnosis, but it seems that you won't have seen this posting until Monday 6th, though it's dated and was written January 31st. Dave's probably already packed his bag for the tutorial.
** Geoff played a vital part in initiating a number of things that now seem a necessary part of our industry (the SIGiST, EuroSTAR, BS7925 and the awful exam, various training organisations and approaches). If that list seems a little old-school, remember that's what initiating means. One reason for the longevity of these bodies is that Geoff built their sustaining communities by seeking out participants whose views were at odds with his own, but who in some sense matched his keenness to communicate and engage with other testers. If the SIGiST, exam and so on now seem moribund, that is perhaps because of the lack of diversity of their current participants. Personally, Geoff has been a crucial and much valued influence – and we taught a class together that was built in part around our fundamental disagreements.
*** Worked with as in: found real bugs in real stuff for real money, working with real people for months at a time. That's what they do.

Known ways of managing ET #05 - Off-Piste Testing

2012-01-17T15:25:00.000Z

tl;dr – scripted guides may not help exploration

Team leaders tell me ‘My testers use manual testing scripts*, but I want them to do more than just plod through – I want them to find problems’. This seems a reasonable idea, but is fundamentally incoherent; ‘Finding a problem’ is not necessarily a complementary goal to the action of ‘following a script’. However, it happens. Let’s look at two common approaches. I’ll call them Iron Script, and (by way of contrast) Marshmallow Script.

Iron Script
The scripts remain authoritative. Testers are expected to deviate at agreed points, typically by changing their data within reasonable limits. Adding new events or changing the path is frowned upon; in extremes, a bug that is found using a non-standard path may be rejected, and the tester asked to reproduce the bug on the accepted path. If you can get some kind of pre-approval for diversions taken through error states and validation checks, you’ll open up a bunch of interesting routes whose valuable information might otherwise be used against you by project pedants.

It’s my experience that scripts in these situations frequently mirror end-user activity, and that the steps focus on triggers at the expense of observations. If your testers must run through the script, then they must, but don’t let them get dull. Remember that you are a team of testers, not users, and that you can still get unassailably-helpful information from querying the database, watching the CPU, intercepting transactions, using a diff tool on the filesystem, or any other neat trick that takes your fancy. Constraints breed creativity.

Marshmallow Script
The scripts are a guide, a collection of hints or waypoints. Testers can deviate wherever they want, using the scripts to get to interesting points, or as a checklist to verify their expectations. The scripts act as fat charters, and by giving their names to paths, help testers to talk about what they’ve done and what they’ve found. This isn’t bad, as far as it goes.

However, the approach puts happy paths – reliable routes that demonstrate limited capability – at the core of decisions about what to test and how to test it. This emphasis can be a terrible drag on the swift revelations that might be desired from unfettered ET. It can wastefully restrict your testers’ imaginations, and seems to reinforce manual testing at the detriment of small cheap tools.

I tend to find that these approaches exist in parallel, but may not be acknowledged as such. It is powerful – sometimes, too powerful – to wonder out loud whether the team as a whole is looking to move away from their scripts or to add to their scripts. This can turn out to be emotive enough to be ignored in polite company; bringing it up in public can make people very impolite indeed.

One might question why the team is writing scripts at all. Scripts are expensive to make and hard to maintain. If they exist to give the testers a framework to explore requirements and product use while writing them, other frameworks might work just as well. If they are primarily a guide for novices or a crutch for the feeble, then perhaps one needs to rethink one’s approach to learning, and possibly to hiring. If they are primarily a way of recording work, then why not record the work with something more unambiguous, or more searchable? If they exist because an environment is hard to automate, then I would wonder if everything scripted is quite so hard to automate. If they exist to keep testers on a leash, then I have no further questions.

These are, however, rationalisations of a generally irrational position. I think the answer lies not in conscious choice, but in habit. The approach seems common in situations where budgets are only available for work that can be measured with reference to requirements and scripts, yet where the test team and its decision makers know that their survival-in-current-form requires fast information about emerging trouble. Maybe it’s endowment bias; if no one wants to chuck away all those scripts they’ve been working on so hard, then the scripts will remain central to the work no matter what the work is. In the first, future plans don’t match current practice. In the second, neither does the past. I often see both. Is it any wonder that the team lead’s goals might not match their means?

As a skier**, I’m drawn to the term ‘Off Piste’, and the term ‘Off-Piste Testing’*** seems a popular metaphor for this approach. Between the mountain top and the snug chalet in the valley floor, there are many paths: some groomed, marked and filled with tourists; others quiet or exposed, with cliffs and bears. There is an implication that off-piste is only for the most skilled, keen and well-equipped skier. The implied kudos is eagerly lapped-up by testers, and off-piste testing can be used as a motivator with two caveats; status should be earned through good work, and good information can gained from diverse approaches. Whatever the rules of the mountain might be, it is perilous to restrict off-piste testing to your elite.

More importantly, off-piste is still downhill. Scripts, whether used as hard or soft guides, bias testers towards a set of activities that most typically follow whatever actions are expected to be valuable to the user, system or owner. These activities are not the only ways to find problems. Those who manage exploratory testing by running after scripts will handicap their team.

* For this blog post, script means a step by step set of instructions to be read through and followed manually by a tester. Some of you may be aghast that such things are still in use. Some of you may see no alternative. For each of you, please believe that each position exists.
** Note to prospective clients – if you're near a properly-skiable mountain and book me to come to you close to a weekend during the season, I may have a seriously-tasty winter offer for you.
*** ‘piste-off testing’, anyone? Just me? Hey ho.

Uncommon Ways of Managing ET #04 - Post-partum Labelling

2012-01-10T16:34:00.000Z

tl;dr – tl;dr your ET notes to see where you've been

I’ve worked with plenty of testers who don’t timebox their time, don’t set out a charter before testing, and don’t do formal debriefs. Clearly, they’re not following session-based testing, but that doesn’t mean they’re necessarily doing bad exploratory work. Indeed, some of the most brilliant exploratory testers I’ve worked with are fully able to do all these things yet choose not to for much of their exploratory testing.

Personally, I almost always have a timebox, and find I prefer my results (but not my activity) if I make good notes. I can find charters trivial or restrictive, and debriefs can lead me to remember only the edited highlights of my exploration – so if my debrief sucks, my memory can be less useful than if I’d not debriefed at all.

Charters, timeboxes, notes and debriefs have a value to the team and the project as well as to the tester. If the team habitually relies on them, but an individual works best without them, then you’re faced with a choice of whether to force that tester towards an ineffective discipline, or whether to damage team performance. Which is no fun.

This then is a simple and obvious alternative, but I’m not aware of much that has been written to describe it. Nonetheless, I’m sure that many readers will recognise the activities below, and I can’t claim that this is in any way novel. Perhaps if no one’s written about it, it doesn’t seem legitimate, so no one writes about it. Perhaps I’ve just forgotten what I’ve read. Anyway, the following is a collection, and to that extent an imaginary extension, of stuff that has worked for me. I’m going to call it Post-partum labelling*. If you’ve got a better name, or know where someone else has named it, super. Comment away.

After a chunk** of testing, the exploratory tester describes their work in a sentence or two. They write this up in public. For example:

8 Jan – 60 minutes: Ed used a javascript profiling tool to analyse the web app.
8 Jan – 120 minutes: Sue spent 2 hours exploring reported instabilities related to switching sort order while autosaving.
9 Jan – 180 minutes: Brin and Rudi spent 30 minutes watching two users interact with the demo app for the first time, and spent the next 60 minutes reviewing and annotating video of the sessions.
10 Jan – 180 minutes: Jonno spent 3 hours on batch input, generating 3088 files that together contained all possible orderings of 5-step transactions.

This works well if you’ve set aside time for experienced and self-directed explorers to test. If you’re expecting a terse diurnal list like the one above, you might find it to be a good fit with daily news. It’s perhaps not such a good fit if you’ve got testers who have problems with focus, or if your test approach means that your list grows by more than half a dozen lines a day.

The list won’t help you know where testing is going, but it’s great to help you know where it’s been. Everyone in the team can see who explored what and when, so you know who to talk to, you know what’s been hit recently, and your memory and understanding of the system’s recent history has enough to help fill in the blanks. The team knows what it is paying attention to, and knows where individual interests lie. I think this is generally more useful than having an obscured testing genius bringing the system to its knees in interminably unfathomable ways.

Writing a post-partum label helps me put most recent test activity behind me, and allows me to think diversely as I enter the next round. I like knowing that I’ll need to write a public one-line summary of my hours of discovery; it helps maintain focus.

While I like a timebox, you might not. I wouldn’t insist on timeboxes if I was doing post-partum labelling. The people in the team know the budget, and they’re already trusted. The exploration is done when it’s done; forcing a timebox is a silly micromanagement. However, if people on your team are prone to pissing away their time and don’t embrace timeboxes or similar tools as part of their personal discipline, they’re probably not the best people to be doing post-partum labelling.

It’s time to change approach when your post-partum labels turn into “looked at login, again” or “checked out last week’s bugfixes”. If your label can be made before exploring, then it probably should be. Post-partum labels arrive after, and may not fit what you would have expected at the start. If you’re exploring, this is often a good thing.

Please, don’t get the impression that the label is an adequate substitute for notes. Sometimes, awfully, unfortunately, that’s what it is. Try to avoid this.

I’ve used similar approaches when I’ve been the exploratory addition to a team that has been relying solely on scripted or massive and confirmatory tests. I found it helpful when we had more test ideas than we could easily manage, and yet had target pathologies, observations and triggers that urgently called for our attention. Post-partum labelling helped me fit my work with other explorers and the rest of the team, helped me gain trust by offering visibility, acted as a trigger and conduit for other people to bring me ideas. It let my team spin very swiftly back through a couple of weeks of exploration, identifying which set of notes might hold relevance. It gave explorers who weren’t happy with SBT fit into a team that was trying to gain the disciplines of SBT. It wasn’t much good for assessing coverage. It didn’t link to requirements. It was rubbish for steering. But I liked it.

I’m very tempted to extend the idea further. I want to capture the information electronically. I want to add tags, to allow me to analyse where we’ve been spending time. I’m keen to describe problems found. I’d like to try using Stefan Butlin’s interesting TestPad web app (and I shall, it’s neat). However, these adjustments change the emphasis of the list. Have a look:

8 Jan – 60 minutes: Ed used a javascript profiling tool to analyse the web app. [code, performance, UX] We’re spending plenty of time inside check_constraints(), which looks recursive.
8 Jan – 120 minutes: Sue spent 2 hours exploring reported instabilities related to switching sort order while autosaving. [instability, UX, autosave] She found a reproducible crashing bug, logged a couple of UX issues, and identified potential exploitation.
9 Jan – 180 minutes: Brin and Rudi spent 30 minutes watching two users interact with the demo app for the first time, and spent the next 60 minutes reviewing and annotating video of the sessions. [UX]We identified and logged UX Issues around context menus, the hiding menu bar, and error messages.
10 Jan – 180 minutes: Jonno spent 3 hours on batch input, generating 3088 files that together contained all possible orderings of 5-step transactions. [batch, instability] The system correctly accepted 182, correctly rejected 2900, but hung on 2 that it should have rejected. No bugs logged yet, as we think this may be to do with a mistake in the configuration data in the test system.

Do you find yourself skipping over stuff now? I do. It’s as if it’s all too much to hold together. You’ll be keeping this information somewhere else, too, I expect – and I think that’s where it should stay. Keep the list single-purpose. You’ll find it lives in people’s heads more easily and more consistently, becoming part of the shared consciousness of the test team. And how cool is that?

* made-up name. Obviously. Post-partum is a latin term used to refer to the mother after giving birth (as opposed to post-natal, which apparently applies to the baby). You know what a label is. I want to get across the idea of a tester giving their work a unique and meaningful title, once it’s done.
** a chunk? What’s a chunk? I find that my mind merrily organises memory and activity, and groups the similar and temporally-close. If you have control over your interruptions, you’ve come to the limits of your chunk when you choose to change task. Sometimes, you don’t choose consciously. It’s still a chunk. My chunks of time testing are often hours. Writing, just minut… hey! A squirrel***!
*** I can see six, right now, in the evergreen oak outside my window. No, seven. Five. A parrot!

Known ways of managing ET #04 - Set Aside Time

2012-01-05T20:21:00.000Z

tl; dr – scheduling ET changes the game. This is how to cope.

The team decides to budget a fixed amount of time for exploratory testing. Of course, that’s not the end of the story. This post is about what happens next.

First some background and disclosure: This sort of decision has been the trigger for a fair proportion of my client engagements since around 1998*. So I see this more often than I might. Generally someone on the team has eloquently persuaded a budget-holder that the project will find value in time spent exploring**, and I get to work with fresh, enthusiastic and newly empowered testers. So I find the situation delightful. Who wouldn’t? I’m sure these two complementary perspectives colour my experiences, and I expect they have coloured my writing too.

Budgeting a chunk of time to explore what you’ve made is a fine idea. As a management approach, however, it’s a bit hands-off. Sometimes, neither sponsor nor enthusiast has worked out the details of what is expected from whom and by when. More particularly, although there’s often a great sense of adventure, there’s not much consideration about the strategies for coping with inevitable change. Here then are some of those changes, a few related pathologies, and some strategic and tactical tweaks that have worked for me.

Dealing with lots of ideas

There will be a monstrous growth in the number of testing ideas; test teams have no problem coming up with new ideas about what and how to test. The practical problems lie in picking the best, dropping the newly-redundant, classifying and slicing and managing the set. Picture people trying to stuff an octopus*** into a rubber sack. This is a natural part of testing; one of the characteristics of wicked problems is that you can’t pin down the solution set.

As with all monstrous growths, the quantity of test ideas will be limited by external factors. If you’re keeping ideas on sticky notes, you’ll run out of wall space – which is perhaps better than putting everything in an expandable database that no one bothers to read. The most usual limits**** are to do with patience and attention span. When working within the set, the team will learn crucial filtering and throwing away skills, but will also run up against endowment bias; it’s hard to let go of something you own, and harder to let go of something you’ve worked hard to gain. There is likely to be conflict or denial – all the more so if the team has to date been under the consensual hallucination of potential completeness which underpins script-only approaches.

The growth in quantity may not be not matched by a growth in diversity or quality of ideas. This is only made worse by a testing guru (sometimes me) who sees it as his or her job to come up with new ideas. A good way to defuse some of this problem is to encourage the team to not only add to the ideas themselves, but to challenge ideas and remove them from play. If you make your influential tester the champion for diversity or quality in the collection, that can help too. I’ve often seen teams hamstrung by an inappropriate attraction to combinatorial collections; given a set of test ideas, someone draws up a table of (say) data type against input, and works through, left to right, top to bottom. Stop this in its tracks; tables and ordered progressions indicate a single real idea, one which is ripe for heavy optimising with a tool. If you can’t automate, decide which combinations are most important to do right now. If you can’t prioritise or optimise, hit the combinations randomly and learn from them as you go.

I like to constrain the numbers of test ideas in play by saying how much time an idea might need, and keeping the total amount of time needed under some reasonable limit. Although I can get as sucked-in by combinatorials as the next tester, I find that I tend to prefer diversity over depth. I try to temper this bias by paying attention to the agreed strategy and the current context – which means it’s good to have talked about what is valuable to the project. If I find myself pouring out ideas in some consultantish denial-of-service attack on the capabilities of the team, I’ll find a corner and write until I’ve calmed down, then see if my list triggers people to put their own ideas up, rather than fill the wall with my collection.

Dealing with fast feedback

Exploratory testing makes some feedback much faster, so a commitment to exploratory testing will change the feedback characteristics of the project. If there is a personal connection between the designers, coders and testers to support this feedback, the consequences can be excellent. Problems get fixed quickly, the team can develop a better understanding of what quality means on the project, and testers swiftly find out what they need to supply in a useful bug report. I’ve often seen palpable increases in respect and communication, which leads to trust and an overall greasing of the machinery of a development team.

Teams who have built massive confirmatory automated test are used to fast feedback, but of a very different flavour from that provided by exploratory testing. Feedback on on novel modes of behaviour and unanticipated failures can deeply challenge a team who thought their design was watertight. I’ve been told that bugs aren’t bugs unless they’re found by a customer, and that any bug without an immediately-obvious fix is a feature. I see both of these reactions as denial, and an indication that I should have prepared the ground better for the kind of feedback I can offer from exploring the product. The situation is made much easier if you are working with a real customer or user, rather than acting as proxy. The cry of pain might also indicate that your testing priorities don’t match the priorities of the people constructing the system – it’s your call whether you need to re-align, or whether you should embrace the difference. I’ve written more about the correspondences and conflicts of exploratory testing on agile teams in Testing in an Agile Environment.

More problematically, some people don’t want feedback when they’re in the throes of making stuff. I’m one; it gets in the way of the fragile extension of the imagination that is at the heart of how I make new things. Some testers are remarkably insensitive to this, others imagine that they need to somehow sweeten the pill. When I have results or questions, I prefer to make it known that I have feedback, and to give that feedback either to a group at a time when everyone is tuned in, or to an individual on invitation. Of course, it’s great to be able to march up to someone’s desk and get an immediate response, but what’s good for you might not be good for your colleague. Decide whether it's the right time to ask before you get between momma bear and her cubs.

As a test team gets closer to its audience, some will imagine that the test team risks losing its independence, and will resist – for instance – exchanging design information or locating the testers a shout away from the coders. Isolating the testers is an obvious but stupid way of encouraging independence of thought. You’ll find more in The Irrational Tester.

Conversely, test teams who have no existing connection with their designers and coders throw their feedback into a void. Swift, accurate and targeted information might seem valuable, but is worthless to the builders if it is delayed by procedure and snowed under by noise. The feedback in this case is primarily for the users (and sometimes the sellers) of the software. It’s crucial to understand your audience.

Some legally-minded people (and some sellers) don’t want information about new failures and will restrict or censor feedback. Some need plausible deniability, and don’t want to even look. If you have this situation as a test manager, messing about with ET won’t fix your problems.

Dealing with decisions and organisational oversight

Groups that are new to ET tend to see a large expansion in the number of problems found before they see a reduction in the number of problems made. More than once, when I’ve been dropped into an agile team as an exploratory tester and customer representative, I’ve had to take the judgement about whether to horribly disrupt the vendor team’s workflow by filling it with bugs. Clearly, if relations are good, blowing the workflow is bad – even if it is an indication of a crisis ignored. So far, I’ve managed to avoid purposefully blocking the flow. However, although it is a decisive and disruptive step, new exploratory testing groups can fatally disrupt the flow easily, unconsciously and even gleefully (which is nauseating, but happens). When bringing ET into a project, it’s vital to have awareness of this dire ability throughout the project team. If the workflow is king but the quality poor, decision makers will need to prepare for compromises on all fronts.

Once exploratory testing is chugging along, you hope to reach a point where fewer bugs are being made. I’ve had complaints from metrics people that fewer bugs are being found. This is a fine demonstration of failure demand, and I find it easier to set it as desired goal at the outset, rather than have to explain it as a side effect. I’ve found it useful to put this in terms of ‘we will not necessarily find more problems, but we will find more useful problems and find them faster’. Similarly, some metrics people count a day when lots of problems have been found as a bad day; it’s easier to help them deal with their pain if you’ve already had a chat about how much worse it would be if all that trouble was revealed later.

A decision to be hands-off can make some managers feel insecure. This feeling may lead them back towards the test team with unexpected needs for oversight and control*****. To avoid this, any team that has been given a degree of autonomy needs voluntarily to help their sponsor feel secure. I find that it helps to make a clear agreement not only about responsibilities, but about material deliverables. For instance: “We will keep a steady pace of exploration, shown by a public counter of time spent. We will display our test ideas and will be able to discuss them at any time with anyone on the project. We will make a visual distinction of those ideas which we have explored, those we are about to do, those we will probably do, those which we have recently eliminated, and those which have recently arrived. All time spent exploratory testing will be documented, and the documentation kept at <link/location>. All bugs logged from exploratory testing will be cross-referenced to their discovery documentation. Where we cannot keep to these commitments, we will make a collected note of the exceptions. We will come to you at the start, end and at regular intervals throughout significant testing efforts to keep you up-to-date. We will come to you with requests for help with obstacles we cannot overcome ourselves and with decisions about changes to budget and staff, but not with decisions about test direction and prioritisation. You will allow time for our regular reports and requests, and will champion our autonomy as set out here. If you are unable to give us the help we ask for, you will let us know swiftly and with reason.”

Budgets change. Sometimes a project wants more exploration, sometimes there’s less available to spend on it. While the test team may have started out with a fixed exploration budget, and may be comfortable cutting or expanding its testing to suit, there may be questions around how it would drive a change and require more (or less) from its sponsors. This is to misunderstand testing as a service – the people to whom one provides a service will be the people who ask for more, who want less, who adjust the balance. Clearly, the test team will be engaged in the negotiation, but I would question the motivation of a test team that prefers its own budgetary decisions over the informed decisions of its customers and sponsors.

Lots of teams seem scared of putting exploratory testing in front of auditors. I’m not sure why; the auditors I’ve met seem to do a lot of exploration in their work, and I’ve always found it helpful to ask the appropriate auditors about what they might expect to see from exploratory testing before we start exploring. If there is, for instance, an unadjustable regulation that stipulates that all tests must be planned, the auditors are not only most likely to know about it, but to be able to give you an indication about what they might accept (i.e. charter+timebox in plan post-hoc). I understand that session-based testing was developed in part to allow exploratory testing to be audited. If auditors have a place in your organisation, then it’s better to expect them****** than to hide; talk to your auditors and negotiate what they need for assurance. I wrote a note about this here on the blog in June 2011: How to Assure Exploratory Testing.

Crisis reveals underlying truths. I can't recall a project that has identified every necessary task, or given each task the time that was needed or budgeted. Testing, especially when considered with a potentially-unlimited discovery element, is eminently squashable and so is usually squashed – which tends to reveal uncomfortable truths about how the overall organisation understands testing. If exploratory testing is squashed out of existence when testing as a whole is squashed, your decision makers see ET as a luxury. If exploratory testing takes the whole pie when (but only when) testing is squashed, decision makers see ET as a short cut. Both these positions are pathologies. You might be able to spot them early by indulging in a spot of scenario planning, or you might trust your instinct. I work from the position that testing is a service – mostly optional, generally valuable – which I find both reasonable and benign, but my position could be a pathology in your organisation.

As a team grows into exploration, it will develop a library of tools. By tool, I don’t mean a downloadable executable, but something that arises from the combination of mindless machinery with data, configuration and conscious application by the minds of the team. A chainsaw needs a lumberjack*******. Some tools arise as testers automate their manual, brain-engaged testing – and as the automation takes over, the tool will change the way a tester triggers problems and their ability to observe surprises, not always for the better. Other tools arise because they enable a whole new class of tests, and using the tool even briefly exposes a new perspective with its own easily accessible bugs. A tool armoury is one of the core assets of an exploratory testing team; exploratory testing without tools is weak and slow. As with any library, it needs to be organised to be useful. If I can, I keep a public list of tools, techniques, tricks and test data, perhaps tagged with general useful areas and knowledgable users. I encourage people to name and share their approaches. I try to get individuals to pair on tool use, or to hold swift training workshops.

One of the strengths of session-based testing is the way that it uses brief and frequent retrospectives to move skills through the team. Any exploration has learning at its heart, otherwise discovery builds nothing. Apart from skills in tools and test approaches, a test team needs to build knowledge of the system they are testing. We all know the truism that by the end of a project, the testers know a system better than anyone else. Exploratory test teams also build their awareness of how the system is used (and abused), and have broad connections throughout the users of their system and across the various sponsors and stakeholders. The edges of the team blur as people are seconded in and out; not all exploratory testers will identify themselves as testers. I have occasionally found easy acceptance of exploratory testing way outside the test team, which can give a neatly circular confirmation that the team made a good decision to set time aside for exploration.

In conclusion…

Test teams setting out to explore need to have a conversation with the overall project that acknowledges that this discovery activity cannot be completed, and seeks to find out what the project most values so that the test team can align its work to the overall desires of its customer and the goals of the project. It’s good to have a one-page strategy outlining goals and responsibilities. Internally, the team will need to work out how to make its work visible and trustable, and how to support its exploration and learning. It will need to organise and constantly refine a collection of tools and test ideas, drawing inspiration from diverse sources. As exploration becomes more understood, used and valued, the test team will broaden its skills and blur its edges, bringing its services closer to the coders and the customers.

* Not that anyone in my circles at that time saw – or named – exploratory testing as a distinct thing.
** I’ve mentioned one way that teams arrive at this point in Known ways of managing ET #03 - The Gamble
*** You’re here? Good for you. Welcome, friend. I know that you are interested in details and diversions. As am I. Between us, I don’t mean an octopus. Imagine Cthulhu.
**** because they’re the smallest, and so are arrived at first.
***** which sounds to me like Marshall McLuhan’s reversal effect. You’ll want to read Michael Bolton’s article for more.
****** It’s not like they’re the Spanish Inquisition.
******* and transport infrastructure, logistics plan, pulp mill, petroleum industry...

Uncommon Ways of Managing ET #03 - Daily News

2011-12-23T15:38:00.000Z

tl;dr – exploration every single day works wonders

Every day, one person explores for 90 minutes, allocating a further 30 minutes to bug logging, diagnosis and follow-up conversations with others involved in the product. The whole team gathers for a 5 minute briefing on the day’s exploration, where the explorer talks about areas covered, concerns and interesting discoveries. Anything is open for exploration; the configured and working product, its data, a group of requirements, known bugs, user manuals, the production trouble logs.

Some days, knowing the area under exploration, the news would be keenly awaited by all. Some days, there would be nothing new to report. Some days, the approach to the exploration would be far more interesting than the results.

The team might include the briefing in a daily standup. They might decide to be briefed first thing in the morning on what had been found in the previous day. They might choose the next area of exploration at the end of the briefing, allow the explorer to be directed by a someone who steers, or give the explorer the initiative. I expect that there would be a big, visible and public compendium of untried ideas.

The time given to exploration is predictable, and should not be seen as a minimum or a maximum, but as a regular activity. The more stable the product, the wider the exploration, the more unstable, the more that the collective exploration will reflect the overall assessment of trouble areas. The group learns as a whole, and individuals learn to take a step beyond the group’s expectations. I expect that individuals would look forward to taking their turn as the explorer, and that the team would rarely keep the same explorer from one day to the next. While competitiveness will lead to diversity and excellence, it stands a chance of causing individuals to hide some of their approaches until they become the explorer. Whoever is managing the team will need to take care to temper competition with shared purpose.

The regular and relentless expenditure of a fixed duration, and the expectation of team scrutiny would encourage each explorer to take a concentrated approach in a fruitful (and so potentially novel) direction. This is an approach to exploration (rather than to testing). It works best if interesting parts of the subject of exploration can be reached swiftly – but as easy parts to reach are exhausted, people will think of new things to explore, or construct mechanisms to get them to a new point of exploration. Collectively, their explorations would be more diverse than might be achieved by a single explorer or dedicated scout. If ever the whole team found they had exhausted their stock of new ideas about what to explore, I would use that as a trigger to ask whether we needed more tools, more skills, a more stable set of artefacts to explore, or whether we knew our application really well.

Known ways of managing ET #03 - The Gamble

2011-12-22T21:00:00.002Z

tl;dr – exploratory testing can be a gamble. And what's the problem with that?

As I write this, it’s coming up to Christmas*. Christmas is a hard deadline**, but there are clear gift-related requirements from one’s nearest and dearest***. As weeks turn to days and days turn to hours, some people find they have no gifts, yet no appetite either for handing over socks or gift vouchers. So they gamble. They give themselves some time, and head off to hunt for presents. Sometimes, not everyone gets a present, sometimes the presents are junk. Sometimes, they’re inspired. Generally, after spending roughly the allotted time and just a touch more than the allotted budget, the giver has something approaching the right number and selection of gift-ish things. We’re all familiar with a gamble, we know that some are more comfortable with a gamble than others, and we know that others positively delight in the last minute sprint-and-bundle.

Exploratory Testing is in great part about discovery. If you’re looking for real surprises, it’s pretty pointless to say how long**** you think it might take you to find them. It is more rational to set limits on how long you’re going to spend looking. This is a gamble. Your budget isn’t necessarily set to somehow match the value of the stuff you’ll find, but may be rather more influenced by what you have available to let you look. If you’re comfortable with a gamble, you may be comfortable with managing ET by lobbying for a budget from the project, and working to find a great way of spending that budget for the project.

Time for a couple of examples: “We want to spend 40 hours in the first week after delivery looking for trouble” is a gamble. “We’ll need someone for three days to prepare the exploratory environments, data, tools and ideas” is an investment in a known deliverable. These aren’t opposites. You will have goals for both. You can schedule both. Nonetheless, some test managers refuse to take gambles to their project managers.

This is not always because the project manager is genuinely uncomfortable with a gamble. Testing is a Wicked Problem, and no one thanks you for dropping a Wicked Problem into their basket of responsibilities. However, many project managers of my acquaintance are rather good gamblers, instinctively and analytically weighing up risk and return. Their management talents extend well beyond the business of juggling durations and dependencies. If you’re comfortable with a gamble, candid about the realities of ET and convincing about your skills, you may well see the PM’s eyes light up. The game is on. You’ll get some proportion of what you’ve asked for, and you’ll go looking for problems.

Of course, managing ET doesn’t end here. But – crucially – this is where it can start.

* or whatever you call it in your tribe.
** although some years I have good reason to send New Year cards.
*** and from those who are remote and unfamiliar, come to that.
**** I’m using long and budget here to mean the aggregate of time, money, people and so on. I don’t simply mean time or money. And we’re all aware that there isn’t a simple equation to switch back and forth between money and people. P ≠ mc2

Uncommon ways of managing ET #02 - Kanban

2011-12-21T17:10:00.000Z

tl;dr – Can Kanban work for ET?

Kanban is a way of managing inventory* – and by making that management visually clear, of helping workers arrive at improvements in the flow of resources. Kanban doesn’t look like a natural fit with testing: It is rather a stretch to say that test teams make things out of stuff in the way that Toyota makes cars out of steel. More to the point perhaps, testing’s inventory problem is sorting, not storage; it’s easy to find vast numbers of things to test, and simple to think of many ways to test them, but hard to find the right tests right now.

Then there’s ‘Kanban in Software Development’**, which is related, but different***, and describes a way to manage not inventories, but software development work in progress. It’s interesting to read Chris McMahon and Matt Heuser’s take on Kanban.

Either way, Kanban helps to visualise flow and to discover how that flow interacts with the capacity to work. As such, I imagine that it might be a reasonable fit with some of the logistics of managing exploratory testing. Perhaps by giving a snapshot of what the exploratory testers are paying attention to (and what they’re not) it might not only intrigue people across the team but also show up process problems that are ripe for fixing.

This isn’t a note about how to fit in with Kanban-driven software development, but a note about how I would use it as a tool within the test team. Also, I’m doubtful about Kanban as a way of managing work in general, and I wouldn’t use it to manage all the work in a test team. So let me give context to the situation in which I think Kanban could be a handy to people managing exploratory testing:

You’re already working with charters (from session-based testing or something similar).
Your project has budgeted enough effort for ET to make it worthwhile managing the flow of the thing****.
You feel the need to visualise and tweak your flow.

Here’s how I would use it:

Set up a chunk of available and visible wall as your Kanban board. Split this up into four columns. A great big fat one on the left for ideas that need to be touched on in the foreseeable***** future. Next right, a skinny column for what you might hope to do today, then a skinnier one that will show the exploratory testing going on right now. Finish up on the far right with a fat one for ideas that don’t need to be considered again in the foreseeable future. You’re going to fill the space with sticky notes. The sticky notes will be moved from left to right. At a glance, anyone in range will be able to see what’s left to do today, what’s being done, what’s been done, and whether testing is going on right now.

Sticky notes start out in the far left column. There will be lots here, probably rather more than the team might be able to test in the available time. Each one will represent a charter that you’re happy to spend a chunk of time on; you’ll write the charter on the note. I think it’s a good idea to put the originator’s name on the charter, so people know who to ask about its history. You’ll also want to represent the time****** you’re giving it, either by note size or with a number.

As a collective, fill up the ‘today’ column with notes. Kanban is a tool to help visualise work, so if you’ve decided that today the team will spend 10 hours exploring but you’ve got 18 hours of charters lined up, it will be obvious that you’ll need to iterate until sanity prevails.

The ‘in progress’ column is to help the team visually manage the flow of work. If you see people as the limiting element of your capacity (ie one person can only do one charter at a time*******), and you’ve got two people testing today, you’ll split the ‘in progress’ column into halves. You’ll give the halves the names of your testers. To give you a sense of today’s capacity, you might even block out chunks so that there is only space for two stickynotes in the column. The two spaces start out empty. When someone starts testing, they move the related stickynote into the ‘in progress’ column.

If that someone was me, I would write my start time and hoped-for end time on the stickynote before I stuck it back onto the Kanban board. Then I would explore, using the charter to direct my testing, within the timebox I had set myself. While testing, I would be very likely to generate more test ideas********. I’d deal with some of these the session, but some would need to become charters themselves. I’ll keep track of these on new stickynotes. When I finished my session, I would move the stickynote out of the space representing me and into the far right hand column, leaving the space empty for a new stickynote.

I would put any new charters on stickynotes into the great mass of notes in the far left hand column. I might even re-jig the work for today, if one of my new charters was more urgent than something else we’d planned. Then, assuming I had more time to explore, I’d move another stickynote from the ‘today’ column into my ‘in progress’ space, and get on with stuff.

That’s the basics. Let’s go one iteration on, and consider some wrinkles.

There’s going to be a surfeit of stickynotes in the left hand column. Too may ideas, and too much to do, is the nature of the testing beast, and I think it’s desirable to show this truth. Having the notes physically available means it’s easy to rearrange them. I suggest that the rearranging happens all the time, by anyone. If activity is dependent on something not-yet-delivered, I’d like to see the stickynotes grouped somehow – perhaps on a sheet that is itself stuck on the the board. If some activity is likely to be done soon, I’d like to reposition its stickynotes on the right of the column, ready to jump into the next day’ work. I’d encourage the team to bubble-sort vertically; to adjust pairs of vertically-adjacent notes from time to time so that more important ones rise. I’d like us to explicitly mark off a ‘pit of pointlessness’ at the bottom of the column containing all the stickynotes that represent things we can’t do*********, won’t do or just don’t want to do.

Over the day’s work, I want to see the number of stickynotes reduce in the ‘today’ column reduce, and the increase in the ‘done’ column. I’d like a second pit of pointlessness in the ‘done’ column for any notes representing a session that went bad. I would want to organise the notes in the ‘done’ column so that I could see, day by day, when something was done. You might want a different organisation. We would talk about it, and the board would have bought out a useful discussion.

The capacity element of this Kanban board only really applies in the ‘today’ and the ‘in progress’ columns. I’ve assumed above that the capacity for ‘today’ (or whatever period you use) is in hours, and having one hour represented by a note of given size might help understand how much time is needed, and has been spent. I’m less comfortable with the capacity of the ‘in progress’ column being named individuals – I’m well aware that an empty slot looks like someone’s not working, and I’m also keen that people can explore together and at times of their choosing. I think that I would prefer to work towards capacity in terms of Test Lab resources; clean data, single links to stubbed-out systems, hand-held devices, or whatever causes our primary bottleneck. Again, that’s something for a self-organising team to sort out for itself.

Frankly, I don’t know what to do about activities and time taken logging my bugs and stats and reports, or how to mark a brief debrief. I’ve worked in teams that have included and excluded plenty of activities from their charters, and I’d suggest consistency of approach within a group is more important the approach itself. Instinct suggests to me that this if board is only for visualising exploratory testing work in progress, that I include time spent doing diagnosis for bug logging and exclude the rest.

No huge surprises here, I hope – but let’s remember that one reason to use Kanban is to optimise away the need for Kanban. In the article referenced in **, Jeffery Liker was quoted in The Toyota Way: “Kanban is something you strive to get rid of, not to be proud of”. The approach I’ve described above should be seen as diagnostic tool rather than a solution to a scheduling problem. I expect that in use, one would see plenty of tweaks to not only to the Kanban board and its processes but – more importantly – to the actual work of managing exploratory testing. I hope that you, dear reader, will sort them out in a way that suits your team, and then will share your solutions (and your context) with the rest of us.

I’ve done bits of this from time to time, but not all of it together. If you’re interested in the ideas above, remember Elisabeth Hendrickson’s mantra: “Empirical evidence trumps speculation. Every. Single. Time.”. Some testers are already on this path, but I can’t find references to their experiences. Perhaps readers will furnish those references in the comments. I want to get to hear Adam Geras’s take on the subject, given that he’s not only lived with, but talked about “A Personal Kanban for Exploratory Testers”. I have a memory, which may be made up but feels as if it arrived in the last six weeks, of a series of pictures posted on twitter, with test activities represented as sticky notes that marched left to right across a board. I can’t find a reference to those pictures in any of my notes or bookmarks. If you recognise that as your work or the work of one of your colleagues, I’d love to hear from you, and to discover what you’ve learnt from the real world, and how badly it beats up my imagination.

* Arrived at by Toyota in the 01950s, refined into a cornerstone of the lean movement since, and now a perhaps just-past-trendy meme in the agile community. Here’s Wikipedia on Kanban. As I understand it, Kanban at Toyota describes a system of signalling that a small, local inventory is empty. It is used not only to manage the flow of components, but also to make that flow explicit, and adjustable. One reason to use Kanban is to optimise away the need for Kanban. This appeals to me. I’ve never seen this kind of Kanban in action, but it’s clearly inspired lots of people.
** see Karl Scotland’s Aspects of Kanban for text, David Anderson’s A Kanban System for Software Engineering for video. Here’s a currently poorly-cited article on Wikipedia so you don’t have to take your hands off the mouse. I have seen it in action, but never the same way twice, and you’re better off going to the sources than reading an inevitably compromised footnote like this one.
*** Tokens indicate presence, not absence. It’s about making inventory, not consuming inventory. Capacity and optimisation seem (in practice) to play second fiddle to visualisation and flow.
**** As a rule of thumb, this is more likely to be true if you’re not just concerned with what to test first, but what might be handy to test next. If you’re jamming ET into the gaps around the edges of your existing testing, I wouldn’t bother managing it with Kanban, because there’s no chance it will flow.
***** My horizon for foreseeable is pretty short. The absolute maximum might be the end of the sprint or the date of software release, and it tends to be less than three days. Your team will have a different attachment to the future. As a group, work out what your horizon for “foreseeable” will be and write it large somewhere obvious. Do change it if you need to.
****** Unless you’re working with fixed-length charters.
******* hmm
******** distractions
********* too big, too difficult, too dependent to consider in the foreseeable future. It’s not just the trivial things that are pointless.

Known ways of managing ET #02 - Bug Bash

2011-12-19T21:08:00.000Z

tl;dr - Bug Bashes are rubbish.

The project gathers people together at an appointed time and place. Everybody splurges on testing for an allotted period, logs some bugs, and stops. If you need examples, see *. It’s a community thing, and there is generally a group hug / doughnut / retrospective before everyone goes back to their day jobs.

I guess one virtue of a bug bash is that it is a concentrated period of work, which may be a good thing in itself. Bug bashes can employ and popularise diversity in a group’s acceptable points of view, which is a plus in my book. You get to meet other people. And maybe a doughnut. But other than these few useful traits, it’s hard to find much that is good.

While the bug hunt room might be a fertile idea-generating ground for small groups in close physical proximity, the format means that many people head into the system for the first time, at the same time. Everyone is testing in parallel for a limited period, so there isn’t much opportunity to learn from each other, to analyse the group’s results, draw conclusions and carry on in a better way. There isn’t much opportunity to appreciate all the different approaches that are being tried, and take a new one – or, indeed, to help the group to explode in variety. There isn’t much opportunity to sling together a swift tool that drastically cuts the manual drudge and finger trouble in later work. And most people, working as novices, will follow a limited gamut of manual paths characterised by learning and exploring the application for the first time**. This could help you predict how your customers*** might interact with the product in the first few hours of use, but it’s not so good for assessing the product in other, more representative ways. A bug bash may be directionless, but that’s not to say that it is diverse.

A bug bash puts strain on available test-environment resources; licenses, batteries for devices, laptops, USB cables, un-damaged data, bandwidth, IP addresses, you name it. I’ve seen a support chap spend days getting the kit together, versioned, charged, addressed, data-filled and working before a push. Even assuming your Test Lab really can support your bashers with hands-on stuff, your back-end and infrastructural resources may not be neatly independent and you’ll end up being stymied by half the group finding the same test-environment bugs. This is great if you’re looking for test-environment or large-group bugs, but again, not so good if you’re looking for a broader or more representative set of interesting issues.

Expect lots of duplicate bugs, as different bug hunters bang into the same low-hanging fruit. Bug bashes often throw up some easily found but hitherto-unseen trouble, but let’s not be unthinkingly self-congratulatory when a third of our crew waste their time investigating, diagnosing and logging the same problem at the same time. More insidiously, duplicate bugs may mean broadly similar paths through the application. If everyone’s doing the same thing, what does that do for coverage? For exploration?

The quality of logged bugs tends to be low, the density high. The brief duration and inevitable peer pressure pushes the people in the room to value their speed to the first bug, and if they don’t get the early bugs, then, hey, there’s kudos to he who logs the most. Bugs are logged at speed, in bulk, with generally poor detail and diagnosis. Some managers are happy that their bug bash has resulted in a great wodge of trouble tickets, but remember that a bulge in the bug rate disturbs the workflow of an agile team like a turtle disturbs the digestion of an anaconda. Finally, the hysterical whoop whoop of competition not only breeds false confidence, but can break the spirit of people with their ego tied up in the code and configuration of the product.

A bug hunt allows a team to throw many people at a problem in a short period. It can appear cheap in elapsed time, while substantial in people time. Wrong. Ten people testing for an afternoon might look like a week’s worth of testing, but it is a week’s worth of testing by someone with 3-hour amnesia. I have seen bug hunts used to demonstrate someone in management’s commitment to testing their product. Diverting half the team off their usual path and into testing for a few hours certainly makes a statement, but the statement is that management is committed to theatrical gestures. Grand Guignol**** testing is a titillation, yet I’ve seen it used to substantiate the assertion that a product is fully tested. Frankly, my arse.

When might I use a bug bash? Perhaps if there was a problem reported frequently in beta testing, which was serious and urgent enough to warrant a diverse group’s concentrated attention but not well-reported-enough to act on directly. I might give the reports to a bug bash group, ask them to find out anything about the problem that isn’t already detailed in the reports, and facilitate their communication by sticking a scribe/steering person at a central and visible whiteboard, equipped with a bell. But I’d prefer to use a small team with big kit.

Getting a large group together for a short period is an expensive way of doing rubbish testing. I’d far rather spend the time and money getting the necessary people together and delivering a test environment that is up, running, connected, data-ready and swiftly-rebuildable. Or delivering a diverse and knowable set of data. Or a collection of reasonable (and less reasonable) user scenarios that stand a chance of saying something interesting and meaningful when tried on the product. Or a couple of hours so we know something about each other beyond name, age, height and title. Actually, pretty much anything is better value than a bug bash.

For a while***** it seemed like every other client wanted to throw most of their exploratory eggs into the bug bash basket. I have no idea who kicked off this ludicrous meme, but I’d still like to tweak their nose. Here’s my position:

Managers: of all the usual gambles you can make with your charges, a Bug Bash is one of the dumbest. Get someone to bring you an alternative, and consider it.

Testers: Bug Bashes might look like fun for you, but they suck for the product and the project. Don’t be fooled.

Clear enough?

* I’ve seen Scrum teams devote the whole team for a couple of hours on the 6-8th working day of a 10 day sprint. I’ve seen waterfall teams drag fifty people into the canteen on a Friday afternoon to batter away at a batch of handhelds. I’ve seen test teams commandeer the boardroom for a day at a time straight after they get the code, every time they get the code. Bet you've seen something similar. Enough examples: back to the polemic.
** Ж: “So what did you do?”
Ю: “Well, I tried logging in, and I’d not logged in before, and it went well, so I tried changing my username and resetting my password, using funny characters.”
Ж: <smacks head on desk>
*** Assuming your insiders are good substitutes for customers…
**** A horror show made up of a series of short pieces. Read Grand Guignol in Wikipedia, and think testing. Compare with Soap Opera Testing.
***** 2004-2008, or so

Uncommon way of managing ET #01 - Scouting

2011-12-17T09:50:00.000Z

tl;dr – skilled, supported, concentrated exploration

The team makes one person* the dedicated explorer for a period. This person, who we’ll call The Scout, spends all their time exploring. Their job is to find as much interesting stuff as they can. They’re supported (and watched) by others who set up environments, log bugs, keep notes, analyse data, suggest and configure tools. Pay attention: These supporting people are not part-time or less skilled; they’re just as engaged as the scout, but they’re not on point.

There are no sessions or formal session-end debriefs, but the team will want to stop and sit back from time to time and come to some conclusions about what they’ve found. The person (or people) on point are switched around regularly – scouting is fatiguing, and diversity is important. People with different specialities are used as required, and The Scout need not be a tester.

Exploration often has a sense of a frontier, a boundary between the known and unknown. The frontier is fundamental to exploration, and The Scout pushes it ever onwards. We understand, of course, that testing has really wiggly and sometimes discontiguous boundaries, and that the territory behind the boundary may not be well-known, and is likely to change unexpectedly. The team will understand this boundary better than anyone else, and will need to come to an understanding about how much they need to be able to notate and share information about the frontier.

This approach is all about discovery. It’s not cheap, nor is it exhaustive, but it is valuable. The project gambles time in return for information, so The Scout needs to know what the project is interested in. I expect there would be tussles about what The Scout would be exploring, and what they would be looking for. So much the better.

Note: This an idea. I’ve not worked (quite) like this. Maybe, though, this idea triggers something that you would like to try with your team. Let me know how you get on.

* or one group, but I’ll write in the singular to keep the grammar simple

Known way of managing ET #01 - Stealthily

2011-12-16T15:19:00.000Z

tl;dr – some people hide their best work from their paymasters

A few people get together to find problems in snatched moments. There's little or no imposed direction, measuring, or task control, and rarely any sense of completeness or coverage. Although the work sometimes gets done with tacit support from one or two individuals in the upper echelons, there is little oversight and it is usually a hidden activity. Testers don’t log time or bugs through the usual channels, and it feels almost like an indulgence, a guilty pleasure.

I've been on a team who hid a pair of stealthy explorers in a corner for a few hours each week. The exploratory testers would tell the rest of the team about the bugs they’d found. Generally those bugs were 're-found' during manual scripted testing to allow them to be logged within the imposed structures of the project – and if no script might find a particular bug, we would assemble one. The customer would not countenance paying for unscripted testing, but was very impressed that we were designing such effective scripts*.

Exploratory testing becomes stealthy typically because those who control and budget for team members' time don't approve of looking for trouble. Look out for rigid 'verification and validation' contracts, consultancy contracts that only allow a small set of explicitly-approved billable activities and legal fears of explicitly-acknowledged defects.

I've seen Exploratory Testing hidden most commonly in teams that focus on user acceptance and regression testing, but I've also seen it in a self-labelled agile team that relied on a (rather sparse) set of low-level confirmatory automated tests. These teams tend to be a mix of self-identified testers and people who may be seconded into the test team or are otherwise keen to avoid the label. However, I’ve even seen stealthy approaches in test teams who were exploring with a degree of management support, but who felt that some of their approaches were beyond what might be accepted.

However nasty such hidden work looks from the outside, it's often rather well supported by individuals within the testing teams. People get the opportunity to work heroically, to subvert management decisions (especially gratifying if those decisions feel irresponsible), and sometimes to have a direct link to someone rather higher up in the tree of project status. The stealthy effort tends to get a geekily-sexy label.

When I meet experienced exploratory testers who have carved themselves a niche in some monolithic institution, they're often proud to be stealthy, and sometimes unhappy to share their approaches. Sometimes their reticence is justified – in groups which insist they don't do ET, acknowledging that you're an the ET specialist doesn't necessarily improve your day.

It’s worth mentioning that some companies consciously take a stealthy approach to discovery work so that they have plausible deniability, for instance while finding bugs to get a company out of a contract, or finding bugs that no one who could be legally-liable should know about. Such activity will challenge your ethics. Call time on these if you must – and sometimes, you must – but be aware that you may be shouting at the waves.

* This was in the early nineties. Don’t think I would put up with it now – but don’t think that the practice has ended, either.

There are Plenty of Ways to Manage Exploratory Testing

2011-12-15T15:41:00.000Z

tl; dr - lots of different ways to manage ET

The key problem that exploratory testing faces, as a viable discipline, is how it is managed. Of course, there are other well-covered interesting areas - the question of whether to do it at all has been debated to death* amongst us pundits (if not with as much fervour in industry), and if you're want to know how to do it, there is a slew of excellent ideas, techniques, disciplines and tricks to choose from**. However, the hairiest problems in actually doing it come from how people organise the work, and how the work and its owning organisation adjust to fit***.

Over the next couple of weeks, I'll post a short series here, snappily entitled "Ten Known Ways to Manage Exploratory Testing" and "Ten Uncommon Ways to Manage Exploratory Testing"****.

Here's a kickoff showing roughly where I'll go:

Ten Known Ways to Manage Exploratory Testing	Ten Uncommon Ways to Manage Exploratory Testing
Stealth Job Traditional Retread Off-Piste (Iron Script) Off-Piste (Marshmallow Script) Bug Hunt Set Aside Time Gambling Script-Substitute Session-Based Test Management (James & Jon Bach, me, others) Questioning (Jon Bach) Thread-Based (James and Jon Bach) Touring (James Whittaker and others) Don't bother (thanks to Dave Liebreich for reminding me...)	Scouting Kanban Following Lenfle Daily News R&D Testing Guru Video Reports Post-Partum Labelling The Summariser GPS Cloudy The Inquiring Metricator

I'll fill you in on what I***** mean by each of these over the next few weeks. Expect about one a day, in no particular order.

And by the way: I'm posting this because it's good stuff, and you're going to find it useful. I'm posting it now because I've got a course in January that I want you to know about. That's January 25-27, in Oxford. A two-day workshop on exploratory testing techniques, followed by one on managing exploratory testing. Book here.

Note: If you're in Scandinavia, I've got one in Copenhagen on 6-8 March through the Morten Hougaard's Pretty Good Testing. Details here.

* And has degenerated as everyone professes to agree with each other's aims while insisting the philosophy's all wrong. Consultants, eh? Welcome to my world.
** Key sources for me****** - both Bachs, Bolton, Carvalho, Edgren, van Eeden, everybody I've ever tested with, Green, Kaner, Hendrikson, Harty, Itkonen, LEWT, me, Richardson, Sabourin, Weinberg, Whittaker. Alphabetical order. A l p h a b e t i c a l. No preference implied. Some are sources for stuff I try not to do...
*** Don't get me wrong – plenty has been written about this, too, over many years. Here's some more.
**** Sorry about the terrible titles. Ten-fer lists wind me up, get me down, piss me off and a whole other bunch of phrasal verbs. But them's the titles. No, I'm not going with The Twelve Days Of Testmas, and yes, obviously there are more than ten of each... I'm not claiming these lists are exhaustive, nor that the items are exclusive. I'll write this up properly once I'm done serialising.
***** You can probably guess some, or most. Sweepstake?
****** Did I miss you out? Apologies. It's a blog posting – half-baked by nature. Email me, and if I forgot you, I'll add you to my list.

I've updated my format...

2011-12-14T00:32:00.000Z

... but not my content.

The old one looked like someone's living room - if that person was living forty years ago. Good riddance.

I understand that lots has changed behind the scenes. Let me know if you spot something that's not what you'd expect.

I've found one fluff so far; the new profile has squashed my headshot.

A couple of hands-on tools

2011-12-12T14:37:00.000Z

tl;dr – two handy tools

On Sunday, I was watching a film of a fine tester testing. I was keeping track of how his testing differed from mine. I realised that I was looking for the functionality of a couple of tools that I sometime use, and that he wasn't using at the time.

Both tools are for the Mac, but I imagine that similar tools are available for PCs too. Although neither are testing tools, they do things that are not only convenient, but by being frictionlessly convenient, allow me to observe and trigger behaviour in usefully-different ways.

The first is Mouseposé from Boinx. Mousposé highlights your on-screen pointer position, and is the close cousin of many tools used by teachers and screencasters to make their actions more obvious.

What makes it useful to me as a tester is that it makes user actions more explicit – not only following the pointer around, but differentiating between one click and double- (and n-) clicks, between right and left button (or however one expresses the peculiar sigil necessary on a trackpad to do the same). It also – and here's the killer – displays the keys you're pressing. These are on the edge of vision, in large letters on a low-screen bezel. They appear very briefly, and captured keys include shift, control, escape, enter and so on. It's great to expose the occasional finger slips that lead to novel behaviour, great for working out what you're doing, and for confirming (or not) that you're doing what you think you're doing. It's especially useful if you, like me, find that your hands don't always do quite what you ask them to do.

If they made a tester version, I'd like it to cope with multi-touch, give me a trail on drag, and ideally have some kind of paper-tape thing to show recent input and save me using a keylogger.

The second is Keycue from Ergonis. Lean on the command key for more than a few seconds, and KeyCue pops up a bezel with every* currently-available command-key combination. Not just the front application, but all the combos that are currently listening. Different options respond as you change your key combo. It's great for ramping up your expert hands-on-keyboard-flying-user tricks, but more than that, it shows you a whole bunch of potential bug triggers.

Key stroke input can cause unusual behaviour not only because it comes through an alternative route (as it happens, the rout is easier to automate, so tends to be covered in developer-side testing/checking) but because it's fast and potentially in conflict with other stuff. Hitting keys in swift succession can expose timing-related bugs (two in Word, one in Excel this morning alone). Hitting meaningful keys that have meaning to something other than the thing you're talking to can also get the pizza spinning. I want to try it on a cyrillic machine and on localised software.

If I had a tester version, I'd prefer that it actually enquired the system internals to find out what was listening, rather than simply parsed menus (and possibly-flaky "User-definable custom shortcut descriptions"). But I don't even know if that's possible.

* if you're not in version 6, every is unfortunately more like most

Something for the Weekend? 006 - Visualisations

2011-12-04T13:29:00.000Z

tl;dr: visual representations are lovely

Christopher Warnow and onformative have worked together to make a movie which gives a visual dimension (not an explanation) for various sorting algorithms.

I came across Warnow because @cunabula had retweeted a link to his visualisation of Amazon's recommended books, A Thousand Milieus. Warnow uses Amazon's recommendations to find a hundred related books - then shows you them as clusters.

Delightfully, Warnow has made his tool available (not open source, but available*) - so here are visualisations** for books*** about testing that I've been known to recommend:

Lessons Learned in Software Testing: 1 and 2
General Systems Thinking: 1 and 2
How to Break Software: 1 and 2

Hope you've had a lovely weekend.

* data walking and munging is done with Processing, which is dead easy to get a handle on, and the graphing side is basically Gephi. So go on - have a play. Here's your map: download the tool (which brings the Processing library for Gephi), download and install Processing, chuck Francis Li's http library into the right place, fire up the tool script (all 12Kb of it) and check that its behaviour seems reasonable. That's the hard stuff done. Now everything is open to you - your first task is to make the tool search for 30 books, rather than 100. Feel satisfied?
** two visualisations per book? Certainly. Running the tool twice produces diagrams with similar content, but very different layout. Compare and contrast.
*** U.S. Amazon store

Exploratory Testing - collected stuff

2011-11-29T14:45:00.000Z

tl;dr: lots about Exploratory Testing

A colleague asked me if I had written articles or blogs about Exploratory Testing. Why yes, I have. It looks like it's time to share them more prominently.

Exploratory Testing is a crucial element that is often poorly integrated, so when I write about testing, I tend to make reference to Exploratory Testing. However, I don't particularly think of ET as the only interesting game in town, so when I write about it, I hope I put it in a relatively rational, consistent and practical testing landscape. Hence this feels like the first time I have consciously made an inventory of the stuff I've written about ET.

Here goes:

Of my long-form ET stuff, the paper that gets the most citations is 'Adventures in Session-Based Testing', which is about managing ET. I wrote it around ten years ago with the very brilliant Niel van Eeden. It won 'Best Paper' at EuroSTAR and STARWest. I updated it, so it occasionally gets called 'Further Adventures...'

The one I'm most fond of is 'Four Exercises for Teaching Exploratory Testing',* but although it went to the Workshop on Teaching Software Testing 5 back in 2006 and should have been part of the online materials, it vanished instead into online limbo. Astonishingly, when I searched just now, it's finally there - but orphaned from the rest of the site. If you can find a path to it from http://www.testingeducation.org/ I'll give you a hug.

You might already recognise the Black Box machines, which get plenty of attention, and are the single biggest cause of random strangers saying hello at conferences. Occasionally someone on a train (typically to or from Paddington) will say "aren't you... didn't you...", which is odd, but nice**. I know of a dozen or so people and organisations who use them for recruitment purposes – so let's call them my attempt at a balance to the useless five years I spent trying to nudge the ISEB exam marginally closer to fit for purpose.

Agile people tend to have come across Elisabeth Hendrickson's excellent Test Heuristics Cheat Sheet, which has my name on it. I contributed when she and I put our Exploratory Testing classes together for a (very enjoyable) 2-hander that we ran in London and in California.

There's plenty of other papers available at http://www.workroom-productions.com/papers.html, and many have related material. A Positive View of Negative Testing has more on techniques, Things Testers Miss is on bug stories, Testing in an Agile Environment covers my experiences of the fit (and friction) of testers - often using ET - on agile projects. In my most recent long-form paper, The Irrational Tester, I appropriated some fashionable ideas from behavioural economics, and I hope not only gave them testing context, but enabled more substantial exploration by drilling back from the pop science to the original research. That one got another 'Best Paper', this time from STAREast.

On that same 'papers' page, you'll find a short series of more conversational short-form things under the heading 'Exploratory Testing Notes'. They go with my Getting a Grip on Exploratory Testing workshop - they're not carefully-checked whitepapers, but nor are they short sharp blog postings. There's yet more that goes with the course, but it changes pretty much every time I do the thing.

As for blog postings, well, you're here - but Blogger is, ironically, a rotten thing for searching.
This is an entry on tools for ET http://workroomprds.blogspot.com/2008/06/tools-for-exploratory-testing.html ,
and this for assurance http://workroomprds.blogspot.com/2011/06/how-to-assure-exploratory-testing.html ,
and two together that may be of interest http://workroomprds.blogspot.com/2008_11_01_archive.html .

Reading this, it's obvious (to me) that lots of the other ET-related stuff I've written over the years has slipped away. That's the problem with the internet - ephemera are eternal, but useful stuff gets drowned. I'll fish some out and post them over the next few weeks.

Finally - I run a workshop from time to time called Getting a Grip on Exploratory Testing. It's all hands-on, and is limited to 12 people. I'm running a public class in Oxford on 25-27 January. Lots of friendly testers have twittered about it, and some (who have been on the thing) have recommended it. You'll need to look now, and book quickly, to get to the early-bird discount by the end of the week.

* Two things to note: firstly, the exercises describes are software, and available to all. Contact me and I'll send you them - plenty of people do, and they're used all over the world. Secondly, they're deep enough exercises to still be part of my workshop. By all means have a play, but if you're thinking of coming on my workshop, be aware that novelty is important to exploration and you won't get as much from the workshop. That said, I always have alternatives available if someone turns out to be familiar with the exercises.
** Sometimes they just want me to sing a song, which is still odd, kind-of-nice, and tends to mean they're a Bulgarian. Once, someone on a train was both a tester and a Bulgarian. We had lots to talk about. I should call him and arrange lunch.

Bias: Illusions and corruption

2011-11-10T00:07:00.000Z

tl;dr we're all nuts

Two recent articles tickled my interest in bias.

In The Observer (How cognitive illusions blind us to reason, an extract from Thinking, Fast and Slow) Daniel Kahneman reminds us that cognitive illusions are stubborn, particularly when one is exercising hard-won, high-level skills. He illustrates this using stock traders, saying that their skill in evaluating the business prospects of a firm is serious work that requires extensive training. "Unfortunately, [this skill] is not sufficient for successful stock trading, where the key question is whether the information about the firm is already incorporated in the price of its stock."

In The Economist, All power tends to corrupt is subheaded "But power without status corrupts absolutely". It describes an experiment in which subjects were asked to select tasks for a colleague to perform. Some of the tasks were demeaning. Before they made their selection, they were given a job they might respect or look down on (the descriptions read a little like fun vs dull testing roles) and a sense of whether they or their colleague had more influence. Those put into the position of having influence but no respect chose significantly more demeaning tasks for their colleague.

Some testers tell me that they do a skilled and difficult job, but don't get much respect. I believe them – and I've found that the articles above have helped me understand my own behaviour a little more clearly.

Byalo Rade - new track from the London Bulgarian Choir

2011-11-07T12:42:00.001Z

I don't usually do this, but apparently you're interested...

Here's a track from the London Bulgarian Choir's new album, Goro Le Goro. The album will be released on November 26th, with a big gig and party in London. I'll be wearing my furry hat, but you'll need to buy a ticket.

<p>&amp;amp;amp;lt;p&amp;amp;amp;gt;&amp;amp;amp;amp;amp;lt;a href="http://londonbulgarianchoir.bandcamp.com/track/byalo-rade"&amp;amp;amp;amp;amp;gt;Byalo Rade by London Bulgarian Choir&amp;amp;amp;amp;amp;lt;/a&amp;amp;amp;amp;amp;gt;&amp;amp;amp;lt;/p&amp;amp;amp;gt;</p>

White Rada is sweeping her yard, her slender figure swaying, her arms like pale wings. As she sings, pearls flow from her mouth. ‘Beautiful Rada, my daughter, don’t leave your yard, don’t lift your eyes, don’t give your flower away.'

Something for the Weekend? 005 - Patterns and evolution

2011-11-06T23:47:00.001Z

Catherine Young sees plenty in clouds.
(which reminds me: Q: what is the sky? A: all of the above from @gimboland/@posh_somme )

I've had a huge print of this sequence over my desk for years. Every time I actually raise my eyes and look at it, I think of change, excellence, strangeness, practice, talent, choice and stopping. Or something. Hope you've had a lovely weekend. Picasso's Bull suite.

A playful exercise for testers

2011-10-31T16:34:00.006Z

tl;dr; here's part of a workshop

This exercise, like many exercises used with testers*, encourages people to discover rules** and build models. I last used it in public at Tony Bruce's excellent London Tester Gathering, and continue to use it at corporate clients. However, unlike most of my stuff, it doesn't involve software. Over the years, I've built up a small collection of bits of landscape. My assembly of palm-sized stones generally adorns one of my monitors in the studio, but occasionally has a second life in workshops. The exercise in which I use them is fun and seems valuable, so I thought I'd spread it about.

----

In a nod of respect to Johanna Rothman***, I call this exercise 'Bring Me a Rock****'.

Here are my instructions:

Set our your stones somewhere where everyone can see and get at them.
Pair up the workshop participants. If you have an odd number of people, pair up with the oddest yourself...

Both of you: write down something to identify a rock. Keep it secret.
Become ASK and FETCH.
- ASK says "Bring me a rock"
- FETCH brings a rock.
- ASK accepts, or rejects the rock, based on the secret they wrote down earlier.
- This continues until ASK accepts a rock
When ASK accepts a rock, FETCH proposes a model
- If FETCH is wrong, swap roles.
- If FETCH is right, ASK writes down new criteria, and FETCH brings another rock
carry on...

Meta-instructions:

find a new pair if bored
observe how you are modelling your partner's models

----

The exercise is fun, but gets a lot more valuable when the participants talk about it afterwards. My approach is to:

discuss the different approaches to test and discovery
pay special attention to the way that patterns are set up and broken
apply learnings about discovery and patterns to testing
avoid too much time spent talking about the different rocks

There's a little fiddle in the instructions that I'd like to draw your attention to. I'm sure you've noticed that the ASK role has a higher status than FETCH. With this in mind, it seems odd that FETCH should stay as FETCH if they correctly predict ASK's model. A playground sense of fairness means that the roles should swap on a 'win'. Do play it this way if you want to. However, I find that swapping on 'lose' introduces interesting, more subversive*****, behaviour. Say hello to the Imp of the Perverse.

If you run an exercise like this, please feel free to change stuff. Also, I would love to hear how the exercise worked for your group. Finally, I'd be grateful if you'd give participants a link to this post.

* I've commonly heard of 'mastermind' being used, but some notable testers use more mutable approaches with lots of rule exploration and discovery. Michael Bolton told me of his new coin meta-game the other day, James Bach and other use cards and dice. If you're a tester interested in games, you'll certainly need to watch Dale Emery and Elisabeth Hendrickson – and try to get along to Elisabeth's Agilistry Studio one of these days.
** There's a whole class of games whose purpose is the discovery of rules - here's Wikipedia's living list of Games with Concealed Rules. You might also want to have a look at Nomic. Nomic's rules aren't concealed, but their evolution is the game. There's a blog post about this around here somewhere...
*** Johanna Rothman's seminal article should be read by all: http://www.jrothman.com/Newsletter/Bringmearock.html
**** Some are smooth, some rough, some round, some flat, some solid, some holey. They're all sorts of different colours and sizes. One is incised with the work 'luck'. One is (was) a plum. Take your pick.
***** ie subverting the game, not each other. I find this approach rewards just-about-guessable reasons for picking a rock, so ultimately this helps people in the pair co-operate in in the learning, rather than allowing one party to stay in the high-status position just by being wilfully obscure. I'm proud of this wrinkle. You might think it's dumb. Your choice.

Something for the Weekend? 004 - Games with unknown rules

2011-10-30T17:35:00.002Z

tl;dr - some games don't have fixed or known rules. Go here or here.

A long time ago, in a wooden house on the shoulder of a snowcovered mountain, I found myself sitting in a circle of people I didn't know at all. We were idly tossing a ball about. The ball was a complex thing with lights and buttons and fresh batteries – and as it moved from person to person, we shared arbitrary rules that were immediately forgotten. We noticed we had the attention of a couple sitting outside the group. They were, it emerged, players in the UK's Go team, and they were simply fascinated.

We continued to throw the ball. Occasionally, another rule would turn up. One of the Go players objected that our rules were inconsistent. Rising to the bait, we gently corrected him by explaining a hitherto unappreciated subtlety to the ruleset. More Go players were drawn towards the circle. As the Go players got down to organising and double-checking, our new rules explored variables beyond buttons and lights; the pause between throws, how hard the chuck, where the target was looking, whose friend they were, whether their name started with a vowel, whether a previous in-game action had temporarily changed their name...

This was, of course, nerd-sniping. But, more interestingly, it was an unexpected kind of game; the kind of game where the rules change. The Go players, on a team trip and consequently pretty much only in contact with other Go players, were more-than-usually locked into a pattern where rules were constant. Believing that all games have fixed rules is an easy habit to fall into.

Nevertheless, games with variable rules aren't unusual – I've spent happy hours playing bar chess, word disassociation, and parroting Mornington Crescent. If you play any games with a five year old, you'll know that rules are (a) important and (b) made up on the spot.

It's tempting to see the world as a game. Some people who explain the world use games correspondingly; as metaphors for the real world. The trouble is that games with fixed, agreed and finite rules are not always a great model. It's all very well to count heads and tails, but we forget that sometimes the coin falls in our tea and the resulting dousing trashes the laptop on which we're attempting to keep score.

The rules of our world are generally local, temporary and inconsistent. Very few rules are universal, very few activities not involving time, energy and accountancy are zero-sum. For me, the joy and value of maths, physics, music, cookery and coding is the discovery of rules; these deeper, emergent, unexpected truths. It's not about playing by the rules, but playing with. So, if you're going to play a game as a metaphor for life, you might consider using a game where the rules are on the unreasonable side of realistic.

All of which is a long and late introduction to: Something for the Weekend? 004 - Games with unknown rules
List of games with mutable rules
List of games with concealed rules

Enjoy.

James Bach moves very fast...

2011-09-24T16:44:00.001+01:00

Something for the Weekend? 003

2011-09-24T16:19:00.001+01:00

This turned up in Donella Meadows' excellent Thinking in Systems: A Primer*.
I've not seen it before, and thought you'd enjoy it.

A system is a big black box
Of which we can’t unlock the locks,
And all we can find out about
Is what goes in and what comes out.

by Kenneth Boulding

...there's more, of course. But I'm not convinced that posting the whole thing is a fair (ab)use of copyright. You'll find it on less fussy people's sites. Go fish.

* That's an Amazon affiliate link... trying it out.

7+3 = 11

2011-07-11T23:26:00.002+01:00

tl;dr – a boring trivial bug is causing me to procrastinate by writing about it
tool – a spreadsheet to help you choose input data to spot the particular pathology

I love moo cards*.

Here's a snap of a recent bill from moo. Spot the bug.

100 cards £9.17 ... Shipping £2.50 ... VAT £2.33 ... Total £14.01 ... new blog post: priceless

The problem is a known pathology**. It's not uncommon to find that basket calculations are sometimes off by a penny; the calculations are done with precision, and those precise numbers are fiddled to fit with our quantum of currency - the penny. The error fits the fiddle.

In this case, the total including delivery and VAT looks as if it should be precisely £14.004. Expecting this to be £14.00, one might be tempted to speculate that the total has been rounded up my mistake, but two things make me not so sure.

Ⴀ) I generally see problems related to truncations (which always go down; £14.004 -> £14.00) and normal rounding (£14.005-> £14.01, but £14.004-> £14.00).
Ⴁ) thinking about it, I had a 10% discount on the normal price as a sop for the knock on effects of a previous bug. Discounts add another layer of complexity.

Let's work the numbers:

£10.19 is the normal price.
After the 10% discount, that would be £9.171, not £9.17.
Add £2.50 delivery to arrive at £11.671.
20% VAT on is £2.3342.
The precise total is £14.0052 - which will be rounded up to £14.01.
The VAT component is £2.3342 - which will be rounded down to £2.33.

That seems more plausible.

Is this a rare combination of numbers? I built a spreadsheet to explore, and it is not; 300 prices between 1p and £10 show this behaviour.

All this is in the context of a 10% discount, 20% tax and £2.50 delivery. But my spreadsheet is a model, so I can change the conditions. Playing with it gives me the following empirical understandings:

you don't see this problem without a discount;
within reasonable ranges, picking alternative discounts doesn't change the incidence much;
within reasonable ranges, changing the tax doesn't change the incidence much - I've seen it go down to 200;
the range of incidence seems to be 200-300 for 'reasonable' ranges of tax and discount
the delivery charge doesn't matter if it's to 2dp (and my model is inaccurate with 3dp)

Constraining myself to a basket with one item, I expect that I can sit down and demonstrate mathematically to my own satisfaction that in order to see a total that rounds up (ie £14.005), and an associated tax that rounds down(ie £2.3342), you need a price with a third decimal place - ie a normal price that has already been adjusted in some way. But that efficiency, while attractive, is a procrastination too far. For now, I'm happy with the general rule of thumb; you only see this problem when at least one thing in your basket can have a price that includes fractions of a penny - but if the potential is there, you'll see if for 20-30% of your possible prices.

Coders: One solution is do all calculations off-screen to full precision, but produce the totals on the bill from the numbers that actually go on the bill. Another is to round your total to 2dp before calculating tax. Of course this can mean having two containers for very similar information.

So far, so fun. For testers.

Frankly, I don't mind paying the extra penny. My problem is what the penny does to my paperwork.

I'm doing my VAT accounts, where I separate the £2.33 from the rest of the total. Moo's fluff on their bill means that stuff that should add up to zero, doesn't. I'll have to fudge the penny, which means introducing a special case. I'll have to be careful, because special cases are where I make accounting mistakes. That's a pain. I hope that you (or Moo) can use that description to advocate a fix for similar bugs.

And I hope that you go out there to find them. Here's the link to that spreadsheet again. I'll use it to generate data to help me reveal this issue***. You may use it and abuse as you wish. Please attribute me if you use it in public. It's got a second page that shows incidence, and a third with instructions, license and known bugs.

* Those of you who have had a business card from me are charmed by them, too. Moo's custom postcards will lend excellent grooviness to a game I have in mind. I want to make special stickers for a bunch of post-it related activities. Moo have always responded swiftly and sweetly to problems, and to top it all, they're local.

** I know this pathology, and I look for it when I test. Indeed, I've got an exercise based on something very similar in one of my classes. Some people question the veracity of that exercise; surely no-one really has obvious errors like this any more. Ha.

*** I tried a google docs version, but it runs like a three legged dog on Safari and Firefox. I was so discouraged I didn't bother with Chrome...

Something for the Weekend? 002

2011-07-08T20:08:00.003+01:00

Briefly – I'm back in the studio for the weekend* – I'm fascinated by the way that technology enables interactive art. Exploration, discovery and emergent properties are desirable, even crucial qualities of the work.

Here are the sites of two people whose work I find especially interesting:

Brendan Dawes – and you should also try MagneticNorth
Robert Hodgin – who has more at his blog, Flight404

I'm thinking of getting to know Processing, before trying to get to grips with Cinder. Any of you got experience to share?

Enjoy the weekend.

* teaboy, mainly

Broken by design

2011-07-06T17:55:00.009+01:00

tl;dr – I can't print or save a filled-in form generated by HMRC software

A pet peeve, involving the Taxman and Adobe.

The Taxmen need a form. They'd like it online, and generously supply free software to help me get the numbers in the boxes. I use their software. It produces a form as a .pdf.

It's a dynamically-filled form*, so if I use one of my usual pdf readers, the boxes are devoid of numbers. Only with Adobe's reader can I see my numbers in the boxes. Adobe's reader is desperately slow and buggy, and I need to explicitly allow it to trust this locally-made form in order to see anything meaningful – but that's not my peeve. My peeve starts when I get to a point where the form is useful, and I'm shown a neat purple message:

        You cannot save data typed into this form. Please print your completed form if you would like a copy for your records.

Well, I would. Note that I've not typed any data into the form; it's been generated for me by the Taxman's tool. I go to print the form. I tend to print to .pdf, as I'm swamped with archived paper as it is, and a .pdf is both searchable and findable. A dialog appears, jauntily sporting the following:

        Saving a PDF file when printing is not supported. Instead, choose File > Save.

I consider printing it to paper**, scanning it in, OCR-ing the thing and calling it quits. Just in case, I try File > Save. No one will be surprised to know that I'm told:

        Data typed into this form will not be saved. Adobe Reader can only save a blank copy of this form.

A blank form? I'm sure that's what the taxman intended. The observant will notice that, as happens so often, following the instructions will put me into a self-defeating infinite loop. I've met this before, and that's my peeve.

It's big guns time. I pull out Acrobat 6 Professional. We're into software-that-costs-money territory here, and indeed have plunged straight into that unhappy valley of software-that-I-need-once-in-a-blue-moon-but-buggers-up-my-machine-to-such-an-extent-that-I-wince. Acrobat Professional is, for those of you unacquainted with Adobe's upgrade paths, ongoingly expensive. It also plays nasty with the other children in the sandpit, and doesn't do anything (except this) that I need.

A minute or two later, after it has managed to load, trashed the screen redraw, bunged the CPU to 100% and asked me to upgrade (not on your nelly, you eight-year-old, tired, hack, although I admit I have considered it), I try printing again.

        Saving a PDF file when printing is not supported. Instead, choose Save from the File menu.

There's that 'not supported' message again. Acrobat aside, I've not yet met an application that can print, but that can't aim it at a pdf. Perhaps I should set up a .pdf printer - but choosing not to address the bristles on that yak for the moment, I choose Save from the File menu, and - astonishingly - I can.

I suspect that by not supported, Adobe actually means restricted to the paid-for version. I suspect (suspicious tester that I am) that Adobe have done this on purpose. The taxman has chosen to provide me with a tool that throws my data away, unless I pay Adobe for the joy of keeping it. I wonder whether the Taxman intended, condoned, or just didn't notice this behaviour.

Post scriptum***: As it happens, the tool turns out to be a dead end. The unprintable form is for my records only. Once I'm done slapping the desk, I fill in the online form in seconds and I'm done.

* For the initiated, this means that the .pdf (empty, pretty) is accompanied by a .fdf (just the numbers).
** Portable Document Format? My arse. Portable when folded up and shoved in a briefcase.
*** As distinct from PostScript. Print joke. Ah ha ha ha, bonk****.
**** Man laughing his head off.

Something for the Weekend? 001 (zero-padded in hope)

2011-07-01T14:55:00.001+01:00

Wil Shipley writes code; I use tools he has had a hand in* at least weekly, more so when I’m onsite**. He seems to be an auteur, involved in all stages of translating ideas into code into cash. He also writes words – copiously, but no longer regularly as far as his blog is concerned. A few years ago, he wrote up a narrative describing his thought processes and discoveries as he worked through a rotten bug. It’s called The Greatest Bug of All, and is packed with meaty goodness.

For those of you who are more visual, those who are interested in variation and multiples, or those fascinated by the anonymous human touch, here is Stephen Wragg’s collection of walking men. Note the specification.

Enjoy.

* typically OmniOutliner and OmniGraffle, although Shipley has moved on since to focus on Delicious Library, which I don’t use so actively.
** If I’m allowed to use my own kit…

How to Assure Exploratory Testing

2011-06-30T16:49:00.002+01:00

Over on LinkedIn, Paul Gerrard has started a "Test Assurance" group, and has asked "How can we assure exploratory testing?"

Lacking in discipline, I found it hard to read every bit of blither posted, but found it easy to respond. In the spirit of reuse, here is my take...

In the small sample of organisations that I’ve met who have an assurance group, each has gone about that work differently*. Some want to see that the team isn't wasting its time, some want to see that the organisation isn't developing false confidence, some want to see that the organisation can substantiate its claims, some want to see that set processes are being followed. If you're asked to assure exploratory testing, it's a good idea to find out what you are expected to judge and influence; those expectations may be in opposition to your own.

However, let’s assume that I am being asked, and I'm being asked to assure exploratory testing in an organisation that isn't doing something I find outrageous**, and I'm being asked to set up some sort of assurance without following the rails of a pre-existing culture. We can all dream.

In that case, I would work in a way that expected 'assurance' to independently assess the degree to which information coming out of a process of work could be trusted, and the degree to which the organisation as a whole trusts it. I’d expect assurance to be a sampling activity, with access to anything but without expectations of touching everything.

For exploratory testing, I would hope to:
- Watch individual exploratory testing activities to judge whether execution was skilled (I’d want a wide variety of business and testing skills on display)
- Watch the team to judge whether their exploratory testing work was supported with tools and information (exploratory testing without tools is weak and slow, exploratory testing in the dark is crippled)
- Gauge whether the team had independence of thought, and to what degree that independence was enabled and encouraged by the wider organisation (bias informs me about trust)
- Read some of the output (reports, notes, bugs) and watch some debriefs (if any) to judge how well the team transmits knowledge about its testing activities.
- Follow unexpected information to see to what extent it was valued or discounted (exploratory testing finds surprises; is that information useful and treated as such?)

I’d hope to do the following less-ET-specific tasks, too.
- Dig into any points where information could be restricted or censored (ie inappropriate sign-off, slow processing, disbelief or denial)
- Observe the use and integration of the team’s information in the wider organisation to judge whether the work was relevant, and accurately understood
- Judge the team’s sense of direction by observing the ways that information found, lessons learned, and feedback from the organisation affect the team’s longer-term choices.

I hope that our hypothetical organisation would use these insights to help jiggle whatever needed jiggling, and that once jiggled, the organisation could feel that they could trust the information from the team even more, and that the team would feel even more relevant and valued. Then I'd kick myself for signing the NDA that stopped me writing about it.

* Some have made the system, some have bought it, some have commissioned it, some have bolted it together. None have used exploratory testing as their sole means of testing. And I've never needed to 'sell' exploratory testing to those organisations that are risk-aware enough to have an assurance group.
** like trying to find a way to avoid clear responsibilities in a legally-defensible way, or trying to avoid the truth about the system under test

Testing, testing, 1 2 3

2011-06-28T18:06:00.001+01:00

Tfft... dffd... is this thing on?*

I'm spending a lot of time, recently, in studios. Some of them not my own. Here are two patterns of behaviour:

А: Silence - "Can you hear me?" - "Yup" - cue playback / recording until finished or otherwise interrupted.
Б: Noises (multiple, some questionable) - "What's that?" - "What?" - "That..." - "Oh. Um..." - stop recording, fiddle until the unexpected is understood.

Hello, world.

* Blogger stopped letting me update the server, the server went pear shaped, and I (copiously) lost enthusiasm. None of these things were connected, but their effect in combination produced a long period of silence. Let's see if this is an intermittent burble, or a gradually-increasing stream.

Has the UK SIGiST lost its way?

2010-09-14T12:22:00.000+01:00

The SIGiST used to be the go-to testing gathering in the South East, if not across the country.

The meetings had genuine energy, the speakers were world-class (and from all over the world), and SIGiST-connected movements led towards successful (in numbers if not acceptance) standards and exams. The group was one of the largest and most active Special Interest groups in the BCS.

Over the last few years, it has come more firmly under the wing of the BCS. It has lost its website, and with the website has gone any degree of facilitated online interaction between attendees. Participants have been disenfranchised; firstly in losing their vote (you now have to be a BCS member to vote), and secondly by the extended term limits of the officers.

In the few years since these changes, London tester gatherings like the TMF and the Tester Meetup have risen in frequency and attendance, SkillsMatter days have taken off with a range of testing themes, and XTC and ACCU regulars extend a warm and knowing welcome to testers. There is constant twitter chatter. Here on the STC, the online community is fertile ground for newspapers, books, groups and ideas.

It's two days to the SIGiST. The program for the event was announced on 24 August for the 16 September - less than four weeks before. The last post on the SIGiST's facebook wall was May. There's no LinkedIn event. Whack SIGiST into twitter, and all you get are references to the Brisbane SIGiST (let's ignore @mpkhosla reminding @MarkCTest that it exists, and my own plaintive self-promotional tweet). Until I set it up twenty minutes ago, there was no listing for the SIGiST as an event on the STC. There are still spaces available.

At this late stage, one would not expect to get time off work, or funding. However, there is a single overriding reason to go, if you care about this once-fine forum: the meeting on Thursday is the AGM.

If you're a BCS member (I'm not), and can spare the time, and want to say something, then please get yourself to the RCoG on Thursday morning and wake the place up. I've got the slot straight after the AGM - I'm more than happy to adjust my talk if you have something (anything) crucial to say. Note: You don't have to pay the entrance fee to attend the AGM.

If you're on the SIGiST committee, and are blowing sparks because I've written this here and on STC, then I can't tell you how relieved I am that you're reading.

Cheers -

James

The Irrational Tester - video and new version of paper

2009-12-13T22:46:00.002Z

Here is version 1.06 of my paper "The Irrational Tester". SQE have kindly posted a video of (one version of) the talk that goes with it; keynote at STARWest 2009.

TestLab stuff coming soon...

TestLab Followup - not yet...

2009-12-09T23:57:00.000Z

Bart Knaack and I ran the "TestLab" at EuroSTAR. I've got lots to write about it - but unfortunately (or fortunately) I came back to an unexpected job in the UK. I'll post conclusions, bugs, pictures and more in a day or two. I've not yet submitted the bugs to OpenEMR, either. It will all happen, but not immediately!

For now, read Michael Bolton's blog postings Bravo! and Best Bug, or Rikard Edgren's Notes.

Cheers -

James

Oh for goodness sake

2009-10-11T23:14:00.002+01:00

The estimable Association for Software Testing isn't exactly approachable via Google, is it?

As it happens, it's reasonable on Bing.

The AST is an association for testers, run by testers. Perhaps it works on their machine.

I shouldn't be so sarky. I've sponsored their conference before now, in 2006 and 2008. I recommend you go.

(now, let's see if my journal-to-blog tool manages picture attachments properly. Expect to see an edit if it's still bust.)

Exploratory Testers - London 3 July - Prince Arthur, Euston.

2009-06-29T23:01:00.000+01:00

As you may know from a previous posting, Michael Bolton and I are going to bring the participants of our separate Exploratory Testing / Rapid Testing classes together, after our classes finish on Friday evening. Please join us!

Date: Friday 3 July, from 5:30 pm
Place: The Prince Arthur pub, near Euston station. Reviews from Beer in the Evening and Fancy a Pint. Here's Google streetview, which should help you get there. Being just next to Euston, it's good for tubes, trains, bicycles and other rational London transport.

I think this is a remarkable opportunity – I hope to see you there!

Cheers - James

London, 3 July – Exploratory testers in a pub

2009-06-18T11:42:00.000+01:00

Tentative plan: Michael Bolton and I are (separately) teaching in London in July. Our courses, mine on Exploratory Testing, his on Rapid Testing, end on the same day. We've talked about bringing the participants together in a London pub at the end of the course. Yesterday, at the SIGiST, we slipped into an announcement.

We don't yet have a pub set, and one or both classes will have to travel (my course is in the City, Mike's is in Westminster). We hope that we will have an enthusiastic, engaged, but not-too-exhausted group of exploratory/rapid testers, and that by bringing the participants in our classes together, we'll all have a lovely time, and learn some stuff.

Details will be posted closer to time, but are, so far: A group of exploratory testers will be meeting in London sometime in the early evening of Friday 3 July. It will be a unique event. Please join us.

PS - a few places are still available on my course. Details here, book here.

More testers in a pub

2009-06-16T00:29:00.000+01:00

If you're interested in testing and based in London, you'll want to be in the Washington pub in Belsize Park tomorrow (Tuesday) night.

http://blogs.imeta.co.uk/RLambert/archive/2009/05/19/the-testing-revolutions-first-meeting-london-tuesday-16th.aspx

The Wash is, funnily enough, only 15 minutes wobbly walk from the LEWT pub (and five minutes by steely steed from Workroom Towers). However, you'll have to tell me all about it – one of my other lives has precedence tomorrow pm. Have a lovely time.

Designing Experiential Exercises

2009-06-03T18:08:00.003+01:00

Readers of this blog may already know that I prefer to learn/teach by doing. If your preferences match mine, you'll be interested to know that Jerry Weinberg and Esther Derby are running a public workshop in Albuquerque on the Design of Experiential Training, from June 22-25 (or 26, depending on which web page you read). Details from Jerry, from Esther.

Jerry is, in addition to his other stuff, the master of the experiential exercise. Or so the people I trust tell me. I'm hugely excited to be going to the workshop. Heaven knows why I've not mentioned it here before. Anyway: I'll be in Albuquerque for midsummer, doing fun stuff.

Have I mentioned my class? Getting a Grip on Exploratory Testing, in London. 2-3 July? Looking forward to that, too. It's all very hands-on and participative – and I imagine I'll still be full of the joys of ideas, and just past my Albuquerque jetlag.

Busses and Exploratory Testing classes

2009-05-28T09:11:00.000+01:00

You wait for ages, then two turn up at once.

I'm running my course in Exploratory Testing in London on 2-3 July. Details here. This is the course I've taught at Nokia, Google and around the world. It is a limited size, practical, hands-on class, and it is all about testing – specifically, how to uncover problems in working systems in a disciplined and efficient manner. I really enjoy teaching this class; lots of lightbulb-on moments for all.

To our mutual frustration, my colleague Michael Bolton is teaching 'Rapid Software Testing' at the same time, also in London. It's a great class, too. The scheduling problem is ours - but the choice is yours!

There's more to say here, but I've got a lovely on-site hands-on testing gig until the end of the week, and the daily stand-up happens in 35 minutes, at the other end of a 30-minute tube ride. Scrums wait for no man, or, at least, no tester...

Best Paper at STAREast

2009-05-10T22:01:00.002+01:00

I'm pleased to announce that The Irrational Tester won "Best Paper" at STAREast. * ** ***

The paper is currently at version 1.0. I'm adding stuff (and sorting out the prose) for version 1.1. Please read it - if you've got any comments, I'd be very interested to hear them.

* The presentation got 9.something out of 10, too.
** In 2002, my paper Adventures in Session-Based Testing (written with Niel vanEeden) won "Best Paper" at STARWest and at EuroSTAR.
*** Three Best Paper prizes! Three! Wahey! ... shutting up now ...

London Exploratory Testing, 2-3 July 2009

2009-04-28T13:15:00.000+01:00

Getting a Grip on Exploratory Testing will happen in London, on 2-3 July.

We're at the lovely Royal Statistical Society, near the Barbican, just north of the City.

I've kept close to last year's prices, and even managed a small reduction (you get a buffet lunch in 2009...), so it's £630+VAT. There's an early-bird discount, so if you book and pay before June 8th, you'll get 10% off (£567+VAT). Email me to register, or get into the online registration here: Online registration

This is the course I teach at corporate clients (recently Google and Nokia). It's very hands-on, with exercises and discussions driving the workshop. You'll discover:

The test design skills to probe a system and trigger a bug
The analysis skills to model the system and understand a bug
The discipline to manage your exploration and sustain your bug rate

Exploratory testing is a disciplined approach used to uncover risks and surprises in real systems. It's a great complement to the massive confirmatory testing found on agile projects, and a necessary skill for agile testers, and for any other tester who needs to do more than simply verify that a system is working as expected. Test managers find this course useful for reconnecting with their test skills, and for understanding the challenges of effectively managing a team that is making good use of ET.

Help me work out when to run the next Exploratory Testing class in the UK.

2009-03-24T17:50:00.000Z

I tend to run an Exploratory Testing course in London once or twice a year. A bunch of people have recently asked me when the next one would be.

Rather than set a date and see who can come, I thought it might be better to gather suggestions for dates. I've set up a survey to help you tell me, and I'm giving contributors a discount. It's a two-minute job, if that. Click here to help.

Note - if you're in mainland Europe, I'll be running a class in Berlin, June 4-5. Details in the most recent Testing Experience.

Cheers - James

Firehose

2009-03-21T18:33:00.001Z

Gotta love* Safari Books Online.

Every book I was about to buy this afternoon, I find I already have**. It's not the first time, either.

No*** Kaner, no Bach, no Whittaker, no Weinberg, so I still need my library. But for everything else**** – from visualising data to Ableton Live, from Python to a guidebook for a 2-versions-old iDVD, there's Safari.

* I agree that it has a frustrating interface, clunky search, single-source issues, is slow for page-flipping, and the 'tokens' make me cross - but I can read those books right now. And there's no new money to find. And I can search. And copy/paste.
** Or, rather, have access to, while I'm online, while my subs continue, and while the service yet lives.
*** I understand the reasons for all these authors not being on SBO, but pure-testing books are generally under-represented. Use this to see the books in the Testing and Debugging section.
**** Of a technical bent.

Funny bug

2009-03-09T01:01:00.001Z

Facebook. Hardly unusual, but made me smile.

Just back from Amsterdam and Copenhagen; enjoyable classes, nice feedback, new contacts. Took the train from A to C - lovely.

Common programming errors (in the news recently)

2009-01-14T12:11:00.000Z

Seen this? 25 Dangerous Programming Errors

It's from SANS, a training provider. I've removed the words "Top" and "Most" to reduce unnecessary hyperbole.

It's culled from this: MITRE's Common Weakness Enumeration. Interesting stuff.

I don't have an opinion on it, yet - but I'd be interested to know who, in the software testing community, does. Answers on a postcard, please. Or a comment here, if it's easier.

XPDay

2008-12-05T00:28:00.000Z

I'm going to XPDay!

Wahey!

Coders, Unit Tests, and Testers

2008-11-28T19:00:00.000Z

On the QAGuild group on LinkedIn, Prasad Narayan asked "Does the Dev team in your organization indulge in Unit Testing? I would appreciate some details, if the answer is in the affirmative.".

Here's my response:

I've worked in plenty of teams where the coders have written unit tests. Typically at organisations which explicitly care about coding. Less so at banks, service or entertainment organisations. On a reasonable proportion, and most especially on agile teams, the coders have written very large numbers of unit tests that act as a scaffold to the code (that is, very large as compared to the expectations of teams that don't use unit tests, so perhaps this is tautological) . More than a simple framework, the tests also act as a way to frame thinking about the code, to consider it before it is made, and to experiment with it when it is under construction.

Such teams are (in my experience) universally proud of their unit tests, and actively show them off. Their code tends to be better, too. When I've worked with teams who are shy of showing me their unit tests, and shy of letting me review them, then (in my experience) the code is universally duff. I'd rather work in a team that has no unit tests than a team that says it does, but won't show me (as a project member who is interested in testing) the tests.

I have worked with teams that write unit tests, but don't run them. This sounds bizarre, but tends to be a problem that creeps up on teams. I've seen it as a result of commenting out unit tests that break because of a non-code-related change (without replacing the test). Be aware also of unit tests that don't test anything important, or which pass with or without the code in place. I have heard of (but not worked on) teams where the testers wrote all the unit tests, and the coders wrote the code. I guess I'd hope that the two groups work closely enough that tests and code could be written together.

It's all to easy to let good unit testing and the resultant relatively-clean code lull one into a false sense of security about the viability of the system as a whole. If I work with a team that is using lots of unit tests, I (very broadly) take this approach:

reviewing the existing unit tests

being sure they're run regularly

trying not to duplicate them too much with my own / the customer's / other team's confirmatory scripted tests

using exploratory / experimental / diagnostic approaches to pick up and dig into all those unexpected risks and surprises

working closely with the coders to enhance, streamline, and otherwise improve their unit tests

Exploration

2008-11-24T11:57:00.000Z

"Do not go where the path may lead; go instead where there is no path and leave a trail"
Ralph Waldo Emerson

Can you start using Exploratory Testing without needing an expert?

2008-11-22T12:29:00.000Z

On Software Testing Club, Anna Baik asked "Can you *bootstrap* your team into using a more exploratory testing approach without needing in-house expertise? Or are you likely to have problems unless you have either a consultant or an already experienced exploratory tester on staff?", which struck me as a reasonable question. Here's my answer (also posted as a reply on Anna's blog, mostly).

You can do exploratory testing without an experienced exploratory tester. Indeed, that's what all exploratory testers do with a new system / technology / customer etc.; start from a position of ignorance and get exploring.

There are a couple of ideas that I'd recommend keeping in mind:

one of the things you'll be doing is gaining experience, and (through reflection) expertise. This is part of all exploration, but it's particularly true when getting going on something new. Expertise will help you find more/better information – and you'll find fewer/poorer without expertise – but it takes time to build. However useful it is, it is neither necessary, nor sufficient, and even expert teams explore better with a newbie in the numbers. This is because...
if a few of you are exploring, there will be great diversity in your approaches. Learn from each other - and try to take those lessons in a way that doesn't flatten that diversity.

With those thoughts, here are a couple of recipes. Adapt as you see fit. For both, you'll need to set aside some time. Regard this time as a gamble, and assess its value when you're done. If I was doing this, I'd prefer to work in a group (so for a 3 person-hour recipe, schedule 2 people for 90 minutes). I might assess value by looking at whether I'd changed my bug rate, or found any particularly useful information / nasty bugs.

1 Exploring via Diagnosis: 3 person-hours

Pick out some current bugs that aren't clear, or that aren't reproducible. Doesn't matter if they're in your bug tracking system or not, but they should be already known, and not yet fixed.
Explore those bugs, seeking information that will clarify them or make them more reliably reproducible. Keep track of your activity.
Review what you've done, collate the information gained. Log new bugs, update clarified bugs.
If you can generalise from the investigative approaches you're used, then do so.
Tell the whole team your results.
Schedule another 3 hours on a different day and repeat!

2 The Usual Suspects: 2 person-hours + 10 minutes preparation

Spend 10 minutes writing down lots of different ways that bugs manifest / get into your software (Use any or all of cause, effect, location, conditions, etc.). Aim for diversity and volume, not completeness or clarity. This might be fun as a whole-team exercise.
Leaving enough time for the review+generalise+share steps at the end, split the remaining time in two.
In one half of the time, pick out problems that you've not yet seen in this release, and look for them. Keep track of your activity.
In the other half, pick out places that haven't yet seen many problems, and look in those for problems. Keep track of your activity.
Review, collate, log, update.
Generalise and tell the whole team your results.
Schedule another chunk of time on a different day and repeat!
Make your trigger list publicly accessible. Invite people to contribute, share, refine etc.

That said, ET is a skilled approach, and it's easier to get those skills into a team with a bit of reading / taking a class / involving a coaching consultant. There are plenty of sources around about getting started with exploration. Niel vanEeden and I wrote a paper called Adventures in Session-Based Testing which may help. It's here: http://www.workroom-productions.com/papers.html. For pithy heuristics, Michael Bolton has recently blogged the results of a fine EuroSTAR workshop here, and you'll also want to check out Elisabeth Hendrickson's cheat sheet, James Bach's mnemonics, Jon Bach's questioning approach, James Whittaker's tours (hopefully turning up on his blog soon).

If you have a favourite bootstrapping-into-exploration source, post it here or on Anna's original posting.

Note - I have an upcoming, no frills, public class on Exploratory Testing, in London, December 8-9. Lots of exercises, discussions, and learning by example.

Checklists

2008-10-30T23:51:00.000Z

A regular correspondent (hello, thanks for triggering this) asked me about checklists and testing. I had a half-written blog entry on some checklist rules of thumb, and shared it with him. Here it is:

Personal

Checklists I make are typically more useful than checklists I get. I expect this is more closely-related to ownership than quality.
I tend to do items at the top first, so ordering may become important even if it isn't meaningful.
I can't take in more than around 20 items on a list without giving it some structure. To help make a list comprehensible, I use outliners, item codes etc, and group items into categories.

General

Laundry-type lists are different from shopping-type lists. Laundry lists tend to be set up as general resources, tend to be long, many items don't apply to current circumstances and so one picks a few, can be inspiring. Shopping lists are often made for a specific need and discarded later, one tries to cover everything on the list. I can use each type to enhance the other, but it's generally a bad thing if someone else uses one of my laundry lists as a shopping list and vice versa. Testing Example: "All my charters" is a laundry-type list. "All my charters for today" is a shopping-type list.
The value given to a laundry-type list can come from an assumption that it is exhaustive, and that its items are mutually exclusive. Few lists are either – there are often missing items, and existing items overlap. Testing example: Lists of non-functional testing qualities. The real value in this laundry-type list is often that it inspires a shopping-type list.
When categorising, it's important not to use categories as proxy items. Even if the list is exhaustive, many items can be re-categorised – so doing none of, or all of, a category can be more arbitrary than it seems. Testing example: charters grouped in a hierarchy.

My correspondent indicated I might be interested in a not-so-recent New Yorker article by Atul Gawande about checklists in medicine. Surfacing briefly into the rolling boil of tester blogs, it turns out that the article has triggered a gentle meme-ripple through the industry.

Gawande's article describes shopping-type lists, to aid memory and set out minimum requirements. It describes how those lists were used by medical professionals to help them do their jobs more reliably. The checklists covered the simple stuff, and called for no judgement or skill in their use (of course, plenty of skill is needed in their construction, and in following whatever comes next to the little tickety box). The results were impressive – but just as impressive was that the medics actually used the simple things.

It seems that the people involved were responsible for making their own lists (perhaps collectively rather than individually) and also for finding out if the lists were working. They were supported at multiple levels – nurses checked that the checklist was in use and the boxes ticked off, executives made sure that necessary supplies were available for the tasks on the list – so that there were few reasons not to actively use the checklist.

I'll add another note to my 'General' list above:

When you're doing a difficult job under pressure, checklists help you concentrate on the job, by allowing you to expend less attention on avoiding mistakes. Testing Example: a list of possible input means (type, mouse, paste, drag in, cause default, undo delete, refresh).

The trick in this counter-intuitive heuristic is the difference between concentrate on, and do. A checklist can let your mind work more freely because the standard stuff isn't going to be forgotten. Indeed, I make shopping lists to go shopping with so I can multitask (for which read listen to The Goons on my iPod).

The article doesn't deal with two important ideas;

Change; when+why do items come off a checklist (important for shopping-type lists).
Use; what kinds of situation are most amenable to lists.

Aside from recommending regular reviews, I have nothing to say here about changing checklists.
Checklists generally help in situations which are:

well-known
busy (in the sense of being dense with stimulus)

It's clear that the massive confirmatory unit tests (and 'acceptance' tests) that characterise agile development can be seen as shopping-type lists, and are all the more powerful for it. The subject is well known (and the tests describe that knowledge) and the environment busy (in the sense of very many tests being run in quick succession). As a list, it helps exactly because it allows one to expend less attention on mistakes. The great (though arguable) strengths of massive confirmatory tests are, however, a special case.

Software development is certainly busy, but much of it is not all that well known. From a test point of view, often we're looking for the unexpected. From an engineering point of view, it's often hard to know what is reliably effective, sufficient and harm-free. Partly as a consequence of this, we tend to start out with laundry-type lists rather than shopping-type lists.

To pick out a few busy+well-known areas specific to testing, one might look at test environment setup, and the list of information on the top of a session form. Both these have a kinship with pre-flight checklists, and if you're not already checklisting in these situations, I expect you would find it valuable.

However helpful checklists are, they are frequently resisted as 'dumbing-down' a skilled task. As one who has resisted in the past, and who is likely resist in the future – I feel this is an entirely fair criticism. Perhaps the best bet is to take an approach similar to that taken by Peter Pronovost, the article's protagonist:

get whoever is making the {mistakes you're trying to avoid} to put together their own mnemonic/minimum standard list

get them to measure themselves with and without the list in use

provide strategic support to help ensure that there's no practical reason why something on the list can't be done, tactical support to help ensure that the list is actually used and used honestly.

Applied Improvisation

2008-10-13T21:23:00.000+01:00

I just got back from the "First in a series of Applied Improvisation Network (AIN) London Events", and thought I'd share some impressions.

The AIN describes itself as "Spreading the Transforming Power of Improvisation". At this particular event, AIN founder Paul Jackson was going to "use improvisation activity to introduce a business theme ... [using] complexity/emergence as the business theme example". I decided to go along because it sounded fun. Re-reading this, I wonder if perhaps my taste for fun has become a little over-sophisticated. Never mind.

The evening was pretty much as described. Just under a dozen people turned up to the usual oddly-shaped room in a re-purposed building. We talked, interacted as individuals and as a group in a set of well-structured exercises, and pushed off for a pint/meal. All was fine and dandy, and I'll go again. I found the exercises interesting, and may adapt* some for my own consultancy and groups - LEWT people, expect to gather in self-selecting groups sometime soon.

However, in this evening's exercises, there was a frustrating focus on game over content. I was reminded of peer events I have attended which degenerate (and I mean degenerate) into good teachers swapping their favourite lessons. Enjoyable and informative, but I took much more away about facilitation exercises and ways to get people to engage in improv than about the structures and ideas of improvisation. Emergent behaviours were discussed, but as personal lessons emerging from an exercise, rather than as properties emerging from a system. Business was discussed, but in terms of getting business people up and interacting in workshops, not in terms of translating improvisational skills into their working environments.

Reacting to this, I revisited some of the ideas about improvisation that I was playing with over the summer, and present them here, tidied and decorated for your amusement.

I'm interested in the improvisation involved in exploring a city, making an extempore speech, singing harmony to an unfamiliar tune. We improvise when we cook a meal with whatever is in the fridge, when we need to get a USB key from behind that hotel radiator, when someone falls off a ladder in front of us, when we get lost - especially when we get lost. To be expert is to be able to improvise with confidence.

It's clear that my interest in improvisation is, in particular, the improvisation we do as individuals under pressure from external circumstances. Perhaps I'm just not a team player; after all, my sports are/were skiing, swimming and fencing. Ask the Choir if they agree before you jump directly to any conclusions...

Improvisation as AIN addresses it is useful, interesting, but seems (on the strength of a single meeting and a swift half) to be biased towards shortform group improvisation under circumstances imposed by the group. This is more complex in at least two ways, and a wonderful field of study - but the interests I list above would be poorly served if this was where improvisation stopped. Conversations indicated that, perhaps, improv was the only improvisation the group could discuss with engagement. I think there's more, and I look forward to interacting with the group and its approaches.

One-line summary: improvisation ≠ improv. Who knew!?

* at the workshop, no ownership was claimed, and no attribution given. One could use the viral meme (pace GPL) and apply the same rules when passing it on, or apply one's own standards if more stringent. I choose to apply my own standards - these exercises were facilitated, and may have been devised, by Paul Z Jackson. However, if it is the practice within this industry to change and neither claim nor attribute, I many yet adjust those standards to fit the context. For those interested in improv exercises, http://improvencyclopedia.org/ is a resource with more than enough (500+) to tickle your fancy.

Decisions and responsibility

2008-10-13T15:10:00.000+01:00

There's plenty of mileage in group decisions, and in the wisdom of crowds. I presume that, as technology enables the convening of groups, we'll see more decisions made collectively. I hope - and believe - that in general those decisions will be better.

However, making a decision is not the whole picture. There is a degree of responsibility that goes with a decision, and I'm coming to the conclusion that a group decision is worth nothing if the individuals in the group are not prepared to take individual responsibility for their decision.

One-line summary: watch out for decisions made by groups whose members are disengaged

Agile2008 (and Glasto 2008)

2008-10-07T10:49:00.000+01:00

Under the inspired guidance of Rachel Davies, Agile2008 modelled itself on Glastonbury Festival. I found myself performing at both, which came as a bit of a surprise.

Aside: I'll leave the muck and general debauchery of Glasto to your imaginations - suffice it to say that I was entirely sober, stayed at my Mum's, and was on-stage with my wife*. With 25 stages and 150000 revellers spread across a Somerset valley, Glastonbury's scale is as staggering now as it was when I first went in 1987 (no wife, no sleep, not sober). It turns out I'm no less impressionable at 40 than I was at 19. I loved it.

With 1500 people, Agile2008 was a couple of orders of magnitude smaller than Glastonbury. Like Glastonbury, it was as fascinating as it was overwhelming. Attendance was about as big as a relatively-technical hotel conference gets, but the truly staggering element was the 25 concurrent tracks. Being a power-law thing, this of course did not mean 60 people in each session, but hundreds in a few, and a small handful of patient listeners in most others. It meant that nobody saw more than a tiny fraction of the material on offer. However, the keynotes** were attended by a vast majority of participants, and served to align the subjects of conversation. With this, and the attention given to breakout areas, triggers for discussion, and informal entertainment/events there was a clear feeling of community.

Performing or not (unless you're headlining), these events are at least as much about going and being with the crowd as they are about seeing the stars. You're as likely to love an act you stumble upon as an act you've waited years to see. I'm perversely proud of wandering away from the mighty (and very favourite) Massive Attack at the height (depth?) of their Other Stage thunder and into the elderly groove of the wonderful (and utterly new to me) Ethiopiques. I'm happy to have voted with my feet in Ron Jeffries / Chet Hendrickson's surprisingly artificial Natural Laws of Software Development and just as happy to have made the temporary acquaintance of that embittered sage, Brian Foote. I crossed paths with Toby Mayer half-a-dozen times, each time coming away with insight and inspiration.

The most interesting element for me was the degree to which the (formal) practices of Agility were not only reinvented by each team, but to a significant extent rejected. Two talks highlighted this particularly well:

Bob Martin's keynote gave the most visceral example, as he asked everyone in the room to put their hands up if they were involved with an agile project, then read a list of common practices and asked people to put their hands down as he listed practices which they did *not* do. By the time he had got about five items down his list, 1500 hands in the air had reduced to just one group, and a couple of dozen isolated hands across the hall. His next point; 'keep your hands up if all your tests are automated' took out the group (oddly enough, a gang of testers from Menlo Innovations) and only very few individuals remained in the game.

Scott Ambler's talk (which packed the 'Questioning Agile' stage/room) put real numbers on this phenomenon with a survey from Dr Dobbs in February. You can read his conclusions ("Agile in Practice: What Is Actually Going On Out There?") on his site, but better yet see a video of the talk (from about 5 feet from where I was squashed in) on InfoQ. You might be interested to know that he's made his data available for analysis. I've been looking through it from a testing point of view for a financial client, and his conclusion seems supported: "The easier practices, particularly those around project management, seem to have a higher adoption rate than the more difficult practices (such as TDD) ... For all the talk about TDD that we see in the agile community, it’s nowhere near as popular as doing a bit of up-front modeling, which we rarely hear anything positive about.". Indeed, I'd be tempted to say that the numbers indicate that practices related to testing are typically among the less-likely to be used. None the less, 80% of respondents felt they had better quality and happier customers.

For those of you who are interested, my own talk (also on the Questioning Agile stage) went well (rather better than the Guardian Stage at Glasto), captured a good audience and generated some fruitful discussions. I took my slightly-jetlaggy part in the pre-conference "functional test tools" workshop (a physical extension of the ongoing discussion on yahoo group aa-ftt) which was worthwhile, but not terribly conclusive.

An excellent event - great for new perspectives, for new people, and for fun. I'd certainly go to Glastonbury again - and with any luck, Agile20xx.

~ o ~

* she leads, and I sing in, the London Bulgarian Choir. The lovely British Sea Power lent us part of their acoustic spot in the Guardian tent, and the girls sang with them for songs on the John Peel stage and the Left Field stage.

** James Surowiecki on diversity/wisdom of crowds, Alan Cooper on engineering user experience and iterative/incremental methods, Bob Martin on a 5th line to the Agile Manifesto ("we value craftsmanship over crap" - although I think there are efforts to make this more boardroom-friendly)

Fun for explorers

2008-09-21T20:37:00.000+01:00

If you like physics, or playing with stuff, you'll like this: FantasticContraption

A perfect example of exploring solutions, alternatives and refinements. For those of you who have been on my exploratory testing course, I couldn't teach ET with it, but I urge any readers interested in exploration to watch themselves - or someone else - working towards solutions, general applications, principles, components. I've just lost a chunk of time myself in a gentle whirl of levers and engines. Reminds me of Meccano, but it's fast, flexible, and never runs out of bits (or seizes up with friction). Must get AquaForest back on to the iPod Touch.

Not every revolution is part of an evolution.

2008-09-17T08:49:00.000+01:00

Evolution is always revolutionary to those caught up in it. Only hindsight makes a revolution part of an ongoing evolution.

If you're not part of the revolution, you won't evolve. If you don't evolve, you're stuck in a dead end of ever-decreasing resources.

Of course, this is to confuse the individual and the tribe.

An alternate way of putting this - no individual survives (Darwinian) evolution. Thank goodness culture is Lamarckian.

Exploration and experimentation

2008-09-15T13:07:00.001+01:00

I thought I'd share this, from The Economist:

In the 19th century it was commonplace to do an experiment simply to see what would happen. That was, in part, because experimenters were often amateurs who were spending private money. In these days of taxpayer-financed science, most experiments are executed with a pretty clear idea of what the outcome ought to be, especially when they are part of wars and campaigns against this or that. The paradox is that, although such efforts do not eliminate Becquerel-like discoveries, they risk limiting the chances of making them.

Tools for Exploratory Testing

2008-06-17T15:54:00.004+01:00

I am often asked what tools I use for Exploratory Testing – most recently, by Patrick O'Beirne while doing a series of talks at SoftTest.

It rather depends on what I need to explore, but my own explorer's toolkit (with a probable unix bias) includes:

- for recording visual stuff: I most often use a camera, which allows screensnaps and brief audio+video without being platform-dependent. For screenshots, on a given platform, I use SnapzPro on Macintosh/OS X, and HyperSnap on Windows. I've been know to rig up a video camera to watch keystrokes and screen. I also use Mouseposé, which makes clicks obvious and shows keystrokes (including modifiers) in a big translucent bezel.

- an aside: I also use Mouseposé when I'm showing people what I'm doing, and even to help me get visual feedback on the buttons I'm pressing. Butterfingers = happy accidents.

- for tools that specifically help with recording manual exploratory testing (ie records keystrokes and timings, allows annotation of screen movies etc.), I've used SiriusSQA's TestExplorer and BlueBerry's BBTestAssistant. For my purposes, they're often a bit heavy and windows-centric, but you may find one or the other is just what you need. You can try them yourselves, as both have a great attitude to limited-use trials. Spector - the outrageously intrusive spy-software - was often put forward as an alternative before tester-oriented tools became available. It seems un-disruptive, and offers comprehensive monitoring, but I found it hard to use for testing. You may find the terms of the license prevent you from using it well in a test environment, and there are difficulties in saving the information you've captured. Finally, it's so very much oriented to sneaking on spouses and employees that having the license makes me feel slightly creepy myself.

- for recording what I do, I was pretty-much paper based until a couple of years ago. I now use a dual system, keeping most of my scribbles and diagrams on paper as an extension of my thought processes, but using OmniOutliner to keep track of bugs found, to put timestamps on observations, arrange stuff hierarchically / foldably, and to allow me to search my notes.

- for monitoring system activity, I use "top" on UNIX-based systems. I've recently discovered the joy of Process Explorer and perfmon (when I got the logging finally turned on) on windows. On the Mac, Activity Monitor is reasonable, but Instruments (a wrapper for DTrace) is for serious work. Crash monitors for specific applications/OSes are good, too, as is a knowledge of where the logs are hidden (and how to extract them).

- for looking at differences, I use unix/windows tools comm and diff. Unix tools can be introduced to windows environments with Cygwin, UnixtUtils or many others. You could also try kdiff, which is pretty comprehensive and runs on everything, or FileMerge (in xCode) on the Mac. It's good to know one's way around regular expressions, so here's a link to ilovejackdaniels' great Regular Expressions cheat sheet. If you need to do specific Windows Registry checks, Process Monitor and TestExplorer will help you out.

- for influencing system activity, it's got to be Holodeck - the freebie on the back of "how to break software security" is less buggy than the version on "how to break software", and (being free) is $$$ less than the retail version. Roll on the day when developers expect this sort of capability to be built into OSs.

- for input data, I often have a library of useful data about the place - large files, empty files, different sizes of different formats of picture, files that have meaning to something I'm testing (ie an Excel file for testing within MS Word). I sometimes use Bach's perlclip for creating data. I use Excel for creating test data to fit requirements, and load it into SQL tables with CSV

- for output data, I primarily use Excel for most parsing/analysis work - with a little bit of fiddling to get in amongst datasets that break Excel's 65K row limit. I very much prefer DataGraph for exploring datasets graphically. It's Mac-only, but IMHO worth buying a mac for.

- for automating , I use whatever's lying about at the client's site. Last time I bought a tool, it was Vermont High Test, so my licence is a bit out of date these days.

- for web testing, I push hard for Firefox to be within the browser gamut, and use FireBug. iMacros was in my toolkit for a while, and I wasn't over-familiar with it, but I've recently tried it out for loading web forms, and it's fine. The firefox version is still free (as in beer).

- for cross-browser testing, I use the free web service netrenderer if I've not got my kit with me. It opens a publicly-accessible web page in the browser of your choice, takes a screensnap, and shows it to you.

- for putting load into an app and measuring output (although not graphing it, because Excel's far better, and DataGraph better still), I use (cross-platform) JMeter. You might want to consider The Grinder.

- for emulators (dead handy), I've had great results with VirtualPC on PCs. The field is moving fast, and these days I'd look specifically at VMWare and Parallels. I've used Parallels on the Mac lots, but not for testing (yet).

I'm sure you use different tools. When I teach Exploratory Testing, the tools workshop is often most eye-opening in closed teams, where one person is often sitting on a tool whose use is immediately apparent to other participants - yet they've never shared their toolsets until the class. Try it yourself – I'd be fascinated to know what tools you use.

[Edited to include OmniOutliner, Mouseposé and to clarify the TextExplorer / BBTA entry]

[Edited to include netrenderer]

Heathrow T5 and software testers

2008-05-13T11:28:00.000+01:00

CIO magazine pulls no punches when they say that inadequate software testing (was) behind T5 problems. I won't make easy assumptions based on that article - you can read it and make your own.

However, if you've been following the story, you'll be interested to know that Huub van der Wouden and Roger Derksen from Transfer Solutions in the Netherlands will be giving the opening keynote on Testing the Heathrow Terminal 5 Baggage Handling System at the London SIGiST on June 18th.

Searchable library - content, not title

2008-05-10T15:20:00.002+01:00

So I was going to buy Gerard Meszaros' book xUnit Test Patterns at StarEast, but the excellent selection from Breakpoint Books had sold out by midway through day 2.

I find that I already rent the online, searchable edition. If it wasn't such a pig to use, I'd love Safari books online.

I'd like to have an online search for my existing luggable library, to let me find what I need in books where I don't have an index, or where the index is rubbish. Google Books gets close, but only gives one return per book. If you know how to get more than one, do let me know.

How much TV does a Wikipedia cost?

2008-05-01T11:22:00.003+01:00

Clay Shirky has written a book called "Here Comes Everybody: The Power of Organizing Without Organizations". Here's a couple of localised Amazon links - amazon.co.uk, amazon.com. Now, I'm not about to recommend the book without reading it. I am about to buy it so I can read it on my way to teach at STAREast.

However, the interconnected world allows us all to read / see what the book's about. Here's a fine blog post of his summarising talk. For those more visually/aurally oriented, here's a the talk itself.

Answer to the question above: Clay says that each year, the people in the USA spend 2,000 times more time than has been needed, so far, to construct Wikipedia. Except that he says it better. Read the article.

Let's work these numbers a tad to give us a feel for their reasonableness. If there are 500,000,000 people in the US, then the work done on Wikipedia so far is equivalent to the TV-watching habits of 250,000 people. That's round about the population of Boise (Idaho) or Daytona Beach. For those of us closer to GMT, think Belfast. If the average citizen spends 10% of their waking time watching TV (The NYT says it's over 4 hours, which is closer to 30%, but let's make life simple and not cut out TV entirely - I assume that many have the TV on while eating, talking, making macramé wallhangings), then it would take the waking time of 25,000 people for around a year. English-only wikipedia has 2.5 million articles, so that's around 3 articles a day per person - I'd have said 1 or less, but we would seem to be around the right kind of numbers.

Of course, this may be a circular analysis if Clay started with 2.5 million articles at around 4 hours a piece to write.

A question for you: Do you know anyone who actually understands the numbers, ie can apply them in their everyday lives? Who understands fundamentally how large numbers of people and small commitments/risks/expenditures actually add up? It doesn't exactly come up in conversation much, but I don't think I know anyone who has a clear feel for this. Perhaps it's my generation. Then again, I know plenty of people who have a fine handle on atomic measurements and cosmological time - perhaps I should get to know some social scientists.

I'm aware that, as humans, we're bad at things outside our direct experience - and have to either put things into scales we can understand, or manipulate the numbers directly. My problem is, perhaps, that we're bad at viscerally understanding how large (small) a thousand (th), a million (th), or a billion (th) actually is - and so when we combine a tiny with a huge we're bad at understanding what that means. In our actual lives, being out by a factor of two is plenty - but when dealing with things beyond our ken, it's much harder to spot. Both the following are out, by a bit - but by how much? 1) A trillion (US) dollars spent on the Iraq war. 2) One in ten million chance of winning the (UK) lottery. To get light-headed, what if the war spend had been aimed at the lottery? Fifty thousand winners, you say, currency conversion being on your mind. Let's say you'd only want one winning ticket a week, if you could help it - that's a winning lottery ticket every week all the way from the battle of Hastings to sometime around my unborn children's late middle age.

Hmm. Let's get back on theme. For the testers among you, another question: How are we sizing our beta tests? At what point might a beta test outweigh a local test team? At what point might a beta test be reasonable expected to have found problems that could surface in the first month of general use?

Justified finger in the air estimates preferred to unsubstantiated formulae.

Fixing 7digital's playback problems in iTunes

2008-04-30T10:00:00.001+01:00

Summary: you'll need to rename .mp4 to .m4a

I just bought The Age of The Understatement by The Last Shadow Puppets, which is rather lovely. As an experiment, I bought it from 7digital - it cost a fiver rather than £7.99 on iTunes.

The beggar doesn't play properly on my kit. Each track pauses for 5 seconds or so, just under 10 seconds into the track - which is hardly conducive to great listening pleasure. As far as maintaining the atmosphere goes, it's a bit like turning the house lights on in a nightclub for a few seconds, just as every track gets going.

This is not normal, so for the configurators among you, I'll describe normal. Music comes from the iTunes store or CDs. I play the songs using iTunes under Leopard on my non-intel Mac Mini. As a tester, I've got CPU monitoring on, always, and as a tester I notice that the machine is pushing as hard as it can during the gap, but not either side. This is also not normal.

Let me also describe the differences - comparing specifically with iTunes Plus, as both deliver AAC audio without DRM protection.

The most obvious difference is that 7digital sent me the files encoded at 320kbps, compared with iTunes' 256kbps. The problem could be to do with the way iTunes codec responds to higher than expected encoding rates. However, I've successfully worked with AAC of my own projects encoded at this rate.

One consequence of this difference is that the files are larger, so perhaps this is at the root of my problem. However, I'm instinctively less than keen on the this as a target for investigation. Digging into that instinct, I can rationalise that 1) the kit is perfectly capable of shifting the quantity of data, 2) the problem is limited to part of a song, 3) the position and duration of the dropout seems unrelated to song length. It doesn't feel like something to do with amount of data.

Another difference: while looking at iTunes' information for the song, I find that it's listed as an MPEG-4 video file. This seems inappropriate - I'd expect it to be an AAC audio file. This is, after all, how it was described at download. I'm aware that this is linked to the extension, and checking the files, find the extension is .mp4. Pretty much everything else is .m4a, including those iTunes Plus files. Other iTunes files are .m4p. Both these extensions indicate Apple's AAC format, one unprotected, the other rights-managed. Those in the know can chastise me for (at least) two reasons. Firstly, MPEG-4 is a container, not a format. Secondly, iTunes should treat .mp4, .m4a and .m4p files the same - if it doesn't, it's iTunes that has the bug, not 7digital.

In the spirit of investigation and experimentation, I change the extension from .mp4 to .m4a, and the problem is gone - no hot-running CPU, the right details in the information. Most importantly, the music now plays without interruption.

Now, this is a hack. It may have corrected a simple mis-configuration. However, sound files are complex containers, and their interactions with various players are not always predictable. Will iTunes be able to burn these tracks onto a one-off mix CD? Will they play nicely with the iPod? If I chuck them into Ableton's Live to mix and mash, will there be a problem? Heaven knows, until I try. Perhaps I should have stuck with their MP3 versions . . .

The immediate problem is solved. I've not resolved the bug, but I've developed an understanding of it which has led to a fix for my machine and my purposes.

Now I'm bug-hunting, here are a few more I noticed along the way. These may help you fix/get round/be prepared for stuff on your own machine.

The transition between Black Plant and I Don't Love You Anymore is stuffed - there's a hole where there should be a transition. This isn't iTunes' infamous truncation bug, it's because it sticks a pause between tracks unless told not to. It's easily fixed - setting the album to "gapless" does the trick.
7digital's site, for reasons best known to itself, gives the tracks un-numbered, and in reverse order - it was only when I played the album that I noticed that due to my own finger trouble, I had two Calm like Us and no Standing Next to Me. That said, re-downloading is trivially easy and (with the server in the UK) pleasantly speedy.
I have to trigger each download individually (dull) if I don't want to install their downloader (I don't, thankyou - and I can't anyway as it's PC only), and when done have to put the tracks into my library (whine whine whine).

Customer impressions of 7digital: they are good value, but slightly troublesome. The trouble may not be their fault, but it means I have to work to use their music on my kit. Not their fault, but I may not buy again.

Customer impressions of The Age of the Understatement: a fine album - good songs, great words. Retro and fun without too much pastiche. Epic, if you're in the mood; insert iPod and become the hero of your supermarket trolley. Better fixed than buggy - I like it enough to mean that I'd buy a working copy if I'd not managed to fix this one, but not without a niggling sense of being conned.

Update: shortly after posting this entry, I was contacted by the CTO of 7digital, who is clearly on the ball. The .mp4/.m4a thing is now with his technical people. Good to help - better to be useful.

Comedy bug

2008-04-14T12:04:00.002+01:00

So: I'm having a problem or two that's likely to be related to a particular software package. I can't say I'm surprised - the software's been buggy and crash-prone from the off, has a UI designed to appeal only to a designer, and the online support is anything but.

I choose to uninstall the software with an application the supplier supplies. Halfway through the install, it tells me that I should have quit out of mail, as it's uninstalling a component with which it vandalised my mail application. It quits its own uninstall and gives me a nice, although temporary log.

I quit mail, and (with a hint of a tester premonition) head off to find the uninstaller, so I can uninstall the final component of this rotten puzzle.

The uninstaller has, naturally, uninstalled itself.

Marvellous.

(for those of you unfamiliar with why this is a problem, it's broadly analogous to locking the keys inside the car)

Running LEWT

2008-04-09T22:55:00.002+01:00

Rikard Edgren's WhereTesting Creativity Grows pointed me to a resource that should be read by anyone trying to create an environment that supports creativity. Nils-Eric Sahlin's article creative environments:
a simple recipe (translated by Linda Schenck) neatly encapsulates and codifies the principles that I try to use for LEWT - and, indeed, extends them.

To borrow Rikard's summary, they include;

generosity
a sense of community
qualifications
cultural diversity
trust and tolerance
equality
curiosity
freedom of spirit
small scale

It's interesting that the facile quality 'communication' is not included. It's also interesting that qualifications (which don't particularly play their part in LEWT) is more subtle that I had perhaps expected. I'll be considering making some minor changes so that people are more confident in their qualification to be part of the group - although I'll be careful about affecting its diversity. My feeling that LAWST's guru-led approach is problematic is neatly caught by the note on equality.

I'll be posting a link to the article to the LEWT discussion group, and hope to start a conversation. Let me know if you'd like to be part of it.

Creativity in Software Testing

2008-04-09T22:41:00.003+01:00

Software testing is frequently - and fundamentally - a creative endeavour. That creativity is closer to that employed for mathematics and music than for statistical analysis or singing.

Although I've talked about this until I've bored all around me, I've never managed to write anything coherent on the subject. So I'm pleased that I came across Rikard Edgren's Where Testing Creativity Grows, as it is one of the very few papers I'm aware of that looks at this important, yet unaccountably neglected perspective.

You can find it here: http://qualtechconferences.arobis.com/upload/documents/REdgren_where_testing_creativity_grows.pdf

I encourage you to have a read. If you know of other papers on the subject, post them here.

James

Question: What testing tool will help me test an Intranet application with lots of complex calculations?

2007-05-25T18:52:00.000+01:00

I'm working for [omitted] software It is a very small comapny. We had developed an Intranet application which consists lot of complex calculation parts. We are looking for a Tool which helps in Testing most complex parts of our application i.e, It should have features of script developing/modifying for regression testing and it should also support Database Testing.

One more thing I would like to inform you that as ours is a very small company we are looking for a tool which comes under $500.00. If any OpenSource tools availble than it is well and good.

Your budget will exclude the larger commercial tools - particularly with ongoing licensing costs. I note that your company provides IT services, and you should consider whether any tool - and the skills acquired to support it - would provide ROI across the company, rather than on a single project.

An organisation's greatest investment in testing with any tool - licensed or open-source - is likely to be the cost of understanding the tool and developing+maintaining the tests. Open source is attractive not only because of its lower initial cost, but because the tools can be modified to suit your purposes. I've detailed some open-source tools below.

Although your email is more detailed than many, it doesn't address the most important question - the purpose of your testing. If you would like to discuss this, call me using skype.

For functional testing of your underlying complex calculations, your best bet might be to write a simple test harness to supply input and check output, avoiding the user interface. This approach allows you to automate testing using powerful design techniques that can go far beyond simple regression scripts, and would potentially give you rather more confidence in the face of change. Look into ideas around combinatorial testing, fuzz testing, and model-based testing.

For testing a web interface, and some functional testing of the software accessed through a browser, I would suggest watir. If by database testing, you mean testing for the purpose of assessing performance or scalability of a database, I would suggest looking at jmeter or grinder.

Watir : http://wtr.rubyforge.org/
JMeter: http://jakarta.apache.org/jmeter/
Grinder: http://grinder.sourceforge.net/

Question: How do I get into testing?

2007-05-25T18:43:00.000+01:00

I completed my graduation in Bachelors of science in 2005.And I have good solving skills in C- language. and I would like to interest to software testing because of that I completed testing tools course also. Actually I am a chemistry graduate. Give any suggestion for entering to testing field.please give reply to this mail.

Are you actively working in IT at the moment?

i am ready to working in software testing and development organisations

Sadly, I didn't reply for a while, until politely nudged a week or so later:

Please give a good suggestion to me

"ready to work", I assume, means that you're not currently working in the industry. The issue then is how to identify organisations that need testing skills, and how to persuade them that you have the appropriate skills and experience.

It's worth considering that organisations that make software have different requirements from those that buy software. However, technical and business skills are necessary in both, and without direct working experience in testing, you will need to persuade such organisations to consider you for a position on the basis of your technical and business knowledge.

You should make a start by identifying those organisations within your area of business that buy software - and the companies that supply those systems. Once you've got a good sense of the software and systems that make up that area, you can concentrate on those which match your technological skillset.

Organisations that make software need new testers toward the end of a project, or at the point where they are customising software for a new client. They may draw their test team from the technical resources, and as a new hire, you would be expected to be able to know enough about the technical stuff to talk to the existing team, and potentially to know more about the business than many of the people you would be joining. You would typically be doing Functional Testing.

Organisations that buy software typically test the systems that they buy only if the system has been customised (or built) for them - this most often happens in large organisations buying 'enterprise' software. These organisations may draw their test teams from current business users and add testing skills. You'd be expected to know enough about the business stuff to talk to the existing team, and potentially to know more about testing and IT than many of the people you would be joining. You would typically be doing User Acceptance Testing.

Qualifications can help - in some [organisational] cultures, you would not be considered for a testing role without a testing qualification, an appropriate business qualification, security classification, or a background in the relevant technology. If you don't have any of these, concentrate on businesses that won't require one.

Put together a CV that details you're relevant experience and shows some understanding of the area you're interested in, and send them off. You can also, if lucky and charming, make great headway networking with people in the business and technical areas.

Question: Does the test strategy vary?

2007-05-25T18:36:00.000+01:00

My question is that does the test strategy vary for both software testing applications and mobile application testing.
Or is it the same followed for all.

And also the test plan and Test matrices are same for mobile application testing or not?

Every good software test strategy I've seen has been influenced by its context. I would expect the strategies to be different - and to reflect the difference between the two contexts you give.

The plan - the list of what happens when - follows from the strategy, inasmuch as the typical purpose of a strategy is to give structure to decisions about what to plan and how to plan.

I've no clear idea what you mean by a test matrix. If I can second-guess for a moment though, I would assume that a test matrix is some combination of individual tests. Different systems may well share a subset of tests. The difference would be in the importance of those tests, and how the information they produce is understood in the context of the project as a whole.

Question: Can you give me a Test Strategy Template?

2007-05-25T18:30:00.000+01:00

I get this a lot, and thought I'd deal with these as a group. For the record: a strategy helps you plan. Good ones tend to arise either from a closely involved team, or after plenty of deep questions from some sort of expert. There are no genuine templates, only examples. Strategies - testing or otherwise - are as different and unique as the projects they serve. The more all-encompassing the strategy, the shorter it should be; no-one needs a 50-page strategy. There is always a strategy, and you can find it by following the decisions - the written strategy may not be the same as the actual strategy.

Here are some examples of the genre . . .

Can you please send me few examples and templetes for Test Strategy....

Thanks for getting in touch. Could you tell me a little more about your project and your goals?

About my project... The project is Retail management(Web-Based Application) and its new project... the development area is J2EE. Now we are in Functional Designing phase... hope after finishing the functional design spec. we are planning to come up with Test Strategy and Test Approach and Plans...

My Ultimate goal is to deliver a Quality product to my Client...

So, can you please send me few sample Test Strategy Templates for it... hope you will do needful to me...

Thanks for this information. It sounds as though you are at a good point in the project to be developing a test strategy. You may already have a workable idea of the technical constraints and potential patterns of failure that could drive aspects of your test design. However, what will be more important to your test strategy are the values that inform your test decisions, and you've not included information about what they might be.

Technical environment and business context are important to a strategy, and it might help to see a strategy for another project in the same environment. However, there are no publicly-available examples I know of that cover this particular pairing.

I've worked on products that have a fair fit with your description. The test strategies for the projects were very different, reflecting the values and priorities of the organisation. For commercial and copyright reasons, I can't supply copies of those strategies to you.

I feel strongly that a test strategy is something that should *not* be created from a template, and I don't supply test strategy templates - either for free, or during a paid interaction. I try to make my position clear here: http://www.workroom-productions.com/strategy.html . The page does include a checklist, which may serve as a stimulus for discussion within your team.

I'm sorry not to be of more help. If you would like to speak with me on this, please call me.

I am working as a Test Manager for one of the big program has teken up my company. Program contains number of projects and applications on different technologies like Java, ATG, Web Services, Data Stage, Mainframe COBOL etc... and it has nearly 250 inbound and outbound systems. Can you please provide inputs for test strategy for these kind of programs? and it will be grate if you provide any sample or template for test strategy.

Thanks for your enquiry. Your project sounds substantial, and I don't think it can be dealt with in an simple email exchange. If you would like to speak with me on this, please call me during London business hours.

Please note: I don't supply test strategy templates - either for free, or during a paid interaction. I try to make my position clear here: http://www.workroom-productions.com/strategy.html . The page does include a checklist, which may serve as a stimulus for discussion within your team.

I'm having a self study training on test strategies and going through your website was wondering if you had Knowledge of examples or references for Test strategies of payroll based systems of which I am conversant with a few. Any help will be appreciated

I can't give you any examples that are specifically tuned to payroll systems. The purpose of a system is important to a strategy, but so are the values of the organisation, the technology of that system, the power and constraints of the potential resources.

If you're working on a self-study course, you could do worse than study the test strategies used on the payroll systems you are familiar with. You are likely to find the actual strategy is very different from the stated strategy - if one even exists. You could also write two or more *different* strategies for the *same* product and context, to expose potential alternatives and underlying rules. Try to keep such strategies short; less than 3 pages or so. I have a rule-of-thumb - the higher-level the strategy, the shorter it should be.

I am a Senior software tester and working for a consultancy company called [obscured supplier]. I am trying to prepare a Test Strategy document for the client [obscured client].Is this any free consulation given by the Work room Productions based on test strategy document .Please let me know this.

Thankyou for your enquiry. Please let me know your ideas so far - and what you might require from consultancy. I notice that [obscured client] are a US-based healthcare company.

A use for my blog

2007-05-25T17:43:00.000+01:00

I can't say I'm a big fan of blogs - their immediacy lends itself to half-baked ideas, and facilitates breathless ranting. Yet they're anything but ephemeral, or private. As an occasional half-baked ranter myself, I much prefer the cut-and-thrust of a conversation.

However, my email address is in the public domain, and I wonder if I can't make use of a blog to communicate some of the questions I receive about software testing, and some of the answers I give. This would be in the nature of an FAQ, although perhaps the F in that acronym is misleading. I get one or two a month, and sometimes wonder what motivates a stranger to ask me for advice.

The questions that are sent to me are typically short, and seem to need a long answer. It's all too easy to put far more information in the answer than is got in the question - a sign that one may be building one's half-baked rant on sand. I try not to let myself get carried away.

Usually, one would need more details to have a decent stab at a good answer. I'll try to engage the questioner; handing out my skype details and looking forward to clarifying emails. It's rare treat when someone actually responds.

My answers are offered for free, with all the attention to quality and detail that implies - but I also see them as a puzzle, and try to write something cogent. I offer them here in a more public forum, to share ideas and invite comments.

I've not got explicit permission to use people's questions, and so I have obscured, cut and amended their words and my answers to suit. I have a small backlog of such questions. I'll post the first shortly. You're welcome to add comments of your own.

Open source test tools

2006-04-12T19:39:00.000+01:00

Here's a website I should have known about. I thought you might want to know, too: http://www.opensourcetesting.com/
Playing with JMeter as I write this.

diagnosis and exploration

2006-04-06T18:20:00.000+01:00

If your coders accept all the problems your testers find - including the intermittent, non-reproducible ones - then they need diagnostic skills.

If not, your testers need diagnostic skills to make the bug more easily reproducible

Who's teaching / writing about diagnostic skills? How are they characteristically different from analysis skills?

Just a thought...

2006-04-06T18:10:00.000+01:00

I've got plenty of half-baked ideas, and I spend hours each week trying to write. It's time I joined the blogosphere, and made something useful of all this gubbins.