Known ways of managing ET #02 - Bug Bash

tl;dr - Bug Bashes are rubbish.

The project gathers people together at an appointed time and place. Everybody splurges on testing for an allotted period, logs some bugs, and stops. If you need examples, see *. It’s a community thing, and there is generally a group hug / doughnut / retrospective before everyone goes back to their day jobs.

I guess one virtue of a bug bash is that it is a concentrated period of work, which may be a good thing in itself. Bug bashes can employ and popularise diversity in a group’s acceptable points of view, which is a plus in my book. You get to meet other people. And maybe a doughnut. But other than these few useful traits, it’s hard to find much that is good.

While the bug hunt room might be a fertile idea-generating ground for small groups in close physical proximity, the format means that many people head into the system for the first time, at the same time. Everyone is testing in parallel for a limited period, so there isn’t much opportunity to learn from each other, to analyse the group’s results, draw conclusions and carry on in a better way. There isn’t much opportunity to appreciate all the different approaches that are being tried, and take a new one – or, indeed, to help the group to explode in variety. There isn’t much opportunity to sling together a swift tool that drastically cuts the manual drudge and finger trouble in later work. And most people, working as novices, will follow a limited gamut of manual paths characterised by learning and exploring the application for the first time**. This could help you predict how your customers*** might interact with the product in the first few hours of use, but it’s not so good for assessing the product in other, more representative ways. A bug bash may be directionless, but that’s not to say that it is diverse.

A bug bash puts strain on available test-environment resources; licenses, batteries for devices, laptops, USB cables, un-damaged data, bandwidth, IP addresses, you name it. I’ve seen a support chap spend days getting the kit together, versioned, charged, addressed, data-filled and working before a push. Even assuming your Test Lab really can support your bashers with hands-on stuff, your back-end and infrastructural resources may not be neatly independent and you’ll end up being stymied by half the group finding the same test-environment bugs. This is great if you’re looking for test-environment or large-group bugs, but again, not so good if you’re looking for a broader or more representative set of interesting issues.

Expect lots of duplicate bugs, as different bug hunters bang into the same low-hanging fruit. Bug bashes often throw up some easily found but hitherto-unseen trouble, but let’s not be unthinkingly self-congratulatory when a third of our crew waste their time investigating, diagnosing and logging the same problem at the same time. More insidiously, duplicate bugs may mean broadly similar paths through the application. If everyone’s doing the same thing, what does that do for coverage? For exploration?

The quality of logged bugs tends to be low, the density high. The brief duration and inevitable peer pressure pushes the people in the room to value their speed to the first bug, and if they don’t get the early bugs, then, hey, there’s kudos to he who logs the most. Bugs are logged at speed, in bulk, with generally poor detail and diagnosis. Some managers are happy that their bug bash has resulted in a great wodge of trouble tickets, but remember that a bulge in the bug rate disturbs the workflow of an agile team like a turtle disturbs the digestion of an anaconda. Finally, the hysterical whoop whoop of competition not only breeds false confidence, but can break the spirit of people with their ego tied up in the code and configuration of the product.

A bug hunt allows a team to throw many people at a problem in a short period. It can appear cheap in elapsed time, while substantial in people time. Wrong. Ten people testing for an afternoon might look like a week’s worth of testing, but it is a week’s worth of testing by someone with 3-hour amnesia. I have seen bug hunts used to demonstrate someone in management’s commitment to testing their product. Diverting half the team off their usual path and into testing for a few hours certainly makes a statement, but the statement is that management is committed to theatrical gestures. Grand Guignol**** testing is a titillation, yet I’ve seen it used to substantiate the assertion that a product is fully tested. Frankly, my arse.

When might I use a bug bash? Perhaps if there was a problem reported frequently in beta testing, which was serious and urgent enough to warrant a diverse group’s concentrated attention but not well-reported-enough to act on directly. I might give the reports to a bug bash group, ask them to find out anything about the problem that isn’t already detailed in the reports, and facilitate their communication by sticking a scribe/steering person at a central and visible whiteboard, equipped with a bell. But I’d prefer to use a small team with big kit.

Getting a large group together for a short period is an expensive way of doing rubbish testing. I’d far rather spend the time and money getting the necessary people together and delivering a test environment that is up, running, connected, data-ready and swiftly-rebuildable. Or delivering a diverse and knowable set of data. Or a collection of reasonable (and less reasonable) user scenarios that stand a chance of saying something interesting and meaningful when tried on the product. Or a couple of hours so we know something about each other beyond name, age, height and title. Actually, pretty much anything is better value than a bug bash.

For a while***** it seemed like every other client wanted to throw most of their exploratory eggs into the bug bash basket. I have no idea who kicked off this ludicrous meme, but I’d still like to tweak their nose. Here’s my position:

Managers: of all the usual gambles you can make with your charges, a Bug Bash is one of the dumbest. Get someone to bring you an alternative, and consider it.

Testers: Bug Bashes might look like fun for you, but they suck for the product and the project. Don’t be fooled.

Clear enough?

* I’ve seen Scrum teams devote the whole team for a couple of hours on the 6-8th working day of a 10 day sprint. I’ve seen waterfall teams drag fifty people into the canteen on a Friday afternoon to batter away at a batch of handhelds. I’ve seen test teams commandeer the boardroom for a day at a time straight after they get the code, every time they get the code. Bet you've seen something similar. Enough examples: back to the polemic.
** Ж: “So what did you do?”
Ю: “Well, I tried logging in, and I’d not logged in before, and it went well, so I tried changing my username and resetting my password, using funny characters.”
Ж: <smacks head on desk>
*** Assuming your insiders are good substitutes for customers…
**** A horror show made up of a series of short pieces. Read Grand Guignol in Wikipedia, and think testing. Compare with Soap Opera Testing.
***** 2004-2008, or so