Fun for explorers

If you like physics, or playing with stuff, you'll like this: FantasticContraption

A perfect example of exploring solutions, alternatives and refinements. For those of you who have been on my exploratory testing course, I couldn't teach ET with it, but I urge any readers interested in exploration to watch themselves - or someone else - working towards solutions, general applications, principles, components. I've just lost a chunk of time myself in a gentle whirl of levers and engines. Reminds me of Meccano, but it's fast, flexible, and never runs out of bits (or seizes up with friction). Must get AquaForest back on to the iPod Touch.

Not every revolution is part of an evolution.

Evolution is always revolutionary to those caught up in it. Only hindsight makes a revolution part of an ongoing evolution.

If you're not part of the revolution, you won't evolve. If you don't evolve, you're stuck in a dead end of ever-decreasing resources.

Of course, this is to confuse the individual and the tribe.

An alternate way of putting this - no individual survives (Darwinian) evolution. Thank goodness culture is Lamarckian.

Exploration and experimentation

I thought I'd share this, from The Economist:

In the 19th century it was commonplace to do an experiment simply to see what would happen. That was, in part, because experimenters were often amateurs who were spending private money. In these days of taxpayer-financed science, most experiments are executed with a pretty clear idea of what the outcome ought to be, especially when they are part of wars and campaigns against this or that. The paradox is that, although such efforts do not eliminate Becquerel-like discoveries, they risk limiting the chances of making them.

Tools for Exploratory Testing

I am often asked what tools I use for Exploratory Testing – most recently, by Patrick O'Beirne while doing a series of talks at SoftTest.

It rather depends on what I need to explore, but my own explorer's toolkit (with a probable unix bias) includes:

- for recording visual stuff: I most often use a camera, which allows screensnaps and brief audio+video without being platform-dependent. For screenshots, on a given platform, I use SnapzPro on Macintosh/OS X, and HyperSnap on Windows. I've been know to rig up a video camera to watch keystrokes and screen. I also use Mouseposé, which makes clicks obvious and shows keystrokes (including modifiers) in a big translucent bezel. 

- an aside: I also use Mouseposé when I'm showing people what I'm doing, and even to help me get visual feedback on the buttons I'm pressing. Butterfingers = happy accidents.

- for tools that specifically help with recording manual exploratory testing (ie records keystrokes and timings, allows annotation of screen movies etc.), I've used SiriusSQA's TestExplorer and BlueBerry's BBTestAssistant. For my purposes, they're often a bit heavy and windows-centric, but you may find one or the other is just what you need. You can try them yourselves, as both have a great attitude to limited-use trials. Spector - the outrageously intrusive spy-software - was often put forward as an alternative before tester-oriented tools became available. It seems un-disruptive, and offers comprehensive monitoring, but I found it hard to use for testing. You may find the terms of the license prevent you from using it well in a test environment, and there are difficulties in saving the information you've captured. Finally, it's so very much oriented to sneaking on spouses and employees that having the license makes me feel slightly creepy myself.

- for recording what I do, I was pretty-much paper based until a couple of years ago. I now use a dual system, keeping most of my scribbles and diagrams on paper as an extension of my thought processes, but using OmniOutliner to keep track of bugs found, to put timestamps on observations, arrange stuff hierarchically / foldably, and to allow me to search my notes.

- for monitoring system activity, I use "top" on UNIX-based systems. I've recently discovered the joy of Process Explorer and perfmon (when I got the logging finally turned on) on windows. On the Mac, Activity Monitor is reasonable, but Instruments (a wrapper for DTrace) is for serious work. Crash monitors for specific applications/OSes are good, too, as is a knowledge of where the logs are hidden (and how to extract them).

- for looking at differences, I use unix/windows tools comm and diff. Unix tools can be introduced to windows environments with Cygwin, UnixtUtils or many others. You could also try kdiff, which is pretty comprehensive and runs on everything, or FileMerge (in xCode) on the Mac. It's good to know one's way around regular expressions, so here's a link to ilovejackdaniels' great Regular Expressions cheat sheet. If you need to do specific Windows Registry checks, Process Monitor and TestExplorer will help you out.

- for influencing system activity, it's got to be Holodeck - the freebie on the back of "how to break software security" is less buggy than the version on "how to break software", and (being free) is $$$ less than the retail version. Roll on the day when developers expect this sort of capability to be built into OSs.

- for input data, I often have a library of useful data about the place - large files, empty files, different sizes of different formats of picture, files that have meaning to something I'm testing (ie an Excel file for testing within MS Word). I sometimes use Bach's perlclip for creating data. I use Excel for creating test data to fit requirements, and load it into SQL tables with CSV

- for output data, I primarily use Excel for most parsing/analysis work - with a little bit of fiddling to get in amongst datasets that break Excel's 65K row limit. I very much prefer DataGraph for exploring datasets graphically. It's Mac-only, but IMHO worth buying  a mac for.

- for automating , I use whatever's lying about at the client's site. Last time I bought a tool, it was Vermont High Test, so my licence is a bit out of date these days.

- for web testing, I push hard for Firefox to be within the browser gamut, and use FireBug. iMacros was in my toolkit for a while, and I wasn't over-familiar with it, but I've recently tried it out for loading web forms, and it's fine. The firefox version is still free (as in beer).

- for cross-browser testing, I use the free web service netrenderer if I've not got my kit with me. It opens a publicly-accessible web page in the browser of your choice, takes a screensnap, and shows it to you.

- for putting load into an app and measuring output (although not graphing it, because Excel's far better, and DataGraph better still), I use (cross-platform) JMeter. You might want to consider The Grinder.

- for emulators (dead handy), I've had great results with VirtualPC on PCs. The field is moving fast, and these days I'd look specifically at VMWare and Parallels. I've used Parallels on the Mac lots, but not for testing (yet).

I'm sure you use different tools. When I teach Exploratory Testing, the tools workshop is often most eye-opening in closed teams, where one person is often sitting on a tool whose use is immediately apparent to other participants - yet they've never shared their toolsets until the class. Try it yourself – I'd be fascinated to know what tools you use.

[Edited to include OmniOutliner, Mouseposé and to clarify the TextExplorer / BBTA entry]

[Edited to include netrenderer]

Heathrow T5 and software testers

CIO magazine pulls no punches when they say that inadequate software testing (was) behind T5 problems. I won't make easy assumptions based on that article - you can read it and make your own.

However, if you've been following the story, you'll be interested to know that Huub van der Wouden and Roger Derksen from Transfer Solutions in the Netherlands will be giving the opening keynote on Testing the Heathrow Terminal 5 Baggage Handling System at the London SIGiST on June 18th.

Searchable library - content, not title

So I was going to buy Gerard Meszaros' book xUnit Test Patterns at StarEast, but the excellent selection from Breakpoint Books had sold out by midway through day 2.

I find that I already rent the online, searchable edition. If it wasn't such a pig to use, I'd love Safari books online.

I'd like to have an online search for my existing luggable library, to let me find what I need in books where I don't have an index, or where the index is rubbish. Google Books gets close, but only gives one return per book. If you know how to get more than one, do let me know.

How much TV does a Wikipedia cost?

Clay Shirky has written a book called "Here Comes Everybody: The Power of Organizing Without Organizations". Here's a couple of localised Amazon links - amazon.co.uk, amazon.com. Now, I'm not about to recommend the book without reading it. I am about to buy it so I can read it on my way to teach at STAREast.

However, the interconnected world allows us all to read / see what the book's about. Here's a fine blog post of his summarising talk. For those more visually/aurally oriented, here's a the talk itself.

Answer to the question above: Clay says that each year, the people in the USA spend 2,000 times more time than has been needed, so far, to construct Wikipedia. Except that he says it better. Read the article.

Let's work these numbers a tad to give us a feel for their reasonableness. If there are 500,000,000 people in the US, then the work done on Wikipedia so far is equivalent to the TV-watching habits of 250,000 people. That's round about the population of Boise (Idaho) or Daytona Beach. For those of us closer to GMT, think Belfast. If the average citizen spends 10% of their waking time watching TV (The NYT says it's over 4 hours, which is closer to 30%, but let's make life simple and not cut out TV entirely - I assume that many have the TV on while eating, talking, making macramé wallhangings), then it would take the waking time of 25,000 people for around a year. English-only wikipedia has 2.5 million articles, so that's around 3 articles a day per person - I'd have said 1 or less, but we would seem to be around the right kind of numbers.

Of course, this may be a circular analysis if Clay started with 2.5 million articles at around 4 hours a piece to write.

A question for you: Do you know anyone who actually understands the numbers, ie can apply them in their everyday lives? Who understands fundamentally how large numbers of people and small commitments/risks/expenditures actually add up? It doesn't exactly come up in conversation much, but I don't think I know anyone who has a clear feel for this. Perhaps it's my generation. Then again, I know plenty of people who have a fine handle on atomic measurements and cosmological time - perhaps I should get to know some social scientists. 

I'm aware that, as humans, we're bad at things outside our direct experience - and have to either put things into scales we can understand, or manipulate the numbers directly. My problem is, perhaps, that we're bad at viscerally understanding how large (small)  a thousand (th), a million (th), or a billion (th) actually is - and so when we combine a tiny with a huge we're bad at understanding what that means. In our actual lives, being out by a factor of two is plenty - but when dealing with things beyond our ken, it's much harder to spot. Both the following are out, by a bit - but by how much? 1) A trillion (US) dollars spent on the Iraq war. 2) One in ten million chance of winning the (UK) lottery. To get light-headed, what if the war spend had been aimed at the lottery? Fifty thousand winners, you say, currency conversion being on your mind. Let's say you'd only want one winning ticket a week, if you could help it - that's a winning lottery ticket every week all the way from the battle of Hastings to sometime around my unborn children's late middle age.

Hmm. Let's get back on theme. For the testers among you, another question: How are we sizing our beta tests? At what point might a beta test outweigh a local test team? At what point might a beta test be reasonable expected to have found problems that could surface in the first month of general use? 

Justified finger in the air estimates preferred to unsubstantiated formulae.

Fixing 7digital's playback problems in iTunes

Summary: you'll need to rename .mp4 to .m4a

I just bought The Age of The Understatement by The Last Shadow Puppets, which is rather lovely. As an experiment, I bought it from 7digital - it cost a fiver rather than £7.99 on iTunes.

The beggar doesn't play properly on my kit. Each track pauses for 5 seconds or so, just under 10 seconds into the track - which is hardly conducive to great listening pleasure. As far as maintaining the atmosphere goes, it's a bit like turning the house lights on in a nightclub for a few seconds, just as every track gets going.

This is not normal, so for the configurators among you, I'll describe normal. Music comes from the iTunes store or CDs. I play the songs using iTunes under Leopard on my non-intel Mac Mini. As a tester, I've got CPU monitoring on, always, and as a tester I notice that the machine is pushing as hard as it can during the gap, but not either side. This is also not normal.

Let me also describe the differences - comparing specifically with iTunes Plus, as both deliver AAC audio without DRM protection.

The most obvious difference is that 7digital sent me the files encoded at 320kbps, compared with iTunes' 256kbps. The problem could be to do with the way iTunes codec responds to higher than expected encoding rates. However, I've successfully worked with AAC of my own projects encoded at this rate.

One consequence of this difference is that the files are larger, so perhaps this is at the root of my problem. However, I'm instinctively less than keen on the this as a target for investigation. Digging into that instinct, I can rationalise that 1) the kit is perfectly capable of shifting the quantity of data, 2) the problem is limited to part of a song, 3) the position and duration of the dropout seems unrelated to song length. It doesn't feel like something to do with amount of data.

Another difference: while looking at iTunes' information for the song, I find that it's listed as an MPEG-4 video file. This seems inappropriate - I'd expect it to be an AAC audio file. This is, after all, how it was described at download. I'm aware that this is linked to the extension, and checking the files, find the extension is .mp4. Pretty much everything else is .m4a, including those iTunes Plus files. Other iTunes files are .m4p. Both these extensions indicate Apple's AAC format, one unprotected, the other rights-managed. Those in the know can chastise me for (at least) two reasons. Firstly, MPEG-4 is a container, not a format. Secondly, iTunes should treat .mp4, .m4a and .m4p files the same - if it doesn't, it's iTunes that has the bug, not 7digital.

In the spirit of investigation and experimentation, I change the extension from .mp4 to .m4a, and the problem is gone - no hot-running CPU, the right details in the information. Most importantly, the music now plays without interruption.

Now, this is a hack. It may have corrected a simple mis-configuration. However, sound files are complex containers, and their interactions with various players are not always predictable. Will iTunes be able to burn these tracks onto a one-off mix CD? Will they play nicely with the iPod? If I chuck them into Ableton's Live to mix and mash, will there be a problem? Heaven knows, until I try. Perhaps I should have stuck with their MP3 versions . . .

The immediate problem is solved. I've not resolved the bug, but I've developed an understanding of it which has led to a fix for my machine and my purposes.

Now I'm bug-hunting, here are a few more I noticed along the way. These may help you fix/get round/be prepared for stuff on your own machine.

• The transition between Black Plant and I Don't Love You Anymore is stuffed - there's a hole where there should be a transition. This isn't iTunes' infamous truncation bug, it's because it sticks a pause between tracks unless told not to. It's easily fixed - setting the album to "gapless" does the trick.
• 7digital's site, for reasons best known to itself, gives the tracks un-numbered, and in reverse order - it was only when I played the album that I noticed that due to my own finger trouble, I had two Calm like Us and no Standing Next to Me. That said, re-downloading is trivially easy and (with the server in the UK) pleasantly speedy.
• I have to trigger each download individually (dull) if I don't want to install their downloader (I don't, thankyou - and I can't anyway as it's PC only), and when done have to put the tracks into my library (whine whine whine).

Customer impressions of 7digital: they are good value, but slightly troublesome. The trouble may not be their fault, but it means I have to work to use their music on my kit. Not their fault, but I may not buy again.

Customer impressions of The Age of the Understatement: a fine album - good songs, great words. Retro and fun without too much pastiche. Epic, if you're in the mood; insert iPod and become the hero of your supermarket trolley. Better fixed than buggy - I like it enough to mean that I'd buy a working copy if I'd not managed to fix this one, but not without a niggling sense of being conned.

Update: shortly after posting this entry, I was contacted by the CTO of 7digital, who is clearly on the ball. The .mp4/.m4a thing is now with his technical people. Good to help - better to be useful.

Comedy bug

So: I'm having a problem or two that's likely to be related to a particular software package. I can't say I'm surprised - the software's been buggy and crash-prone from the off, has a UI designed to appeal only to a designer, and the online support is anything but.

I choose to uninstall the software with an application the supplier supplies. Halfway through the install, it tells me that I should have quit out of mail, as it's uninstalling a component with which it vandalised my mail application. It quits its own uninstall and gives me a nice, although temporary log.

I quit mail, and (with a hint of a tester premonition) head off to find the uninstaller, so I can uninstall the final component of this rotten puzzle.

The uninstaller has, naturally, uninstalled itself.

Marvellous.

(for those of you unfamiliar with why this is a problem, it's broadly analogous to locking the keys inside the car)

Running LEWT

Rikard Edgren's WhereTesting Creativity Grows pointed me to a resource that should be read by anyone trying to create an environment that supports creativity. Nils-Eric Sahlin's article creative environments:
a simple recipe (translated by Linda Schenck) neatly encapsulates and codifies the principles that I try to use for LEWT - and, indeed, extends them.

To borrow Rikard's summary, they include;

• generosity
• a sense of community
• qualifications
• cultural diversity
• trust and tolerance
• equality
• curiosity
• freedom of spirit
• small scale

It's interesting that the facile quality 'communication' is not included. It's also interesting that qualifications (which don't particularly play their part in LEWT) is more subtle that I had perhaps expected. I'll be considering making some minor changes so that people are more confident in their qualification to be part of the group - although I'll be careful about affecting its diversity. My feeling that LAWST's guru-led approach is problematic is neatly caught by the note on equality.

I'll be posting a link to the article to the LEWT discussion group, and hope to start a conversation. Let me know if you'd like to be part of it.

Creativity in Software Testing

Software testing is frequently - and fundamentally - a creative endeavour. That creativity is closer to that employed for mathematics and music than for statistical analysis or singing.

Although I've talked about this until I've bored all around me, I've never managed to write anything coherent on the subject. So I'm pleased that I came across Rikard Edgren's Where Testing Creativity Grows, as it is one of the very few papers I'm aware of that looks at this important, yet unaccountably neglected perspective.

You can find it here: http://qualtechconferences.arobis.com/upload/documents/REdgren_where_testing_creativity_grows.pdf

I encourage you to have a read. If you know of other papers on the subject, post them here.

James

Question: What testing tool will help me test an Intranet application with lots of complex calculations?

I'm working for [omitted] software It is a very small comapny. We had developed an Intranet application which consists lot of complex calculation parts. We are looking for a Tool which helps in Testing most complex parts of our application i.e, It should have features of script developing/modifying for regression testing and it should also support Database Testing.

One more thing I would like to inform you that as ours is a very small company we are looking for a tool which comes under $500.00. If any OpenSource tools availble than it is well and good.

Your budget will exclude the larger commercial tools - particularly with ongoing licensing costs. I note that your company provides IT services, and you should consider whether any tool - and the skills acquired to support it - would provide ROI across the company, rather than on a single project.

An organisation's greatest investment in testing with any tool - licensed or open-source - is likely to be the cost of understanding the tool and developing+maintaining the tests. Open source is attractive not only because of its lower initial cost, but because the tools can be modified to suit your purposes. I've detailed some open-source tools below.

Although your email is more detailed than many, it doesn't address the most important question - the purpose of your testing. If you would like to discuss this, call me using skype.

For functional testing of your underlying complex calculations, your best bet might be to write a simple test harness to supply input and check output, avoiding the user interface. This approach allows you to automate testing using powerful design techniques that can go far beyond simple regression scripts, and would potentially give you rather more confidence in the face of change. Look into ideas around combinatorial testing, fuzz testing, and model-based testing.

For testing a web interface, and some functional testing of the software accessed through a browser, I would suggest watir. If by database testing, you mean testing for the purpose of assessing performance or scalability of a database, I would suggest looking at jmeter or grinder.

Watir : http://wtr.rubyforge.org/
JMeter: http://jakarta.apache.org/jmeter/
Grinder: http://grinder.sourceforge.net/