Is it dangerous to measure ROI for test automation?

I have been a fan of trying to show ROI for automation in a way that is simple enough to understand easily, and provides a way of showing the benefit of automation compared to its cost.

I have been developing a spreadsheet with sample calculations (sparked off initially by Molly Mahai, including an example from Mohacsi & Beer's chapter in the new book, and some other people have also been influential - thanks). I have sent this spreadsheet out to around 300 people, including most who have attended my automation tutorials.

The problem with showing ROI is that it's hard to quantify some of the important factors, so I have focused on showing ROI using what is the most straight-forward to quantify - people's effort and/or time. This can be converted into money, using some kind of salary cost, if desired, and either effort or money can then be plugged into the ROI calculation = (benefit - cost) / cost.

So basically, this is showing how a set of tests requires less human effort when those tests are automated than would be required if those same tests were run manually.

This is great, right? We have a clear business case showing savings from automation than are greater than the cost of developing the automation, so our managers should be happy.

Recently, however, I have been wondering whether this approach can be dangerous.

If we justify automation ONLY in terms of reduced human effort, we run the risk of implying that the tools can replace the people, and this is definitely not true! Automation supports testing, it does not replace testers. Automation should free the testers to be able to do better testing, designing better tests, having time to do exploratory testing, etc.

So should we abandon ROI for automation? I don’t think that’s a good idea – we should be gaining business benefit from automation, and we should be able to show this.

Scott Barber told me about Social ROI – a way of quantifying some of the intangible benefits – I like this but haven’t yet seen how to incorporate it into my spreadsheet.

In our book, there are many success stories of automation where ROI was not specifically calculated, so maybe ROI isn’t as critical as it may have seemed.

I don’t know the answer here – these are just my current thoughts!

How long does it take to write a book?

A number of people have asked me this, since our new book is now out.

This book took us 2 and a half years. This doesn’t include the effort put in by the case study authors and other contributors, so this book represents a lot of work! What exactly had we been doing all that time? I wondered that too, so here is where the time went.

In August 2009, I have my first note of our plan to solicit contributions for a new book on automation experience. We sent emails, put a call for contributions on my web site, and talked to people at conferences, and began gathering potential contributions.

I started keeping track of the hours we spent from December 2009. We had a big initial “push” (the first peak on the graph) and produced a “protobook” – 4 chapters with an introduction. We were sure this would be snapped up by a publisher!

We submitted to the publisher of our previous book in mid-February, but initially they weren’t very keen! This was a blow, as we were convinced this would be a great book! I tried several other publishers over the next few months, and got rejected; I continued to try and convince Pearson/Addison Wesley that they should publish our book.

They eventually relented in July and we signed a contract. We worked steadily on the book over the rest of that year, a total of around 400 hours between us. The complete draft manuscript, ready for independent review was due on the 15^th April, and the final manuscript on the 15^th October. “No problem”, we thought.

However, we found that we did need to work at a more intense level - we spent another 300 hours to the end of April (the double peak with some time out in February), and another 300 hours to the end of 2011. The final peak was editing the final page proofs and doing the index, more work that we had anticipated at that stage. The total for us comes to just under 1000 hours. We don't know how much time the contributors spent, but it their time was equivalent to ours, the collaboration represents around 2000 hours of time - that's more than one working year.

We enjoyed working on this book and reading all the stories as they came in. There are many useful lessons, some heartfelt pain, and many gratifying successes among the stories.

You can follow the book on Twitter on @AutExpBook. The book tweets tips and good points every few days!

Thanks to all the book contributors, and to co-author Mark Fewster.