Thursday 21 June 2012

Is it dangerous to measure ROI for test automation?


I have been a fan of trying to show ROI for automation in a way that is simple enough to understand easily, and provides a way of showing the benefit of automation compared to its cost.

I have been developing a spreadsheet with sample calculations (sparked off initially by Molly Mahai, including an example from Mohacsi & Beer's chapter in the new book, and some other people have also been influential - thanks). I have sent this spreadsheet out to around 300 people, including most who have attended my automation tutorials.

The problem with showing ROI is that it's hard to quantify some of the important factors, so I have focused on showing ROI using what is the most straight-forward to quantify - people's effort and/or time. This can be converted into money, using some kind of salary cost, if desired, and either effort or money can then be plugged into the ROI calculation = (benefit - cost) / cost.

So basically, this is showing how a set of tests requires less human effort when those tests are automated than would be required if those same tests were run manually.

This is great, right? We have a clear business case showing savings from automation than are greater than the cost of developing the automation, so our managers should be happy.

Recently, however, I have been wondering whether this approach can be dangerous.

If we justify automation ONLY in terms of reduced human effort, we run the risk of implying that the tools can replace the people, and this is definitely not true! Automation supports testing, it does not replace testers. Automation should free the testers to be able to do better testing, designing better tests, having time to do exploratory testing, etc.

So should we abandon ROI for automation? I don’t think that’s a good idea – we should be gaining business benefit from automation, and we should be able to show this.

Scott Barber told me about Social ROI – a way of quantifying some of the intangible benefits – I like this but haven’t yet seen how to incorporate it into my spreadsheet.

In our book, there are many success stories of automation where ROI was not specifically calculated, so maybe ROI isn’t as critical as it may have seemed.

I don’t know the answer here – these are just my current thoughts!