Thursday, 21 June 2012

Is it dangerous to measure ROI for test automation?


I have been a fan of trying to show ROI for automation in a way that is simple enough to understand easily, and provides a way of showing the benefit of automation compared to its cost.

I have been developing a spreadsheet with sample calculations (sparked off initially by Molly Mahai, including an example from Mohacsi & Beer's chapter in the new book, and some other people have also been influential - thanks). I have sent this spreadsheet out to around 300 people, including most who have attended my automation tutorials.

The problem with showing ROI is that it's hard to quantify some of the important factors, so I have focused on showing ROI using what is the most straight-forward to quantify - people's effort and/or time. This can be converted into money, using some kind of salary cost, if desired, and either effort or money can then be plugged into the ROI calculation = (benefit - cost) / cost.

So basically, this is showing how a set of tests requires less human effort when those tests are automated than would be required if those same tests were run manually.

This is great, right? We have a clear business case showing savings from automation than are greater than the cost of developing the automation, so our managers should be happy.

Recently, however, I have been wondering whether this approach can be dangerous.

If we justify automation ONLY in terms of reduced human effort, we run the risk of implying that the tools can replace the people, and this is definitely not true! Automation supports testing, it does not replace testers. Automation should free the testers to be able to do better testing, designing better tests, having time to do exploratory testing, etc.

So should we abandon ROI for automation? I don’t think that’s a good idea – we should be gaining business benefit from automation, and we should be able to show this.

Scott Barber told me about Social ROI – a way of quantifying some of the intangible benefits – I like this but haven’t yet seen how to incorporate it into my spreadsheet.

In our book, there are many success stories of automation where ROI was not specifically calculated, so maybe ROI isn’t as critical as it may have seemed.

I don’t know the answer here – these are just my current thoughts!

7 comments:

Jonathan Kohl said...

Interesting post Dot, I have a different view about ROI and testing, makes me uncomfortable. (Maybe it was all those accounting and finance courses I had to take in university.) This article explains how I feel about ROI in a services field better than I can:
http://www.copyblogger.com/social-media-marketing-roi/

As they point out in the piece:
"The problem for marketing professionals is that marketing activity is not an investment.
An investment is an asset that you purchase and place on your Balance Sheet. Like an office building or a computer system. It’s something you could sell later if you didn’t need it any more.
Marketing is an expense, and goes on the Profit & Loss statement."

Testing is similar. It is an activity, and we develop tools to support it. In general, our automation code doesn't get treated as an asset from an accounting perspective, it helps us perform our testing service much better than manual testing alone. It helps increase the value of code and product assets by helping us make them better.

The key point in the article for me:
"Sales generate revenue. Marketing generates profits."
My testing view: Testing generates better product value.

It isn't a simple calculation, because testing brings so many intangibles that are vital to the success of the product or projects we work on, just as marketing helps sales. If we measure it too simply, we run the risk of undermining what we deliver.

Goran Begic said...

Hi Dot,Thank you for bringing this topic up. The idea that test automation can replace people (testers) resides on the faulty premise that we test enough and that we don't need to test more, or that we don't need to test better. More products need to fail before there is enough awareness that increase in software complexity requires a quantum leap on the quality side (and I don't mean certification).

Btw. there are many poor ROI studies floating around, many based on cost of fixing of defects and their number (like all defects are the same). The best ROI study I could find on this topic is the paper from Doug Hoffman from long time ago: Cost Benefits Analysis of Test Automation ---the above is my personal opinion and not the opinion, or policy of my employer---

testastic said...

Great Post.

In my opinion, explaining Testing activities in terms of ROI is almost always a GOOD thing.

I would argue that the cost/benefit equation is hard to do in Testing, because the benefit is actually "Risk Mitigation" and while it can be measured (think Insurance industry), it is not something the Test community has gain muscle in.

I go into this more in my blog,
What is the ROI of Test?.

In addition, in my own experience, Automation has fantastically great ROI, but *not* due to reducing headcount, but in reducing Time to get a result. In my experience, automation start up cost is really short, but it's the maintenance cost that kills you.

Thanks for blogging this really important topic!

Dot Graham said...

Thanks very much for your very interesting comments! Sorry I didn't respond earlier - been having trouble with internet access.

Jonathan, a very interesting perspective on ROI with regard to marketing. I certainly don't have a background in financial stuff, and I suspect a lot of testers / test managers don't either. I liked the analogy in the article about trying to quantify the ROI of having email - as they say, it's just a necessary part of doing business now.

However, I wasn't quite sure how to apply the ideas to test automation, though. Is testing an expense or an investment? I also think of it as insurance - you may not "win" unless you have a disaster, but then you'd be very glad to have it.

Test automation is doubly indirect - testing is done on code, and automation is done on the testing.

But because automation is seen as an investment, even if this isn't correct according to financial models, it should IMHO make things visibly better, and this is difficult to do in a way that is simple enough to be understood but realistic enough.

Goran, I enjoyed looking through Doug Hoffman's paper (though I didn't study all the maths). His approach is very similar to the simple ROI spreadsheet that I have been giving away, though mine is based on mainly variable costs without fixed costs.

I fully agree with you that justifying automation on the basis of bugs found is not a good idea - it's tests that find bugs, not the manner in which tests are run. Did you see my blog post in May 2010 about this?

Brent (testastic), thanks! I like your blog and risk mitigation ideas - yes, I agree it is like insurance. Interesting that your friend's company was getting rid of QA. Here's something I have said (but I don't know if anyone actually did this): So, your boss want to get rid of testing/QA/testers. Just say, "Ok - but just sign here on this paper stating that this was your decision." If you think testing is expensive, try not testing. But as you say, it's a lot less expensive with a 3-weekly delivery cycle than a 3-yearly one.

Thanks very much for your comments and interesting links!

Dot Graham said...

Thanks to Tim Fretwell for discovering another blog that responded to this one, though not here. See http://exploringuncertainty.com/blog/ - the entry "R. O. Why" He has a very nice example of what to automate and what not to automate.

James said...

I am not automation testing guy, but most of the times I have heard people saying that automation testing is one of the costly job to do. If it is, I am sure most of the managers will perfer to have a cost effective tool to evaluate their performance.

Raju. said...

Interesting posts and comments here.
I like the analogy of 'Insurance'. Yes, ROI (short or long-term) is must before investing in Test Automation.
What i am afraid is not enough thought is spent on ROI, Risks Identification and in hedging all critical risks identified.

Hence unfortunately many a times automation is enforced to replace manual work without understanding ROI !!