Test automation as an orchard

At StarEast in May 2016, I was kindly invited to give a lightning keynote, which I did on this analogy. Hope you find it interesting and useful!

-----------------------------------------------------------------------

Automation is SO easy.

Let me rephrase that - automation often seems to be very easy.

When you see your first demo, or run your first automated test, it’s like magic - wow, that’s good, wish I could type that fast.

But good automation is very different to that first test.

If you go into the garden and see a lovely juicy fruit hanging on a low branch, and you reach out and pick it, you think, "Wow, that was easy - isn’t it good, lovely and tasty".

But good test automation is more like building an orchard to grow enough fruit to feed a small town.

Where do you start?

First you need to know what kind of fruit you want to grow - apples? oranges? (oranges would not be a good choice for the UK). You need to consider what kind of soil you have, what kind of climate, and also what will the market be - you don’t want to grow fruit that no one wants to buy or eat.

In automation, first you need to know what kind of tests you want to automate, and why. You need to consider the company culture, other tools, what the context is, and what will bring lasting value to your business.

Growing pains?

Then you need to grow your trees. Fortunately automation can grow a lot quicker than trees, but it still takes time - it’s not instant.

While the trees are growing, you need to prune them and prune them hard especially in the first few years. Maybe you don’t allow them to fruit at all for the first 3 years - this way you are building a strong infrastructure for the trees so that they will be stronger and healthier and will produce much more fruit later on. You may also want to train them to grow into the structure that you want from the trees when they are mature.

In automation, you need to prune your tests - don’t just let them grow and grow and get all straggly. You need to make sure that each test has earned its place in your test suite, otherwise get rid of it. This way you will build a strong infrastructure of worthwhile tests that will make your automation stronger and healthier over the years, and it will bring good benefits to your organisation. You need to structure your automation (a good testware architecture) so that it will give lasting benefits.

Feeding, pests and diseases

Over time, you need to fertilise the ground, so that the trees have the nourishment they need to grow to be strong and healthy.

In automation, you need to nourish the people who are working on the automation, so that they will continue to improve and build stronger and healthier automation. They need to keep learning, experimenting, and be encouraged to make mistakes - in order to learn from them.

You need to deal with pests - bugs - that might attack your trees and damage your fruit.

Is this anything to do with automation? Are there bugs in automated scripts? In testing tools? Of course there are, and you need to deal with them - be prepared to look for them and eradicate them.

What about diseases? What if one of your trees gets infected with some kind of blight, or suddenly stops producing good fruit? You may need to chop down that infected tree and burn it, because it you don’t, this blight might spread to your whole orchard.

Does automation get sick? Actually, a lot of automation efforts seem to decay over time - they take more and more effort to maintain. technical debt builds up, and often the automation dies. If you want your automation to live and produce good results, you might need to take drastic action and re-factor the architecture if it is causing problems. Because if you don’t, your whole automation may die.

Picking and packing

What about picking the fruit? I have seen machines that shake the trees so they can be scooped up - that might be ok if you are making cider or applesauce, but I wouldn’t want fruit picked in that way to be in my fruit bowl on the table. Manual effort is still needed. The machines can help but not do everything (and someone is driving the machines).

Test execution tools don’t do testing, they just run stuff. The tools can help and can very usefully do some things, but there are tests that should not be automated and should be run manually. The tools don’t replace testers, they support them.

We need to pack the fruit so it will survive the journey to market, perhaps building a structure to hold the fruit so it can be transported without damage.

Automation needs to survive too - it needs to survive more than one release of the application, more than one version of the tool, and may need to run on new platforms. The structure of the automation, the testware architecture, is what determines whether or not the automated tests survive these changes well.

Marketing, selling, roles and expectations

It is important to do marketing and selling for our fruit - if no one buys it, we will have a glut of rotting fruit on our hands.

Automation needs to be marketed and sold as well - we need to make sure that our managers and stakeholders are aware of the value that automation brings, so that they want to keep buying it and supporting it over time.

By the way, the people who are good at marketing and selling are probably not the same people who are good at picking or packing or pruning - different roles are needed. Of course the same is true for automation - different roles are needed: tester, automator, automation architect, champion (who sells the benefits to stakeholders and managers).

Finally, it is important to set realistic expectations. If your local supermarket buyers have heard that eating your fruit will enable them to leap tall buildings at a single bound, you will have a very easy sell for the first shipment of fruit, but when they find out that it doesn’t meet those expectations, even if the fruit is very good, it may be seen as worthless.

Setting realistic expectations for automation is critical for long-term success and for gaining long-term support; otherwise if the expectations aren’t met, the automation may be seen as worthless, even if it is actually providing useful benefits.

Summary

So if you are growing your own automation, remember these things:

• -      it takes time to do it well
• -      prepare the ground
• -      choose the right tests to grow
• -      be prepared to prune / re-factor
• -      deal with pests and diseases (see previous point)
• -      make sure you have a good structure so the automation will survive change
• -      different roles are needed
• -      sell and market the automation and set realistic expectations
• -      you can achieve great results

I hope that all of your automation efforts are very fruitful!

The wrong question: What percentage of tests have you automated?

At a couple of recent conferences, I became aware that people are asking the wrong question with regard to automation. There was an ISTQB survey that asked “How many (what percentage of) test cases do you automate?”. In talking to a delegate after my talk on automation at another conference, she said that her manager wanted to know what percentage of tests were automated; she wasn’t sure how to answer, and she is not alone. It is quite common for managers to ask this question; the reason it is difficult to answer is because it is the wrong question.

Why do people ask this? Probably to get some information about the progress of an automation effort, usually when automation is getting started. This is not unreasonable, but this question is not the right one to ask, because it is based on a number of erroneous assumptions:

Wrong assumption 1)  All manual tests should be automated. “What percentage of tests” implies that all existing tests are candidates for automation, and the percentage will measure progress towards the “ideal” goal of 100%.

It assumes that there is a single set of tests, and that some of them are manual and some are automated. Usually this question actually means “What percentage of our existing manual tests are automated?”

But your existing manual tests are not all good candidates for automation – certainly some manual tests can and should be automated, but not all of them!

Examples: if you could automate “captcha”, then the “captcha” isn’t working, as it’s supposed to tell the difference between a human and a computer. “Do these colours look nice?” or “Is this exactly what a real user would do”? And tests that take too long to automate such as tests that are not run very often or those that. are complex to automate.

Wrong assumption 2) Manual tests are the only candidates for automation. “What percentage of tests” also implies that the only tests worth automating are existing manual tests, but this is also incorrect. There are many things that can be done using tools that are impossibly or infeasible to do when testing manually.

Examples: additional verification or validation of screen objects – are they in the correct state? When testing manually, you can see what is on the screen, but you may not know its state or whether the state is displaying correctly.

Tests using random inputs and heuristic oracles, which can be generated in large volume and checked automatically.

Wrong assumption 3) A manual test is the same as an automated test. “What percentage of tests” also assumes that a manual test and an automated test are the same - but they are not. A manual test consists of a set of directions for a human being to follow; it may be rather detailed (use customer R Jones), or it could be quite vague (use an existing customer). A manual test is optimised for a human tester. When tests are executed manually, they may vary slightly each time, and this can be both an advantage (may find new bugs) and a disadvantage (inconsistent tests, not exactly repeated each time).

An automated test should be optimized for a computer to run. It should be structured according to good programming principles, with modular scripts that call other scripts. It shouldn’t be one script per test, but each test should use many scripts (most of them shared) and most scripts should be used in many tests. An automated test is executed in exactly the same way each time, and this can be an advantage (repeatability, consistency) and a disadvantage  (won’t find new bugs).

One manual test may be converted into 3, 5, 10 or more automated scripts. Take for example a manual test that starts at the main menu, navigates to a particular screen and does some tests there, then returns to the main menu. And suppose you have a number of similar tests for the same screen, say 10. If you have one script per test, each will do 3 things: navigate to the target area, do tests, navigate back. If the location of the screen changes, all of those tests will need to be changed – a maintenance nightmare (especially if there are a lot more than 10 tests)! Rather, each test should consist of at least 3 scripts: one to navigate to the relevant screen, one (or perhaps many) scripts to perform specific tests, and one script to navigate back to the main menu. Note that the same “go to screen” and “return to main menu” script is used by all of these tests. Then if the screen is re-located, only 2 scripts need to be changed and all the automated tests will still work.

But now the question is: how many tests have you automated? Is it the 10 manual tests you started with? Or should you count automated scripts? Then we have at least 12 but maybe 20. Suppose you now find that you can very easily add another 5 tests to your original set, sharing the navigation scripts and 4 of the other scripts. Now you have 15 tests using 13 scripts – how many have you automated? Your new tests never were manual tests, so have you automated 10 tests (of the original set) or 15?

Wrong assumption 4) Progress in automation is linear (like testing). A “what percent completed” measure is fine for an activity that is stable and “monotonic”, for example running sets of tests manually. But when you automate a test, especially at first, you need to put a lot of effort in initially to get the structure right, and the early automated tests can’t reuse anything because nothing has been built yet. Later automated tests can be written / constructed much more quickly than the earlier ones, because there will (should) be a lot of reusable scripts that can just be incorporated into a new automated test. So if your goal is to have say 20 tests automated in 2 weeks, after one week you may only have automated only 5 of those tests, but the other 15 can easily be automated in week 2. So after week 1 you have automated 25% of the tests, but you have done 50% of the work.

Eventually it should be easier and quicker to add a new automated test than to run that test manually, but it does take a lot of effort to get to that point.

Good progress measures. So if these are all reasons NOT to measure the percentage of manual tests automated, what would be a good automation progress measure instead? Here are three suggestions:

1)            Percentage of automatable tests that have been automated. Decide first which tests are suitable for automation, and/or that you want to have as automated tests, and measure the percentage automated compared to that number, having taken out tests that should remain manual and tests that we don’t want to automate now. This can be done for a sprint, or for a longer time frame (or both). As Alan Page says, "Automate 100% of the tests that should be automated."

2)            EMTE: Equivalent Manual Test Effort: Keep track of how much time a set of automated tests would have taken if they had been run manually. Each time those tests are run (automatically), you “clock up” the equivalent of that manual effort. This shows that automation is running tests now that are no longer run manually, and this number should increase over time as more tests are automated.

3)            Coverage: With automation, you can run more tests, and therefore test areas of the application that there was never time for when the testing was done manually. This is a partial measure of one aspect of the thoroughness of testing (and has its own pitfalls), but is a useful way to show that automation is now helping to test more of the system.

Conclusion So if your manager asks you “What percent of the tests have you automated?” you need to ask something like: Percent of what? Out of existing tests that could be automated or that we decide to automate? What about additional tests that would be good to automate that we aren’t doing now? Do you want to know about progress in time towards our automation goal, or literally only the tests, as this will be different because automated tests are structured differently to manual tests.

It might be a good idea to find out why he or she has asked that question – what is it that they are trying to see? They need to have some visibility for automation progress, and it is up to you to agree something that would be useful and helpful, honest and reasonably easy to measure. Good luck! And let me know how you measure your progress in automation!

If you want more advice on automation, see the wiki that I am doing with Seretta Gamba at TestAutomationPatterns.org.

Is it dangerous to measure ROI for test automation?

I have been a fan of trying to show ROI for automation in a way that is simple enough to understand easily, and provides a way of showing the benefit of automation compared to its cost.

I have been developing a spreadsheet with sample calculations (sparked off initially by Molly Mahai, including an example from Mohacsi & Beer's chapter in the new book, and some other people have also been influential - thanks). I have sent this spreadsheet out to around 300 people, including most who have attended my automation tutorials.

The problem with showing ROI is that it's hard to quantify some of the important factors, so I have focused on showing ROI using what is the most straight-forward to quantify - people's effort and/or time. This can be converted into money, using some kind of salary cost, if desired, and either effort or money can then be plugged into the ROI calculation = (benefit - cost) / cost.

So basically, this is showing how a set of tests requires less human effort when those tests are automated than would be required if those same tests were run manually.

This is great, right? We have a clear business case showing savings from automation than are greater than the cost of developing the automation, so our managers should be happy.

Recently, however, I have been wondering whether this approach can be dangerous.

If we justify automation ONLY in terms of reduced human effort, we run the risk of implying that the tools can replace the people, and this is definitely not true! Automation supports testing, it does not replace testers. Automation should free the testers to be able to do better testing, designing better tests, having time to do exploratory testing, etc.

So should we abandon ROI for automation? I don’t think that’s a good idea – we should be gaining business benefit from automation, and we should be able to show this.

Scott Barber told me about Social ROI – a way of quantifying some of the intangible benefits – I like this but haven’t yet seen how to incorporate it into my spreadsheet.

In our book, there are many success stories of automation where ROI was not specifically calculated, so maybe ROI isn’t as critical as it may have seemed.

I don’t know the answer here – these are just my current thoughts!

How long does it take to write a book?

A number of people have asked me this, since our new book is now out.

This book took us 2 and a half years. This doesn’t include the effort put in by the case study authors and other contributors, so this book represents a lot of work! What exactly had we been doing all that time? I wondered that too, so here is where the time went.

In August 2009, I have my first note of our plan to solicit contributions for a new book on automation experience. We sent emails, put a call for contributions on my web site, and talked to people at conferences, and began gathering potential contributions.

I started keeping track of the hours we spent from December 2009. We had a big initial “push” (the first peak on the graph) and produced a “protobook” – 4 chapters with an introduction. We were sure this would be snapped up by a publisher!

We submitted to the publisher of our previous book in mid-February, but initially they weren’t very keen! This was a blow, as we were convinced this would be a great book! I tried several other publishers over the next few months, and got rejected; I continued to try and convince Pearson/Addison Wesley that they should publish our book.

They eventually relented in July and we signed a contract. We worked steadily on the book over the rest of that year, a total of around 400 hours between us. The complete draft manuscript, ready for independent review was due on the 15^th April, and the final manuscript on the 15^th October. “No problem”, we thought.

However, we found that we did need to work at a more intense level - we spent another 300 hours to the end of April (the double peak with some time out in February), and another 300 hours to the end of 2011. The final peak was editing the final page proofs and doing the index, more work that we had anticipated at that stage. The total for us comes to just under 1000 hours. We don't know how much time the contributors spent, but it their time was equivalent to ours, the collaboration represents around 2000 hours of time - that's more than one working year.

We enjoyed working on this book and reading all the stories as they came in. There are many useful lessons, some heartfelt pain, and many gratifying successes among the stories.

You can follow the book on Twitter on @AutExpBook. The book tweets tips and good points every few days!

Thanks to all the book contributors, and to co-author Mark Fewster.

Automation benefit measured by EMTE - good or bad?

Being able to run tests that we would not have had time to run manually is one of the benefits of automated testing; we should be able to measure this benefit in some way.

What is EMTE?

EMTE stands for "Equivalent Manual Test Effort" and is a way of measuring the benefit of running automated tests.

If an automated test (A) would have taken 4 hours to run manually, then its EMTE is 4 hours; another test (B) that would have taken 7.5 hours to run manually has an EMTE of 7.5 hrs.

In a test cycle, if Test A is run five times, and Test B is run twice, then the EMTE for that cycle is 5*4 hrs + 2*7.5 hrs = 20 + 15 = 35 hours EMTE.

What is EMTE used for?

EMTE can be used as a way to measure a benefit of test automation (automated running of functional tests).

When tests are automated, they can be run in much less time than they could be run manually. Our tests A and B may be able to run in 5 and 10 minutes respectively, for example. So we can achieve "4 hours' worth" of manual testing in 5 minutes of automated testing. Whenever we run Test A, we can "clock up" 4 hours of EMTE.

Is EMTE a good thing?

Yes, because it is a way to show the benefit of automation.

Costs (of automation as well as other things) tend to become visible by themselves - managers see that people are spending time on the automation. But what is the benefit of this automation? If you don't make benefits visible to managers, there is a risk that they will not see the benefits, and may eventually conclude that there are no benefits. EMTE is one way to make an automation benefit visible.

So how could it be a bad thing?

I have had discussions with a couple of people recently (thanks Julian and Wade) about abusing EMTE, and yes, it can be abused (as any metric can). Here is how it could be mis-used:

"Test A takes 5 minutes, so let's run it 12 times every hour for 2 hours. This gives 24*4 hours of EMTE = 96 hours. This will make us look really great!"

The problem is that after the first run, the other 23 runs are being done just for the sake of looking good, not for a valuable benefit in terms of running that test. This is an abuse of EMTE, and is a bad thing.

What to do about it?

Use EMTE (and other measures of the benefit of test automation) sensibly.

Perhaps only "count" EMTE once a day, however many times a test is run? (e.g. in continuous integration testing)

In what other ways can the benefit of automation be shown? (e.g. more coverage, freeing testers to find more bugs, number of times tests are run, more variety of data used?)

Have you encountered the abuse of this (or other automation) measures? How have you solved the problem? (Please comment, thanks!)

Dot Graham

Automated tests should find bugs? No!

I have recently been having what seems like the same discussion with a number of different people.

"Automated tests should find bugs" or "find more bugs" is a very common misconception. Basically this says that finding bugs is a valid objective for automation. I don't agree - I think this is generally a very poor objective for test automation. The reasons are to do with the nature of testing and of automation.

Testing is an indirect activity, not a direct one. We don't just "do testing", we "test something". (Testing is like a transitive verb which requires an object to be grammatically correct.) This is why the quality of the software we test has a large impact on testing: if a project is delayed because testing finds lots of bugs, we shouldn't blame the testing! (I hope that most people realize this by now, but do have my doubts at times!) Testing is not responsible for the bugs inserted into software any more than the sun is responsible for creating dust in the air. Testing is a way of assessing the software, whatever the quality of that software is.

Test automation is doubly indirect. We don't "do automation", we "automate tests that test something".

Automation is a mechanism for executing tests, whatever the quality of those tests are (that assess the software, whatever the quality of that software is).

Bugs are found by tests, not by automation.

It is just as unfair to hold automation responsible for the quality of the test, as it is to hold the testing responsible for the quality of the software.

This is why "finding bugs" is not a good objective for test automation. But there are a couple more points to make.

Most people automate regression tests. Regression tests by their nature are tests that have been run before and are run many times. The most likely time a test will find a bug is the first time it is run, so regression tests are less likely to find bugs than say exploratory tests. In addition the same test run for a second time (and more) is even less likely to find a bug. Hence the main purpose of regression tests (whether automated or not) is to give confidence that what worked before is still working (to the extent that the tests cover the application).

Of course, this is complicated by the fact that because automated tests can be run more often, they do sometimes find bugs that wouldn't have been found otherwise. But even this is not because those tests are automated, it is because they were run. If the tests that are automated had been run manually, then those manual tests would have found the bugs. So even this bug-finding is a characteristic of the tests, not of the automation.

So should your goal for automation be to find bugs? No! At least not if you are planning to automate your existing regression tests.

I have been wondering if there may be two exceptions: Model-Based Testing (where tests are generated from a model), and mature Keyword-driven automation, i.e. using a Domain Specific Test Language. In both cases, the first time a test is run is in its automated form.

But hang on, this means that again it is the tests that are finding the bugs, not the fact that those tests are automated!

"Finding bugs" is a great objective for testing - but it is not a good objective for automation.

ROI from automation

How do you measure Return on Investment from automation?

According to http://www.investopedia.com/terms/r/returnoninvestment.asp, return on investment is (Gain - Cost) / Cost.

So if we have a set of automated tests that save 5 hours of manual testing, then if those tests took one hour to build and run, our ROI would be (5 - 1) / 1 = 4 (if my arithmetic is correct). So that would be a four-fold ROI for those automated tests.

Of course in the initial stages of automation, before our automation regime is mature, it might take us 10 hours to build and run those same tests. This would give an ROI of (5 - 10) / 10, i.e. a negative return on investment of -0.5.

Negative ROI is not generally a good idea, as it means you are losing money. But you may have negative ROI in the early stages of automation, but a very good positive ROI long-term. It might also make more sense to look at ROI over a larger sample, say all of the automation build and run in a month, and monitor that month by month.

How do you measure ROI for automation?

Don't blame the bread-making machine - thoughts on test automation

If you have a bread-making machine, it may or may not make good bread.

If it doesn't, the reason might be the ingredients you put into the machine. If you put in too much salt (or no salt at all), the bread won't be edible. Is that the fault of the bread-making machine?

Some people blame the test automation for things which are actually the responsibility of the testing. For example, if you mistakenly think that finding lots of bugs is the main aim of your regression test automation, you are setting yourself up for disappointment, if not failure of your automation effort. Finding bugs is what testing (and testers) do - it is not the job of automation - the job of automation is to run tests.

The tests are the ingredients that determine whether the testing is effective or not. The effectiveness of the tests is the same whether those tests are run manually or using an automation tool. 

The tool is no more responsible for the quality of the testing than the bread-making machine is responsible for the taste of the bread.