Thursday, 26 May 2016

Test automation as an orchard

At StarEast in May 2016, I was kindly invited to give a lightning keynote, which I did on this analogy. Hope you find it interesting and useful!


Automation is SO easy.

Let me rephrase that - automation often seems to be very easy.
When you see your first demo, or run your first automated test, it’s like magic - wow, that’s good, wish I could type that fast.

But good automation is very different to that first test.

If you go into the garden and see a lovely juicy fruit hanging on a low branch, and you reach out and pick it, you think, "Wow, that was easy - isn’t it good, lovely and tasty".

But good test automation is more like building an orchard to grow enough fruit to feed a small town.

Where do you start?
First you need to know what kind of fruit you want to grow - apples? oranges? (oranges would not be a good choice for the UK). You need to consider what kind of soil you have, what kind of climate, and also what will the market be - you don’t want to grow fruit that no one wants to buy or eat.

In automation, first you need to know what kind of tests you want to automate, and why. You need to consider the company culture, other tools, what the context is, and what will bring lasting value to your business.

Growing pains?
Then you need to grow your trees. Fortunately automation can grow a lot quicker than trees, but it still takes time - it’s not instant.

While the trees are growing, you need to prune them and prune them hard especially in the first few years. Maybe you don’t allow them to fruit at all for the first 3 years - this way you are building a strong infrastructure for the trees so that they will be stronger and healthier and will produce much more fruit later on. You may also want to train them to grow into the structure that you want from the trees when they are mature.

In automation, you need to prune your tests - don’t just let them grow and grow and get all straggly. You need to make sure that each test has earned its place in your test suite, otherwise get rid of it. This way you will build a strong infrastructure of worthwhile tests that will make your automation stronger and healthier over the years, and it will bring good benefits to your organisation. You need to structure your automation (a good testware architecture) so that it will give lasting benefits.

Feeding, pests and diseases
Over time, you need to fertilise the ground, so that the trees have the nourishment they need to grow to be strong and healthy.

In automation, you need to nourish the people who are working on the automation, so that they will continue to improve and build stronger and healthier automation. They need to keep learning, experimenting, and be encouraged to make mistakes - in order to learn from them.

You need to deal with pests - bugs - that might attack your trees and damage your fruit.

Is this anything to do with automation? Are there bugs in automated scripts? In testing tools? Of course there are, and you need to deal with them - be prepared to look for them and eradicate them.

What about diseases? What if one of your trees gets infected with some kind of blight, or suddenly stops producing good fruit? You may need to chop down that infected tree and burn it, because it you don’t, this blight might spread to your whole orchard.

Does automation get sick? Actually, a lot of automation efforts seem to decay over time - they take more and more effort to maintain. technical debt builds up, and often the automation dies. If you want your automation to live and produce good results, you might need to take drastic action and re-factor the architecture if it is causing problems. Because if you don’t, your whole automation may die.

Picking and packing
What about picking the fruit? I have seen machines that shake the trees so they can be scooped up - that might be ok if you are making cider or applesauce, but I wouldn’t want fruit picked in that way to be in my fruit bowl on the table. Manual effort is still needed. The machines can help but not do everything (and someone is driving the machines).

Test execution tools don’t do testing, they just run stuff. The tools can help and can very usefully do some things, but there are tests that should not be automated and should be run manually. The tools don’t replace testers, they support them.

We need to pack the fruit so it will survive the journey to market, perhaps building a structure to hold the fruit so it can be transported without damage.

Automation needs to survive too - it needs to survive more than one release of the application, more than one version of the tool, and may need to run on new platforms. The structure of the automation, the testware architecture, is what determines whether or not the automated tests survive these changes well.

Marketing, selling, roles and expectations
It is important to do marketing and selling for our fruit - if no one buys it, we will have a glut of rotting fruit on our hands.

Automation needs to be marketed and sold as well - we need to make sure that our managers and stakeholders are aware of the value that automation brings, so that they want to keep buying it and supporting it over time.

By the way, the people who are good at marketing and selling are probably not the same people who are good at picking or packing or pruning - different roles are needed. Of course the same is true for automation - different roles are needed: tester, automator, automation architect, champion (who sells the benefits to stakeholders and managers).

Finally, it is important to set realistic expectations. If your local supermarket buyers have heard that eating your fruit will enable them to leap tall buildings at a single bound, you will have a very easy sell for the first shipment of fruit, but when they find out that it doesn’t meet those expectations, even if the fruit is very good, it may be seen as worthless.

Setting realistic expectations for automation is critical for long-term success and for gaining long-term support; otherwise if the expectations aren’t met, the automation may be seen as worthless, even if it is actually providing useful benefits.

So if you are growing your own automation, remember these things:
  • -      it takes time to do it well
  • -      prepare the ground
  • -      choose the right tests to grow
  • -      be prepared to prune / re-factor
  • -      deal with pests and diseases (see previous point)
  • -      make sure you have a good structure so the automation will survive change
  • -      different roles are needed
  • -      sell and market the automation and set realistic expectations
  • -      you can achieve great results

I hope that all of your automation efforts are very fruitful!


Unknown said...

I like the analogy, but we've come across teams that take this bit too literally:

> Maybe you don’t allow them to fruit at all for the first 3 years - this way you are building a strong infrastructure for the trees so that they will be stronger and healthier and will produce much more fruit later on.

We have seen a team spend a year (no exaggeration) building elaborate test infrastructure without producing any test results. They thought that once this infrastructure was complete, they would be able to automate all the testcases "in a week". But they built the infrastructure without building testcases at the same time, to validate & guide the design of the infrastructure.

As you might expect, that year of work turned out to be essentially worthless.

Maybe it's that some developers find infrastructure work to be more fun than writing tests. Whatever the reason, some guidance is needed. We agree that it's important to invest in test infrastructure, but a healthy dose of "agile" can prevent you from building the *wrong* infrastructure.

We (stb-tester) try to encourage our users to automate a single testcase, no matter how simple, and then set up a CI system, or at least automatic nightly test runs, before automating any further testcases. Run your tests!

Dot Graham said...

Hi David,

Thanks very much for your very perceptive comment! I didn't mean to imply that work on automation should take years (or even months) before there are any tests to run, so thanks for showing how that could be mis-interpreted. I fully agree with you that an agile approach to automation is the best way, and to get tests running quickly. I too have seen organisations spend months trying to decide which tool to get rather than just getting started and experimenting.

Perhaps a better parallel would be something like this: It may take a few years to get your automation into the "right shape" for your organisation. The first attempt at automation (as with the first attempt at doing anything new) will not be your best effort, so don't expect it to be perfect. Throw away what doesn't work and continue to re-factor and improve what you have, learning what works best as you go along.

Thanks for the feedback!

Regards, Dot

Shirley Perkins said...

Hi Dot, I very much like your thoughtful approach when starting with test automation.
Also, before implementing test automation it’s important to determine the targets and problems it should solve for you and your organization.

Below is an article that demonstrates that, you are welcome to have a look:

Dot Graham said...

Hi Shirley,

Thanks, and I totally agree with you that the place to start with automation is to ask why you want it, and set realistic, achievable and clear goals or objectives. We talk about this in the Test Automation Patterns wiki:

Thanks for the article link - Mark makes some very useful points.

Dot Graham said...

Although I appreciate complimentary comments (thank you), I do not permit comments on my blog that are advertising services. Chennai, please take note, thanks.