tag:blogger.com,1999:blog-8378239747319086089.post8648592186750265067..comments2024-03-14T07:42:11.367-07:00Comments on Dorothy Graham Blog: The wrong question: What percentage of tests have you automated?Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.comBlogger11125tag:blogger.com,1999:blog-8378239747319086089.post-39183805441461482982016-02-04T03:46:22.289-08:002016-02-04T03:46:22.289-08:00Thank you so much for sharing your thought with us...Thank you so much for sharing your thought with us.Farhana Munahttps://youtu.be/20gT7stk5asnoreply@blogger.comtag:blogger.com,1999:blog-8378239747319086089.post-527629346692315272015-11-11T11:33:30.909-08:002015-11-11T11:33:30.909-08:00I absolutely agree, time gained and time spent are...I absolutely agree, time gained and time spent are 2 factors in a bigger picture called business value. Also factors like reduced number of escapes, higher coverage, improved reporting, improved cooperation and much more are relevant. Some can be predicted, some are very hard to express in numbers.Anonymoushttps://www.blogger.com/profile/05500007510662174897noreply@blogger.comtag:blogger.com,1999:blog-8378239747319086089.post-5531324851057294162015-11-11T04:59:40.655-08:002015-11-11T04:59:40.655-08:00Thanks everyone for your great comments!
@Viktor:...Thanks everyone for your great comments!<br /><br />@Viktor: I agree. What I am attempting to address is the most common misunderstanding of this question, but clearly you are looking at this in the right way, looking at percentage of automatable tests for example.<br /><br />@Martin: Yes! Maintenance is critical for continuing automation, as is minimising technical debt. In Lisa Crispin's chapter in the Experiences book, one of their success factors was re-factoring the automation in a separate Sprint every 6 months, so technical debt was addressed.<br /><br />@Ruben: Yes! Business value is what we should be delivering, but I find it difficult to find a way to measure this - any suggestions? You mention time spent and saved, but that's not the same thing as value, right?<br /><br />@Conrad: I was suggesting EMTE and/or coverage as alternatives - I agree they shouldn't be confused. If automated tests are easily de-railed by changes, then IMHO, they haven't been designed well enough, and the testware architecture needs attention. If the automate tests are built so that the most likely changes are the easiest for the automated tests to cope with, then the automation is flexible and likely to be long-lived. I also agree that automated support should not be limited to regression tests/checks - there is much more scope for useful tool support for testing. I like the idea of an automation "appliance" - sounds interesting.<br /><br />@Michael: Good to hear that! Tool vendors have a definite role to play in educating about what can and can't be done with tools. Whether or not the level of manual testing is significantly reduced depends on the amount of manual testing that was suitable for automation - sometimes it will be a lot, other times not so much. <br /><br />@Dave: Yes I well remember those days too!<br /><br />Thanks for your very useful and insightful comments.Dot Grahamhttps://www.blogger.com/profile/07991717205107510005noreply@blogger.comtag:blogger.com,1999:blog-8378239747319086089.post-25470122333039418952015-11-07T12:19:43.835-08:002015-11-07T12:19:43.835-08:00Thanks for the great post. As a automation tool v...Thanks for the great post. As a automation tool vendor, (Tricentis), we are often asked how to get to 100% automation. Our answer is always the same, "that is the wrong question." Where we see the future of Continuous Testing, or testing as a direct component of a CI/CD environment is a shift to highly automated API tests, with UI testing primarily focused on automated end to end, system integration testing, and a SIGNIFICANTLY reduced level of manual testing, primarily driven by exploratory testing. A base assumption of achieving this vision is a clear understanding of what test should be performed at all, and which are candidates for automation.<br /><br />Much appreciation for keeping this discussion going!<br /><br />Michael Eckhoffhttp://www.tricentis.comnoreply@blogger.comtag:blogger.com,1999:blog-8378239747319086089.post-44580972984076808612015-11-06T08:07:16.393-08:002015-11-06T08:07:16.393-08:00Yes the EMTE (Estimated Manual test Effort) that a...Yes the EMTE (Estimated Manual test Effort) that automation gives you needs to be a metric - but not to be confused or just bundled together with the "coverage". Automation does include lots of unit-testing in applications that have been designed to be tested at the component and even at the feature level - and that's where the magic sauce goes bad - Automation is best when used to test "components", and in scenarios where the scope is narrow. automated tools are not great at feature testing because feature changes kill dumb automated scripts which are typically just testing a computer-observable-behaviour and not an "outcome". Manual testing is thus better at feature breadth or new features. But automation should never be limited to regression-testing only because of this flaw.<br /><br />So I like to keep EMTE in sight, but prefer to keep technical debt low and speed up automated test execution and use automation up and down the entire fabrication/assembly-line. Development happens in teams right? teams around the globe benefit if you place your automated tests into an "appliance" and let them run it anywhere in the world, anytime - but without having to own the appliance. In other words the developers use the test-appliances, and only ask you for help when the test-runs fail.Conrad Braamhttps://www.blogger.com/profile/14516128061510018135noreply@blogger.comtag:blogger.com,1999:blog-8378239747319086089.post-69131316657331243892015-11-06T05:02:23.750-08:002015-11-06T05:02:23.750-08:00Great reflection, Dorothy! Thanks!
I guess the bes...Great reflection, Dorothy! Thanks!<br><br />I guess the best question instead that should be asked is about the business value that the automation initiative brought. This involves time that has been saved, technical debt (assuming user stories have been created for that), time it took to write scripts, but also time to setup a framework and infrastructure.Anonymoushttps://www.blogger.com/profile/05500007510662174897noreply@blogger.comtag:blogger.com,1999:blog-8378239747319086089.post-29388140848763171342015-11-03T13:30:42.275-08:002015-11-03T13:30:42.275-08:00Actually, it is a perfectly valid question and man...Actually, it is a perfectly valid question and managers should be asking it all the time. It does not mean that people should only automate so called manual tests (bad name to begin with) or that people should aim to automate everything there is in the test cases repository (if you don't have one, get one). It means that people try to automate test cases when it makes sense and that they keep track of what's automated vs what's not. If you don't understand why, well, you do not understand QA.Anonymoushttps://www.blogger.com/profile/11745227657927440221noreply@blogger.comtag:blogger.com,1999:blog-8378239747319086089.post-25319914808788207542015-11-03T11:05:08.273-08:002015-11-03T11:05:08.273-08:00Hi Dot,
very valid points, worth revisiting every...Hi Dot,<br /><br />very valid points, worth revisiting every now and then. Thanks.<br /><br />If I may add to them, the importance of the related topic of maintenance is hard to overstate, and it could be considered the flip side of the question you address. (You do in fact refer to it already in your article.) The business value of any number of working automated test cases is very quickly reduced to zero if they start failing because they do not keep up with reality. So having any number of test cases automated, in whatever way, usually amounts to close to nothing unless their continuity is also ensured. There are several ways of addressing it, and much more text can and has been devoted to it, but the essential point is that keeping maintenance effort low needs very serious attention.<br /><br />When I read the Agile manifesto again recently, I found it interesting that one of the principles behind it calls for 'continuous attention to technical excellence.' It struck me that it does not say 'quality' but 'excellence.' The principle above it mentions the related topic of sustainable development, being able to maintain a constant pace indefinately. This requires that technical debt is not allowed to pile up. I for one make no distinction between production code and automation code here. If something is worth doing, it is worth doing well. If automation code is treated as a second-class citizen, it will quickly start to behave like one.<br /><br /><br />Regards,<br /><br />Martin GijsenAnonymoushttps://www.blogger.com/profile/12261434685272275268noreply@blogger.comtag:blogger.com,1999:blog-8378239747319086089.post-32711008392061170192015-11-03T03:31:17.781-08:002015-11-03T03:31:17.781-08:00Takes me back to a test automation exercise in a c...Takes me back to a test automation exercise in a certain company in Tewkesbury!daver22https://www.blogger.com/profile/06745366009000679845noreply@blogger.comtag:blogger.com,1999:blog-8378239747319086089.post-53326853488912175772015-11-03T00:55:01.537-08:002015-11-03T00:55:01.537-08:00Hi Kobi,
Thanks very much for you comment - yes y...Hi Kobi,<br /><br />Thanks very much for you comment - yes you are right, and I think that coverage is something that is also often mis-understood. Just because you have one test for something, as you say, does not mean that it is "completely" tested. <br /><br />I have increased the font size (and changed font) so I hope it is more readable now.<br /><br />PS I also added in "Page's Law", thanks to Alan's tweet.Dot Grahamhttps://www.blogger.com/profile/07991717205107510005noreply@blogger.comtag:blogger.com,1999:blog-8378239747319086089.post-15307566595839962452015-11-02T22:07:23.539-08:002015-11-02T22:07:23.539-08:00Hi Dorothy - Thanks for this interesting post,
Ano...Hi Dorothy - Thanks for this interesting post,<br />Another issue not mentioned above is management of the percentage of each manual or expected test content coverage vs. what was eventually implemented by automation.<br />Managers and others seem to instinctively think that if Feature X was automated, the full extent of what we used to test manually is now automated,<br />But in many cases only a fraction of the "expected" test is automated, either since automation cannot see all that manual tester do, or since some parts are just too hard and wasteful to automate.<br />In most companies these deviations are not managed, so we are left with an assumption of "full" test, while actually there should be a list of additional items to be run manually.<br /><br />BTW - please consider increasing the font size in your site (while we can zoom, it's best to have it that way in 1st place).<br /><br />Thanks,<br />@halperinko - Kobi Halperinhalperinkohttps://www.blogger.com/profile/02746116081985463537noreply@blogger.com