tag:blogger.com,1999:blog-83782397473190860892024-03-14T07:42:11.499-07:00Dorothy Graham BlogDot Graham, software testing consultant,
www.DorothyGraham.co.ukDot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.comBlogger19125tag:blogger.com,1999:blog-8378239747319086089.post-52896888799731644672019-05-01T14:44:00.000-07:002019-05-01T16:30:32.610-07:00My favourite techniques.<br />
<br />
As my lightning keynote at Star East today, I put in a plea not to forget some of the "old things", even as we embrace new things.<br />
<br />
Over my nearly 50 years in the industry, I have seen many new things come along, most of which were supposed to solve "all of our problems". They never do, of course, but often there is something good that comes from them and often that lasts.<br />
<br />
But don't forget that the old things are still useful, even if they aren't new. Remember the classic techniques - they still have value.<br />
<br />
But rather than talk about this, I have a song, to the tune of "These are a few of my favourite things".<br />
<br />
<div class="MsoNormal" style="font-family: "Times New Roman"; margin: 12pt 0in;">
<span lang="EN-US" style="font-size: large;"><i>Boundary Analysis, and State Transitions,<br />Trees of decisions, Equivalence Partitions,<br />Coverage of statements is not just for freaks,<br />These are a few of my favourite techniques.<o:p></o:p></i></span></div>
<div class="MsoNormal" style="font-family: "Times New Roman"; margin: 6pt 0in;">
<span lang="EN-US" style="font-size: large;"><i>Walkthroughs, Reviews, and Inspections are awesome<br />They belong in your toolbox and not in a museum<br />Welcome the feedback as useful critiques<br />These are a few of my favourite techniques<o:p></o:p></i></span></div>
<div class="MsoBodyTextIndent" style="font-family: "Times New Roman"; margin: 12pt 0in 12pt 9pt;">
<span lang="EN-US" style="font-size: large;"><i>When the bugs bite, when my build breaks<br />When I'm feeling sad,<br />I simply remember my favorite techniques<br />And then I don't feel so bad.<o:p></o:p></i></span></div>
<div class="MsoBodyTextIndent" style="font-family: "Times New Roman"; margin: 12pt 0in;">
<span style="font-size: large;"><i><br /></i></span></div>
<div class="MsoBodyTextIndent" style="font-family: "Times New Roman"; margin: 12pt 0in;">
<span lang="EN-US" style="font-size: large;"><i>DevOps and Agile, integration continuous<br />Manual testing should not be superfluous<br />Techniques that work well are not antiques<br />They are a few of my favourite techniques.<o:p></o:p></i></span></div>
<div class="MsoNormal" style="font-family: "Times New Roman"; margin: 6pt 0in;">
<span lang="EN-US" style="font-size: large;"><i>If you want your testing to be highest quality<br />You’d better investigate testing exploratory<br />Start with a charter and give it some tweaks<br />Just use these lovely heuristic techniques.<o:p></o:p></i></span></div>
<div class="MsoBodyTextIndent" style="font-family: "Times New Roman"; margin: 12pt 0in 12pt 9pt;">
<span lang="EN-US" style="font-size: large;"><i>When the users find a defect,<br />Is it such a crime?<br />I simply remember my favorite techniques<br />will help me find more next time.<o:p></o:p></i></span></div>
<div class="MsoNormal" style="font-family: "Times New Roman"; margin: 6pt 0in;">
<span lang="EN-US" style="font-size: large;"><i><br /><o:p></o:p></i></span></div>
<div class="MsoNormal" style="font-family: "Times New Roman"; margin: 6pt 0in;">
<span lang="EN-US" style="font-size: large;"><i>Issues and Patterns – automated regression<br />Testware architecture should be an obsession<br />Test in production and analytics<br />These are a few of my favourite techniques.<o:p></o:p></i></span></div>
<div class="MsoNormal" style="font-family: "Times New Roman"; margin: 6pt 0in;">
<span lang="EN-US" style="font-size: large;"><i>Artificial Intelligence is hot stuff at Star East<br />Are testing careers becoming deceased?<br />Not if you listen to test conference geeks,<br />They’ll recommend we still need these techniques.<o:p></o:p></i></span></div>
<div class="MsoBodyTextIndent" style="font-family: "Times New Roman"; margin: 12pt 0in 12pt 9pt;">
<span lang="EN-US"><span style="font-size: large;"><i>When they tell me, it's a feature<br />And I get depressed<br />I simply remember my favorite techniques<br />And oh how I love to test!</i></span><span style="font-size: large;"><o:p></o:p></span></span></div>
Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com2tag:blogger.com,1999:blog-8378239747319086089.post-13778709696009960502016-05-26T04:46:00.000-07:002016-05-26T04:46:25.051-07:00Test automation as an orchardAt StarEast in May 2016, I was kindly invited to give a lightning keynote, which I did on this analogy. Hope you find it interesting and useful!<br />
<br />
-----------------------------------------------------------------------<br />
<br />
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">Automation
is SO easy.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">Let me
rephrase that - automation often seems to be very easy.<o:p></o:p></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">When you
see your first demo, or run your first automated test, it’s like magic - wow, that’s
good, wish I could type that fast.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">But good
automation is very different to that first test.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">If you go into the garden and see a lovely juicy fruit hanging on a low branch,
and you reach out and pick it, you think, "Wow, that was easy - isn’t it good, lovely and
tasty".<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">But good
test automation is more like building an orchard to grow enough fruit to feed a
small town.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;"><b>Where do
you start?</b></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">First you
need to know what kind of fruit you want to grow - apples? oranges? (oranges
would not be a good choice for the UK). You need to consider what kind of soil
you have, what kind of climate, and also what will the market be - you don’t
want to grow fruit that no one wants to buy or eat.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">In
automation, first you need to know what kind of tests you want to automate, and
why. You need to consider the company culture, other tools, what the context
is, and what will bring lasting value to your business.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;"><b>Growing pains?</b></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">Then you
need to grow your trees. Fortunately automation can grow a lot quicker than
trees, but it still takes time - it’s not instant.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">While the
trees are growing, you need to prune them and prune them hard especially in the
first few years. Maybe you don’t allow them to fruit at all for the first 3
years - this way you are building a strong infrastructure for the trees so that
they will be stronger and healthier and will produce much more fruit later on.
You may also want to train them to grow into the structure that you want from
the trees when they are mature.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">In
automation, you need to prune your tests - don’t just let them grow and grow
and get all straggly. You need to make sure that each test has earned its place
in your test suite, otherwise get rid of it. This way you will build a strong
infrastructure of worthwhile tests that will make your automation stronger and
healthier over the years, and it will bring good benefits to your organisation.
You need to structure your automation (a good testware architecture) so that it
will give lasting benefits.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;"><b>Feeding, pests and diseases</b></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">Over time,
you need to fertilise the ground, so that the trees have the nourishment they
need to grow to be strong and healthy.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">In
automation, you need to nourish the people who are working on the automation,
so that they will continue to improve and build stronger and healthier
automation. They need to keep learning, experimenting, and be encouraged to
make mistakes - in order to learn from them.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">You need to
deal with pests - bugs - that might attack your trees and damage your fruit.<o:p></o:p></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;"><br /></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">Is this
anything to do with automation? Are there bugs in automated scripts? In testing
tools? Of course there are, and you need to deal with them - be prepared to
look for them and eradicate them.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">What about
diseases? What if one of your trees gets infected with some kind of blight, or
suddenly stops producing good fruit? You may need to chop down that infected
tree and burn it, because it you don’t, this blight might spread to your whole
orchard.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">Does
automation get sick? Actually, a lot of automation efforts seem to decay over
time - they take more and more effort to maintain. technical debt builds up,
and often the automation dies. If you want your automation to live and produce
good results, you might need to take drastic action and re-factor the
architecture if it is causing problems. Because if you don’t, your whole
automation may die.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;"><b>Picking and packing</b></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">What about
picking the fruit? I have seen machines that shake the trees so they can be
scooped up - that might be ok if you are making cider or applesauce, but I
wouldn’t want fruit picked in that way to be in my fruit bowl on the table. Manual
effort is still needed. The machines can help but not do everything (and
someone is driving the machines).<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">Test
execution tools don’t do testing, they just run stuff. The tools can help and
can very usefully do some things, but there are tests that should not be
automated and should be run manually. The tools don’t replace testers, they
support them.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">We need to
pack the fruit so it will survive the journey to market, perhaps building a
structure to hold the fruit so it can be transported without damage.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">Automation
needs to survive too - it needs to survive more than one release of the
application, more than one version of the tool, and may need to run on new
platforms. The structure of the automation, the testware architecture, is what
determines whether or not the automated tests survive these changes well.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;"><b>Marketing, selling, roles and expectations</b></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">It is
important to do marketing and selling for our fruit - if no one buys it, we
will have a glut of rotting fruit on our hands. <o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">Automation
needs to be marketed and sold as well - we need to make sure that our managers
and stakeholders are aware of the value that automation brings, so that they
want to keep buying it and supporting it over time.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">By the way,
the people who are good at marketing and selling are probably not the same
people who are good at picking or packing or pruning - different roles are
needed. Of course the same is true for automation - different roles are needed:
tester, automator, automation architect, champion (who sells the benefits to
stakeholders and managers).<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">Finally, it
is important to set realistic expectations. If your local supermarket buyers have heard that eating your fruit will enable them to leap tall buildings at a single
bound, you will have a very easy sell for the first shipment of fruit, but when
they find out that it doesn’t meet those expectations, even if the fruit is
very good, it may be seen as worthless.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">Setting
realistic expectations for automation is critical for long-term success and for
gaining long-term support; otherwise if the expectations aren’t met, the
automation may be seen as worthless, even if it is actually providing useful
benefits.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;"><b>Summary</b></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">So if you
are growing your own automation, remember these things:<o:p></o:p></span></div>
<div class="MsoListParagraphCxSpFirst" style="mso-list: l0 level1 lfo1; text-indent: -.25in;">
</div>
<ul>
<li><span lang="EN-GB" style="text-indent: -0.25in;">-<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span><span lang="EN-GB" style="text-indent: -0.25in;">it
takes time to do it well</span></li>
<li><span lang="EN-GB" style="text-indent: -0.25in;">-<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span><span lang="EN-GB" style="text-indent: -0.25in;">prepare
the ground</span></li>
<li><span lang="EN-GB" style="text-indent: -0.25in;">-<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span><span lang="EN-GB" style="text-indent: -0.25in;">choose
the right tests to grow</span></li>
<li><span lang="EN-GB" style="text-indent: -0.25in;">-<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span><span lang="EN-GB" style="text-indent: -0.25in;">be
prepared to prune / re-factor</span></li>
<li><span lang="EN-GB" style="text-indent: -0.25in;">-<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span><span lang="EN-GB" style="text-indent: -0.25in;">deal
with pests and diseases (see previous point)</span></li>
<li><span lang="EN-GB" style="text-indent: -0.25in;">-<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span><span lang="EN-GB" style="text-indent: -0.25in;">make
sure you have a good structure so the automation will survive change</span></li>
<li><span lang="EN-GB" style="text-indent: -0.25in;">-<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span><span lang="EN-GB" style="text-indent: -0.25in;">different
roles are needed</span></li>
<li><span lang="EN-GB" style="text-indent: -0.25in;">-<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span><span lang="EN-GB" style="text-indent: -0.25in;">sell
and market the automation and set realistic expectations</span></li>
<li><span lang="EN-GB" style="text-indent: -0.25in;">-<span style="font-family: 'Times New Roman'; font-size: 7pt; line-height: normal;"> </span></span><span lang="EN-GB" style="text-indent: -0.25in;">you
can achieve great results</span></li>
</ul>
<!--[if !supportLists]--><br />
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-GB" style="mso-ansi-language: EN-GB;">I hope that
all of your automation efforts are very fruitful!<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<!--[if gte mso 9]><xml>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
<o:PixelsPerInch>96</o:PixelsPerInch>
</o:OfficeDocumentSettings>
</xml><![endif]-->
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-US</w:LidThemeOther>
<w:LidThemeAsian>X-NONE</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="false"
DefSemiHidden="false" DefQFormat="false" DefPriority="99"
LatentStyleCount="380">
<w:LsdException Locked="false" Priority="0" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 9"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="header"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footer"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index heading"/>
<w:LsdException Locked="false" Priority="35" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of figures"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope return"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="line number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="page number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of authorities"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="macro"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="toa heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 5"/>
<w:LsdException Locked="false" Priority="10" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Closing"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Signature"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="true"
UnhideWhenUsed="true" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Message Header"/>
<w:LsdException Locked="false" Priority="11" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Salutation"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Date"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Block Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Hyperlink"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="FollowedHyperlink"/>
<w:LsdException Locked="false" Priority="22" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Document Map"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Plain Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="E-mail Signature"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Top of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Bottom of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal (Web)"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Acronym"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Cite"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Code"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Definition"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Keyboard"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Preformatted"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Sample"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Typewriter"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Variable"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Table"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation subject"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="No List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Contemporary"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Elegant"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Professional"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Balloon Text"/>
<w:LsdException Locked="false" Priority="39" Name="Table Grid"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Theme"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Level 9"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" QFormat="true"
Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" QFormat="true"
Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" QFormat="true"
Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" QFormat="true"
Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" QFormat="true"
Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" QFormat="true"
Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" SemiHidden="true"
UnhideWhenUsed="true" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="TOC Heading"/>
<w:LsdException Locked="false" Priority="41" Name="Plain Table 1"/>
<w:LsdException Locked="false" Priority="42" Name="Plain Table 2"/>
<w:LsdException Locked="false" Priority="43" Name="Plain Table 3"/>
<w:LsdException Locked="false" Priority="44" Name="Plain Table 4"/>
<w:LsdException Locked="false" Priority="45" Name="Plain Table 5"/>
<w:LsdException Locked="false" Priority="40" Name="Grid Table Light"/>
<w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="46" Name="List Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 6"/>
</w:LatentStyles>
</xml><![endif]-->
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin:0in;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:Calibri;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;}
</style>
<![endif]-->
<!--StartFragment-->
<!--EndFragment--><br />
<div class="MsoNormal">
<br /></div>
Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com5tag:blogger.com,1999:blog-8378239747319086089.post-86485921867502650672015-11-01T09:24:00.000-08:002015-11-03T00:48:29.458-08:00The wrong question: What percentage of tests have you automated?<div class="MsoNormal">
<span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;">At a couple of recent conferences, I became
aware that people are asking the wrong question with regard to automation.
There was an ISTQB survey that asked “How many (what percentage of) test cases
do you automate?”. In talking to a delegate after my talk on automation at
another conference, she said that her manager wanted to know what percentage of
tests were automated; she wasn’t sure how to answer, and she is not alone. It
is quite common for managers to ask this question; the reason it is difficult
to answer is because it is the wrong question.</span></div>
<div class="MsoNormal">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoNormal">
<span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;">Why do people ask this? Probably to get
some information about the progress of an automation effort, usually when
automation is getting started. This is not unreasonable, but this question is
not the right one to ask, because it is based on a number of erroneous
assumptions:</span></div>
<div class="MsoNormal">
<span lang="EN-US" style="text-indent: -18pt;"><span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></span></div>
<div class="MsoNormal">
<span lang="EN-US" style="text-indent: -18pt;"><span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></span></div>
<div class="MsoNormal" style="text-indent: 0px;">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><span lang="EN-US" style="text-indent: -18pt;"><b>Wrong assumption 1) <span style="font-family: "times new roman"; font-size: 7pt;"> </span></b></span><b style="text-indent: -18pt;"><u><span lang="EN-US">All</span></u><span lang="EN-US"> manual tests should be automated.</span></b><span lang="EN-US" style="text-indent: -18pt;"> “What percentage of tests” implies that all existing tests are
candidates for automation, and the percentage will measure progress towards the
“ideal” goal of 100%.</span></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;">It assumes that there is a
single set of tests, and that some of them are manual and some are automated.
Usually this question actually means “What percentage <b style="mso-bidi-font-weight: normal;"><i style="mso-bidi-font-style: normal;">of our existing manual tests</i></b>
are automated?”</span></div>
<div class="MsoListParagraphCxSpMiddle">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;">But your existing manual
tests are not all good candidates for automation – certainly some manual tests
can and should be automated, but not all of them!</span></div>
<div class="MsoListParagraphCxSpMiddle">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;">Examples: if you could
automate “captcha”, then the “captcha” isn’t working, as it’s supposed to tell
the difference between a human and a computer. “Do these colours look nice?” or
“Is this exactly what a real user would do”? And tests that take too long to
automate such as tests that are not run very often or those that. are complex
to automate.</span></div>
<div class="MsoListParagraphCxSpMiddle">
<b style="text-indent: -18pt;"><span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></b></div>
<div class="MsoListParagraphCxSpMiddle" style="text-indent: 0px;">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><span style="text-indent: -18pt;"><span lang="EN-US"><b>Wrong assumption 2)</b> <b>Manual tests are the only candidates for automation.</b></span></span><span lang="EN-US" style="text-indent: -18pt;"> “What percentage of tests” also implies that the only tests worth
automating are existing manual tests, but this is also incorrect. There are
many things that can be done using tools that are impossibly or infeasible to
do when testing manually.</span></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;">Examples: additional
verification or validation of screen objects – are they in the correct state?
When testing manually, you can see what is on the screen, but you may not know
its state or whether the state is displaying correctly.</span></div>
<div class="MsoListParagraphCxSpMiddle">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;">Tests using random inputs
and heuristic oracles, which can be generated in large volume and checked
automatically.</span></div>
<div class="MsoListParagraphCxSpMiddle">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoListParagraphCxSpMiddle" style="text-indent: 0px;">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><span style="text-indent: -18pt;"><span lang="EN-US"><b>Wrong assumption 3) A manual test is the same as an automated test.</b></span></span><span lang="EN-US" style="text-indent: -18pt;"> “What percentage of tests” also assumes that a manual test and an
automated test are the same - but they are not. A manual test consists of a set
of directions for a human being to follow; it may be rather detailed (use
customer R Jones), or it could be quite vague (use an existing customer). <b><i>A manual
test is optimised for a human tester.</i></b> When tests are executed manually,
they may vary slightly each time, and this can be both an advantage (may find
new bugs) and a disadvantage (inconsistent tests, not exactly repeated each
time).</span></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><b style="mso-bidi-font-weight: normal;"><i style="mso-bidi-font-style: normal;"><span lang="EN-US">An automated test should be
optimized for a computer to run</span></i></b><span lang="EN-US">. It should be
structured according to good programming principles, with modular scripts that
call other scripts. It shouldn’t be one script per test, but each test should
use many scripts (most of them shared) and most scripts should be used in many
tests. An automated test is executed in exactly the same way each time, and
this can be an advantage (repeatability, consistency) and a disadvantage (won’t
find new bugs).</span></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;">One manual test may be
converted into 3, 5, 10 or more automated scripts. Take for example a manual
test that starts at the main menu, navigates to a particular screen and does
some tests there, then returns to the main menu. And suppose you have a number
of similar tests for the same screen, say 10. If you have one script per test,
each will do 3 things: navigate to the target area, do tests, navigate back. If
the location of the screen changes, all of those tests will need to be changed
– a maintenance nightmare (especially if there are a lot more than 10 tests)!
Rather, each test should consist of at least 3 scripts: one to navigate to the relevant
screen, one (or perhaps many) scripts to perform specific tests, and one script
to navigate back to the main menu. Note that the same “go to screen” and “return
to main menu” script is used by all of these tests. Then if the screen is
re-located, only 2 scripts need to be changed and all the automated tests will
still work.</span></div>
<div class="MsoListParagraphCxSpMiddle">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;">But now the question is:
how many tests have you automated? Is it the 10 manual tests you started with?
Or should you count automated scripts? Then we have at least 12 but maybe 20. Suppose
you now find that you can very easily add another 5 tests to your original set,
sharing the navigation scripts and 4 of the other scripts. Now you have 15
tests using 13 scripts – how many have you automated? Your new tests never were
manual tests, so have you automated 10 tests (of the original set) or 15? </span></div>
<div class="MsoListParagraphCxSpMiddle">
<b style="text-indent: -18pt;"><span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></b></div>
<div class="MsoListParagraphCxSpMiddle" style="text-indent: 0px;">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><span style="text-indent: -18pt;"><span lang="EN-US"><b>Wrong assumption 4) Progress in automation is linear (like testing).</b></span></span><span lang="EN-US" style="text-indent: -18pt;"> A “what percent completed” measure is fine for an activity that is
stable and “monotonic”, for example running sets of tests manually. But when
you automate a test, especially at first, you need to put a lot of effort in
initially to get the structure right, and the early automated tests can’t reuse
anything because nothing has been built yet. Later automated tests can be written
/ constructed much more quickly than the earlier ones, because there will
(should) be a lot of reusable scripts that can just be incorporated into a new
automated test. So if your goal is to have say 20 tests automated in 2 weeks,
after one week you may only have automated only 5 of those tests, but the other 15
can easily be automated in week 2. So after week 1 you have automated 25% of
the tests, but you have done 50% of the work.</span></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoListParagraphCxSpMiddle">
<span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;">Eventually it should be
easier and quicker to add a new automated test than to run that test manually,
but it does take a lot of effort to get to that point.</span></div>
<div class="MsoListParagraphCxSpLast">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoNormal">
<span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;"><b>Good progress measures. </b>So if these are all reasons NOT to measure
the percentage of manual tests automated, what would be a good automation
progress measure instead? Here are three suggestions:</span></div>
<div class="MsoNormal">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoListParagraphCxSpFirst" style="margin-left: 54.0pt; mso-add-space: auto; mso-list: l1 level1 lfo2; text-indent: -36.0pt;">
<!--[if !supportLists]--><span style="font-family: "verdana" , sans-serif; font-size: large;"><span lang="EN-US" style="mso-bidi-font-family: Cambria; mso-bidi-theme-font: minor-latin;"><span style="mso-list: Ignore;">1)<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]--><span lang="EN-US">Percentage of <b style="mso-bidi-font-weight: normal;">automatable</b> tests that have been
automated. Decide first which tests are suitable for automation, and/or that you
want to have as automated tests, and measure the percentage automated compared
to that number, having taken out tests that should remain manual and tests that
we don’t want to automate now. This can be done for a sprint, or for a longer
time frame (or both). As Alan Page says, "Automate 100% of the tests that should be automated."</span></span></div>
<div class="MsoListParagraphCxSpMiddle" style="margin-left: 54.0pt; mso-add-space: auto;">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoListParagraphCxSpLast" style="margin-left: 54.0pt; mso-add-space: auto; mso-list: l1 level1 lfo2; text-indent: -36.0pt;">
<!--[if !supportLists]--><span style="font-family: "verdana" , sans-serif; font-size: large;"><span lang="EN-US" style="mso-bidi-font-family: Cambria; mso-bidi-theme-font: minor-latin;"><span style="mso-list: Ignore;">2)<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]--><span lang="EN-US">EMTE: Equivalent Manual Test
Effort: Keep track of how much time a set of automated tests would have taken
if they had been run manually. Each time those tests are run (automatically),
you “clock up” the equivalent of that manual effort. This shows that automation
is running tests now that are no longer run manually, and this number should
increase over time as more tests are automated.</span></span></div>
<div class="MsoNormal">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoListParagraph" style="margin-left: 54.0pt; mso-add-space: auto; mso-list: l1 level1 lfo2; text-indent: -36.0pt;">
<!--[if !supportLists]--><span style="font-family: "verdana" , sans-serif; font-size: large;"><span lang="EN-US" style="mso-bidi-font-family: Cambria; mso-bidi-theme-font: minor-latin;"><span style="mso-list: Ignore;">3)<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]--><span lang="EN-US">Coverage: With automation, you
can run more tests, and therefore test areas of the application that there was
never time for when the testing was done manually. This is a partial measure of
one aspect of the thoroughness of testing (and has its own pitfalls), but is a
useful way to show that automation is now helping to test more of the system.</span></span></div>
<div class="MsoNormal">
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoNormal">
<span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;"><b>Conclusion</b> So if your manager asks you “What percent of
the tests have you automated?” you need to ask something like: Percent of what?
Out of existing tests that could be automated or that we decide to automate?
What about additional tests that would be good to automate that we aren’t doing
now? Do you want to know about progress in time towards our automation goal, or
literally only the tests, as this will be different because automated tests are
structured differently to manual tests.</span></div>
<div class="MsoNormal">
<span lang="EN-US" style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoNormal">
<span style="font-family: "verdana" , sans-serif; font-size: large;">It might be a good idea to find out why he
or she has asked that question – what is it that they are trying to see? They
need to have some visibility for automation progress, and it is up to you to
agree something that would be useful and helpful, honest and reasonably easy to
measure. Good luck! And let me know how you measure your progress in
automation!</span><br />
<span style="font-size: large;"><span style="font-family: "verdana" , sans-serif;"><br /></span>
<span style="font-family: "verdana" , sans-serif;">If you want more advice on automation, see the wiki that I am doing with Seretta Gamba at <a href="http://testautomationpatterns.org/">TestAutomationPatterns.org</a>.</span></span><br />
<span style="font-family: "verdana" , sans-serif; font-size: large;"><br /></span></div>
<div class="MsoNormal">
<span lang="EN-US"><br /></span></div>
Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com11tag:blogger.com,1999:blog-8378239747319086089.post-53507373255794330552014-04-08T08:10:00.000-07:002014-04-08T08:10:25.189-07:00Testers should learn to code?It seems to be the "perceived wisdom" these days that if testers want to have a job in the future, they should learn to write code. Organisations are recruiting "developers in test" rather than testers. Using test automation tools (directly) requires programming skills, so the testers should acquire them, right?<br />
<br />
I don't agree, and I think this is a dangerous attitude for testing in general.<br />
<br />
Here's a story of two testers:<br />
<br />
<ul>
<li><span style="font-family: Cambria; font-size: 12pt;">Les has degree in Computer Science, started out in a traditional test team, and now
works in a multi-disciplinary agile team. Les is a person who likes to turn a
hand to whatever needs doing, and enjoys a technical challenge. Les is very
happy to write code, and has recently starting coding for a recently acquired
test automation tool, making sure that good programming practices are applied
to the testware and test code. Les is very happy as a developer-tester.</span></li>
<li><span style="font-family: Cambria; font-size: 12pt;">Fran came into testing through the business. Started out being a user
who was more interested in any new release from IT than the other users, so
became the “first user”. Got drawn into the user acceptance test group and
enjoyed testing – found things that the technical people missed, due to a good
business background. With training in testing techniques, Fran became a really
good tester, providing great value to the organization. Probably saved them
hundreds of thousand pounds a year by advising on new development and testing
from a user perspective. Fran never wanted anything to do with code.</span></li>
</ul>
<br />
<span lang="EN-US" style="font-family: Cambria; font-size: 12.0pt; mso-ansi-language: EN-US; mso-ascii-theme-font: minor-latin; mso-bidi-font-family: "Times New Roman"; mso-bidi-theme-font: minor-bidi; mso-fareast-font-family: Cambria; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;"><span lang="EN-US" style="font-size: 12pt;">
<!--[if gte mso 9]><xml>
<o:DocumentProperties>
<o:Template>Normal.dotm</o:Template>
<o:Revision>0</o:Revision>
<o:TotalTime>0</o:TotalTime>
<o:Pages>1</o:Pages>
<o:Words>43</o:Words>
<o:Characters>247</o:Characters>
<o:Company>Dorothy Graham</o:Company>
<o:Lines>2</o:Lines>
<o:Paragraphs>1</o:Paragraphs>
<o:CharactersWithSpaces>303</o:CharactersWithSpaces>
<o:Version>12.0</o:Version>
</o:DocumentProperties>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
</o:OfficeDocumentSettings>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:WordDocument>
<w:Zoom>0</w:Zoom>
<w:TrackMoves>false</w:TrackMoves>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:DrawingGridHorizontalSpacing>18 pt</w:DrawingGridHorizontalSpacing>
<w:DrawingGridVerticalSpacing>18 pt</w:DrawingGridVerticalSpacing>
<w:DisplayHorizontalDrawingGridEvery>0</w:DisplayHorizontalDrawingGridEvery>
<w:DisplayVerticalDrawingGridEvery>0</w:DisplayVerticalDrawingGridEvery>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:DontGrowAutofit/>
<w:DontAutofitConstrainedTables/>
<w:DontVertAlignInTxbx/>
</w:Compatibility>
</w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" LatentStyleCount="276">
</w:LatentStyles>
</xml><![endif]-->
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman";
mso-ascii-font-family:Cambria;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:"Times New Roman";
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Cambria;
mso-hansi-theme-font:minor-latin;
mso-ansi-language:EN-US;}
</style>
<![endif]-->
<!--StartFragment-->
<!--EndFragment--></span></span><br />
<div class="MsoNormal">
<span lang="EN-US">What will happen when the CEO hears:
“Testers should learn to code”? Les’s job is secure, but what about Fran? I suspect that Fran is already feeling less valued by the organisation and is worried about job security, in spite of having provided a great service for years as an excellent software tester.</span></div>
<div class="MsoNormal">
<span lang="EN-US"><br /></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
So what’s wrong with testers who write
code?</div>
<div class="MsoListParagraphCxSpFirst" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
</div>
<ul>
<li><span lang="EN-US" style="text-indent: -18pt;">-<span style="font-family: 'Times New Roman'; font-size: 7pt;">
</span></span><span lang="EN-US" style="text-indent: -18pt;">absolutely nothing</span></li>
<li><span lang="EN-US" style="text-indent: -18pt;">-<span style="font-family: 'Times New Roman'; font-size: 7pt;">
</span></span><span lang="EN-US" style="text-indent: -18pt;">for testers who want to code,
who enjoy it, who are good at it</span></li>
<li><span lang="EN-US" style="text-indent: -18pt;">-<span style="font-family: 'Times New Roman'; font-size: 7pt;">
</span></span><span lang="EN-US" style="text-indent: -18pt;">for testers in agile teams</span></li>
</ul>
<!--[if !supportLists]--><br />
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Why is this a dangerous attitude for
testing in general?</div>
<div class="MsoListParagraphCxSpFirst" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
</div>
<ul>
<li><span lang="EN-US" style="text-indent: -18pt;">-<span style="font-family: 'Times New Roman'; font-size: 7pt;"> </span></span><span lang="EN-US" style="text-indent: -18pt;">it reads as “<u>all</u> testers
should write code” and is taken as that by managers who are looking to get rid of people</span></li>
<li><span lang="EN-US" style="text-indent: -18pt;">-<span style="font-family: 'Times New Roman'; font-size: 7pt;">
</span></span><span lang="EN-US" style="text-indent: -18pt;">not all testers will be good at
it or want to become developers (maybe that's why they went into testing)</span></li>
<li><span lang="EN-US" style="text-indent: -18pt;">-<span style="font-family: 'Times New Roman'; font-size: 7pt;">
</span></span><span lang="EN-US" style="text-indent: -18pt;">it implies that “the only good tester
is one who can write code”</span></li>
<li><span lang="EN-US" style="text-indent: -18pt;">-<span style="font-family: 'Times New Roman'; font-size: 7pt;">
</span></span><span lang="EN-US" style="text-indent: -18pt;">it devalues testing skills
(now want coders, not [good] testers. In fact, if coders can test, why do we need
specialist testers anyway?)</span></li>
<li><span lang="EN-US" style="text-indent: -18pt;"><span lang="EN-US" style="text-indent: -18pt;">-<span style="font-family: 'Times New Roman'; font-size: 7pt;"> </span></span><span lang="EN-US" style="text-indent: -18pt;">tester-developers may "go native" and be pushed into development, so we lose more testing skills</span></span></li>
<li><span lang="EN-US" style="text-indent: -18pt;">-<span style="font-family: 'Times New Roman'; font-size: 7pt;"> </span></span><span lang="EN-US" style="text-indent: -18pt;">it's not right to force good testers
out of our industry</span></li>
</ul>
<div style="text-indent: -24px;">
<div style="text-indent: 0px;">
So I say, let's stand up for testing skills, and for non-developer testers!</div>
<div>
<br /></div>
</div>
<br />Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com1tag:blogger.com,1999:blog-8378239747319086089.post-81727392506485659852012-06-21T10:57:00.003-07:002012-06-21T10:57:37.325-07:00Is it dangerous to measure ROI for test automation?<br />
<div class="MsoNormal" style="mso-layout-grid-align: none; mso-pagination: none; text-autospace: none;">
<span class="Apple-style-span" style="font-family: Georgia; font-size: 21px;">I have been a fan of trying to show ROI for
automation in a way that is simple enough to understand easily, and provides a
way of showing the benefit of automation compared to its cost.</span></div>
<div class="MsoNormal" style="mso-layout-grid-align: none; mso-pagination: none; text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="mso-layout-grid-align: none; mso-pagination: none; text-autospace: none;">
<span lang="EN-US" style="font-family: Georgia; font-size: 16pt;">I have been developing a spreadsheet with sample
calculations (sparked off initially by Molly Mahai, including an example from
Mohacsi & Beer's chapter in the new book, and some other people have also
been influential - thanks). I have sent this spreadsheet out to around 300
people, including most who have attended my automation tutorials.<o:p></o:p></span></div>
<div class="MsoNormal" style="mso-layout-grid-align: none; mso-pagination: none; text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="mso-layout-grid-align: none; mso-pagination: none; text-autospace: none;">
<span lang="EN-US" style="font-family: Georgia; font-size: 16pt;">The problem with showing ROI is that it's hard to
quantify some of the important factors, so I have focused on showing ROI using
what is the most straight-forward to quantify - people's effort and/or time.
This can be converted into money, using some kind of salary cost, if desired,
and either effort or money can then be plugged into the ROI calculation =
(benefit - cost) / cost.<o:p></o:p></span></div>
<div class="MsoNormal" style="mso-layout-grid-align: none; mso-pagination: none; text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="mso-layout-grid-align: none; mso-pagination: none; text-autospace: none;">
<span lang="EN-US" style="font-family: Georgia; font-size: 16pt;">So basically, this is showing how a set of tests
requires less human effort when those tests are automated than would be
required if those same tests were run manually.<o:p></o:p></span></div>
<div class="MsoNormal" style="mso-layout-grid-align: none; mso-pagination: none; text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="mso-layout-grid-align: none; mso-pagination: none; text-autospace: none;">
<span lang="EN-US" style="font-family: Georgia; font-size: 16pt;">This is great, right? We have a clear business
case showing savings from automation than are greater than the cost of
developing the automation, so our managers should be happy.<o:p></o:p></span></div>
<div class="MsoNormal" style="mso-layout-grid-align: none; mso-pagination: none; text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="mso-layout-grid-align: none; mso-pagination: none; text-autospace: none;">
<span lang="EN-US" style="font-family: Georgia; font-size: 16pt;">Recently, however, I have been wondering whether
this approach can be dangerous.<o:p></o:p></span></div>
<div class="MsoNormal" style="mso-layout-grid-align: none; mso-pagination: none; text-autospace: none;">
<br /></div>
<div class="MsoNormal">
<span lang="EN-US" style="font-family: Georgia; font-size: 16pt;">If we justify automation ONLY in terms of reduced
human effort, we run the risk of implying that the tools can replace the
people, and this is definitely not true! Automation supports testing, it does
not replace testers. Automation should free the testers to be able to do better
testing, designing better tests, having time to do exploratory testing, etc.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-US" style="font-family: Georgia; font-size: 16pt;">So should we abandon ROI for automation? I don’t
think that’s a good idea – we should be gaining business benefit from automation, and we should be able to show this.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-US" style="font-family: Georgia; font-size: 16pt;">Scott Barber told me about Social ROI – a way of
quantifying some of the intangible benefits – I like this but haven’t yet seen
how to incorporate it into my spreadsheet.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-US" style="font-family: Georgia; font-size: 16pt;">In our book, there are many success stories of
automation where ROI was not specifically calculated, so maybe ROI isn’t as
critical as it may have seemed.<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span lang="EN-US" style="font-family: Georgia; font-size: 16pt;">I don’t know the answer here – these are just my
current thoughts!<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com6tag:blogger.com,1999:blog-8378239747319086089.post-54307820357703763522012-01-25T07:53:00.000-08:002012-01-25T08:38:34.655-08:00How long does it take to write a book?<a href="http://4.bp.blogspot.com/-R3JfD1J_KTU/TyAoVe1sqDI/AAAAAAAAACU/pDdFpsnZdP8/s1600/book%2Bhours.tiff" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 320px; height: 133px;" src="http://4.bp.blogspot.com/-R3JfD1J_KTU/TyAoVe1sqDI/AAAAAAAAACU/pDdFpsnZdP8/s320/book%2Bhours.tiff" border="0" alt="" id="BLOGGER_PHOTO_ID_5701601477771700274" /></a><p class="MsoNormal" style="margin-top:6.0pt;margin-right:0cm;margin-bottom:6.0pt; margin-left:0cm"><span lang="EN-US">A number of people have asked me this, since our new <a href="http://www.dorothygraham.co.uk/automationExperiences/index.html">book</a> is now out.</span></p><p class="MsoNormal" style="margin-top:6.0pt;margin-right:0cm;margin-bottom:6.0pt; margin-left:0cm">This book took us 2 and a half years. This doesn’t include the effort put in by the case study authors and other contributors, so this book represents a lot of work! What exactly had we been doing all that time? I wondered that too, so here is where the time went.</p> <p class="MsoNormal" style="margin-top:6.0pt;margin-right:0cm;margin-bottom:6.0pt; margin-left:0cm"><span lang="EN-US">In August 2009, I have my first note of our plan to solicit contributions for a new book on automation experience. We sent emails, put a call for contributions on my web site, and talked to people at conferences, and began gathering potential contributions.</span></p> <p class="MsoNormal" style="margin-top:6.0pt;margin-right:0cm;margin-bottom:6.0pt; margin-left:0cm"><span lang="EN-US">I started keeping track of the hours we spent from December 2009. We had a big initial “push” (the first peak on the graph) and produced a “protobook” – 4 chapters with an introduction. We were sure this would be snapped up by a publisher! </span></p> <p class="MsoNormal" style="margin-top:6.0pt;margin-right:0cm;margin-bottom:6.0pt; margin-left:0cm"><span lang="EN-US">We submitted to the publisher of our previous book in mid-February, but initially they weren’t very keen! This was a blow, as we were convinced this would be a great book! I tried several other publishers over the next few months, and got rejected; I continued to try and convince Pearson/Addison Wesley that they should publish our book. </span></p> <p class="MsoNormal" style="margin-top:6.0pt;margin-right:0cm;margin-bottom:6.0pt; margin-left:0cm"><span lang="EN-US">They eventually relented in July and we signed a contract. We worked steadily on the book over the rest of that year, a total of around 400 hours between us. The complete draft manuscript, ready for independent review was <span class="Apple-style-span" style="font-size:100%;">due</span> on the 15<sup>th</sup> April, and the final manuscript on the 15<sup>th</sup> October. “No problem”, we thought.</span></p><p class="MsoNormal" style="margin-top:6.0pt;margin-right:0cm;margin-bottom:6.0pt; margin-left:0cm"><span lang="EN-US">However, we found that we did need to work at a more intense level - we spent another 300 hours to the end of April (the double peak with some time out in February), and another 300 hours to the end of 2011. The final peak was editing the final page proofs and doing the index, more work that we had anticipated at that stage. The total for us comes to just under 1000 hours. We don't know how much time the contributors spent, but it their time was equivalent to ours, the collaboration represents around 2000 hours of time - that's more than one working year.</span></p><div><span lang="EN-US" style="font-family:Cambria;mso-fareast-font-family: Cambria;mso-bidi-Times New Roman";mso-ansi-language:EN-US; mso-fareast-language:EN-USfont-family:";"><span class="Apple-style-span" style="font-size:100%;"> <!--[if gte mso 9]><xml> <o:documentproperties> <o:template>Normal.dotm</o:Template> <o:revision>0</o:Revision> <o:totaltime>0</o:TotalTime> <o:pages>1</o:Pages> <o:words>22</o:Words> <o:characters>129</o:Characters> <o:company>Dorothy Graham</o:Company> <o:lines>1</o:Lines> <o:paragraphs>1</o:Paragraphs> <o:characterswithspaces>158</o:CharactersWithSpaces> <o:version>12.0</o:Version> </o:DocumentProperties> <o:officedocumentsettings> <o:allowpng/> </o:OfficeDocumentSettings> </xml><![endif]--><!--[if gte mso 9]><xml> <w:worddocument> <w:zoom>0</w:Zoom> <w:trackmoves>false</w:TrackMoves> <w:trackformatting/> <w:punctuationkerning/> <w:drawinggridhorizontalspacing>18 pt</w:DrawingGridHorizontalSpacing> <w:drawinggridverticalspacing>18 pt</w:DrawingGridVerticalSpacing> <w:displayhorizontaldrawinggridevery>0</w:DisplayHorizontalDrawingGridEvery> <w:displayverticaldrawinggridevery>0</w:DisplayVerticalDrawingGridEvery> <w:validateagainstschemas/> <w:saveifxmlinvalid>false</w:SaveIfXMLInvalid> <w:ignoremixedcontent>false</w:IgnoreMixedContent> <w:alwaysshowplaceholdertext>false</w:AlwaysShowPlaceholderText> <w:compatibility> <w:breakwrappedtables/> <w:dontgrowautofit/> <w:dontautofitconstrainedtables/> <w:dontvertalignintxbx/> </w:Compatibility> </w:WordDocument> </xml><![endif]--><!--[if gte mso 9]><xml> <w:latentstyles deflockedstate="false" latentstylecount="276"> </w:LatentStyles> </xml><![endif]--> <!--[if gte mso 10]> <style> /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"Times New Roman"; mso-ascii-font-family:Cambria; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Cambria; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:EN-US;} </style> <![endif]--> <!--StartFragment--> </span><p class="MsoNormal" style="margin-top: 6pt; margin-right: 0cm; margin-bottom: 6pt; margin-left: 0cm; "><span class="Apple-style-span" style="font-size:100%;">We enjoyed working on this book and reading all the stories as they came in. There are many useful lessons, some heartfelt pain, and many gratifying successes among the stories.</span></p><p class="MsoNormal" style="margin-top: 6pt; margin-right: 0cm; margin-bottom: 6pt; margin-left: 0cm; "><span class="Apple-style-span" style="font-size:100%;">You can follow the book on Twitter on @AutExpBook. The book tweets tips and good points every few days!</span></p><p class="MsoNormal" style="margin-top:6.0pt;margin-right:0cm;margin-bottom:6.0pt; margin-left:0cm">Thanks to all the book contributors, and to co-author <a href="http://www.grove.co.uk">Mark Fewster</a>.</p> <!--EndFragment--></span></div>Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com0tag:blogger.com,1999:blog-8378239747319086089.post-36983930493605616942011-02-05T04:48:00.001-08:002011-02-05T05:03:43.774-08:00Part 3. Certification schemes do not assess tester skill?(Continuing from Parts 1 and 2, certification is evil? and some history about ISTQB)<div><!--StartFragment--> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">Now I would like to say something about the criticism that the current schemes do not assess tester skill. I am thinking mainly of the Foundation level, as </span><span class="Apple-style-span" style=" ;font-family:Times;font-size:21px;">this seems to be where the criticism is mainly directed, and </span><span class="Apple-style-span" style=" ;font-family:Times;font-size:21px;">I am more familiar with that than the Advanced Levels.</span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">In the main, I agree. There is a modicum of skill needed in testing techniques to be able to answer multiple-choice questions about them, but that is not the same as being able to test well in practice. And I also agree that the current ISTQB Foundation level is based more on learning facts than on practicing the craft. Why is that? Because the current scheme was designed to meet a different need, a basic ignorance about testing in general; it was not designed to assess testing skill.<o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">I feel it is unfair for people to criticize a scheme because it doesn’t conform to what they think assessment of testers should be today, when the scheme was never meant to be that kind of assessment. It’s a bit like criticizing a bicycle for not powering you up a hill by itself – it’s not intended to do that.</span><span lang="EN-US" style=" font-family:Georgia;mso-bidi-font-family:Georgia;font-size:16.0pt;"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">When we were developing the original Foundation Syllabus with ISEB, I remember many discussions about what was possible and practical, including ways of assessing tester skills beyond basic concepts and vocabulary: <o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;text-indent:36.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none"><span lang="EN-US" style="font-family:Times;mso-bidi-font-family:Times;font-size:16.0pt;">- interviews? <o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;text-indent:36.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none"><span lang="EN-US" style="font-family:Times;mso-bidi-font-family:Times;font-size:16.0pt;">- looking at projects submitted from their workplace?<o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;text-indent:36.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none"><span lang="EN-US" style="font-family:Times;mso-bidi-font-family:Times;font-size:16.0pt;">- observing them at work? <o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;text-indent:36.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none"><span lang="EN-US" style="font-family:Times;mso-bidi-font-family:Times;font-size:16.0pt;">- substantial pieces of testing work to be done in a supervised exam-like setting or as a project to be given in within a time frame?<o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">All of these have significant challenges. For example, how to ensure fairness if different people interview in different ways, ensuring that the work being assessed was actually done by the person submitting it, time commitment, scope and fair comparison of observation at work, designing a testing task that would be applicable to people from different industries. <o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">We decided that the place to start was with something very basic that could be built on later, something that would try to cover common ground that all testers should know and build on - hence it was called "Foundation".<o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">Criticism is good – we all learn by having our ideas challenged. But current qualification schemes are not “evil”, even if there are aspects of their current implementation that are not as they should be. So let’s take the context of the certification schemes into account, and remember that what may be ideal for today was not possible 12 or 13 years ago.<o:p></o:p></span></p> <!--EndFragment--> </div>Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com14tag:blogger.com,1999:blog-8378239747319086089.post-6406143807597436842011-02-05T04:44:00.000-08:002011-02-05T04:59:22.869-08:00Part 2. A bit of history about ISTQB certification<!--StartFragment--> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">(Continuing from Part 1 where I was surprised at reactions to certification as "evil")</span></p><p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">In the early 1990s, software testing was not a respected profession; in fact many thought of testing at best as a “necessary evil” (if they thought of testing at all!). There were few people who specialized in testing, and it was seen as a “second-class” activity. There was a general perception that testing was easy, that anyone could do it, and that you were rather strange if you liked it. </span></p><p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">It was then that I decided to specialize in testing, seeing great scope for improvement in testing activities in industry, not only in imparting fundamental knowledge about testing (basic principles and techniques), but also in improving the view testers had of themselves, and the perceptions of testers in their companies. I developed training courses in testing, and began Grove Consultants, named after my house in Macclesfield. One of my most popular talks at the time was called “Test is a four-letter word”, reflecting the prevailing culture about testing. The UK’s Specialist Interest Group in Software Testing (SIGIST) was started by Geoff Quentin in 1989, and was the only gathering of testers in the UK.</span><span lang="EN-US" style="font-family:Georgia;mso-bidi-font-family:Georgia;font-size:16.0pt;"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">It was into this context that the initiative to create a qualification for testers was born. Although I was not the initiator, I was involved from the first meeting (called by Paul Gerrard at a STARWest conference in 1997) and the earliest working groups that developed the first Foundation Syllabus, donating many hours of time to help progress this effort. This work was carried out with support from ISEB (Information Systems Examination Board) of the British Computer Society (meeting rooms, travel expenses and admin help). The testing qualification was modeled on ISEB’s qualifications in Project Management and Information Systems Infrastructure, which were perceived as useful and valuable in their respective sectors. One of the aims was to give people a common vocabulary to talk about testing, since at the time people seemed to be using many different terms for the same thing.</span><span lang="EN-US" style="font-family:Georgia;mso-bidi-font-family:Georgia;font-size:16.0pt;"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">The first course based on the ISEB Foundation Syllabus was given in October 1998 and the first Foundation Certificates in Software Testing were awarded at that time. But an important aspect of the scheme was that it was not necessary to take a training course in order to get the qualification; you could just take the exam. (Some other schemes were based on attendance at courses which seemed too training-provider profit-oriented to us.)</span><span lang="EN-US" style="font-family:Georgia; mso-bidi-font-family:Georgia;font-size:16.0pt;"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">The success of the Foundation qualification took everyone by surprise – there seemed to be a hunger for something that gave testers more respect, both for themselves and from their employers. It also gave testers a common vocabulary and more confidence in their work. The Foundation qualification was meeting its main objective of “removing the bottom layer of ignorance” about software testing.</span><span lang="EN-US" style="font-family:Georgia;mso-bidi-font-family:Georgia;font-size:16.0pt;"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">Work then began on extending the ISEB qualification to a more advanced level (which became the ISEB Practitioner qualification) and also to extending it to other countries, as news of the qualification spread in the international community. I was a facilitator at the meeting that formed ISTQB in 2001 in Sollentuna, Sweden.</span><span lang="EN-US" style="font-family:Georgia;mso-bidi-font-family:Georgia;font-size:16.0pt;"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">I became a member of the working party that produced the first ISTQB Foundation Syllabus in 2005, and I am amazed at how ISTQB has grown; it has certainly changed over the past six years. While working on the update to the Foundation book, I was rather surprised and disappointed at the apparent lack of review the before release of the 2010 Syllabus.<o:p></o:p></span></p><p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">In Part 3 I return to the criticism that current certification schemes do not address tester skill.</span></p> <!--EndFragment-->Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com9tag:blogger.com,1999:blog-8378239747319086089.post-79470513151950507832011-02-05T04:40:00.000-08:002011-02-05T04:47:59.633-08:00Part 1. Certification is evil?<!--StartFragment--> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><b style="mso-bidi-font-weight:normal"><span lang="EN-US" style="font-family:Times;mso-bidi-font-family:Times;font-size:16.0pt;">Certification is evil?<o:p></o:p></span></b></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">This is a 3-part blog (because it ended up rather longer than I anticipated when I started).<o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">1: My reaction to “certification is evil”<o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">2: A bit of history about ISTQB certification<o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">3: The criticism that current schemes do not assess tester skill<o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">I enjoyed reading the Software Testing Club's publication "A Tester is for Life, not just for Christmas" (http://blog.softwaretestingclub.com/2010/11/a-tester-is-for-life-not-just-for-christmas/) which consists of interviews with a number of people, all of whom answered the same questions.</span><span lang="EN-US" style="font-family: Georgia;mso-bidi-font-family:Georgia;font-size:16.0pt;"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">But I was quite shocked at the strength of feeling shown by quite a few against certification. The question asked was "What are your views on Testing Certification Schemes?" Although there are a few older testing qualifications from the US, it seems that most people assumed the ISTQB qualifications were being asked about.<o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">A couple of people called the ISTQB scheme a "scam", one person said it was "downright dangerous", and someone said it was "the devil in disguise" and talked about the "evil axis" of certification institutes, training providers and HR departments. Some mentioned "profit" and "money-making". Several criticized any exam based on memorizing, saying that current exams are too easy, even "trivial"; one said the exam "has no value". <o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">The basis of this criticism (if there is a rational basis) seems to be that current schemes do not assess tester skills, i.e. how well you can actually do testing. (I will return to this point at the end of this blog.)</span><span lang="EN-US" style="font-family: Georgia;mso-bidi-font-family:Georgia;font-size:16.0pt;"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">(On the other hand, many said that the current schemes were a good starting point for people new to testing, that it gave knowledge of basic terminology, helped them get their first job, and can demonstrate that you are serious enough about testing to take an exam.)</span><span lang="EN-US" style="font-family:Georgia;mso-bidi-font-family:Georgia;font-size:16.0pt;"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:16.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">I was not surprised to find people who didn't like the ISTQB certification schemes - I have been to conferences over the past few years where this has become clear. But some of the "opposition" to certification is, I believe, based on a false premise about what it was intended to be, and this I feel is somewhat unfair.</span><span lang="EN-US" style="font-family:Georgia;mso-bidi-font-family:Georgia;font-size:16.0pt;"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:12.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">Just to state my own position: I am not involved in any certification scheme at the moment. I no longer provide accredited ISTQB training courses in software testing. However, I was involved in writing the initial Foundation Syllabus, and I am currently in the process of updating our book that supports the ISTQB Foundation Syllabus.</span><span lang="EN-US" style="font-family:Georgia;mso-bidi-font-family:Georgia;font-size:16.0pt;"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:12.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">It seems to me that people have lost sight of (or perhaps were never aware of) the origins and purpose of the current certification scheme, at least the ISTQB Foundation (which is the one I am most familiar with).</span><span lang="EN-US" style="font-family: Georgia;mso-bidi-font-family:Georgia;font-size:16.0pt;"><o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:12.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">I have been involved in software testing all my working life, for over 40 years. The situation before the current ISTQB certification began was very different to what testing is like today!<o:p></o:p></span></p> <p class="MsoNormal" style="margin-bottom:12.0pt;mso-pagination:none;mso-layout-grid-align: none;text-autospace:none"><span lang="EN-US" style="font-family: Times;mso-bidi-font-family:Times;font-size:16.0pt;">See my next entry for a bit of history about ISTQB.</span></p> <!--EndFragment-->Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com2tag:blogger.com,1999:blog-8378239747319086089.post-25964298342404814892010-10-07T05:03:00.000-07:002010-10-13T03:03:42.762-07:00Automation benefit measured by EMTE - good or bad?<div><b><span class="Apple-style-span" style="font-weight: normal; ">Being able to run tests that we would not have had time to run manually <b>is</b> one of the benefits of automated testing; we should be able to measure this benefit in some way.</span></b></div><div><b><br /></b></div><div><b>What is EMTE?</b></div><div>EMTE stands for "Equivalent Manual Test Effort" and is a way of measuring the benefit of running automated tests.</div><div><br /></div><div>If an automated test (A) would have taken 4 hours to run manually, then its EMTE is 4 hours; another test (B) that would have taken 7.5 hours to run manually has an EMTE of 7.5 hrs.</div><div><br /></div><div>In a test cycle, if Test A is run five times, and Test B is run twice, then the EMTE for that cycle is 5*4 hrs + 2*7.5 hrs = 20 + 15 = 35 hours EMTE.</div><div><br /></div><div><b>What is EMTE used for?</b></div><div>EMTE can be used as a way to measure a benefit of test automation (automated running of functional tests). </div><div><br /></div><div>When tests are automated, they can be run in much less time than they could be run manually. Our tests A and B may be able to run in 5 and 10 minutes respectively, for example. So we can achieve "4 hours' worth" of manual testing in 5 minutes of automated testing. Whenever we run Test A, we can "clock up" 4 hours of EMTE.</div><div><br /></div><div><b>Is EMTE a good thing?</b></div><div>Yes, because it is a way to show the benefit of automation. </div><div><br /></div><div>Costs (of automation as well as other things) tend to become visible by themselves - managers see that people are spending time on the automation. But what is the benefit of this automation? If you don't make benefits visible to managers, there is a risk that they will not see the benefits, and may eventually conclude that there are no benefits. EMTE is one way to make an automation benefit visible.</div><div><br /></div><div><b>So how could it be a bad thing?</b></div><div>I have had discussions with a couple of people recently (thanks Julian and Wade) about abusing EMTE, and yes, it can be abused (as any metric can). Here is how it could be mis-used:</div><div><br /></div><div><blockquote>"Test A takes 5 minutes, so let's run it 12 times every hour for 2 hours. This gives 24*4 hours of EMTE = 96 hours. This will make us look really great!"</blockquote></div><div><br /></div><div>The problem is that after the first run, the other 23 runs are being done just for the sake of looking good, not for a valuable benefit in terms of running that test. This is an abuse of EMTE, and is a bad thing.</div><div><br /></div><div><b>What to do about it?</b></div><div>Use EMTE (and other measures of the benefit of test automation) sensibly. </div><div><br /></div><div>Perhaps only "count" EMTE once a day, however many times a test is run? (e.g. in continuous integration testing)</div><div><br /></div><div>In what other ways can the benefit of automation be shown? (e.g. more coverage, freeing testers to find more bugs, number of times tests are run, more variety of data used?)</div><div><br /></div><div>Have you encountered the abuse of this (or other automation) measures? How have you solved the problem? (Please comment, thanks!)</div><div><br /></div><div>Dot Graham</div><div><br /></div><div><br /></div>Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com9tag:blogger.com,1999:blog-8378239747319086089.post-5351667889423521872010-05-27T05:13:00.000-07:002010-05-27T05:54:39.199-07:00Automated tests should find bugs? No!I have recently been having what seems like the same discussion with a number of different people. <div><br /></div><div>"Automated tests should find bugs" or "find more bugs" is a very common misconception. Basically this says that finding bugs is a valid objective for automation. I don't agree - I think this is generally a very poor objective for test automation. The reasons are to do with the nature of testing and of automation.</div><div><br /></div><div>Testing is an indirect activity, not a direct one. We don't just "do testing", we "test something". (Testing is like a transitive verb which requires an object to be grammatically correct.) This is why the quality of the software we test has a large impact on testing: if a project is delayed because testing finds lots of bugs, we shouldn't blame the testing! (I hope that most people realize this by now, but do have my doubts at times!) Testing is not responsible for the bugs inserted into software any more than the sun is responsible for creating dust in the air. Testing is a way of assessing the software, whatever the quality of that software is.</div><div><br /></div><div>Test automation is doubly indirect. We don't "do automation", we "automate tests that test something".</div><div><br /></div><div>Automation is a mechanism for executing tests, whatever the quality of those tests are (that assess the software, whatever the quality of that software is).</div><div><br /></div><div>Bugs are found by <b>tests</b>, not by automation. </div><div><br /></div><div>It is just as unfair to hold automation responsible for the quality of the test, as it is to hold the testing responsible for the quality of the software.</div><div><br /></div><div>This is why "finding bugs" is not a good objective for test automation. But there are a couple more points to make.</div><div><br /></div><div>Most people automate regression tests. Regression tests by their nature are tests that have been run before and are run many times. The most likely time a test will find a bug is the first time it is run, so regression tests are less likely to find bugs than say exploratory tests. In addition the same test run for a second time (and more) is even less likely to find a bug. Hence the main purpose of regression tests (whether automated or not) is to give confidence that what worked before is still working (to the extent that the tests cover the application).</div><div><br /></div><div>Of course, this is complicated by the fact that because automated tests can be run more often, they do sometimes find bugs that wouldn't have been found otherwise. But even this is not because those tests are <b>automated</b>, it is because they were <b>run</b>. If the tests that are automated <span class="Apple-style-span" style="font-weight: bold; ">had <span class="Apple-style-span" style="font-weight: normal; ">been run manually, then those manual tests would have found the bugs. So even this bug-finding is a characteristic of the tests, not of the automation.</span></span></div><div><br /></div><div>So should your goal for automation be to find bugs? No! At least not if you are planning to automate your existing regression tests.</div><div><br /></div><div>I have been wondering if there may be two exceptions: Model-Based Testing (where tests are generated from a model), and mature Keyword-driven automation, i.e. using a Domain Specific Test Language. In both cases, the first time a test is run is in its automated form. </div><div><br /></div><div>But hang on, this means that again it is the <span class="Apple-style-span" style="font-weight: bold; ">tests <span class="Apple-style-span" style="font-weight: normal; ">that are finding the bugs, not the fact that those tests are automated!</span></span></div><div><br /></div><div>"Finding bugs" is a great objective for testing - but it is not a good objective for automation.</div>Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com17tag:blogger.com,1999:blog-8378239747319086089.post-60995690959992690062010-01-14T02:00:00.000-08:002010-01-14T02:33:20.057-08:00Finding 40% of your own mistakesI used to be fond of quoting a statistic that says you can only find around 40% of your own mistakes.<br /><br />Michael Stahl emailed me to ask where this number came from. Interesting question - first thought - I don't remember! I'm sure I must have read it somewhere at some time, but where, by whom and was it based in a study?<br /><br />I checked with Mark Fewster, one of my former colleagues, and he thinks it might have come from a study done by the Open University in the UK.<br /><br />I checked with Tom Gilb, as he uses an estimate of around a third (33%) for the effectiveness of an initial inspection - which is probably more effective than an individual anyway! Tom has demonstrated an effectiveness of 33% repeatedly by experimentation with early Inspections; he said it also agrees with Capers Jones' data.<br /><br />I think we used the figure of 40% only because people found it more believable than 33%.<br /><br />The frightening consequence is that if you don't have anyone else review your work, you are guaranteed to leave in two thirds of your own mistakes!Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com8tag:blogger.com,1999:blog-8378239747319086089.post-42454940597117707782010-01-02T03:09:00.000-08:002010-01-02T03:30:47.792-08:00DDP Discussions and challengesSeveral people have asked about benchmarks for DDP. I have actually blogged about this, but my comments are "buried" in the comments to the post about "Starting with DDP". Please have a look at <a href= "https://www.blogger.com/comment.g?blogID=8378239747319086089&postID=140185916164272907"> the comments for that post</a>, which include:<div><br /></div><div>- benchmarking DDP with other organisations (raised by Bernhard Burger)</div><div><br /></div><div>- Paul Herzlich's challenges about the seriousness of defects, getting data, DDP being hard to use and code-based metrics (all of which I have replied to in my following comment)</div><div><br /></div><div>- using DDP to improve development (raised by Ann-Charlotte)</div><div><br /></div><div>- Michael Bolton's challenges: </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>5 examples to show when it doesn't work (which I reply to in my first comment following his) (including some Dilbert Elbonian testers ;-)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>7 "problems" - some of which I agree with, some I don't understand, and some I think illustrate the benefit of DDP rather than being problems with it (replied to in my second comment after his)</div><div><br /></div><div>Thanks to Michael B's comments, I also formulated 3 Rules for when to use DDP:</div><div><span class="Apple-style-span" style=" color: rgb(51, 51, 51); line-height: 18px; font-family:'Trebuchet MS', Verdana, Arial, sans-serif;font-size:small;">DDP is appropriate to use when:<br />1) you keep track of defects found during testing<br />2) you keep track of defects found afterwards<br />3) there are a reasonable number of defects – for both 1) and 2)</span></div><div><br /></div><div>These are not the whole story (as illustrated by Michael's examples) but I think are a pre-requisite to sensible use of DDP.</div><div><br /></div>Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com0tag:blogger.com,1999:blog-8378239747319086089.post-46060067037072026672009-12-18T01:57:00.000-08:002009-12-18T04:17:00.035-08:00Can Agile be Context-Driven?"They say they are doing Agile, but they're not doing all of the practices, so they aren't really Agile." <div><br /></div><div>This seems to be a popular view, implying that Agile needs to be done "properly". A search on "agile best practice" brings up many entries. Is there such a thing as Agile Best Practice?<div><br /></div><div>The Context-driven principles (www.context-driven-testing.com) include:</div><div>1. The value of any practice depends on its context.</div><div>2. There are good practices in context, but there are <b>no best practices</b>. (emphasis mine)</div><div><br /></div><div>So if you adapt agile practices to suit your context, as was described by Gitte Ottosen at EuroSTAR, isn't that context-driven agile? And isn't that the best way to practice it?</div><div><br /></div><div><br /></div><div><br /></div></div>Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com2tag:blogger.com,1999:blog-8378239747319086089.post-54509266564267178572009-06-19T08:34:00.000-07:002009-06-19T08:43:25.338-07:00ROI from automationHow do you measure Return on Investment from automation? <div><br /></div><div>According to <span class="Apple-style-span" style=" ;font-family:Helvetica, -webkit-fantasy;font-size:medium;">http://www.investopedia.com/terms/r/returnoninvestment.asp, <span class="Apple-style-span" style=" ;font-family:Georgia, fantasy;font-size:16px;">return on investment is (Gain - Cost) / Cost.</span></span></div><div><span class="Apple-style-span" style="font-family:Georgia, -webkit-fantasy;"><br /></span></div><div><span class="Apple-style-span" style="font-family:Georgia, -webkit-fantasy;">So if we have a set of automated tests that save 5 hours of manual testing, then if those tests took one hour to build and run, our ROI would be (5 - 1) / 1 = 4 (if my arithmetic is correct). So that would be a four-fold ROI for those automated tests.</span></div><div><span class="Apple-style-span" style="font-family:Georgia, -webkit-fantasy;"><br /></span></div><div><span class="Apple-style-span" style="font-family:Georgia, -webkit-fantasy;">Of course in the initial stages of automation, before our automation regime is mature, it might take us 10 hours to build and run those same tests. This would give an ROI of (5 - 10) / 10, i.e. a negative return on investment of -0.5.</span></div><div><span class="Apple-style-span" style="font-family:Georgia, -webkit-fantasy;"><br /></span></div><div><span class="Apple-style-span" style="font-family:Georgia, -webkit-fantasy;">Negative ROI is not generally a good idea, as it means you are losing money. But you may have negative ROI in the early stages of automation, but a very good positive ROI long-term. It might also make more sense to look at ROI over a larger sample, say all of the automation build and run in a month, and monitor that month by month.</span></div><div><span class="Apple-style-span" style="font-family:Georgia, -webkit-fantasy;"><br /></span></div><div><span class="Apple-style-span" style="font-family:Georgia, -webkit-fantasy;">How do you measure ROI for automation?</span></div><div><span class="Apple-style-span" style=" ;font-family:Helvetica;font-size:medium;"><div><br /></div></span></div>Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com6tag:blogger.com,1999:blog-8378239747319086089.post-73990601905014567832009-03-16T02:47:00.000-07:002009-03-16T02:59:50.661-07:00Make testing fun?<span class="Apple-style-span" style="font-style: italic;">On the LinkedIn group "Software Testing Club", the question was asked: "How can we make testing more fun?" They are running 1300 scripts over 4 months.</span><div><span class="Apple-style-span" style="font-style: italic;"><br /></span></div><div><span class="Apple-style-span" style="font-style: italic;">There were some interesting answers about holding weekly competitions, creating visibility with management, and having a movie in work time. There were also a couple of mentions about automation also. Here is my response to the thread:</span></div><div><br /></div><div>--------------------</div><div><br /></div><div>Draw a graph of the "boringness" of the testing - on a scale of 1 to 10 - of different things the testers are doing. The more boring the activity, the more ripe it is for automation. But don't think of automation as all-or-nothing. Pick out the most boring things for people to do and automate them. You can get real benefit by automating only 2% of the scripts, for example.</div><div><br /></div><div>You don't need to purchase expensive tools to automate either. Use what you already have or look into open source tools (see the FreeTest Conference link from my web site).</div><div><br /></div><div>However, do be warned that a free tool is not free! In order to be successful in automation, you must plan and prepare how you will do it.</div><div><br /></div><div>In fact there is so much to say about how to go about automation that I have written a book about it with Mark Fewster (see link on my web site). Although the case studies are now getting a little "long in the tooth", the main body of the book covers the principles of automation - whatever tool you use - that will help you creat a long-lasting "regime".</div><div><br /></div><div>Manual testing is not fun if you are just following a script; but manual testing is tremendous fun if you are doing exploratory testing, for example. Get the computers (tools) to do what they are good at, and free the people to do what they are good at.</div><div><br /></div><div><br /></div><div><br /></div><div><br /></div>Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com0tag:blogger.com,1999:blog-8378239747319086089.post-25970083293679915592009-02-23T21:17:00.000-08:002010-01-02T03:08:09.090-08:00Don't blame the bread-making machine - thoughts on test automationIf you have a bread-making machine, it may or may not make good bread.<div><br /></div><div>If it doesn't, the reason might be the ingredients you put into the machine. If you put in too much salt (or no salt at all), the bread won't be edible. Is that the fault of the bread-making machine?</div><div><br /></div><div>Some people blame the test automation for things which are actually the responsibility of the testing. For example, if you mistakenly think that finding lots of bugs is the main aim of your regression test automation, you are setting yourself up for disappointment, if not failure of your automation effort. Finding bugs is what testing (and testers) do - it is not the job of automation - the job of automation is to run tests.</div><div><br /></div><div>The tests are the ingredients that determine whether the testing is effective or not. The effectiveness of the tests is the same whether those tests are run manually or using an automation tool. </div><div><br /></div><div>The tool is no more responsible for the quality of the testing than the bread-making machine is responsible for the taste of the bread.</div>Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com4tag:blogger.com,1999:blog-8378239747319086089.post-1401859161642729072008-12-29T10:21:00.000-08:002010-01-02T03:07:39.871-08:00DDP<div><b>Getting started with DDP </b></div><div>You probably already have the information you need to calculate your own DDP - the number of defects found in testing, and the number that were found later (e.g. a later stage of testing or reported to Support).</div><div><br /></div><div>For example, if system testing (ST) found 50 bugs, and user acceptance testing (UAT) found 25, the DDP of ST was 67% (50 / 75). </div><div><br /></div><div>Take a project that finished 3 to 6 months ago and calculate what the DDP was for that. Now you have a starting point to see how your testing is changing over time.</div><div><br /></div>Not all bugs are created equal! When you start measuring DDP, you might want to use only the highest one or two severity ratings. Or you could have one DDP for high severity, and one for all bugs.<div><br /></div><div>If you have any questions about DDP please ask!</div>Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com9tag:blogger.com,1999:blog-8378239747319086089.post-65095166315104876812008-12-28T00:00:00.000-08:002008-12-28T14:17:51.035-08:00<div>DDP is my favourite metric. It is very easy to measure, easy to understand, and gives you an idea of the quality of the testing, in terms of defects found. Many people have found it very useful. If you are one of those people, please respond to this blog and tell me about your experiences!<br /></div><div><br /></div><div><b>What is DDP?</b></div><div>DDP is Defect Detection Percentage. It is a measure of the effectiveness of testing (or reviewing) at finding defects. The calculation is very simple: The number of defects (bugs) found in testing* divided by the number of defects found in total so far**.</div><div><br /></div><div>*DDP can be calculated for different iterations, different testing stages (e.g. integration and system test) or testing by a defined set of testers.</div><div><br /></div><div>**You can choose any point in time to calculate DDP - for example, one month after an iteration is released (or when the next iteration is released).</div><div><br /></div><div>There are lots of variations on DDP - if you have any questions, please ask!</div><div><br /></div><div>Dot</div>Dot Grahamhttp://www.blogger.com/profile/07991717205107510005noreply@blogger.com4