Saturday, 5 February 2011

Part 3. Certification schemes do not assess tester skill?

(Continuing from Parts 1 and 2, certification is evil? and some history about ISTQB)

Now I would like to say something about the criticism that the current schemes do not assess tester skill. I am thinking mainly of the Foundation level, as this seems to be where the criticism is mainly directed, and I am more familiar with that than the Advanced Levels.

In the main, I agree. There is a modicum of skill needed in testing techniques to be able to answer multiple-choice questions about them, but that is not the same as being able to test well in practice. And I also agree that the current ISTQB Foundation level is based more on learning facts than on practicing the craft. Why is that? Because the current scheme was designed to meet a different need, a basic ignorance about testing in general; it was not designed to assess testing skill.

I feel it is unfair for people to criticize a scheme because it doesn’t conform to what they think assessment of testers should be today, when the scheme was never meant to be that kind of assessment. It’s a bit like criticizing a bicycle for not powering you up a hill by itself – it’s not intended to do that.

When we were developing the original Foundation Syllabus with ISEB, I remember many discussions about what was possible and practical, including ways of assessing tester skills beyond basic concepts and vocabulary:

- interviews?

- looking at projects submitted from their workplace?

- observing them at work?

- substantial pieces of testing work to be done in a supervised exam-like setting or as a project to be given in within a time frame?

All of these have significant challenges. For example, how to ensure fairness if different people interview in different ways, ensuring that the work being assessed was actually done by the person submitting it, time commitment, scope and fair comparison of observation at work, designing a testing task that would be applicable to people from different industries.

We decided that the place to start was with something very basic that could be built on later, something that would try to cover common ground that all testers should know and build on - hence it was called "Foundation".

Criticism is good – we all learn by having our ideas challenged. But current qualification schemes are not “evil”, even if there are aspects of their current implementation that are not as they should be. So let’s take the context of the certification schemes into account, and remember that what may be ideal for today was not possible 12 or 13 years ago.


Pete Walen said...

Thanks for this series. I found it interesting.

In my case, much of the objection to /most/ certifications is how some people believe they confer a form of superiority over the unwashed masses. This is neither the fault of the certification organization, the examiners nor those who worked hard to develop the certification process.

I shall keep your insights in mind in the future.

Thank you.

Simon Morley said...

Very interesting reading and background - it's refreshing and important to get the perspectives that existed as some of the original motivators behind the certifications. I started my testing in the early 90's too and recognise many of the perceptions about testing that you stated.

A question: If one of the motivations in those days was "removing the bottom layer of ignorance about software testing." (talking about the foundation level certification) - is there the same need today?

If so, how would you rate the foundation level certification in the basket of opportunities that exist for testers today (and there are many more opportunities than when I started in the early 90's)?

A realted question: Was the perceived ignorance about software testing related to only software testers or non-testers also (including managers and execs)? If so, how was the certification rated in terms of removing that ignorance for non-testers (including managers/execs)?

I too see many opportunities in the need for understanding, appreciating and communicating about software testing. I'm not sure that a certification is the best way to communicate this to non-testing managers and executives - I'm still working and searching for that.

Thanks for the write-ups.

Stephen Hill said...

Thank you for writing this series Dorothy. I, for one, did not know much about the history of the certification and its original intentions.

I have a question, though: the original vision of ISEB was to help gain an understanding of testing when the craft was still relatively unknown in its own right; do you think the schemes as they currently stand are still as relevant and giving that broad spectrum of knowledge that was previously envisaged?

I fear that testing has moved on rapidly but the cert schemes have not moved on at the same rate and this is giving a false confidence so I wondered what your opinions were on this.



richjuggle said...

As a graduate developer who has found himself by surprise in a testing role, the ISTQB foundation syllabus has provided me with more respect of Testing as a career option and it's necessity.

Dot Graham said...

Thanks for your comments!

Richjuggle, nice to hear that it is still able to do this, thanks.

Pete, good point, and probably a problem with many types of qualification. I have known good testers who find great difficulty passing an exam, and poor testers who do well on exams. Passing the exam only means you have passed an exam!

Simon and Stephen, you asked a similar question - is it still relevant today? I believe it is (see Richjuggle for example) because there are always people who are new to testing who need to have a basic understanding about it.

But the industry does move on, and inevitably, once a scheme is established, it takes effort, time and mutual agreement from many people to move it on - a cruise ship handles very differently to a rowboat.

Simon, yes, there are more opportunities today for testers to educate themselves, including free training from AST, for example, as well as online courses, books, blogs, etc. However I think the ISTQB Foundation does have a role to play, at least for now and the next few years, partly because it is now so well established and accepted.

When we started, we had hoped that the testing qualification would also be used by project managers, developers etc to educate them about testing also. For a time, one of the ISEB developer qualification schemes allowed the software testing qualification as one of its options. But I don't think it ever penetrated the project management area as much as it should have, and I think there is still a fair amount of ignorance about testing from higher level management and executives.

I agree, Simon, that it is still a challenge to communicate to managers and executives what testing is and is not, what it can and can't do. I don't think this will be solved by any testing qualification; a better approach would be to infiltrate management qualifications.

Thanks again for your comments!

Stephen Hill said...


Thanks for replying. The issues surrounding updating and maintaining the certifications are important ones. It takes a lot of time and energy to keep things fresh and current - we see that in all walks of life - and that seems to be where the contention lies.

That said it is great to know that the schemes are still helping people but perhaps it is because there are several competing schemes using their own definitions for testing terms that yet more confusion has crept in across the industry. It is almost as though we have too many standards.

I should also point out that I personally have gained a lot of benefit from ISEB certified courses (I have both of the new Practitioner certs) and have passed the exams going alongside those courses. As I have progressed in my testing career, though, I see dangers in the assumptions being made by top management about what the certificate holder has actually demonstrated.

Many thanks again,

Stephen Hill

Devon Smith said...

Thanks for this post- it was interesting to get a bit of history and information behind the certification process. I am always surprised that people are so opposed to it, since it seems like - as you said- it can help us all speak the same language and shows the people who care enough to take a test.

I just think that it shouldn't be considered a badge that means you don't still have to demonstrate your testing skills in other ways.

Paul Gerrard said...

Hi Dot,

I thought I'd refer you to a blog I posted some time ago here: Certification - a personal history. It was an attempt to set the record straight on the 'ancient history'.

Are certifications evil? Well, you could ask the same question of anything. Do people or the industry need 'protection' from these things or do we treat people as being intelligent enough to recognise good or bad aspects? I favour the latter. However...

In 1997/8 when the whole thing got started, I think we had a reasonable excuse for 'not getting it right first time'. What pains me is that after so many years (and presumably years of effort) is that the quality of what is 'out there' is so poor and the inertia of the whole empire means that not even typos get fixed years after they are identified.

In April 2008, I posted a series of short reviews of sections of the Advanced Syllabus available here. Here is one example of the type of comment: The open source tools arena is advancing rapidly of course, but the principles, as set out in the syllabus are timeless aren't they?

I started writing these critiques because I thought it wouldn't take me very long and also that, ISTQB being suitably embarrassed, would pay some attention to these problems and fix them. I was wrong on both counts. I gave up very quickly.

The "Version 2007" Advanced Level Syllabus, four years after it was written, is uncorrected and still the downloadable version.

Three years after my comments, (which I know were noted by the authors), the content of this document is still, well... poor.

For that, there is no excuse.


Markus Gärtner said...

Hi Dorothy,

thanks for the explanation of the origins of the certification scheme. As one of the initial "certification is evil" proponents, I wanted to put up a reaction, and when I noticed this might take longer than a usual comment, I made it a blog entry in its own:

I appreciate the efforts the people put up into the syllabus. Yet, I am still puzzled why many of the claims I hear about the program do not match my experience.

My key points are, that many of the claims of the certification proponents - common language, respect for testing, etc. - do not hold. My experiences with testing teams do not see these claims fulfilled.

Stuart Taylor said...

Hi Dorothy.

I caught up with you during a SIGIST event, and lamented about how sad I am that the certification gets used as a CoRgi gas fiiters badge. "I have the certificate, therefore I'm a professional tester", sadly many of the candidates I interview hold this belief, then struggle to draw a V model.

ISTQB isn't evil, but the peddlers of "learn in 24hrs" courses and "answer banks" have a lot to answer for.

As you know, I preferred ISEB, but at the end of the day, it's all about skill and aptitude.


Timothy is a natural by the way.

halperinko said...

Sorry for the delayed response, Just saw this series which was referred to in another source, Thank you for describing the history of ISTQB.

Few months after starting my testing career back in 1994, I was lucky to receive a testing basics course training with the rest of the testing group.
These kind of courses existed till the ISTQB foundation arrived, and I as a test manager I was one of the followers, not because I thought it brings anything new, but exactly as you say - it was an opportunity to align the language in compare to the Babylon mix which existed earlier (hope I am using the right term, English is not my mother tongue).
Over 10 years later, I do believe ISTQB courses served their purpose on that - though there are many veterans in many positions which still keep the mix of terms going, by tutoring youngsters and coworkers with confusing language of testing terms.
BUT while the original courses were fully aimed at learning the testing profession, I have noticed that the current ISTQB foundation courses in some organizations, were overly aimed at passing the exam - I see that as a sad side affect.

In order to reduce the profit claims against ISTQB Foundation courses, I call all test managers, to verify their company subsidises the course for newcomers lacking that basic training (as was done in old format of testing courses), rather than using it as a recruitment filter.

Anonymous said...

Hi Dot,
I thought it was worth writing about this as I have been one of the "ISTQB critics" over the years and only last year took the ANZTB to task at their own Conference in Auckland by publicly stating that the ISTQB/ANZTB was LAZY with respect to it's role in Tester certification. The Conference Chair (David Hayman) even wrote to me after the Conference and asked me to modify my material so that it could be added to the Conference archive (yes I reluctantly changed it)!!!
Now thanks to reading your Blog I will re-think my views and comments. I have always been a supporter of certification, but have also found it difficult to reconcile this with my growing frustration with the ISTQB syllabus and exams.

Marco said...

My observation is that the ISQTB approach is dogmatic and over simplified.

The emphasis on strict delineation of roles and responsibilities, heavy documentation, up front planning and ceremonious processes would not work well in my actual test practice.

While, the latest syllabus mentions that context is important the course then continues to describe an approach which assumes all contexts are the same.

To my mind, the advanced courses which are aimed at managers are the most dangerous because it mandates specific approaches based on very simplified models of test problems.

For example, to solve problem A, the only 'correct' approach must be the sequence of actions X, Y and Z.

In real life, test management problems are not so simple to solve.

Not every constraint or challenge can be covered by a generic approach and sometimes a more tailored approach would be more effective.

Unfortunately, the situation with ISQTB in 2013 is that experienced testers taking ISQTB exams are giving some answers from the ISQTB playbook which they know in their heart do not work or apply in real test practice.

For test managers, I believe a standard project manager course would be much more applicable to the real world than ISQTB.

Thank you for raising this important topic and I would appreciate any feedback you may have to my comments.

Dot Graham said...
This comment has been removed by the author.
Dot Graham said...

Marco, thanks for your comment and apologies for the delay in responding.

Whenever you begin to learn about something, you have to start somewhere. And the place to start is generally with a simple approach, rather than exposure to the full complexity of the topic. In addition, if you are looking to assess a level of knowledge over a wide range of people, this needs to be based on some common ground. So an introduction to any topic tends to be more towards the prescriptive and simple, rather than complicated by all the permutations of possibility inherent in the real world, and the ISTQB qualifications are no exception.

As your experience grows, your learning deepens, you realize that things are not as “black and white” as they first seemed, you begin to see exceptions to the rules, and you learn (sometimes through mistakes) how to do your work more effectively and efficiently. At this point in your learning journey, things that were fine at the start may now appear over-simplified and dogmatic.

Although I have not been involved with syllabus development for many years, certainly in the early days we were careful to say that one size does not fit all situations. I would like to see the specific wording that you are referring to, where you say that a specific management approach is “mandated” and where there is only one “correct” approach. Those who I have consulted about this do not think that this is actually in the syllabus (and I don’t know if you are referring to the older Advanced syllabus or the recently-released one).

Of course, in real life, problems are not simple to solve. But isn’t it better to learn solutions in a safe environment with feedback from those who have had more experience, rather than making expensive mistakes for real? Airline pilots make mistakes in simulators so they are less likely to make them in practice.

The ISTQB syllabi are aimed at a large audience and must be general. Yes a tailored approach for each organization is what will work better, but how could that be taught and assessed? If it were possible, it would only apply to a very narrow audience, and there would be a great multiplicity of variations if not qualifications. Surely addressing the simplest common ground is a better approach for this scheme?

If you object to something that is taught in a Syllabus because it doesn’t apply to you, that doesn’t make it invalid, it makes it inappropriate (for your context). But that same idea may be just what another organization needs. Not all organizations are the same (or even groups within the same organization). A tester who appreciates the variety of possible solutions has a broader base for their career.

Of course, the syllabi are not perfect – what is? Yes it takes a long time to correct mistakes, there are disagreements about topics, and perhaps it has become “political”. But I still believe that having the ISTQB qualifications as a starting point is an improvement over the vacuum that existed before it came about.

In terms of project management courses, a friend of mine who has been on both disagrees – there was very little about testing at all, let alone the particular problems that are unique to testing.

I look forward to any further comments you have – thanks for contributing to this discussion!