Part 3. Certification schemes do not assess tester skill?

(Continuing from Parts 1 and 2, certification is evil? and some history about ISTQB)

Now I would like to say something about the criticism that the current schemes do not assess tester skill. I am thinking mainly of the Foundation level, as this seems to be where the criticism is mainly directed, and I am more familiar with that than the Advanced Levels.

In the main, I agree. There is a modicum of skill needed in testing techniques to be able to answer multiple-choice questions about them, but that is not the same as being able to test well in practice. And I also agree that the current ISTQB Foundation level is based more on learning facts than on practicing the craft. Why is that? Because the current scheme was designed to meet a different need, a basic ignorance about testing in general; it was not designed to assess testing skill.

I feel it is unfair for people to criticize a scheme because it doesn’t conform to what they think assessment of testers should be today, when the scheme was never meant to be that kind of assessment. It’s a bit like criticizing a bicycle for not powering you up a hill by itself – it’s not intended to do that.

When we were developing the original Foundation Syllabus with ISEB, I remember many discussions about what was possible and practical, including ways of assessing tester skills beyond basic concepts and vocabulary:

- interviews?

- looking at projects submitted from their workplace?

- observing them at work?

- substantial pieces of testing work to be done in a supervised exam-like setting or as a project to be given in within a time frame?

All of these have significant challenges. For example, how to ensure fairness if different people interview in different ways, ensuring that the work being assessed was actually done by the person submitting it, time commitment, scope and fair comparison of observation at work, designing a testing task that would be applicable to people from different industries.

We decided that the place to start was with something very basic that could be built on later, something that would try to cover common ground that all testers should know and build on - hence it was called "Foundation".

Criticism is good – we all learn by having our ideas challenged. But current qualification schemes are not “evil”, even if there are aspects of their current implementation that are not as they should be. So let’s take the context of the certification schemes into account, and remember that what may be ideal for today was not possible 12 or 13 years ago.

Part 2. A bit of history about ISTQB certification

(Continuing from Part 1 where I was surprised at reactions to certification as "evil")

In the early 1990s, software testing was not a respected profession; in fact many thought of testing at best as a “necessary evil” (if they thought of testing at all!). There were few people who specialized in testing, and it was seen as a “second-class” activity. There was a general perception that testing was easy, that anyone could do it, and that you were rather strange if you liked it.

It was then that I decided to specialize in testing, seeing great scope for improvement in testing activities in industry, not only in imparting fundamental knowledge about testing (basic principles and techniques), but also in improving the view testers had of themselves, and the perceptions of testers in their companies. I developed training courses in testing, and began Grove Consultants, named after my house in Macclesfield. One of my most popular talks at the time was called “Test is a four-letter word”, reflecting the prevailing culture about testing. The UK’s Specialist Interest Group in Software Testing (SIGIST) was started by Geoff Quentin in 1989, and was the only gathering of testers in the UK.

It was into this context that the initiative to create a qualification for testers was born. Although I was not the initiator, I was involved from the first meeting (called by Paul Gerrard at a STARWest conference in 1997) and the earliest working groups that developed the first Foundation Syllabus, donating many hours of time to help progress this effort. This work was carried out with support from ISEB (Information Systems Examination Board) of the British Computer Society (meeting rooms, travel expenses and admin help). The testing qualification was modeled on ISEB’s qualifications in Project Management and Information Systems Infrastructure, which were perceived as useful and valuable in their respective sectors. One of the aims was to give people a common vocabulary to talk about testing, since at the time people seemed to be using many different terms for the same thing.

The first course based on the ISEB Foundation Syllabus was given in October 1998 and the first Foundation Certificates in Software Testing were awarded at that time. But an important aspect of the scheme was that it was not necessary to take a training course in order to get the qualification; you could just take the exam. (Some other schemes were based on attendance at courses which seemed too training-provider profit-oriented to us.)

The success of the Foundation qualification took everyone by surprise – there seemed to be a hunger for something that gave testers more respect, both for themselves and from their employers. It also gave testers a common vocabulary and more confidence in their work. The Foundation qualification was meeting its main objective of “removing the bottom layer of ignorance” about software testing.

Work then began on extending the ISEB qualification to a more advanced level (which became the ISEB Practitioner qualification) and also to extending it to other countries, as news of the qualification spread in the international community. I was a facilitator at the meeting that formed ISTQB in 2001 in Sollentuna, Sweden.

I became a member of the working party that produced the first ISTQB Foundation Syllabus in 2005, and I am amazed at how ISTQB has grown; it has certainly changed over the past six years. While working on the update to the Foundation book, I was rather surprised and disappointed at the apparent lack of review the before release of the 2010 Syllabus.

In Part 3 I return to the criticism that current certification schemes do not address tester skill.

Part 1. Certification is evil?

Certification is evil?

This is a 3-part blog (because it ended up rather longer than I anticipated when I started).

1: My reaction to “certification is evil”

2: A bit of history about ISTQB certification

3: The criticism that current schemes do not assess tester skill

I enjoyed reading the Software Testing Club's publication "A Tester is for Life, not just for Christmas" (http://blog.softwaretestingclub.com/2010/11/a-tester-is-for-life-not-just-for-christmas/) which consists of interviews with a number of people, all of whom answered the same questions.

But I was quite shocked at the strength of feeling shown by quite a few against certification. The question asked was "What are your views on Testing Certification Schemes?" Although there are a few older testing qualifications from the US, it seems that most people assumed the ISTQB qualifications were being asked about.

A couple of people called the ISTQB scheme a "scam", one person said it was "downright dangerous", and someone said it was "the devil in disguise" and talked about the "evil axis" of certification institutes, training providers and HR departments. Some mentioned "profit" and "money-making". Several criticized any exam based on memorizing, saying that current exams are too easy, even "trivial"; one said the exam "has no value".

The basis of this criticism (if there is a rational basis) seems to be that current schemes do not assess tester skills, i.e. how well you can actually do testing. (I will return to this point at the end of this blog.)

(On the other hand, many said that the current schemes were a good starting point for people new to testing, that it gave knowledge of basic terminology, helped them get their first job, and can demonstrate that you are serious enough about testing to take an exam.)

I was not surprised to find people who didn't like the ISTQB certification schemes - I have been to conferences over the past few years where this has become clear. But some of the "opposition" to certification is, I believe, based on a false premise about what it was intended to be, and this I feel is somewhat unfair.

Just to state my own position: I am not involved in any certification scheme at the moment. I no longer provide accredited ISTQB training courses in software testing. However, I was involved in writing the initial Foundation Syllabus, and I am currently in the process of updating our book that supports the ISTQB Foundation Syllabus.

It seems to me that people have lost sight of (or perhaps were never aware of) the origins and purpose of the current certification scheme, at least the ISTQB Foundation (which is the one I am most familiar with).

I have been involved in software testing all my working life, for over 40 years. The situation before the current ISTQB certification began was very different to what testing is like today!

See my next entry for a bit of history about ISTQB.