DDP Discussions and challenges

Several people have asked about benchmarks for DDP. I have actually blogged about this, but my comments are "buried" in the comments to the post about "Starting with DDP". Please have a look at the comments for that post, which include:

- benchmarking DDP with other organisations (raised by Bernhard Burger)

- Paul Herzlich's challenges about the seriousness of defects, getting data, DDP being hard to use and code-based metrics (all of which I have replied to in my following comment)

- using DDP to improve development (raised by Ann-Charlotte)

- Michael Bolton's challenges:

5 examples to show when it doesn't work (which I reply to in my first comment following his) (including some Dilbert Elbonian testers ;-)

7 "problems" - some of which I agree with, some I don't understand, and some I think illustrate the benefit of DDP rather than being problems with it (replied to in my second comment after his)

Thanks to Michael B's comments, I also formulated 3 Rules for when to use DDP:

DDP is appropriate to use when:
1) you keep track of defects found during testing
2) you keep track of defects found afterwards
3) there are a reasonable number of defects – for both 1) and 2)

These are not the whole story (as illustrated by Michael's examples) but I think are a pre-requisite to sensible use of DDP.

DDP

Getting started with DDP

You probably already have the information you need to calculate your own DDP - the number of defects found in testing, and the number that were found later (e.g. a later stage of testing or reported to Support).

For example, if system testing (ST) found 50 bugs, and user acceptance testing (UAT) found 25, the DDP of ST was 67% (50 / 75).

Take a project that finished 3 to 6 months ago and calculate what the DDP was for that. Now you have a starting point to see how your testing is changing over time.

Not all bugs are created equal! When you start measuring DDP, you might want to use only the highest one or two severity ratings. Or you could have one DDP for high severity, and one for all bugs.

If you have any questions about DDP please ask!

DDP is my favourite metric. It is very easy to measure, easy to understand, and gives you an idea of the quality of the testing, in terms of defects found. Many people have found it very useful. If you are one of those people, please respond to this blog and tell me about your experiences!

What is DDP?

DDP is Defect Detection Percentage. It is a measure of the effectiveness of testing (or reviewing) at finding defects. The calculation is very simple: The number of defects (bugs) found in testing* divided by the number of defects found in total so far**.

*DDP can be calculated for different iterations, different testing stages (e.g. integration and system test) or testing by a defined set of testers.

**You can choose any point in time to calculate DDP - for example, one month after an iteration is released (or when the next iteration is released).

There are lots of variations on DDP - if you have any questions, please ask!

Dot