An Ingredients List for Testing - Part Two
Friday, August 27, 2010
By James Whittaker
When are you finished testing? It’s the age old quality question and one that has never been adequately answered (other than the unhelpful answer of never). I argue it never will be answered until we have a definition of the size of the testing problem. How can you know you are finished if you don’t fully understand the task at hand?
Answers that deal with coverage of inputs or coverage of code are unhelpful. Testers can apply every input and cover every line of code in test cases and still the software can have very serious bugs. In fact, it’s actually likely to have serious bugs because inputs and code cannot be easily associated with what’s important in the software. What we need is a way to identify what parts of the product can be tested, a bill of materials if you will, and then map our actual testing back to each part so that we can measure progress against the overall testing goal.
This bill of materials represents everything that can be tested. We need it in a format that can be compared with actual testing so we know which parts have received enough testing and which parts are suspect.
We have a candidate format for this bill of materials we are experimenting with at Google and will be unveiling at GTAC this year.
When are you finished testing? It’s the age old quality question and one that has never been adequately answered (other than the unhelpful answer of never). I argue it never will be answered until we have a definition of the size of the testing problem. How can you know you are finished if you don’t fully understand the task at hand?
Answers that deal with coverage of inputs or coverage of code are unhelpful. Testers can apply every input and cover every line of code in test cases and still the software can have very serious bugs. In fact, it’s actually likely to have serious bugs because inputs and code cannot be easily associated with what’s important in the software. What we need is a way to identify what parts of the product can be tested, a bill of materials if you will, and then map our actual testing back to each part so that we can measure progress against the overall testing goal.
This bill of materials represents everything that can be tested. We need it in a format that can be compared with actual testing so we know which parts have received enough testing and which parts are suspect.
We have a candidate format for this bill of materials we are experimenting with at Google and will be unveiling at GTAC this year.
Testers can apply every input and cover every line of code in test cases and still the software can have very serious bugs.
ReplyDeleteYes, testers can cover every single line of code by different techniques but the problem we face is related to estimates, budgeting, priortization etc. I never saw anyone considering all of them without making any compromises or trade-offs.
What we need is a way to identify what parts of the product can be tested, a bill of materials if you will, and then map our actual testing back to each part so that we can measure progress against the overall testing goal.
James, are you referring to Scope of Testing, Test Coverage and Risk Based Testing?
Who should actually define these things, a tester or a product expert (because a tester may not be the correct person initially)?
I am looking forward to attend GTAC at Hyderabad this year and wish to gain more understanding on the subject from you.