Coverage analysis can only tell you how the code that exists has been exercised. It cannot tell you how code that ought to exist would have been executed. Consider the following:
error_code = FunctionCall();
// returns kFatalError, kRecoverableError, or kSuccess
if (error_code == kFatalError) {
// handle fatal error, exit
} else {
// assume call succeeded
}
This code is only handling two out of the three possible return values (a bug!). It is missing code to do error recovery when kRecoverableError is returned. With tests that generate only the values kFatalError and kSuccess, you will see 100% coverage. The test case for kRecoverableError does not increase coverage, and appears “redundant” for coverage purposes, but it exposes the bug!
nice explanation that code coverage is not the same as path coverage
ReplyDeleteI would have liked the article to mention object (assembly) level testing. If you define your test cases using breakpoints at every branch (or jump) instruction you can observe every path. I.E. if(a||b) will be at least 2 branch instructions and force you to test both. Testing at object level will also highlight any dead code or logic inserted (or optimized) by the compiler/linker. This is the kind of testing required by DO-178B level A certification. FYI: DO-178B is an FAA specification required for civilian equipment that will fly in US Airspace.
ReplyDeleteNice post, Dave; code coverage indeed isn't test coverage.
ReplyDeleteHere are a couple of points that I think are important and necessary for to ask about test coverage, where code coverage simply begs the questions:
1) Will the code in question work reliably?
2) Do you care about performance in the system under test?
3) So you've got code coverage--but isn't it testing only your code? It's not testing the code of the operating system, third-party libraries, hardware or firmware, or anything else on which your product depends? That is, code coverage indicates what code you've touched in the product--but we have to test not only the product, but the system--the product and all the things around it.
4) Does your test coverage tell you about supportability, testability, maintainability, portability, and localizability?
5) Does your test coverage address the issue of value? Code coverage can tell you if something has been done--but was that something useful?
Cheers,
---Michael B.
Amazing example on code coverage. But looks as if better said than done. Really wish if we lived in a perfect world. Blogging initiatives like the current one keeps the spirits live. Good guys.
ReplyDeletegreat explanation, besides code coverage it also made me many more things to understand.
ReplyDeleteSachin
great explanation, besides code coverage it also made me understand many more things.
ReplyDeleteSachin
great article for code coverage and perfect testing logic
ReplyDelete