Google Testing Blog: TotT: A Matter of Black and White

TotT: A Matter of Black and White

Thursday, August 07, 2008

The progressive developer knows that in this complex modern world, things aren't always black-and-white, even in testing. Sometimes we know the software won't return the best answer, or even a correct answer, for all input. You may be tempted to write only test cases that your software can pass. After all, we can't have automated tests that fail in even one instance. But, this would miss an opportunity.Speaking of black-and-white, take decoding of two-dimensional barcodes, like QR Codes. From a blurry, skewed, rotated image of a barcode, software has to pick it out, transform it, and decode it:

http://google.com/gwt/n?u=bluenile.comEven the best software can't always find that barcode. What should tests for such software do?We have some answers, from experience testing such software. We have two groups of black-box tests that verify that images decode correctly: must-have and nice-to-have. Tests verify that the must-have set – the easy images – definitely decode correctly. This is what traditional tests would include, which typically demand a 100% pass rate. But we also see how well we do on the more difficult nice-to-have set. We might verify that 50% of them decode, and fail otherwise.The advantage? We can include tougher test cases in unit tests, instead of avoiding them. We can observe small changes – improvements as well as degradations – in decode accuracy over time. It doubles as a crude quality evaluation framework.Where can this progressive thinking be applied? Maybe when your code...Only needs to be correct in most cases. As here, write tests to verify easy cases work, but also that some hard cases pass too.Needs to be fast. You write unit tests that verify it runs "fast enough" on simple input. How about writing tests that make sure it runs "fast enough" on most of some larger inputs too?Is heuristic. You write unit tests that verify that the answer is “really close” to optimal on simple input, but also that it's “kind of close” on difficult input.By the way, did we mention project ZXing, Google's open-source decoder project? Or that Print Ads is already helping clients place these two-dimensional barcodes in the New York Times? Or that there are other formats like Data Matrix? or that you can put more than just a URL in these barcodes? This is a technology going global, so, time to read up on it.

4 comments :

UnknownAugust 8, 2008 at 11:02:00 AM PDT
Is there anything in Junit that allows you to say "report that this test passed or failed" but don't include it as a fatal failure? I'm curious how I could implement this....
ReplyDelete
Replies
srowenAugust 11, 2008 at 6:22:00 AM PDT
You can see what we do in the source -- we write a test that just asserts that at least x of y images decoded.

http://code.google.com/p/zxing/source/browse/trunk/core/test/src/com/google/zxing/common/AbstractBlackBoxTestCase.java
ReplyDelete
Replies
dastelsAugust 14, 2008 at 8:50:00 AM PDT
john: the Google C++ Testing Framework supports this idea, but jUnit does not (to my knowledge, I haven't looked at jUnit in close detail since 4.0 came out).
ReplyDelete
Replies
PascalAschwandenAugust 28, 2008 at 12:33:00 PM PDT
The most important thing is to look at the test case in terms of ROI. You get good ROI when the test case prevents errors from going unnoticed and when the test case has a high percentage of finding errors. As such, its best to start with system tests, then go down to unit tests.

Explore, Learn, Review: (programming languages)
http://www.codesplunk.com
ReplyDelete
Replies

Add comment

The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.

Testing Blog

TotT: A Matter of Black and White

4 comments :

Labels

Archive

Feed