Posted by Antoine Picard We have become test hoarders. Our focus on test-driven development, developer testing and other testing practices has allowed us to accumulate a large collection of tests of various types and sizes. Although this is valiant and beneficial, it is too easy to forget that each test, whether a unit test or a manual test has a cost as well. This cost should be balanced against the benefits of the test when deciding whether a test should be deleted or whether it should be written in the first place.
anonymized but were found at various time in our code tree. Take this test:def testMyModule (self):mymodule .main() mymodule , this test will only fail if main throws an exception. Certainly this is a useful condition to detect but it is wasteful to consume the time of a full run of mymodule without verifying its output. Let's look at another low-value test:def testFooInitialization (self):bak ') Foo's constructor, catching all exceptions means that the test will never fail. It creates coverage but never catches any bugs.QA is not testing what the PM cares about. Some even apply the broken-window principle to these tests saying that tests that don't get fixed give the impression that testing is not valuable.
Our final specimen is slightly higher value but still not very high. Consider the function Bar: def Bar():SlowBarHelper 1()SlowBarHelper 2()SlowBarHelper 3()
We could employ stubs or mocks to write a quick unit test of Bar but all we could assert is that the three helpers got called in the right order. Hardly a very insightful test. In non-compiled languages, this kind of test does serve as a substitute syntax-checker but provides little value beyond that.refactor , ...) and slightly lesser benefits (confidence about a small pieces of the system as opposed to the overall system). By contrast, regression tests tend to incur a greater cost (since most regression tests are large tests) but attempt to maximize their benefits by targeting areas of previous failures under the assumption that those are most likely to have bugs in the future.
Great post! Another benefit of unit testing is increased development speed. Developers attempt more diverse solutions when they have the confidence that a well-rounded unit test suite will catch them if they fall.
ReplyDeleteJust enlarging on high valuable tests. Their value should be measured (if it is possible) by how many errors it does manage to find before the tested feature is released.
ReplyDeleteThe trouble with this is that is very hard to account for this kind of events. Imagine a developer that has done a unit test for his method and found that a parameter for this method is not correctly checked for constraint violations declared on the documentation. This all is done @ development time. The A-HA moment could measured only if, by good will, the developer remembered to write it down.