Since Jan 1, 2010: 75% of the tests for my project have failed 0 times. 20 % have failed between 1 and 100 times. 5% have failed more than 100 times.
My diagnosis: The 20% are live tests, doing important work. The 5% are mostly flaky. The 75% probably can be broken into 2 sets, the 'keeping us honest' tests. They cover very basic, core components of our systems that change very, very rarely. BUT, if something were to change, we'd want to know immediately. The other set is probably dead code, we're testing because we like tests, not to protect our code base.
What does this mean? I'm not sure, but certainly we've got a lot of stale tests wasting CPU cycles and cluttering our test suites. For sure, this makes it difficult to understand our real tests.
Perhaps looking at some of these older, stale tests and modifying them could be a nice way to prevent test/code rot .
I agree with this if you are adding tests or getting rid of useless tests. But randomly changing test is probably not good. Hopefully you have a testing architecture that allows you to run a lot of tests and keep them organized.
I really appreciate your post and it was superb .Thanks for sharing.I would like to hear more about this in future. Regards: http://www.blackitsoft.com/inventory-pos-software.aspx
Since Jan 1, 2010:
ReplyDelete75% of the tests for my project have failed 0 times.
20 % have failed between 1 and 100 times.
5% have failed more than 100 times.
My diagnosis:
The 20% are live tests, doing important work.
The 5% are mostly flaky.
The 75% probably can be broken into 2 sets, the 'keeping us honest' tests. They cover very basic, core components of our systems that change very, very rarely. BUT, if something were to change, we'd want to know immediately. The other set is probably dead code, we're testing because we like tests, not to protect our code base.
What does this mean? I'm not sure, but certainly we've got a lot of stale tests wasting CPU cycles and cluttering our test suites. For sure, this makes it difficult to understand our real tests.
Perhaps looking at some of these older, stale tests and modifying them could be a nice way to prevent test/code rot .
Where is Part Five? =)
ReplyDeleteI agree with this if you are adding tests or getting rid of useless tests. But randomly changing test is probably not good. Hopefully you have a testing architecture that allows you to run a lot of tests and keep them organized.
ReplyDeleteI really appreciate your post and it was superb .Thanks for sharing.I would like to hear more about this in future.
ReplyDeleteRegards:
http://www.blackitsoft.com/inventory-pos-software.aspx