While there is no question that key functional code should have unit tests, the more pragmatic development groups I work with say, "Why should I test an obscure UI method? It will take longer to make the test than to write the six or eight lines of code for the whole method and besides, if I use Eclipse right, five of those lines are written for me before I write anything." The other line I hear is, "We're a manufacturing company, not a software vendor! Of course I expect our vendors to use cutting edge practices, but that's not us." How would you respond to these?
For some reason it’s a challenge to convince developers to test. The Return on Investment isn’t obvious to many developers.
I think one reason is that people mix the concepts of unit testing and integration testing, partly because a unit testing framework is used for both.
People new to unit testing frameworks often start writing integration tests because they’re the easiest to write. Unfortunately they’re also the most brittle, and tend to fail for no apparent reason — when some dependency isn’t properly configured. After a few such false positives developers grow tired of testing because they find the maintenance cost too high.
Another reason is that unit testing is hard. Getting used to think in terms of testability doesn’t come natural to many developers. The same goes for dependency injection and mocking frameworks.
In addition many frameworks used by the application aren’t written with testability in mind. And so extra effort must go into mocking their APIs.
Finally, in my experience, organizing tests so that a team of developers are comfortable with the structure is paramount to success. Basically, a common set of guidelines should be in place from early on and at least one team member must be passionate about testing for it to take off.
I agree strongly with your main points, about the value of both unit testing and test-driven development. But I do have some worries about subsidiary points, and also think you need to distinguish the two more emphatically. Unit testing when you don’t already have a framework to use can be hard, perhaps even impractical—but even in such cases test-driven development is still useful.
And I think we all need to be wary of Beizer’s Pesticide Paradox and Whittaker’s False Sense of Thoroughness. Google, for all its devotion to unit testing, and the overall quality level of its products, is not immune to this. (Specifics on request.) You call some simple high-level testing “A complete waste of my time when I could have known the same thing by just running my unit tests.” But even though unit tests may well be better than high-level tests, they do tell you different things. I’ve seen a lot of products which passed their unit tests but had glaring high-level bugs.
@Jay Mumper, for the first one, while I don't argue that it takes more time, the six or eight lines more often than not grows into a much bigger / longer method on an active project. And before long, you have this long, delicate method where any mucking around leads to bugs. Wouldn't spending a few extra minutes writing some tests around that save you hours later down ? As for the second one, if you are the one developing a software, I don't are what company you are, you are a software engineer, and it becomes your responsibility to make sure you test your application. Now whether you want to drain thousands of dollars in manual testing or write some unit tests and reduce the strain on qa efforts down the line, thats a call you will have to make.
@Flash, I completely agree that you do need higher levels of test. There is no way unit tests are going to catch integration issues and issues in interaction between components. What I do strongly hate is the lack of either and then just running your app and manually testing it. That is what I was complaining about in that line you quoted. Focussing just on higher level testing and ignoring unit testing is a huge glaring mistake in my opinion.
@Liu, check out the Testability Explorer project (http://testability-explorer.googlecode.com/) which has a pretty good hygiene of writing tests along with all their code, as well as integration tests for some of the more complicated interactions. In particular, check out the core plugin under src.
@Dave, if you are planning to work on an entirely new codebase and want to know how to tdd in such an environment, then Kent Beck's TDD book ( http://www.amazon.com/Test-Driven-Development-Addison-Wesley-Signature/dp/0321146530) comes to mind. But it doesn't talk about working with legacy code bases much and assumes you are familiar with the XUnit pattern. Most of the stuff I learnt has been from pairing with other developers and trying out different techniques and seeing what works and what doesn't. In the end, it does become as simple as, write a test, write the code to get it to pass, and iterate. And as long as you follow the base principles listed at (http://misko.hevery.com/code-reviewers-guide/), then you should be good to go.
Request for an article. How would you unit test a form class. Example code: $form->text('username', array('class' => 'users')); This creates an input text field called username Thankyou
This is all fine and great, but what do you do in the case where you have code that is inherently not machine testable, because the output requires human judgement? This is often the case in my projects as I work on Natural Language Processing - unit testing this is entirely nontrivial.
Second, my biggest gripe with unit testing is that it can lull you into a false sense of security. You can be inclined to think your code works and does what it should, when in reality it only behaves well for the test cases covered in your unit tests. For more complex systems it is very easy to miss out on many corner cases in your unit tests, and as a developer it is very frustrating to be handed code that "works because the unit tests say so" only to find out that the unit tests didn't cover my use cases.
@Toby This is an interesting issue (the false sense of security). What I have found is that by adopting more traditional QA processes, such as test plans, it's much easier to see unit tests in context. This tends to help in avoiding any particular type of test becoming "god". Now, it can sometimes be challenging to adapt existing organizational practices to incorporate automated testing organically, because they are often tuned for manual testers. But the rewards are usually worth it, especially if there are a wide variety of tests in use.
Good post. I've posted a "made me think" response in the Typemock Blog . We're doing a webinar next week about this very topic - how to motivate people to start unit testing in an organization.
Unit tests are basically written and executed by software developers to make sure that code meets its design and requirements and behaves as expected. The goal of unit testing is to segregate each part of the program and test that the individual parts are working correctly.
While there is no question that key functional code should have unit tests, the more pragmatic development groups I work with say, "Why should I test an obscure UI method? It will take longer to make the test than to write the six or eight lines of code for the whole method and besides, if I use Eclipse right, five of those lines are written for me before I write anything." The other line I hear is, "We're a manufacturing company, not a software vendor! Of course I expect our vendors to use cutting edge practices, but that's not us." How would you respond to these?
ReplyDeleteRight on!
ReplyDeleteI’d like to supplement your reasoning with a couple of guidelines for writing and organizing tests:
http://www.bugfree.dk/blog/2009/06/19/my-unit-testing-guidelines
For some reason it’s a challenge to convince developers to test. The Return on Investment isn’t obvious to many developers.
I think one reason is that people mix the concepts of unit testing and integration testing, partly because a unit testing framework is used for both.
People new to unit testing frameworks often start writing integration tests because they’re the easiest to write. Unfortunately they’re also the most brittle, and tend to fail for no apparent reason — when some dependency isn’t properly configured. After a few such false positives developers grow tired of testing because they find the maintenance cost too high.
Another reason is that unit testing is hard. Getting used to think in terms of testability doesn’t come natural to many developers. The same goes for dependency injection and mocking frameworks.
In addition many frameworks used by the application aren’t written with testability in mind. And so extra effort must go into mocking their APIs.
Finally, in my experience, organizing tests so that a team of developers are comfortable with the structure is paramount to success. Basically, a common set of guidelines should be in place from early on and at least one team member must be passionate about testing for it to take off.
I agree strongly with your main points, about the value of both unit testing and test-driven development. But I do have some worries about subsidiary points, and also think you need to distinguish the two more emphatically. Unit testing when you don’t already have a framework to use can be hard, perhaps even impractical—but even in such cases test-driven development is still useful.
ReplyDeleteAnd I think we all need to be wary of Beizer’s Pesticide Paradox and Whittaker’s False Sense of Thoroughness. Google, for all its devotion to unit testing, and the overall quality level of its products, is not immune to this. (Specifics on request.) You call some simple high-level testing “A complete waste of my time when I could have known the same thing by just running my unit tests.” But even though unit tests may well be better than high-level tests, they do tell you different things. I’ve seen a lot of products which passed their unit tests but had glaring high-level bugs.
@Jay Mumper, for the first one, while I don't argue that it takes more time, the six or eight lines more often than not grows into a much bigger / longer method on an active project. And before long, you have this long, delicate method where any mucking around leads to bugs. Wouldn't spending a few extra minutes writing some tests around that save you hours later down ? As for the second one, if you are the one developing a software, I don't are what company you are, you are a software engineer, and it becomes your responsibility to make sure you test your application. Now whether you want to drain thousands of dollars in manual testing or write some unit tests and reduce the strain on qa efforts down the line, thats a call you will have to make.
ReplyDelete@Flash, I completely agree that you do need higher levels of test. There is no way unit tests are going to catch integration issues and issues in interaction between components. What I do strongly hate is the lack of either and then just running your app and manually testing it. That is what I was complaining about in that line you quoted. Focussing just on higher level testing and ignoring unit testing is a huge glaring mistake in my opinion.
Can someone give an best example for these guidelines in a real opensource project? That would be a lot of help if exists. Great Thanks!
ReplyDeleteThis is really good stuff, thanks for posting!
ReplyDeleteDoes your team have a "Top 10" list of books that discuss the art of TDD that you could recommend?
Good question Liu. I'm convinced that thorough unit tests are something that are written about, but don't exist in real life.
ReplyDelete@Liu, check out the Testability Explorer project (http://testability-explorer.googlecode.com/) which has a pretty good hygiene of writing tests along with all their code, as well as integration tests for some of the more complicated interactions. In particular, check out the core plugin under src.
ReplyDelete@Dave, if you are planning to work on an entirely new codebase and want to know how to tdd in such an environment, then Kent Beck's TDD book ( http://www.amazon.com/Test-Driven-Development-Addison-Wesley-Signature/dp/0321146530) comes to mind. But it doesn't talk about working with legacy code bases much and assumes you are familiar with the XUnit pattern. Most of the stuff I learnt has been from pairing with other developers and trying out different techniques and seeing what works and what doesn't. In the end, it does become as simple as, write a test, write the code to get it to pass, and iterate. And as long as you follow the base principles listed at (http://misko.hevery.com/code-reviewers-guide/), then you should be good to go.
Request for an article.
ReplyDeleteHow would you unit test a form class.
Example code:
$form->text('username', array('class' => 'users'));
This creates an input text field called username
Thankyou
This is all fine and great, but what do you do in the case where you have code that is inherently not machine testable, because the output requires human judgement? This is often the case in my projects as I work on Natural Language Processing - unit testing this is entirely nontrivial.
ReplyDeleteSecond, my biggest gripe with unit testing is that it can lull you into a false sense of security. You can be inclined to think your code works and does what it should, when in reality it only behaves well for the test cases covered in your unit tests. For more complex systems it is very easy to miss out on many corner cases in your unit tests, and as a developer it is very frustrating to be handed code that "works because the unit tests say so" only to find out that the unit tests didn't cover my use cases.
@Shyam Seshadri, Great Thanks!
ReplyDeleteMisko Hevery is a great Coach! Thanks all your guys hard work!
@Toby
ReplyDeleteThis is an interesting issue (the false sense of security).
What I have found is that by adopting more traditional QA processes, such as test plans, it's much easier to see unit tests in context. This tends to help in avoiding any particular type of test becoming "god".
Now, it can sometimes be challenging to adapt existing organizational practices to incorporate automated testing organically, because they are often tuned for manual testers. But the rewards are usually worth it, especially if there are a wide variety of tests in use.
Hi Shyam,
ReplyDeleteGood post. I've posted a "made me think" response in the Typemock Blog . We're doing a webinar next week about this very topic - how to motivate people to start unit testing in an organization.
Unit tests are basically written and executed by software developers to make sure that code meets its design and requirements and behaves as expected.
ReplyDeleteThe goal of unit testing is to segregate each part of the program and test that the individual parts are working correctly.