@InjectStoreServiceImpl storeService;
public void testStorePortal() {... storeService.doSomething();...}
def testInternMakesCoffee(self): self.caffeinated = False def DrinkCoffee(): self.caffeinated = True DelegateToIntern(work=Intern().MakeCoffee, callback=DrinkCoffee) self.assertFalse(self.caffeinated, "I watch YouTubework; intern brews") time.sleep(60) # 1min should be long enough to make coffee, right? self.assertTrue(self.caffeinated, "Where's mah coffee?!?")
def testInternMakesCoffee(self): is_started, can_finish, is_done = Event(), Event(), Event() def FakeCoffeeMaker(): is_started.set() # Allow is_started.wait() to return. # Wait up to 1min for can_finish.set() to be called. The timeout # prevents failures from hanging, but doesn't delay a passing test. can_finish.wait(timeout=60) # .await() in Java DelegateToIntern(work=FakeCoffeeMaker, callback=lambda:is_done.set()) is_started.wait(timeout=60) self.assertTrue(is_started.isSet(), "FakeCoffeeMaker should have started") self.assertFalse(is_done.isSet(), "Don't bug me before coffee's made") can_finish.set() # Now let FakeCoffeeMaker return. is_done.wait(timeout=60) self.assertTrue(is_done.isSet(), "Intern should ping when coffee's ready")
by Miško Hevery
So you join a new project, which has an extensive mature code base. Your new lead asks you to implement a new feature, and, as a good developer, you start by writing a test. But since you are new to the project, you do a lot of exploratory "What happens if I execute this method" tests. You start by writing this:
testCreditCardCharge() {CreditCard c = new CreditCard( "1234 5678 9012 3456", 5, 2008);c.charge(100);}
This code:
Now, I want to focus on the last point. How in the world did the test cause an actual charge on my credit card? Charging a credit card is not easy. The test needs to talk to a third party credit card web-service. It needs to know the URL for the web-service. It needs to authenticate, pass the credentials, and identify who the merchant is. None of this information is present in the test. Worse yet, since I don't even know where that information is present, how do I mock out the external dependencies so that every run does not result in $100 actually being charged? And as a new developer, how was I supposed to know that what I was about to do was going to result in me being $100 poorer? That is "Spooky action at a distance!"
But why do I get NullPointerException in isolation while the test works fine when run as part of the suite? And how do I fix it? Short of digging through lots of source code, you go and ask the more senior and wiser people on the project. After a lot of digging, you learn that you need to initialize the CreditCardProcessor.
testCreditCardCharge() {CreditCardProcessor.init();CreditCard c = new CreditCard( "1234 5678 9012 3456", 5, 2008);c.charge(100);}
You run the test again; still no success, and you get a different exception. Again, you chat with the senior and wiser members of the project. Someone tells you that the CreditCardProcessor needs an OfflineQueue to run.
testCreditCardCharge() {OfflineQueue.init();CreditCardProcessor.init();CreditCard c = new CreditCard( "1234 5678 9012 3456", 5, 2008);c.charge(100);}
Excited, you run the test again: nothing. Yet another exception. You go in search of answers and come back with the knowledge that the Database needs to be initialized in order for the Queue to store the data.
testCreditCardCharge() {Database.init();OfflineQueue.init();CreditCardProcessor.init();CreditCard c = new CreditCard( "1234 5678 9012 3456", 5, 2008);c.charge(100);}
Finally, the test passes in isolation, and again you are out $100. (Chances are that the test will now fail in the suite, so you will have to surround your initialization logic with "if not initialized then initialize" code.)
The problem is that the APIs are pathological liars. The credit card pretends that you can just instantiate it and call the charge method. But secretly, it collaborates with the CreditCardProcessor. The CreditCardProcessor API says that it can be initialized in isolation, but in reality, it needs the OfflineQueue. The OflineQueue needs the database. To the developers who wrote this code, it is obvious that the CreditCard needs the CreditCardProcessor. They wrote the code that way. But to anyone new on the project, this is a total mystery, and it hinders the learning curve.
But there is more! When I see the code above, as far as I can tell, the three init statements and the credit card instantiation are independent. They can happen in any order. So when I am re-factoring code, it is likely that I will move and rearrange the order as a side-effect of cleaning up something else. I could easily end up with something like this:
testCreditCardCharge() {CreditCardProcessor.init();OfflineQueue.init();CreditCard c = new CreditCard( "1234 5678 9012 3456", 5, 2008);c.charge(100);Database.init();}
The code just stopped working, but I had no way to knowing that ahead of time. Most developers would be able to guess that these statements are related in this simple example, but on a real project, the initialization code is usually spread over many classes, and you might very well initialize hundreds of objects. The exact order of initialization becomes a mystery.
How do we fix that? Easy! Have the API declare the dependency!
testCreditCardCharge() {Database db = Database();OfflineQueue q = OfflineQueue(db);CreditCardProcessor ccp = new CreditCardProcessor(q);CreditCard c = new CreditCard( "1234 5678 9012 3456", 5, 2008);c.charge(ccp, 100);}
Since the CreditCard charge method declares that it needs a CreditCardProcessor, I don't have to go ask anyone about that. The code will simply not compile without it. I have a clear hint that I need to instantiate a CreditCardProcessor. When I try to instantiate the CreditCardProcessor, I am faced with supplying an OfflineQueue. In turn, when trying to instantiate the OfflineQueue, I need to create a Database. The order of instantiation is clear! Not only is it clear, but it is impossible to place the statements in the wrong order, as the code will not compile. Finally, explicit reference passing makes all of the objects subject to garbage collection at the end of the test; therefore, this test can not cause any other test to fail when run in the suite.
The best benefit is that now, you have seams where you can mock out the collaborators so that you don't keep getting charged $100 each time you run the test. You even have choices. You can mock out CreditCardProcessor, or you can use a real CreditCardProcessor and mock out OfflineQueue, and so on.
Singletons are nothing more than global state. Global state makes it so your objects can secretly get hold of things which are not declared in their APIs, and, as a result, Singletons make your APIs into pathological liars.
Think of it another way. You can live in a society where everyone (every class) declares who their friends (collaborators) are. If I know that Joe knows Mary but neither Mary nor Joe knows Tim, then it is safe for me to assume that if I give some information to Joe he may give it to Mary, but under no circumstances will Tim get hold of it. Now, imagine that everyone (every class) declares some of their friends (collaborators), but other friends (collaborators which are singletons) are kept secret. Now you are left wondering how in the world did Tim got hold of the information you gave to Joe.
Here is the interesting part. If you are the person who built the relationships (code) originally, you know the true dependencies, but anyone who comes after you is baffled, since the friends which are declared are not the sole friends of objects, and information flows in some secret paths which are not clear to you. You live in a society full of liars.
The progressive developer knows that in this complex modern world, things aren't always black-and-white, even in testing. Sometimes we know the software won't return the best answer, or even a correct answer, for all input. You may be tempted to write only test cases that your software can pass. After all, we can't have automated tests that fail in even one instance. But, this would miss an opportunity.
Speaking of black-and-white, take decoding of two-dimensional barcodes, like QR Codes. From a blurry, skewed, rotated image of a barcode, software has to pick it out, transform it, and decode it:
Even the best software can't always find that barcode. What should tests for such software do?
We have some answers, from experience testing such software. We have two groups of black-box tests that verify that images decode correctly: must-have and nice-to-have. Tests verify that the must-have set – the easy images – definitely decode correctly. This is what traditional tests would include, which typically demand a 100% pass rate. But we also see how well we do on the more difficult nice-to-have set. We might verify that 50% of them decode, and fail otherwise.
The advantage? We can include tougher test cases in unit tests, instead of avoiding them. We can observe small changes – improvements as well as degradations – in decode accuracy over time. It doubles as a crude quality evaluation framework.
Where can this progressive thinking be applied? Maybe when your code...
Only needs to be correct in most cases. As here, write tests to verify easy cases work, but also that some hard cases pass too.
Needs to be fast. You write unit tests that verify it runs "fast enough" on simple input. How about writing tests that make sure it runs "fast enough" on most of some larger inputs too?
Is heuristic. You write unit tests that verify that the answer is “really close” to optimal on simple input, but also that it's “kind of close” on difficult input.
By the way, did we mention project ZXing, Google's open-source decoder project? Or that Print Ads is already helping clients place these two-dimensional barcodes in the New York Times? Or that there are other formats like Data Matrix? or that you can put more than just a URL in these barcodes? This is a technology going global, so, time to read up on it.
Remember to download this episode of Testing on the Toilet and post it in your office.