By Philip Zembrod
In my quest to explore TDD I recently found another propery of TDD-written code that I hadn't expected: When reviewing or just reading such code, it's often best to first read the tests.
When I look at new code or a code change, I ask: What is this about? What is it supposed to do? Questions that tests often have a good answer for. They expose interfaces and state use cases. This is cool, I thought, and decided to establish test-first reading as my code-reviewing routine. Of course this just applies the specification aspect of tests: Reading the specs before reading the code.
Only it didn't always work. From some tests I just failed to learn the point and intention of the tested code. Often, though not always, these were tests that were heavy with mocks and mock expectations.
Mocks aren't always a helpful tool, was my first conclusion. The phrase "Good mocks, bad mocks" popped up in my mind. I began to appreciate fakes again - and the people who write them. But soon I realized that this was about more than mocks vs. fakes vs. dummies vs. other Friends You Can Depend On. I was really looking at how well tests fulfill their role as specification.
TDD teaches that tests are a better specification than prose. Tests are automatically enforced, and get stale less easily. But not all tests work equally well as specification! That's what test driven code reviewing taught me.
I began to call them well-specifying tests and poorly-specifying tests. And the specification aspect isn't just some additional benefit, it's a really crucial property of tests. The more I thought about it, the more I saw: It is connected to a lot of things that first weren't obvious to me:

If tests are poorly-specifying, then possibly the tested product is poorly specified or documented. After all, it's the tests that really make sure how a product behaves. If they don't clearly state what they test, then it's less clear how the product works. That's a problem.

Well-specifying tests are more robust. If a test just does and verifies things of which the architect or product manager will readily say "yes, we need that" then the test will survive refactorings or new features. Simply because "yes, we need that." The test's use case is needed, its conditions must hold. It needn't be adapted to new code, new code must pass it. False positives are less likely.

Corollary: Well-specifying tests have higher authority. If a test fails, a natural reaction is to ask "is this serious?" If a test is poorly-specifying, if you don't really understand what it is testing, then you may say "well, maybe it's nothing". And you may even be right! If a test is well-specifying, you'll easily see that its failing is serious. And you'll make sure the code gets fixed.
I'm now thinking about an authority rank between 0 and 1 as a property of tests. It could be used to augment test coverage metrics. Code that is just covered by poorly-specifying tests would have poor authority coverage, even if the coverage is high. Quantifying an authority rank would be a conceptual challenge, of course, but part of it could be how well test driven code reviewing works with a given test.
P.S. If anyone suspects that I'm having some fun inventing terms beginning with "test driven," I'll plead guilty as charged. :-)

6 comments

Google

Labels: Philip Zembrod

Test Driven Integration

Tuesday, June 22, 2010

By Philip Zembrod

In an earlier post on trying out TDD I wrote how my mindset while coding changed from fear of bugs in the new code to eager anticipation to see the new code run through and eventually pass the already written tests. Today I want to tell about integrating components by writing integration tests first.

In a new project we decided to follow TDD from the start. We happily created components, “testing feature after feature into existence” (a phrase I love; I picked it up from a colleague), hitting a small test coverage of around 90% from the start. Obviously, when it came to integrating the components into a product, the obvious choice was to do that test-driven, too. So how did that go?

What I would have done traditionally was select a large enough set of components that, once integrated, should make up something I could play with. Since at least a minimum UI would be needed, plus something that does visible or useful things, preferably both, this something would likely have been largish, integrating quite a few components. With the playing around and tryout, I’d enter debugging, because of course it wouldn’t work at first attempt. The not-too-small number of integrated components would make tracking the cause of failures hard, and anticipating all this while coding, I’d have met the well-known fearful mindset again, slowing me down, as I described in my initial TDD post.

How did TDI change this game for me? I realized: With my unit test toolbox that can test any single component, I can also test an integration of 2 components regardless of whether they have a UI or do something visible. That was the key to a truly incremental process of small steps.

First, write the test for 2 components, run it and see it fail to make sure the integration code I’m about to write is actually executed by the test. Write that bit of code, run the test and see it succeed. If it still fails, fix what's broken and repeat. Finding what's broken in this mode is usually easy enough because the increments are small. If the test failure doesn’t make obvious what’s wrong, adding some verifications or some logging does the trick. A debugger should never be needed; automated tests are, after all, a bit like recorded debugging sessions that you can replay any time in the future.

Repeating this for the 3rd component added, the 4th, etc., I could watch my product grow, with new passing tests every day. Small steps, low risks in each, no fear of debugging; instead, continuous progress. Every day this roaring thrill: It works, it works! Something’s running that didn’t run yesterday. And tomorrow morning I’ll start another test that will run tomorrow evening, most likely. Or already at lunchtime. Imagine what kind of motivation and acceleration this would give you. Better, try it out for yourself. I hope you’ll be as amazed and excited as I am.

What are the benefits? As with plain TDD, I find this fun-factor, this replacement of dread of debugging by eagerness for writing the next test to be able to write and run the next code the most striking effect of TDI.

The process is also much more systematic. Once you have specified your expectations at each level of integration, you’ll verify them continuously in the future, just by running the tests. Compare that to how reproducible, thorough and lasting your verification of your integration would be if you’d done it manually.

And if you wrote an integration test for every function or feature that you cared about during integration, then you can make sure each of them is in shape any time by just running the tests. I suspect one can’t appreciate the level of confidence in the code that creates until one has experienced it. I find it amazing. I dare you to try it yourself!

P.S. On top of this come all the other usual benefits of well-tested code that would probably be redundant to enumerate here, so I won’t. ;-)

6 comments

Google

Labels: Philip Zembrod

GUI Testing: Don't Sleep Without Synchronization

Tuesday, October 28, 2008

Posted by Philip Zembrod, Software Engineer in Test, SwedenLife would be good if only the GUI tests weren't so flaky.GUIs are not designed to synchronize with other computer programs. They are designed to synchronize with human beings

Humans act much more slowly. Well-honed GUI test robots drive GUIs at near theoretical maximum speed.

Humans are much better at observing the GUI, and they react intelligently to what they see.

Humans extract more meaningful information from a GUI.

a GUI application usually lacks these means of synchronizationIs the app under test finished with what it's doing?Maybe you want to test something while "it" is underwayBut usually, you want to wait until "it" is doneHow to find out whether "it" is done? Ask!provide one or several test hooks suitable for your synchronization needswait while a process has pending input eventsIf the GUI app has pending input, it's surely not ready for new input.Is the GUI app's message queue empty?If the GUI app still has messages in its queue, it's not yet ready for new input.more like sending a probe than a questionthe test framework checks periodicallyto see whether the flag has been set.When the posted message (the probe) has been processed, then surely messages and events sent earlier to the GUI app have been processed.fast and stable test execution, without any test flakiness due to timing issues. All without sleeps,I'd be interested to hear if someone has done something similar

10 comments

Google

Labels: Philip Zembrod

Test first is fun!

Monday, September 08, 2008

Posted by Philip ZembrodWrite a failing test for each featureThen implement the feature until the test succeeds.a way to higher productivitygetting tests written was indeed one of the less important reasons to write tests first!leads you to think about the interface firstfocus on how the new interface will be usedregard the test as a coded specificationa technical safeguard against growing stale: if it does, the test will probably failTests are no longer a necessary chore, but the place and time where I start to design something new. see my code running the minute it is written and compiled!writing new code for complex systems was always accompanied by fearslowed me downNow, this fear is gone!It will cost me just a few keystrokes to run my finished code, and I will immediately see what it does.It's the same old enthusiasm that I felt more than 20 years ago when I wrote and ran my first programs.