Google Testing Blog: September 2009

TotT: Literate Testing With Matchers

Tuesday, September 29, 2009

By Zhanyong G. Mock Wan in Google Kirklandmatchersare predicates that know how to describe themselvesContainsRegex"Ahcho+!""Ahcho+!""Ahchoo!""Ahchoooo! Sorry.""Aha!"matcherslend themselves easily to an assertion style that resembles natural languages EXPECT_THAT(value, matcher);valuematcher #include <gmock/gmock.h>
using ::testing::Contains;
...
EXPECT_THAT(GetUserList(), Contains(admin_id));GetUserList()when an EXPECT_THAT assertion fails, it will print an informative message that includes the expression being validated, its value, and the property we expect it to have Value of: GetUserList()
Expected: contains "yoko"
  Actual: { "john", "paul", "george", "ringo" } std::vector<std::string> users = GetUserList();
EXPECT_TRUE(VectorContains(users, admin_id))
    << " GetUserList() returns " << users
    << " and admin_id is " << admin_id;http://code.google.com/p/googlemock/http://code.google.com/p/googlemock/wiki/CheatSheetToilet-friendly version

2 comments

Checked exceptions I love you, but you have to go

Wednesday, September 16, 2009

By Miško Hevery
Empirical EvidenceLost in NoiseUnreachable CodebytesToString(byte[] bytes) { ByteArrayOutputStream out = new ByteArrayOutputStream(); try { out.write(bytes); out.close() return out.toSring(); } catch (IOException e) { // This can never happen! // Should I rethrow? Eat it? Print Error? } }Closures Don't Like YouLost FidelityYou can't do Anything AnywayHow I deal with the code

Always catch all checked exceptions at source and rethrow them as LogRuntimeException.

LogRuntimeException is my runtime un-checked exception which says I don't care just log it.

Here I have lost Exception fidelity.

All of my methods do not declare any exceptions

As I discover that I need to deal with a specific exception I go back to the source where LogRuntimeException was thrown and I change it to <Specific>RuntimeException (This is rarer than you think)

I am restoring the exception fidelity only where needed.

Net effect is that when you come across a try-catch clause you better pay attention as interesting things are happening there.

Very few try-catch calluses, code is much easier to read.

Very close to 100% test coverage as there is no dead code in my catch blocks.

34 comments

The Plague of Entropy

Monday, September 14, 2009

Mathematically entropy is a measure of uncertainty. If there are, say, five events then maximum entropy occurs when those five events are equally likely and minimum entropy when one of those events is certain and the other four impossible.
The more uncertain events you have to consider, the higher measured entropy climbs. People often think of entropy as a measure of randomness: the more (uncertain) events one must consider, the more random the outcome becomes.
Testers introduce entropy into development by adding to the number of things a developer has to do. When developers are writing code, entropy is low. When we submit bugs, we increase entropy. Bugs divert their attention from coding. They must now progress in parallel on creating and fixing features. More bugs means more parallel tasks and raises entropy. This entropy is one reason that bugs foster more bugs ... the entropic principle ensures it. Entropy creates more entropy! Finally there is math to show what is intuitively appealing: that prevention beats a cure.
However, there is nothing we can do to completely prevent the plague of entropy other than create developers who never err. Since this is unlikely any time soon we must recognize how and when we are introducing entropy and do what we can to manage it. The more we can do during development the better. Helping out in code reviews, educating our developers about test plans, user scenarios and execution environments so they can code against them will reduce the number of bugs we have to report. Smoking out bugs early, submitting them in batches and making sure we submit only high quality bugs by triaging them ourselves will keep their mind on development. Writing good bug reports and quickly regressing fixes will keep their attention where it needs to be. In effect, it maximizes the certainty of the 'development event' and minimizes the number and impact of bugs. Entropy thus tends toward it's minimum.
We can't banish this plague but if we can recognize the introduction of entropy into development and understand its inevitable effect on code quality, we can keep it at bay.

4 comments

It is not about writing tests, its about writing stories

Wednesday, September 02, 2009

by Miško Hevery

the transmission should allow the output shaft to be locked, move in same direction (D) as the input shaft, move in opposite (R) or move independently (N)

the transmission should allow the ratio between input and output shaft to be [-1, 0, 1, 2, 3, 4]

the transmission should start in 1st and than switch to higher gear before the engine reaches maximum revolutions.

the transmission should down shift whenever the engine RPM fall bellow 1000 RPMs

the transmission should up-shift whenever the estimated fuel consumption at a higher gear ration is better than the current one.

the transmission should down-shift whenever the gas pedal is depressed more than 50% and the RPM is lower than the engine's peak output RPM.

test to make sure all of the gears work

test to make sure that the engine is not allowed to over-rev

at 3000 RPM input shaft, apply 100% throttle and assert that the transmission goes to 2nd gear.

creating a story which is forcing a particular design decision.

tests are a collection of assumptions which needs to be true at all times.

miss a lot of reasons why things are done in particular way even if we have 100% coverage

test are often brittle because they are tied to particulars of the current implementation

tests are just snapshots and don't tell a story of why the component does something, only that it does.

8 comments

The 7th Plague and Beyond

Wednesday, September 02, 2009

Sorry I haven't followed up on this, let the excuse parade begin: A) My new book just came out and I have spent a lot of time corresponding with readers. B) I have taken on leadership of some new projects including the testing of Chrome and Chrome OS (yes you will hear more about these projects right here in the future). C) I've gotten just short of 100 emails suggesting the 7th plague and that takes time to sort through.
This is clearly one plague-ridden industry (and, no, I am not talking about my book!)
I've thrown out many of them that deal with a specific organization or person who just doesn't take testing seriously enough. Things like the Plague of Apathy (suggested exactly 17 times!) just doesn't fit. This isn't an industry plague, it's a personal/group plague. If you don't care about quality, please do us all a favor and get out of the software business. Go screw someone else's industry up, we have enough organic problems we have to deal with. I also didn't put down the Plague of the Deluded Developer (suggested by various names 22 times) because it dealt with developers that as a Googler I no longer have to deal with ... those who think they never write bugs. Our developers know better and if I find out exactly where they purchased that clue I will forward the link.
Here's some of the best. As many of them have multiple suggesters I have credited the persons who were either first or gave the most thoughtful analysis. Feel free, if you are one of these people, to give further details or clarifications in the comments of this post as I am sure these summaries do not do them justice.
The Plague of Metrics (Nicole Klein, Curtis Pettit plus 18 others): Metrics change behavior and once a tester knows how the measurement works, they test to make themselves look good or say what they want it to say ignoring other more important factors. The metric becomes the goal instead of measuring progress. The distaste for metrics in many of these emails was palpable!
The Plague of Semantics (Chris LeMesurier plus 3 others): We misuse and overuse terms and people like to assign their own meaning to certain terms. It means that designs and specs are often misunderstood or misinterpreted. This was also called the plague of assumptions by other contributors.
The Plague of Infinity (Jarod Salamond, Radislav Vasilev and 14 others): The testing problem is so huge it's overwhelming. We spend so much time trying to justify our coverage and explain what we are and are not testing that it takes away from our focus on testing. Every time we take a look at the testing problem we see new risks and new things that need our attention. It randomizes us and stalls our progress. This was also called the plague of endlessness and exhaustion.
The Plague of Miscommunication (Scott White and 2 others): The language of creation (development) and the language of destruction (testing) are different. Testers write a bug report and the devs don't understand it and cycles have to be spent explaining and reexplaining. A related plague is the lack of communication that causes testers to redo work and tread over the same paths as unit tests, integration tests and even the tests that other testers on the team are performing. This was also called the plague of language (meaning lack of a common one).
The Plague of Rigidness (Roussi Roussev, Steven Woody, Michele Smith and 5 others): Sticking to the plan/process/procedure no matter what. Test strategy cannot be bottled in such a manner yet process heavy teams often ignore creativity for the sake of process. We stick with the same stale testing ideas product after product, release after release. This was also called the plague of complacency. Roussi suggested a novel twist calling this the success plague where complacency is brought about through success of the product. How can we be wrong when our software was so successful in the market?
And I have my own 7th Plague that I'll save for the next post. Unless anyone would like to write it for me? It's called the Plague of Entropy. A free book to the person who nails it.

8 comments

Testing Blog

TotT: Literate Testing With Matchers

Checked exceptions I love you, but you have to go

The Plague of Entropy

It is not about writing tests, its about writing stories

The 7th Plague and Beyond

Labels

Archive

Feed