class Database { // some fields declared here boolean isDirty(Cache cache, Object obj) { for (Object cachedObj : cache.getObjects) { if (cachedObj.equals(obj)) return false; } return true; } }
class Authenticator { Ldap ldap; Cookie login(User user) { if (user.isSuperUser()) { if ( ldap.auth(user.getUser(), user.getPassword()) ) return new Cookie(user.getActingAsUser()); } else (user.isAgent) { return new Cookie(user.getActingAsUser()); } else { if ( ldap.auth(user.getUser(), user.getPassword()) ) return new Cookie(user.getUser()); } return null; } }
class User { String user; String password; boolean isAgent; boolean isSuperUser; String actingAsUser; Cookie login(Ldap ldap) { if (isSuperUser) { if ( ldap.auth(user, password) ) return new Cookie(actingAsUser); } else (user.isAgent) { return new Cookie(actingAsUser); } else { if ( ldap.auth(user, password) ) return new Cookie(user); } return null; } }
class User { String user; String password; Cookie login(Ldap ldap) { if ( ldap.auth(user, password) ) return new Cookie(user); return null; } } class SuperUser extends User { String actingAsUser; Cookie login(Ldap ldap) { if ( ldap.auth(user, password) ) return new Cookie(actingAsUser); return null; } } class AgentUser extends User { String actingAsUser; Cookie login(Ldap ldap) { return new Cookie(actingAsUser); } }
by Juergen Allgayer, Conference Chair
Testing for the Web is the theme of the 4th Google Test Automation Conference (GTAC), to be held in Zurich, October 21-22. We are happy to announce that we are now accepting applications for attendance. The success of the conference depends on active participation of the attendees. Because the available spaces for the conference are limited, we ask each person to apply for attendance. Since we aim for a balanced audience of seasoned practitioners, students and academics, we ask the applicants to provide a brief background statement.How to applyPlease visit http://www.gtac.biz/call-for-attendance to apply for a attendance.DeadlinePlease submit your application until August 28, 2009 at the latest.Registration FeesThere are no registration fees. We will send out detailed registration instructions to each invited applicant. We will provide breakfast and lunch. There will be a reception on the evening of October 21.CancellationIf you applied but can no longer attend the conference please notify usimmediately by sending an email to gtac-2009-cfa@google.com sosomeone from the waiting list can get the opportunity instead.Further informationGeneral website: http://www.gtac.biz/Call for proposals: http://www.gtac.biz/call-for-proposalsCall for attendance: http://www.gtac.biz/call-for-attendanceAccommodations: http://www.gtac.biz/accomodations
By James A. Whittaker
And now for the last plague in this series. I hope you enjoyed them (the posts ...not the plagues!)
Imagine playing a video game blindfolded or even with the heads up display turned off. You cannot monitor your character's health, your targeting system is gone. There is no look ahead radar and no advance warning of any kind. In gaming, the inability to access information about the campaign world is debilitating and a good way to get your character killed.
There are many aspects of testing software that fall into this invisible spectrum. Software itself is invisible. We see it only through the UI with much of what is happening doing so under the covers and out of our line of sight. It’s not like building a car in which you can clearly see missing pieces and many engineers can look at a car and get the exact same view of it. There is no arguing whether the car has a bumper installed, it is in plain sight for everyone involved to see. Not so with software which exists as magnetic fluctuations on storage media. It’s not a helpful visual.
Software testing is much like game playing while blindfolded. We can't see bugs; we can't see coverage; we can't see code changes. This information, so valuable to us as testers, is hidden in useless static reports. If someone outfitted us with an actual blindfold, we might not even notice.
This blindness concerning our product and its behavior creates some very real problems for the software tester. Which parts of the software have enjoyed the most unit testing? Which parts have changed from one build to the next? Which parts have existing bugs posted against them? What part of the software does a specific test case cover? Which parts have been tested thoroughly and which parts have received no attention whatsoever?
Our folk remedy for the blindness plague has always been to measure code coverage, API/method coverage or UI coverage. We pick the things we can see the best and measure them, but do they really tell us anything? We’ve been doing it this way for years not because it is insightful, but simply because it is all our blindness will allow us to do. We’re interacting with our application under test a great deal, but we must rely on other, less concrete senses for any feedback about our effort.
Software testers could learn a lot from the world of gaming. Turn on your heads up display and see the information you've been blind to. There's power in information.
There are two communities who regularly find bugs, the testers who are paid to find them and the users who stumble upon them quite by accident. Clearly the users aren’t doing so on purpose, but through the normal course of using the software to get work (or entertainment, socializing and so forth) done failures do occur. Often it is the magic combination of an application interacting with real user data on a real user’s computing environment that causes software to fail. Isn’t it obvious then that testers should endeavor to create such data and environmental conditions in the test lab in order to find these bugs before the software ships?
Actually, the test community has been diligently making an attempt to do just that for decades. I call this bringing the user into the test lab, either in body or in spirit. My own PhD dissertation was on the topic of statistical usage testing and I was nowhere near the first person to think of the idea as my multi-page bibliography will attest. But there is a natural limit to the success of such efforts. Testers simply cannot be users or simulate their actions in a realistic enough way to find all the important bugs. Unless you actually live in the software you will miss important issues.
It’s like homeownership. It doesn’t matter how well the house is built. It doesn’t matter how diligent the builder and the subcontractors are during the construction process. The house can be thoroughly inspected during every phase of construction by the contractor, the homeowner and the state building inspector. There are just some problems that will only be found once the house is occupied for some period of time. It needs to be used, dined in, slept in, showered in, cooked in, partied in, relaxed in and all the other things homeowners do in their houses. It’s not until the teenager takes an hour long shower while the sprinklers are running that the septic system is found deficient. It’s not until a car is parked in the garage overnight that we find out the rebar was left out of the concrete slab. The builder won't and often can't do these things.
And time matters as well. It takes a few months of blowing light bulbs at the rate of one every other week to discover the glitch in the wiring and a year has to pass before the nail heads begin protruding from the drywall. These are issues for the homeowner, not the builder. These are the software equivalents of memory leaks and data corruption, time is a necessary element in finding such problems.
These are some number of bugs that simply cannot be found until the house is lived in and software is no different. It needs to be in the hands of real users doing real work with real data in real environments. Those bugs are as inaccessible to testers as nail pops and missing rebar are to home builders.
Testers are homeless. We can do what we can do and nothing more. It’s good to understand our limitations and plan for the inevitable “punch lists” from our users. Pretending that once an application is released the project is over is simply wrong headed. There is a warranty period that we are overlooking and that period is still part of the testing phase.
by Patrick Copeland
We get questions once in a while about our readership. Here's a brief summary of the last 30 days...
Page view count: 34,140Time on each page: 2:52Most popular day to read: Tuesday's.Most traffic (top 5 in order): US, India, UK, Germany, CanadaNumber of countries with readers: 131
Most popular posts...
"Testing is boring." Don’t pretend for a moment that you’ve never heard a developer, designer or other non tester express that sentiment and take a moment to search your own soul for the truth. Even the most bigoted tester would have to admit to contracting the boredom plague at some point. The day-in, day-out execution of tests and the filing of bug reports simply doesn't hold the interest of most technical people who are drawn to computing for its creative and challenging reputation. Even if you find yourself immune to the boredom, you have to admit there are many monotonous and uncreative aspects of testing.
It doesn’t begin that way though. Early in a tester’s career, the thrill of the bug hunt can keep a tester going for many months. It can be as intoxicating as playing a video game and trying to find some elusive prize. And lots of progress in terms of skill is made in those early years with testers going from novice to pretty good in no time flat. Who can argue with a career that offers such learning, advancement and intellectually interesting problems?
But as the achievement curve levels out, the task of testing can get very repetitive and that quickly turns to monotony. I think, promotion concerns aside, this is why many testers switch to development after a few years. The challenge and creativity gets eclipsed by the monotony.
I think bored testers are missing something. I submit that it is only the tactical aspects of software testing that become boring over time and many turn to automation to cure this. Automation as a potion against the tedium of executing test cases and filing bugs reports is one thing, but automation is no replacement for the strategic aspects of the testing process and it is in this strategy that we find salvation from this plague. The act of test case design, deciding what should and shouldn’t be tested and in what proportion, is not something automation is good at and yet it is an interesting and intellectually challenging task. Neither is the strategic problem of monitoring tests and determining when to stop. These are hard and interesting strategic problems that will banish the plague of boredom. Testers can succumb to the plague of boredom or they can shift their focus from mostly tactical to a nice mix of tactical work and strategic thinking.
Make sure that in your rush to perform the small tactical testing tasks you aren't dropping the strategic aspects of your work because therein are the interesting technical challenges and high level thinking that will hold your interest and keep this plague at bay.
There are two types of amnesia that plague software testers. We have team amnesia that causes us to forget our prior projects, our prior bugs, tests, failures and so forth. It’s time we developed a collective memory that will help us to stop repeating our mistakes. Every project is not a fresh start, it's only a new target for what is now a more experienced team. The star ship Enterprise keeps a ship's log. A diary that documents its crews’ adventures and that can be consulted for details that might help them out of some current jam. I'm not advocating a diary for test teams, but I do want a mechanism for knowledge retention. The point is that as a team we build on our collective knowledge and success. The longer the Enterprise crews' memory, the better their execution.
Quick, tell me about the last major failure of your teams’ product? Does your team have a collective memory of common bugs? Do you share good tests? If one person writes a test that verifies some functionality does everyone else know this so they can spend their time testing elsewhere? Do problems that break automation get documented so the painstaking analysis to fix them doesn’t have to be repeated? Does the team know what each other is doing so that their testing overlaps as little as possible? Is this accomplished organically with dashboards and ongoing communication or are the only sync points work-stopping, time-wasting meetings? Answer honestly. The first step to recovery is to admit you have a problem.
The second type of memory problem is industry amnesia. When I mentioned Boris Beizer and the pesticide paradox in my last post, how many of you had to look it up? And those who did know about it, how's your AJAX skills? Be honest...there are folks with both a historical perspective and modern skills but they are far too rare. Our knowledge, it seems, isn’t collective, it’s situational. Those who remember Beizer's insights worked in a world where AJAX didn’t exist, those who webspeak without trouble missed a lot of foundational thinking and wisdom. Remembering only what is in front of us is not really memory.
Industry amnesia is a real problem. Think about it this way: That testing problem in front of you right now (insert a problem that you are working on here) has been solved before. Are you testing an operating system? Someone has, many of them. Web app? Yep, that’s been done. AJAX? Client-server? Cloud service? Yes, yes and yes. Chances are that what you are currently doing has been done before. There are some new testing problems but chances are the one in front of you now isn’t one of them. Too bad the collective memory of the industry is so bad or it would be easy to reach out for help.
Let me end this column by pointing my finger inward: How will we (Google) test the newly announced Chrome operating system? How much collective knowledge have we developed from Chrome and Android? How much of what we learned testing Android will help? How much of it will be reusable? How easily will the Chrome and Andriod test teams adapt to this new challenge? Certainly many test problems are ones we've faced before.
Will we remember?
Take your average developer and ask "do you know language/technology X?" None of us will feel any shame in admitting that we do not know X. After all there are so many languages, frameworks and technologies, how could you know them all? But what if X is writing testable code? Somehow we have trouble answering the question "do you know how to write tests?" Everyone says yes, whether or not we actually know it. It is as if there is some shame in admitting that you don't know how to write tests.
Now I am not suggesting that people knowingly lie here, it is just that they think there is nothing to it. We think: I know how to write code, I think my code is pretty good, therefore my code is testable!
I personally think that we would do a lot better if we would recognize testability as a skill in its own right. And as such skills are not innate and take years of practice to develop. We could than treat it as any other skill and freely admit that we don't know it. We could than do something about it. We could offer classes, or other materials to grow our developers, but instead we treat it like breathing. We think that any developer can write testable code.
It took me two years of writing tests first, where I had as much tests as production code, before I started to understand what is the difference between testable and hard to test code. Ask yourself, how long have you been writing tests? What percentage of the code you write is tests?
Here is a question which you can ask to prove my point: "How do you write hard to test code?" I like to ask this question in interviews and most of the time I get silence. Sometimes I get people to say, make things private. Well if visibility is your only problem, I have a RegExp for you which will solve all of your problems. The truth is a lot more complicated, the code is hard to test doe to its structure, not doe to its naming conventions or visibility. Do you know the answer?
We all start at the same place. When I first heard about testing I immediately thought about writing a framework which will pretend to be a user so that I can put the app through its paces. It is only natural to thing this way. This kind of tests are called end-to-end-tests (or scenario or large tests), and they should be the last kind of tests which you write not the first thing you think of. End-to-end-tests are great for locating wiring bugs but are pretty bad at locating logical bugs. And most of your mistakes are in logical bugs, those are the hard ones to find. I find it a bit amusing that to fight buggy code we write even more complex framework which will pretends to be the user, so now we have even more code to test.
Everyone is in search of some magic test framework, technology, the know-how, which will solve the testing woes. Well I have news for you: there is no such thing. The secret in tests is in writing testable code, not in knowing some magic on testing side. And it certainly is not in some company which will sell you some test automation framework. Let me make this super clear: The secret in testing is in writing testable-code! You need to go after your developers not your test-organization.
Now lets think about this. Most organizations have developers which write code and than a test organization to test it. So let me make sure I understand. There is a group of people which write untestable code and a group which desperately tries to put tests around the untestable code. (Oh and test-group is not allowed to change the production code.) The developers are where the mistakes are made, and testers are the ones who feel the pain. Do you think that the developers have any incentive to change their behavior if they don't feel the pain of their mistakes? Can the test-organization be effective if they can't change the production code?
It is so easy to hide behind a "framework" which needs to be built/bought and things will be better. But the root cause is the untestable code, and until we learn to admit that we don't know how to write testable code, nothing is going to change...
We all have separation anxiety. Its a human tendency. We love inertia, and we don’t like change. But why should your code have separation anxiety ? Its not human (even though it might try and grow a brain of its own at times)!
I bring this up because I got the chance to work with someone who had some questions on how to test UI code. Fairly innocuous question, and in most cases, my response would have been, keep the UI code simple and free of any logic, and don’t worry too much about it. Or you could write some nice end to end / integration tests / browser based tests. So with that response set in mind, I set off into the unknown. Little was I to know was that was the least of my concerns. As I sat down to look at the code, I saw that there were already tests for the code. I was optimistic now, I mean, how bad could things be if there are already tests for it ?
Well, I should remember next time to actually look at the tests first. But anyways, there were tests, so I was curious what kinds of tests they wanted to write and what kind of help I could provide, if any. So it turns out, the class was some sort of GUI Component, which basically had some fields, and depending on whether they were set of not, displayed them as widgets inside of it. So there were, say, 5 fields, and differing combinations of what was set would produce different output. The nice thing was that the rendered data was returned as a nice Java object, so you could easily assert on what was set, and get a handle on the widgets inside of it, etc. So the method was something along the following lines :
public RenderedWidgetGroup render() { RenderedWidgetGroup group = createWidgetGroup(this.componentId, this.parent); if (this.name = null) { return group; } group.addWidget(new TextWidget(this.name)); group.addWidget( new DateWidget( this.updatedTimestamp == null ? this.createdTimestamp : this.updatedTimestamp)); return group;}
So far, so good, right ? Nice, clean, should be easy to test if we can set up this component with the right fields. After that, it should just be a few tests based on the different code paths defined by the conditionals. Yeah, thats what I thought too. So, me, being the naive guy that I was, said, yeah, that looks nice, should be easy to test. I don’t see the problem.
Well, then I was taken to the tests. The first thing I see is a huge test. Or atleast thats what I think it is. Then I read it more closely, and see its a private method. It seems to take in a bunch of fields (Fields with names that I distinctly remembered from just a while ago) and churn out a POJO (Plain Old Java Object). Except this POJO was not the GUI Component object I expected. So obviously, I was curious (and my testing sensors were starting to tingle). So I followed through to where it was called (which wasn’t very hard, it was a private method) and lo and behold, my worst fears confirmed.
The POJO object generated by the private method was passed in to the constructor of the GUI component, which (on following it further down the rabbit hole) in its constructor did something like the following :
public MyGUIComponent(ComponentId id, Component parent, MyDataContainingPOJO pojo) { super(id, parent); readData(pojo);}
And of course, readData, as you would expect, is a :
And of course, without writing the exact opposite of this method in the unit test, it just wasn’t possible to write unit tests. And sudddenly, it became clear why they were having trouble unit testing their Gui Components. Because if they wanted to test render (Which was really their aim), they would have to set up this POJO with the correct fields (which of course, to make things more interesting / miserable, had sub POJOs with sub POJOs of their own. Yay!) to be exercised in their test.
The problem with this approach is two fold :
This also adds, as I saw, obviously complicated pre test setup logic which should not be required to test something completely different. This is a HUGE code smell. Unit testing a class should not be an arduous, painful task. It should be easy as setting up a POJO and testing a method. The minute you have to perform complicated setup to Unit test a class (Note, the keyword is unit test. You can have integration tests which need some non trivial setup.), stop! There is something wrong.
The problem here is that there is a mixing of concerns in the MyGuiComponent class. As it turns out, MyGuiComponent breaks a few fundamental rules of testability. One, it does work in the constructor. Two, it violates the law of demeter, which states that the class should ask for what it needs, not what it doesn’t need to get what it needs (Does that even make sense ?). Thirdly, it is mixing concerns. That is, it does too much. It knows both how to create itself as well as do the rendering logic. Let me break this down further :
If you scroll back up and look at the constructor for MyGuiComponent, you will see it calling readData(pojo). Now, the basic concept to test any class is that you have to create an instance of the class under test (unless it has static methods. Don’t get me started on that…). So now, every time you create an instance of MyGuiComponent, guess what ? readData(pojo) is going to get called. Every. Single. Time ! And it cannot be mocked out. Its a private method. And god forbid if you really didn’t care about the pojo and passed in a null. Guess what ? It most probably will blow up with a NullPointerException. So suddenly, that innocuous method in the constructor comes back to haunt you when you write yours tests (You are, aren’t you ?).
Furthermore, if you look at what readData(pojo) does, you would be even more concerned. At its base, MyGuiComponent only cares about 6 fields which it needs to render. Depending on the state of each of these fields, it adds widget. So why does the constructor ask for something totally unrelated ? This is a fundamental violation of the Law of Demeter. The Law of Demeter can be summed up as “Ask for what you need. If you need to go through one object to get what you need, you are breaking it.”. A much more fancier definition can be found on the web if you care, but the minute you see something like A.B().C() or something along those lines, there is a potential violation.
In this case, MyGuiComponent doesn’t really care about the POJO. It only cares about some fields in the POJO, which it then assigns to its own fields. But the constructor still asks for the POJO instead of directly asking for the fields. What this means is that instead of just creating an instance of MyGuiComponent with the required fields in my test, I now have to create the POJO with the required fields and pass that in instead of just setting it directly. Convoluted, anyone ?
Finally, what could be considered an extension of the previous one, but deserves a rant of its own. What the fundamental problem with the above class is that it is mixing concerns. It knows both how to create itself as well as how to render itself once it has been created. And the way it has been coded enforces that the creation codepath is executed every single time. To provide an analogy for how ridiculous this is, it is like a a Car asking for an Engine Number and then using that number to create its own engine. Why the heck should a car have to know how to create its engine ? Or even a car itself ? Similarly, why should MyGuiComponent know how to create itself ? Which is exactly what is happening here.
Now that we had arrived at the root of the problem, we immediately went about trying to fix it. My immediate instinct was to pull out MyGuiComponent into the two classes that I was seeing. So we pulled out a MyGuiComponentFactory, which took up the responsibility of taking in the POJO and creating a GuiComponent out of it. Now this was independently testable. We also added a builder pattern to the MyGuiComponent, which the factory leveraged.
class MyGuiComponentFactory { MyGuiComponent createFromPojo(ComponentId id, Component parent, MyDataContainingPOJO pojo) { // Actual logic of converting from pojo to // MyGuiComponent using the builder pattern }}class MyGuiComponent { public MyGuiComponent(ComponentId id, Component parent) { super(id, parent); } public MyGuiComponent setName(String name) { this.name = name; return this; }}
And now, suddenly (and expectedly, I would like to add), the constructor for MyGuiComponent becomse simple. There is no work in the constructor. The fields are set up through setters and the builder pattern. So now, we started writing the unit tests for the render methods. It took about a single line of setup to instantiate MyGuiComponent, and we could test the render method in isolation now. Hallelujah!!
Disclaimer :No code was harmed / abused in the course of the above blog post. There were no separation issues whatsoever in the end, it was a clean mutual break!
