If aimlessness is the result of ‘just doing it’ then repetitiveness is the result of ‘just doing it some more.’ Pass after pass, build after build, sprint after sprint, version after version we test our product. Developers perform reviews, write unit tests and run static analyzers. But we have little insight into this effort and can't take credit for it. Developers test but then we retest. We can’t assume anything about what they did so we retest everything. As our product grows in features and bug fixes get applied, we continue our testing. It isn’t long until new tests become old tests and all of them eventually become stale.
This is Boris Beizer’s pesticide paradox. Pesticide will kill bugs, but spray the same field enough times with the same poison and the remaining bugs will grow immune. Rinse and repeat is a process for cleaning one’s hair, not testing software. The last thing we need is a build full of super-bugs that resist our ‘testicide.’ Even worse, all that so-called successful testing will give us a false sense of thoroughness and make our completeness metrics a pack of very dangerous lies. When you aren't finding bugs it's not because there are no bugs, it's the repetition that's creating the pesticide paradox.
Farmers know to adjust their pesticide formula from time to time and also to adjust the formula for the specific type of pest they expect in their fields. They do this because they understand the history of pesticide they used and know they can't get by with brute force repetition of they same old poison. Testers must pay attention to their test results too and watch for automation that isn’t adding value. A healthy injection of variation into automation is called for. Change the order of tests, change their data source, find new environments, modify input values do something the bugs aren’t prepared for.
The Plague of Aimlessness
Lore. It’s more than just a cool word. It conjures up a sorcerous image in one’s mind of ancient spell books and learned wizards with arcane and perilously attained knowledge.
And it’s exactly what we lack in software testing. Testing lore? Are you kidding me? Where is it? Who’s bogarting it? Can I have a hit?
The software testing industry is infected with the plague of aimlessness. We lack lore; we lack a body of knowledge that is passed from wizard to apprentice and written down in spell books for the industrious to study. Our apprentices are without their masters. We must all reinvent the wheel in the privacy of our offices only to have other testers the world over reinvent it in theirs.
I suggest we stop this nonsense. Testing is far too aimless. We test because we must or our manager tells us to do so. We automate because we can or because we know how to, not because it is part of some specific and proven strategy and certainly not because our lore dictates it. Is there a plan or some documented wisdom that guides our testing or are we just banging on the keyboard hoping something will fail? Where are the testing spell books? Surely the perilously attained knowledge of our tester forebears is something that we can access in this age of readily available information?
When a hunter makes a kill, they remember the terrain and circumstances. They pass this knowledge on to their successors. Over time they understand the habits of their prey and the collective knowledge of many hunters makes the job of future hunters far easier. When you see this terrain, you can expect game to behave in this manner. Can we say the same of testing? How well do we learn from each other? Do our ‘eureka moments’ get codified so that future testers will not have to suffer the aimless thrashing that we suffered? Can we say when you see functionality like that, the best way to test it is like this?
The plague of aimlessness is widespread. The need for testing lore is acute. Nike tells us to ‘just do it’ but what applies to exercise is not good advice for software testing. The next time you find yourself ‘just doing’ testing, pause for a moment and ask yourself ‘what is my goal?’ and ‘is there a purpose to this test?’ If the answer doesn’t immediately come to mind, you’re aimless, just doing it, and relying on luck and the sheer force of effort to find your quarry.
Luck has no place in sorcery or hunting and it has no place in testing. Luck is a nice happenstance, but it cannot be our plan A. Watch for the plague of aimlessness. Document your successes, scrutinize your failures and make sure you pass on what you learn from this introspection to your colleagues.
Be their wizard. Build a testing spell book and share it with others on your team. Over time you’ll banish the plague of aimlessness.
The 4th Google Test Automation Conference brings together a selected set of industry practitioners around the topic of software testing and automation. This annual conference provides a forum for presentations and connects professionals with each other. To increase outreach, presentations are published online for everybody to see.
This years theme is Testing for the Web, topics may include:
Presentations are targeted at experienced engineers actively working on problems of quality, test automation and techniques, but also include students and academics. We encourage innovative ideas, controversial experiences, problems, and solutions that further the discussion of software engineering and testing. Presentations are 45 min in length and speakers should be prepared for an active question and answer session following their presentation. While ideas are good, ideas refined by experience are even more interesting to participants at GTAC.
The conference is a two day event comprised of a single track of talks. Our philosophy is to engage a small set of active participants who all experience the same topics carrying the discussions into lightning talks, speaker Q&A, and topical discussion groups. Each year we have worked to identify a location that has a unique profile of technology professionals. This year the conference will be held at the Google office in Zurich, Switzerland on October 21 and 22, 2009.Submission of ProposalsPlease email a detailed and extended abstract (one page at most) to gtac-2009-cfp@google.com. Your submission must include the name of topic, author(s), affiliation, and an outline of the proposed talk. We strongly recommend you to also submit one or two highlight slides of the talk. Submit your proposal before August 1, 2009. We will acknowledge reception within one business day. Where employer or disclosure authorization is needed, authors need to obtain it prior to submitting. The program committee will evaluate proposals based on quality and relevance. All submissions will be held confidentially prior to contacting the selected presenters.Notification of AcceptanceNotification of acceptance will be sent out on or before August 8, 2009. Authors of accepted proposals will present at the conference and their talk will be made available to the public on YouTube.CopyrightGTAC requires authors to present at the conference and permit their presentation to be made available on YouTube.AttendeesTo ensure active participation and provide a variety of technical perspectives, we select applying attendees. Further information will be published via a call for participation at a later time.Important DatesAugust 1 - Deadline for presentation proposalsAugust 8 - Notification of acceptanceOctober 21+22 - GTAC conference (Zurich, Switzerland)QuestionsIf you have questions regarding the submission process or potential topics please email us at: gtac-2009@google.comWe will add more information to the Google Testing Blog as we get closer to the dates.
Why the change to Google? I’ve been asked that over and over again. As I reflect on the whole process, I have to admit that I like change. I like the challenge it brings, the creativity it sparks and the potential that I might fail at some new endeavor is simply intoxicating. Change, I think, is good. After all, if Robert Plant and Jimmy Page had never ventured beyond their comfortable British borders, they would have never written Kashmir and the planet is far better off for having that song.
My first week at Google has been a whirlwind of activity. I had the distinction of being (at a ripe of old age of 43) the oldest person at new employee orientation. I passed a billionaire in the hallway. I sat in a room with some of the best testing minds in Silicon Valley and walked across campus with a young engineer whose biggest problem is that she can’t learn enough fast enough. I’ve signed dozens of books.
There’s much to learn and much to do. I’ll catalog the results here if anyone is interested in following it. Coming from a company like Microsoft, I am used to mind-bogglingly complex problems and comfortable with partial solutions that point toward a better but still imperfect future. My role at Google will be to continue to thwart the impossible. Innovation as a main course is what brought me here. I hope to continue my work on testing tours and envisioning the future, but I am even more excited about the things I can’t yet see. Given the team that I am working with here, I think it is safe to expect big things.
In case you are interested, I am located in the Kirkland WA office and not yet assigned to a project. If I am lucky I will manage to get my hands into everything. I’ll try and be careful not to spill the secret sauce over my nicest shirt…
