The Plague of Aimlessness
Lore. It’s more than just a cool word. It conjures up a sorcerous image in one’s mind of ancient spell books and learned wizards with arcane and perilously attained knowledge.
And it’s exactly what we lack in software testing. Testing lore? Are you kidding me? Where is it? Who’s bogarting it? Can I have a hit?
The software testing industry is infected with the plague of aimlessness. We lack lore; we lack a body of knowledge that is passed from wizard to apprentice and written down in spell books for the industrious to study. Our apprentices are without their masters. We must all reinvent the wheel in the privacy of our offices only to have other testers the world over reinvent it in theirs.
I suggest we stop this nonsense. Testing is far too aimless. We test because we must or our manager tells us to do so. We automate because we can or because we know how to, not because it is part of some specific and proven strategy and certainly not because our lore dictates it. Is there a plan or some documented wisdom that guides our testing or are we just banging on the keyboard hoping something will fail? Where are the testing spell books? Surely the perilously attained knowledge of our tester forebears is something that we can access in this age of readily available information?
When a hunter makes a kill, they remember the terrain and circumstances. They pass this knowledge on to their successors. Over time they understand the habits of their prey and the collective knowledge of many hunters makes the job of future hunters far easier. When you see this terrain, you can expect game to behave in this manner. Can we say the same of testing? How well do we learn from each other? Do our ‘eureka moments’ get codified so that future testers will not have to suffer the aimless thrashing that we suffered? Can we say when you see functionality like that, the best way to test it is like this?
The plague of aimlessness is widespread. The need for testing lore is acute. Nike tells us to ‘just do it’ but what applies to exercise is not good advice for software testing. The next time you find yourself ‘just doing’ testing, pause for a moment and ask yourself ‘what is my goal?’ and ‘is there a purpose to this test?’ If the answer doesn’t immediately come to mind, you’re aimless, just doing it, and relying on luck and the sheer force of effort to find your quarry.
Luck has no place in sorcery or hunting and it has no place in testing. Luck is a nice happenstance, but it cannot be our plan A. Watch for the plague of aimlessness. Document your successes, scrutinize your failures and make sure you pass on what you learn from this introspection to your colleagues.
Be their wizard. Build a testing spell book and share it with others on your team. Over time you’ll banish the plague of aimlessness.
Very true.
ReplyDeleteAs a tester, while testing we need to pause and ponder over the direction, capture and go on building the knowledge, which helps the process (apart from the individuals)...not just for the team, or organisation but for the testing community from where we have learned many many things.
Writing about it is much easier than done. However being aware is like half battle won.
Knowledge building will be a continuous process. Sometimes, for its applicability, we need to remove specificness w.r.t. bug, project, domain, etc. and make the content more generic to begin with. That's how other disciplines of engineering have been able to come this far.
You suggest grabbing a 'testing spell book', but from a 'Grand Wizard' like you, what spell books should we pick up?
ReplyDeleteAn excellent post, I'm sharing this with my colleagues as we speak. Some good "a-ha" moments in here when I realize that I sometimes "just do it".
ReplyDeleteSo very very true. I wish I had that 'lore' during my early days. I struggled and am struggling still. I have tried to pass on my knowledge/collections at Software Testing Fundamentals Blog. I feel we, Software Testers, need to open up more and feel dignified.
ReplyDelete"Be their wizard. Build a testing spell book and share it with others on your team. Over time you’ll banish the plague of aimlessness."
ReplyDeleteNo thanks!
Perhaps for you, testing is sorcery. Perhaps incantations are what you teach. Perhaps spells and numerology should be future subjects for your writings.
But for me (and I suspect most of us), testing is all about our business.
Instead of incantations, I want people to think.
Instead of spells, I'd suggest knowledge and practice.
Instead of praying to the mystical, invisible "crowd", I'd suggest hiring professionals.
Testing isn't magic. It's hard work, knowledge and skill. Magical thinking cannot help move testing forward as a profession.
I feel sorry if your testing is aimless. Mine isn't. My team's testing is far from aimless. And that of most testers I know isn't aimless.
-joe
I have to disagree. I think there is plenty of lore. In my case, using this "lore" is extremely important.
ReplyDeleteI manage a very small group of testers. Half of my group is made up of interns who, most of the time, have absolutely no experience in testing at all. Many times I can persuade them to come back for another term, but how can I teach you how to test in a matter of a month or less? So you can be effective for me?
Well, aside from mentoring (Wizard and Apprentice), my team sits down for a half hour or 45 minutes every day for a "brag session". To put it in terms of your lore, think of it as a group of warriors sitting down at a wodden bar table and telling their tales of battle over a dirty mug of grog.
We don't have a lack of lore in testing, we just don't spread the information as well as some. Soem people I talk to claim they don't have a half hour a day to spend in a brag session with their team, but, from my experience, I can't afford NOT to have a half an hour a day with my team.
I have to jump on the disagreement bandwagon, I've not resorted to lore since I started in the mid-90's. Since then I have pursued a body of knowledge that helps me find the business and customer, needs of the product I am testing. To me its all been about the communication, we have plenty of voices speaking out there about testing, and how to test, but its a cacophony, Maybe if people get off their own bandwagons, and dogma, and speak to each other instead of over each other we'll get that body of knowledge.
ReplyDeleteThere are points you bring up that everyone who works in this industry should be aware of.
"We test because we must or our manager tells us to do so."
Well it is what we are hired to do, but the reason we were hired is because we have the aptitude to look for issues and think outside of the code. My manager may tell me to do the work, but its work I want to do, and am excited to do it.
"We automate because we can or because we know how to, not because it is part of some specific and proven strategy and certainly not because our lore dictates it."
I think this is the problem with most automation, it is not planned and haphazard, unless you treat it like a software project its going to fail. There is plenty out there about this already.
"When a hunter makes a kill, they remember the terrain and circumstances."
When I test a product I should have known what my prey and terrain are through careful planning, the Test Plan, and as I have learned from my hunt, my updates to the plan, it should be able to help me next time. This is what I think of when it comes to test experience.
Sorcery isn't as easy as you make it out to be. Wizards spend their lives studying and practicing their discipline. It only seems like magic because you haven't seen all the practice that goes into it. Magic is the art of making knowledge and skill look easy. If your processes are more mature in this manner then call it magic/lore/knowledge/discipline but share it with the rest of the world please. I like the idea of a 'brag session' as a concept but I don't like the name.
ReplyDeleteThose who think most testing is aimless, or that sorcery, illusion, spells, incantations, folklore, magic, hocus-pocus, or wizardry are needed for teaching testers should get their hands on:
ReplyDelete"Lessons Learned in Software Testing", "How We Test Softare at Microsoft", or "Effective Software Testing: 50 Specific Ways to Improve Your Testing"
No magic there, just real-world, practical advice.
A lot of this talk of magic makes me think of the line "any sufficiently advanced technology is indistinguishable from magic".
ReplyDeleteActually one of the first things I do when I join a company, as the first tester, or as part of a group is find the wiki - local document repository. Everything I do, or can do, is documented in some form for others to use, after I left one company I heard from a Developer months later who thanked me for that as he needed to do something I rarely did and no one remembered how to do; yet my document gave step by step instructions. I've done monthly QA talks as a way to share with others, even getting others on the team to sign up and share.
I guess I wouldn't so much say that its all Lore, but rather isn't it better not to hoard the information? I like the Agile concept of Information Radiators, and try to implement them in any fashion possible whenever I can.
Well, what the gist of what James says and the 'disagreement bandwagon' say is the same, I think: We need to communicate, we need to share our knowledge.
ReplyDeleteHow we choose to do that might differ but a Tome and an Information Radiator are the same in essence.
Books such as "Lessons Learned in Software Testing" codify some of the lore learnt by software testing professionals.
ReplyDeleteI'd like to encourage more of the blog readers to share experiences of their testing techniques and practices.
Hey James!
ReplyDeleteThanks for the comment. Yeah, the "brag session" thing has actually worked well for our team. I haven't tested it in a larger environment, where I think it may be less feasible to do in a half hour.
In the case of a smaller team, maybe a "brag box" could be another idea. So you can still gather the team for a half hour a day. However, throughout the day, if someone finds something and it makes them laugh or they think they did something extraordinary in order to find the issue, have a "brag box" accompanied with some 4x6 cardstock cards that they can write it down on and drop it in the box. Then the Manager should decide what is appropriate for that day, or ever for that matter, and they can organize the submissions into something more fluid, so there are no 10 minute ventures off into the woods by any one person.
I agree, "brag" is a little sucky. I just haven't come up with a better name for it as of yet.
As for your comment on sorcery, I found it pretty interesting. I mean I've always just found these scrolls and tomes lying around or in treasure chests and stuff like that. Then I read them and BAM! I know something new. However, you need to actually look above your head really quick so you know what you learned because the text will disappear after 3 or 4 seconds. I mean there would be nothing more embarassing than thinking you don't know magic missle and immitating some other wizard in the bar and you're like, "muahahahaha, I cast MAGIC MISSLE!" and they you send a magic missle right through the side of the wall or something, or worse kill someone.
You know, because if you kill someone who's good, then you're automatically relegated to black magic and they have this WHOLE set of other complex rules, like you must know Curse Level 2 and you need to wear the skull of some dead animal around your neck, or you need to be open to the idea of summoning the evil lord. Being good is just so much easier.
This is a great post. Recently some members on my team were fighting the fact that we had all these stub controllers for testing the user interface. The controllers did nothing and created unnecessary maintenance, they said. Soon I realized that I had not explained well enough how the stub controllers were there in order isolate the user interface from the rest of the system so we could pinpoint failures easier. Aha, they said.
ReplyDeleteSpellbook = WIKI
ReplyDeleteThe testing slogan should be "Just do it yourself (with a little help from your friends!)".
ReplyDeleteI see the point on sorcery, magic, and lore. I have rarely worked with selfish individuals so I have had great experiences working to obtain knowledge from others.
However, it gets to the point where you need to learn on your own. Isn't that what we're all supposed to do?
I like the book recommendation. Lessons Learned is a good book but doesn't provide anything close to all the answers. To be a great tester you must be a great software engineer. Too often we focus on just testing or its techniques and lose sight of the greater goal: The Software System .
Sharing is great and you should always allow time to provide guidance when needed. I currently assist several junior testers and some not so with information they need to do their job correctly. Do I have the time to do so? Not really. However, it's magical to be able to assist others.
The best learning is obtained while teaching.
This feeds into a topic that I've been pondering for while.
ReplyDeleteTest automation, by and large, ends up being a very domain-specific exercise. This is because software developed to solve a problem becomes a domain of it's own, and then we create another layer of software (automated tests) to solve the problems of that domain.
So most test automation "lore" ends up being very domain specific. For example, we often have to scrape screenshots for text or bitmap fragments, and there are dozens of things that can cause this to fail. Almost all of them are specific to the software being tested. So if we adapt our tests to compensate for these, they become more and more specific to the problem domain.
As this progresses, our "lore" becomes inherently more arcane, and seemingly less useful, which means it is less likely to be recorded properly.
Now, much of this occurs because automated testing software is not treated as software itself, but rather as a side-effect. So maintainability and extensibility tend to be second class citizens when it comes to development.
The net effect of this, in large organizations, is that there is very little re-use across automated testing projects, and they tend to get scrapped once the problem domain they are solving becomes less important, or is rewritten.
We are currently working on a project to establish a common platform for automated testing, and I may attempt to give a GTAC presentation about it. Anyone interested?
This is actually quite interesting to me as my first book on software testing was actually written as a sort of spellbook on testing security.
ReplyDeleteThe reasoning behind what was being tested, then case studies, then suggestions of testing techniques and resources.
I would put forth, however, that as long as testing is regarded as something anyone can do if they can write code (as it is by some companies), the less the need for this passing of knowledge is acknowledged.
Very true. Very true. Well said and you took the guts to say just that. There are many masters out there who reinvented the wheel themselves but have shied away from sharing with others the intricacies of testing. Software development basically being a team effort, we should see to it that like minded people could come together and share their experiences (both good and bad) among the testing fraternity.
ReplyDeleteKudos to you for this post. For all those testers out there, you could refer to my post for any basics of testing you may need.
Go to Software testing Basics
Any analogy fails if you push it too far, but the best analogies give real insight; perhaps even unexpected insight.
ReplyDeleteIf we look away from the pages of the Dungeon Master's Guide for knowledge on how sorcery was actually used we will see a history of ever more bizarre failure. Most of the time, magic does not work, some of the time it at least appears to and so the practitioners try to determine what was important, what contributed to the success, what could make it better. Testing is too much like that. There is a body of lore, but little empirical evidence. Just look at Software Inspections, running full inspections is as difficult as high magic and just as expensive and yet how much real evidence is there to support all of these arcane practises?
As with hitorical magical schools, the would-be testing adept will happily jump on whatever new craze comes along, desperate for effect. Has anybody tried Test Point Analysis? Smoke and mirrors comes to mind. What about Exploratory Testing? Obviously I can't see the benefits because I'm not a good enough tester, insufficiently refined.
How much test effort is wasted effort? How much time do we spend on validation/positive testing when we should be looking for problems? How much effort do we put into automation? What returns have we seen? Why do we not know? Why don't we collect real, useful data?
As surely as alchemy became chemistry software testing must improve, I just hope we don't have to sacrifice all the unicorns before we get there.
I am on the disagree bandwagon myself. Testing may be "magic" in some organizations but not mine. Every test we write and run has a well defined purpose. Every day we learn new things that we share in our daily meetings and every project produces lessons learned that feed our knowledgebase and are used to refine and improve our internal best practices.
ReplyDeleteMy read of James' testing sorcery analogy is that we need to continue to move from the days of the Alchemy of Testing, to the Chemistry of Testing.
ReplyDeleteThe testing community has clearly come a long way since Glenford Myer's "The Art of Software Testing".
That said, there will always, IMO, be a certain level of magic involved in testing. That thing that cannot be defined. Call it inspiration. Call it creativity. Call it sorcery. There is always that something special that separates a master practitioner, from a mere dilettante.
completely do not agree as a software tester myself. When I test, I do it with a purpose and with the knowledge of the people who went before me as every professional should (be it testing, development or design). Maybe this is the way that Google likes to test (we need to test; oe, lets program something to test it with, not caring what the coverage or thought behind it is).
ReplyDeleteDon't get me wrong; I like google, but not their view of testing (what do you mean testing engeneer? That's just ike a developer who does testing....). They should take a look at what the profession of software testing really means and that is to test if the requirements of the client are being met; Will he get the program he asked for with the quality required...nothing more, nothing less....
There are some lores around the world as far as I know (Take TMap for example), but as it is with any profession, there is always someone who does not agree with this and goes his own way....