Testivus, Testability and Dr. Jekyll and Mr. Hyde

Thursday, July 01, 2010

by Alberto Savoia
Note: Apparently, there were lots of downloads of the Testivus booklet and I hit some kind of quota on my personal account. If you have problems with reaching the original link below, please try this new download link or this one.
A major topic at this year's GTAC conference is going to be testability: "We also want to highlight methodologies and tools that can be used to build testability into our products." That's great!
Testability is one of the most important, yet overlooked, attributes of code – and one that is not discussed enough. That's unfortunate, because by the time the issue of testability comes up in a project it's usually too late. As preparation and seeding for GTAC, I though it would be fun and useful to get some discussions on testability going. So here we go, feel free to chime in with your thoughts.

A few years ago, after watching one too many episodes of Kung Fu, I was inspired to write a pretentious and cryptic little booklet about testing called "The Way of Testivus" (PDF).

Testivus addresses the issue of testability in a few places, but I would like to start the discussion with this maxim:

To me, "Think of code and tests as one" is the very foundation of testability. If you don't think about testing as you design and implement your code, you are very likely to make choices that will impair testability when the time comes. This position seemed obvious and non-controversial to me at the time I wrote it, and I still stand by it. Most people seem to agree with it as well, and more than one person told me that it's their favorite and most applicable maxim from all of Testivus. There are however three groups of people who found issue with it.
Some of the people, mostly from the TDD camp, think that my choice of words leaves too much wiggle room: "Thinking about the tests is not enough, they should be writing and running those tests at the same time."
Others think that code and tests should not be thought of as one at all, but they should be treated independently – ideally as adversaries: "I don't want code and tests to be too "friendly". Production code should not be changed or compromised to make the testing easier, and tests should not trust the hooks put in the code to make it more testable." Most of the people in this camp are not big fans of unit/developer testing in the first place, but not all. One person, a believer in developer testing, told me that he gets the best results with a Dr. Jekyll and Mr. Hyde approach. He assumes two different roles and personalities based on whether he's coding or testing his own code. When coding, he's the constructive Dr. Jekyll who focuses on elegant and efficient design and algorithms – and does not worry about testability. When testing, he turns into the destructive Mr. Hyde; he tries to forget that it's his code or how he implemented it, and puts all his energy and anger into trying to break it. Sounds like it could work quite well – though I don't think I'd want this person as an office mate during the Mr. Hyde phase.
A third group, thought that the maxim was fine for unit tests, but not applicable to other types of tests that were best served by an adversarial black-box approach.
What are your thoughts? Is it enough to think about testability when designing or writing the code, or must you actually write and run some tests in parallel with the code? Does anyone agree with the position that code and tests should be designed and developed in isolation? Are there other Dr. Jekylls and Mr. Hydes out there?
Alberto

14 comments

Update! GTAC 2010 Keynote Speakers

Tuesday, June 29, 2010

We are thrilled to announce that GTAC 2010 keynotes are finalized. We are very fortunate to have three world-renowned software testing icons: Robert Binder, Dr. Jeff Offutt and Dr. James Whittaker. Robert, Jeff and James bring to GTAC a powerful combination of practical and theoretical experience to help us address key aspect of this year's theme: Test to Testability. Robert will kick-off Day 1 with a keynote on the critical role of testability. On Day 2, Jeff will share his experiences on Automatic Test Generation from source code and about a technique he invented, Bypass Testing, for black-box testing of web applications. James will close the conference with a keynote focusing on the Test(ing) challenges and how to get ahead of them. More details on the specifics of their talk coming soon!
Here are brief bios of our esteemed speakers:
Robert BinderRobert V. Binder is a software entrepreneur and technologist with over 34 years of systems engineering experience. His 1994 analysis of software testability http://portal.acm.org/citation.cfm?id=184077 has had a continuing influence on research and practice in this area. As principal of System Verification Associates, he leads teams that deliver advanced IT assurance solutions. He was recently awarded a U.S. Patent for a unique approach to model-based testing of mobile systems. He is a member of the Editorial Board of Software Testing, Verification, and Review and internationally recognized as the author of the definitive Testing Object-Oriented Systems: Models, Patterns, and Tools. Robert holds an MS in Electrical Engineering and Computer Science from the University of Illinois at Chicago and a MBA from the University of Chicago. He is an IEEE Senior Member.
Dr. Jeff OffuttDr. Jeff Offutt is Professor of Software Engineering at George Mason University. He has invented numerous test strategies, published over 120 refereed research papers, and is co-author of the textbook Introduction to Software Testing. He is editor-in-chief of Wiley's journal of Software Testing, Verification and Reliability; steering committee chair for the IEEE International Conference on Software Testing, Verification, and Validation; and program chair for ICST 2009. He has consulted with numerous companies on software testing, usability, and software intellectual property issues. Offutt is on the web at http://www.cs.gmu.edu/~offutt/
Dr. James WhittakerDr. Whittaker is currently the Engineering Director over engineering tools and testing for Google's Seattle and Kirkland offices. He holds a PhD in computer science from the University of Tennessee and is the author or coauthor of four acclaimed textbooks. How to Break Software, How to Break Software Security (with Hugh Thompson) and How to Break Web Software (with Mike Andrews). His latest is Exploratory Software Testing: Tips, Tricks, Tours and Techniques to Guide Test Design and he's authored over fifty peer-reviewed papers on software development and computer security. He holds patents on various inventions in software testing and defensive security applications and has attracted millions in funding, sponsorship, and license agreements while a professor at Florida Tech. He has also served as a testing and security consultant for dozens of companies and spent 3 years as an architect at Microsoft.
Reminder: Call for proposalsIf you would like to present at this year’s GTAC please remember to submit your proposal by the July 9th deadline. Please visit http://www.gtac.biz/call-for-proposals for details.
Sujay Sahni for the GTAC 2010 Committee

2 comments

Test Driven Integration

Tuesday, June 22, 2010

By Philip Zembrod

In an earlier post on trying out TDD I wrote how my mindset while coding changed from fear of bugs in the new code to eager anticipation to see the new code run through and eventually pass the already written tests. Today I want to tell about integrating components by writing integration tests first.

In a new project we decided to follow TDD from the start. We happily created components, “testing feature after feature into existence” (a phrase I love; I picked it up from a colleague), hitting a small test coverage of around 90% from the start. Obviously, when it came to integrating the components into a product, the obvious choice was to do that test-driven, too. So how did that go?

What I would have done traditionally was select a large enough set of components that, once integrated, should make up something I could play with. Since at least a minimum UI would be needed, plus something that does visible or useful things, preferably both, this something would likely have been largish, integrating quite a few components. With the playing around and tryout, I’d enter debugging, because of course it wouldn’t work at first attempt. The not-too-small number of integrated components would make tracking the cause of failures hard, and anticipating all this while coding, I’d have met the well-known fearful mindset again, slowing me down, as I described in my initial TDD post.

How did TDI change this game for me? I realized: With my unit test toolbox that can test any single component, I can also test an integration of 2 components regardless of whether they have a UI or do something visible. That was the key to a truly incremental process of small steps.

First, write the test for 2 components, run it and see it fail to make sure the integration code I’m about to write is actually executed by the test. Write that bit of code, run the test and see it succeed. If it still fails, fix what's broken and repeat. Finding what's broken in this mode is usually easy enough because the increments are small. If the test failure doesn’t make obvious what’s wrong, adding some verifications or some logging does the trick. A debugger should never be needed; automated tests are, after all, a bit like recorded debugging sessions that you can replay any time in the future.

Repeating this for the 3rd component added, the 4th, etc., I could watch my product grow, with new passing tests every day. Small steps, low risks in each, no fear of debugging; instead, continuous progress. Every day this roaring thrill: It works, it works! Something’s running that didn’t run yesterday. And tomorrow morning I’ll start another test that will run tomorrow evening, most likely. Or already at lunchtime. Imagine what kind of motivation and acceleration this would give you. Better, try it out for yourself. I hope you’ll be as amazed and excited as I am.

What are the benefits? As with plain TDD, I find this fun-factor, this replacement of dread of debugging by eagerness for writing the next test to be able to write and run the next code the most striking effect of TDI.

The process is also much more systematic. Once you have specified your expectations at each level of integration, you’ll verify them continuously in the future, just by running the tests. Compare that to how reproducible, thorough and lasting your verification of your integration would be if you’d done it manually.

And if you wrote an integration test for every function or feature that you cared about during integration, then you can make sure each of them is in shape any time by just running the tests. I suspect one can’t appreciate the level of confidence in the code that creates until one has experienced it. I find it amazing. I dare you to try it yourself!

P.S. On top of this come all the other usual benefits of well-tested code that would probably be redundant to enumerate here, so I won’t. ;-)

6 comments

GTAC: Call for Attendance & Proposals

Sunday, May 16, 2010

Google Test Automation Conference (GTAC) 2010

Call for Attendance & Proposals

We are happy to announce that the application process is now open for Attendance and Proposals for the Fifth Google Test Automation Conference (GTAC), to be held in Hyderabad, India on October 28 - 29th.

As in previous years, GTAC is an invitation only conference where we enable sharing of great ideas and active participation to challenge and refine our thoughts and experiences. As such the the application process expects you to share your ideas and insights that you would bring to the conference and how these would further the discussion about this year’s theme of Test to Testability. This information will help the committee select a balanced audience of seasoned practitioners, students and academics.

Also this year, we are introducing a participant-driven format that will give the power to the attendees to select and voice their opinion on the speakers and the content! To make these changes, we are opening up proposals and attendance applications simultaneously. Once the initial set of participants are finalized, we will conduct online viewing and voting by the participants for presentations.

How to apply
For Attendance: Please visit http://www.gtac.biz/call-for-attendance
For Proposals (to present): Please visit http://www.gtac.biz/call-for-proposals

Deadline
The due date for both categories of applications is July 9th, 2010.

Registration Fees
There are no registration fees. Please check the FAQ page for more information.

Further information
General website: http://www.gtac.biz/
Call for proposals: http://www.gtac.biz/call-for-proposals
Call for attendance: http://www.gtac.biz/call-for-attendance
FAQ: http://www.gtac.biz/faq
Questions: Email us at gtac-2010@google.com

We look forward to your applications and a great GTAC!
Finally we would appreciate your help in helping us spread the word about this event.

Regards
Sujay Sahni on behalf of the GTAC 2010 Committee

1 comment

Do Know Evil

Thursday, May 06, 2010

Web Application Exploits and Defenses by Bruce Leban in Google Kirkland http://google-gruyere.appspot.com/ If you want your application to be as secure as possible, you need to learn how Evil People think. And you'll want to use that knowledge to do penetration testing: attacking your own application to try to find bugs. To help you understand how applications can be attacked and how to protect them from attack, we've created the “Web Application Exploits and Defenses” codelab. The codelab uses Gruyere, a small, cheesy, web application that is full of real world bugs. In the codelab, you'll learn how to:

Attack a web application to find and exploit common web security vulnerabilities.

Avoid and fix these common bugs.

Gruyere is chock full of cool features, and the more features an application has the larger the attack surface. Your application probably has features just like these: Can you match each feature to the vulnerability that it exposes and the exploit it enables?

Feature

New template language
HTML allowed in snippets
File upload capability
AJAX
Web-based admin console

Vulnerability

Cross Site Scripting (XSS)
Cross Site Request Forgery (XSRF)
Cross Site Script Inclusion (XSSI)
Path traversal
Client-state manipulation

Exploit

Information disclosure
Elevation of privilege
Denial of Service (DoS)
Spoofing
Code execution

Ha! Tricked you! Each of these features introduces multiple vulnerabilities. And each vulnerability can be exploited in multiple ways. The codelab walks you step by step through each vulnerability, with progressive hints guiding you on how to find them, how to exploit them and how to avoid them. Here are some examples of fictitious attacks against Google applications. Do you recognize them? (answers below)

http://www.gmail.com/?search=in:spam+%3Cscript%3EmoveToInbox(selectAll())%3C/script%3E
http://www.blogger.com/delete-blog.g
http://www.picasa.com/../../../../../../../etc/passwd
http://www.youtube.com/admin?v=Vr0oK3gMzK&action=rickroll
http://checkout.google.com/buy?order=4815162342&total=0.01

Are you sure that your application isn't vulnerable to similar attacks!?Toilet-Friendly Version for the answers

7 comments

Testing Blog

Testivus, Testability and Dr. Jekyll and Mr. Hyde

Update! GTAC 2010 Keynote Speakers

Test Driven Integration

GTAC: Call for Attendance & Proposals

Do Know Evil

Labels

Archive

Feed