by Patrick Copeland
We get questions once in a while about our readership. Here's a brief summary of the last 30 days...
Page view count: 34,140Time on each page: 2:52Most popular day to read: Tuesday's.Most traffic (top 5 in order): US, India, UK, Germany, CanadaNumber of countries with readers: 131
Most popular posts...
Testing, once a marginalized function at Google, is now an integral part of Google’s innovation machine. Patrick Copeland describes how this vital transformation took place. As he was analyzing how to be more efficient and better align his testing team with the needs of the company, Patrick realized they had to move beyond “just testing” and become a force for change. His approach was based on five powerful principles: (1) Building feature factories rather than products, (2) Embedding testing at the grass roots level, (3) Solving problems of scale with technology, (4) Automating at the right level of abstraction, (5) Only doing what the team can do well. Learn how Google test teams used these principles to shift from a “service group” composed predominantly of exploratory testers to an “engineering group” with technical skills. Their focus became “delivering innovation” rather than “testing product.” Learn how Patrick led a cultural shift where product teams saw testing and continuous improvement, not as alien concepts driven by someone else, but as a tool for them to meet their own goals of delivering features quickly and with fewer problems. Discover how you can incorporate the lessons of Google to make your test team a vital force for change.
The “Gamer Generation” is here. According to the Washington Post, in 2006 at least 40% of Americans play video games on a computer or a console. The largest demographic is 18-34 year-olds, many of whom are working as our friends and peers as software developers and testers. There is a great book, Got Game, which describes the impact of games on the workforce of today and in the future.
Many businesses are wondering these days how to use simple concepts popular in video games and apply them to work.
It’s happening already. From a construction site with a sign posted with “days since last accident” to the car dealer with salespersons leader board scrawled out in chalk. Simple competition helps work get done in all kinds of industries. The success of sales contests is a great example of how well competition works.
“In every job that must be done, there is an element of fun. You find the fun and—SNAP—the job’s a game!” —Mary Poppins
A company called Seriosity has made interesting use of video game concepts at work with their application of virtual currency to email. To help with information overload from an overflowing inbox, email senders can “spend” virtual currency (Serios) to help prioritize their messages. Spending to denote the importance of a message is not a whole lot different than buying new golf clubs with virtual money earned in a Tiger Woods game, buying new tires in a car racing game – or using gamer points to upgrade your armor in an RPG (Role-playing Game). Wired Magazine had a great article on the use of WOW as a leadership training game. Playing games at work is more than just a quick game of online Hearts with one finger on the “boss key”. Games at work can have real impact.
With other industries using productivity games, what about software development? How can productivity games help testers improve the quality of the products they test? One of the more famous test techniques is the bug bash. Teams pick a certain date, and everyone bangs on a certain area of the product to find bugs. At the end, prizes are awarded while everyone enjoys pizza and drinks. That sounds a lot like game, doesn't it? It’s also a perfect example of a simple productivity game.
As the world of software gets more complex and the test matrix grows, there is no shortage of opportunities for testers to do more testing. Quality is always “improved”, it’s never completed, fixed or finished. Therefore, testers must choose from a variety of techniques and activities to pursue tasks with the highest return and value. In 1990, Harry Markowitz won the Nobel Prize for his work on Portfolio Selection (that he began in the 1950’s). The theory, familiar in financial markets and mutual funds, is that a portfolio of diverse assets with varying risk and returns actually reduces risk in time uncertain conditions. Testers can benefit from the work on portfolio selection by varying the techniques they use to find defects. However, some techniques are more expensive, more difficult, or less attractive than others. Some teams, in some situations, can rely on the altruism and volunteer efforts – organizational citizenship behaviors – of their members to take on the more difficult tasks. However, that doesn't always work, and other incentives may be required. That’s where productivity games can help.
When a project needs a bit of a push towards a certain behavior, building a game around it can help. For example, if a test team wants to improve the usage of unit tests by developers, encourage the team to “eat their own dogfood” or run automated tests on personal machines overnight, then creating a game will help people’s motivation can motivate people to participate in those activities.
Productivity games differ from traditional games in a few distinct ways. The goal of a productivity game is to attract players. These games don’t need winners as much as they need players. In many cases, productivity games don’t even need prizes. Productivity games can be simple. There’s no need to invest in expensive graphics or 3-D modeling – people play the game to compete around a work activity, not because of the appeal of the user interface. A simple leader board might be all that’s needed.
Using games at work is a powerful and effective method to influence change in organizational behavior, and therefore requires care in the design and use. It is possible to overdo the use of games to a point where they are counter-productive or ineffective. Successful game design is best achieved through experience and experimentation, and the goal should be to keep things interesting enough to always attract players. A game where one player leaps out to an insurmountable lead is not effective, because that will discourage others from playing. When a team wants to encourage testers to invest in fault injection techniques, setting up a leader board with bug totals found via fault injection techniques will attract attention of other testers, often just by word-of-mouth. You should encourage team-wide competition, smack talk, and public embarrassment as a means to draw attention. Games should be short in duration. Many testers remember the Dilbert “I’m gonna write me a new minivan” from 11/13/95. This is what results from a poorly designed productivity game.
Productivity games are great at motivating people to do things and change behavior. And as Uncle Ben said to Spiderman, “With great power comes great responsibility”. It’s critical to have specific goals for the games, and to understand the impact before deploying a game – it’s pretty likely that people will play, at least initially, and the impact may not be what you intended. Keep the duration of games short in order to be able to adjust. And finally, keep the games focused on volunteer or “extra” activities. A game designed around an individual’s job or portions of everyone’s regular job can introduce unusual feelings when it comes to rewards and performance evaluation. It’s easier to steer clear of that than to try to figure out the right balance.
Shigeru Miyamoto, who designed Super Mario Bros., talks about games as “miniature gardens”, metaphorically representing the cultural values, humanity, and challenges of everyday life the way a miniature garden might represent a real one.
The whole idea, for me, is about the exploration of a hakoniwa, a miniature garden. It's like a garden in a box, where if you look at it from different angles, you can see different plants and arrangements. And you have that sense of surprise and exploration. There's always things you can find. [2]
Games are a lot like testing. Software quality assurance and testing efforts represent a proxy for real users in much the same way that games are a proxy for the real world. The challenges that Mario faces with fire-breathing dragons and knife throwing turtles are balanced by the allure of coins and princesses. A testing organization has a good idea of the tools available in their portfolio to improve quality and eliminate those “fire-breathing dragons” for their customers. Productivity games allow teams to offer coins and position rewards around their equivalents of Markowitz’s “diverse assets” at their disposal, in the form of defect detection and removal techniques, to improve quality.
For the players, productivity games offer people the ability to learn, solve problems, and earn a reward. They offer a way to challenge players to compete, to explore and discover, and to establish a surrogate identity or status via a leader board. There are many fundamental motivations to play a game. “If you build it, they will come [3]." A good productivity game designer can build a game around “work that needs to be done that no one wants to do” – and like the lone teenager who plays Halo for weeks on end, people will take on that work. Experiment and explore – the results can be surprising.
I’d be interested in any comments about Productivity Games that you might have.
Thanks,
Ross Smith
http://www.defectprevention.org/
Interesting Links:
When Work becomes a Game
Serious Games Initiative
Got Game - How the Gamer Generation is Reshaping Business Forever
Chapter 5 – Productivity Games - The Practical Guide to Defect Prevention
2008 Serious Games Summit presentation
Miniature Gardens and Magic Crayons: Games, Spaces, & Worlds
HBR Leadership’s Online Labs
NPR: Software lets users assign value to email
Game Tycoon blog (How Video Games are transforming the Business World)
Productivity Games Blog
Special thanks to Joshua Williams for his help with this post.
[1] Robert Pirsig, Zen and the Art of Motorcycle Maintenance[2] Wired Magazine Interview http://blog.wired.com/games/2007/12/interview-super.html[3] Field of Dreams, 1989
[Side note: Even before his first day at the Googleplex, Jason showed an amazing dedication to Google. After leaving Chicago with the big moving truck, he and his family had to stop after just a few hours because of an ice storm. Cars sliding off the road left and right. Further along, in Kansas, one of his kids caught pneumonia. His family stayed in the local hospital while Jason drove on. Heading west, there was a big snow storm in Colorado, which he wanted to avoid. He drove further south and ended up in a rare (but really real) dust storm over the Border States. He promised us some great video footage of his drive through tumbleweeds. He finally arrived and has settled in and is looking forward to calmer times in the Bay Area. After that trip, he isn't even worried about earthquakes, fires, mud slides, or traffic on the 101.]
Couple of GTAC Videos with Jason: SeleniumRC, Selenium vs WebDriver
CG: Why did you invent Selenium? What was the motivation? Huggins: Selenium was extracted from a web-based (Python + Plone!) time-and-expense (T&E) application my team and I were building for my previous employer. One of the mandates for the new T&E app was that it needed to be "fast, fast, fast." The legacy application was a client-server Lotus Notes application and wasn't scalable to the current number of offices and employees at the time. To live up to the "fast, fast, fast" design goal, we tried to improve and speed up the user experience as much as possible. For example, expense reports can get pretty long for people who travel a lot. No matter how many default rows we put in a blank expense form, people often needed to add more rows of expense items to their reports. So we added an "Add row" button to the expense report form. To make this "fast, fast, fast," I decided to use a button that triggered JavaScript to dynamically add one blank row to the form. At the time (Spring 2004), however, JavaScript was considered buggy and evil by most web developers, so I caught a lot of flak for not going with the classic approach of POSTing the form and triggering a complete page refresh with the current form data, plus one blank row.
Going down the road of using JavaScript had consequences. For one, we had a really, really difficult time testing that little "Add row" button. And sadly, it broke often. One week "Add row" would be working in Mozilla (Firefox was pre-1.0), but broken in Internet Explorer. And of course, nothing was working in Safari since few developers were allowed to have Macs. ;-) The following week, we'd open a bug saying "'Add row' is broken in IE!!" The developer assigned to the issue would fix and test it in IE, but not test for regressions in Mozilla. So, "Add row" would now be broken in Mozilla, and I'd have to open a ticket saying "'Add row' is now broken in Mozilla!!!". Unlike most other corporate IT shops, we didn't have the luxury of telling all employees to use a single browser, and developers didn't want to manually test every feature in all supported browsers every time. Also, we had a very tiny budget and commercial testing tools were -- and still are -- ridiculously over-priced on a per-seat basis. The T&E project was done the "Agile Way" -- every developer does testing -- so shelling out thousands of dollars per developer for a testing tool wasn't going to happen. Never mind the fact that there were no commercial tools that did what we needed anyway!
After many months of trial and error and various code workarounds, I came to the conclusion I needed a testing tool that would let me functional-test JavaScript-enhanced web user interfaces (aka "DHTML" or now "Ajax"). More specifically, I needed to test the browser UIs: Internet Explorer, Mozilla Firefox, and Safari. There were no commercial apps at the time that could do this, and the only open source option was JsUnit, which was more focused on unit testing pure JavaScript functions, rather than being a higher-level black-box/smoke-test walk through a web app. So we needed to write our own tool. Our first attempt was a Mozilla extension called "driftwood" (never released), coded by Andrew McCormick, another co-worker of mine at the time. It did make testing the "Add row" button possible, but since it was Mozilla-only, it wasn't what we needed for testing in all browsers. Paul Gross and I started over, and I started to educate myself on functional testing tools and techniques and stumbled upon Ward Cunningham's Framework for Integrated Testing (FIT). I originally set out to implement "FIT for JavaScript," but quickly realized we were drifting away from the FIT API, so Selenium became its own thing.
CG: Why does the world need another test tool?Huggins: At the time I created Selenium, had there been another testing tool that could test JavaScript UI features in all browsers on all platforms, believe me, I would have saved lots of time *not* writing my own tool.
CG: What's special about it?Huggins: Well, maybe the right question is "What's “lucky” about it? Selenium was created in a time when JavaScript was considered "bad" and generally avoided by most professional web developers. Then Google Maps hit the scene a year later, the term "Ajax" was coined, and BOOM! Overnight, JavaScript became "good." Also, Firefox started stealing market share from IE. The combination of needing to test 1) JavaScript features 2) in several browsers (not just IE) was a "right place, right time" moment for Selenium.
CG: When did you realize that Selenium was a big deal? What was the tipping point? Huggins: When I started being asked to give presentations or write about Selenium by people I didn't know. The tipping point for Selenium technically relied on two things: 1) the Selenium IDE for Firefox, written by Shinya Kasatani, which made installation and the first-hour experience tons better for new users. And 2), Selenium Remote Control (RC) created by Paul Hammant, and extended by Dan Fabulich, Nelson Sproul, and Patrick Lightbody, which let developers write their tests in Java, C#, Perl, Python, or Ruby, and not have to write all their tests in the original FIT-like HTML tables. Socially, if Google Maps or Gmail never existed and thus the whole Ajax gold rush, I wonder if JavaScript would still be considered "bad," with a similar "why bother?" attitude to testing it.
CG: Have you discovered any interesting teams using Selenium in ways you'd never intended? Huggins: At my previous company, I did see some developers write Selenium scripts to create their time and expense reports for them from YAML or XLS files. Since we hadn't exposed a back-end API, automating the browser for data entry was the next best thing. It was never designed for this purpose, but I started (ab)using it as coded bug reports. Asking users for steps on how to reproduce a bug naturally lends itself to looking like a Selenium test for that bug. Also, I've used the Selenium IDE Firefox plug-in to enter NBC's "Deal or No Deal" contest on their website from home, but I stopped doing that when I read in the fine print that the use of automation tools to enter their contest was grounds for disqualification.
CG: What advice do you have to offer Google groups interested in Selenium?Huggins: Well, one of the biggest drawbacks with user interface testing tools is that they're slow for various reasons. One way to bring the test run times down is to run them in parallel on a grid of servers, instead of sequentially. Of course, that isn't news to your average Googler. Engineers would be more likely to run automated browser UI tests if they could run 1000 tests in 1 minute total time on 1000 machines instead of 1000 tests in 1000 minutes on 1 machine. Sadly, though, most projects allocate only one machine, maybe two, to browser testing. I'm really excited to come to Google with the resources, the corporate interest, and the internal client base to make a large scale Selenium UI test farm possible. Eventually, I’d like to take Selenium in some new directions that we’ll talk about in later blog posts. But I'm getting ahead of myself. I have to survive Noogler training first.
