Google Testing Blog: November 2008

Guide to Writing Testable Code

Wednesday, November 26, 2008

Guide to Writing Testable CodeTo keep our code at Google in the best possible shape we provided our software engineers with these constant reminders. Now, we are happy to share them with the world.Many thanks to these folks for inspiration and hours of hard work getting this guide done:

Jonathan Wolter

Russ Ruffer

Miško Hevery

Flaw #1: Constructor does Real WorkWarning Signs

new keyword in a constructor or at field declaration

Static method calls in a constructor or at field declaration

Anything more than field assignment in constructors

Object not fully initialized after the constructor finishes (watch out for initialize methods)

Control flow (conditional or looping logic) in a constructor

Code does complex object graph construction inside a constructor rather than using a factory or builder

Adding or using an initialization block

Flaw #2: Digging into CollaboratorsWarning Signs

Objects are passed in but never used directly (only used to get access to other objects)

Law of Demeter violation: method call chain walks an object graph with more than one dot (.)

Suspicious names: context, environment, principal, container, or manager

Flaw #3: Brittle Global State & SingletonsWarning Signs

Adding or using singletons

Adding or using static fields or static methods

Adding or using static initialization blocks

Adding or using registries

Adding or using service locators

Flaw #4: Class Does Too MuchWarning Signs

Summing up what the class does includes the word “and”

Class would be challenging for new team members to read and quickly “get it”

Class has fields that are only used in some methods

Class has static methods that only operate on parameters

5 comments

Online Machine Learning Testing == Extreme Testing

Monday, November 24, 2008

Posted by Alek Icev As you may know our core vision is to build "The perfect search engine that would understand exactly what you mean and give back exactly what you want.". In order to do that we learn from our data, we learn from the past and we love Machine Learning. Everyday we are trying to answer the following questions.

Is this email spam?

Is this search result relevant?

What product category does that query belong to?

What is the ad that users are most likely to click on for the query “flowers”?

Is this click fraudulent?

Is this ad likely to result in a purchase (not merely a click)?

Is this image pornographic?

Does this page contain malware? –Should this query bring up a maps onebox?

Solving many problems require Machine Learning techniques. On all of them we can build prediction models that will learn from the past and try to give the most precise answers to our users. We use variety of Machine Learning algorithms at Google and we are experimenting with numerous old and new advancements in this field in order to find the most accurate, fast and reliable solution for the different problems that we are attacking. Of course one the biggest challenges that we are facing in the Test Engineering community is how are we going to test these algorithms. The amount of the data that Google generates goes beyond all of the known boundaries of environments where the current Machine Learning Solutions were being crafted and tested. We want to open discussion around the ideas how to test different online machine algorithms. From time to time we will present an algorithm and some ideas how to test it and solicit the feedback from the wider audience i.e. try to build a wisdom of the crowds over the testing ideas.
So let's look at the Stochastic Gradient Descent Algorithm

Where X is the set of input values of X_i ,W is set of the importance factors(weights) of every value X_i. A positive weight means that that risk factor increases the probability of the outcome, while a negative weight means that that risk factor decreases the probability of that outcome. t is the target output value, η is the learning rate(the role of the learning rate is to control the level to which the weights are modified at every iteration and f(z) is the output generated by the function that maps large input domain to a small set of output values in this case. The function f(z) in this case is the logistic function:

$f(z)=\frac{1}{1+e^{-z}}$

z = x0w0 + x1w1 + x2w2 + ... + xkwk The logistic function has nice characteristics since it can take any input, and basically squash it to 0 or 1. Ideal for predicting probabilities on events that are dependent on multiple factors(X_i) each with different importance weights(W_i).The Stochastic Gradient Descent provides fast convergence to find the optimal minimums of the error(E) that the function is making on the prediction as well as if there are multiple local minimums the algorithms guarantees converging to the global minimum of the prediction error. So let’s go back now into the real online world where we want to give answers (predictions) to our users in milliseconds and ask the question how are we going to design automated tests for the Stochastic Gradient Descent Algorithm embedded into a live online prediction system. The environment is pretty agile and dynamic, the code is being changed every hour, you want your tests to run on 24/7 basis, you want to detect errors upstream in the development process, but you don’t want to block the development process with tests that are running days, on the other side you want to release new features fast, but the release process has to be error prone(imagine the world with google being down for 5 mins, that is a global catastrophe, isn’t it?! So let’s look at some of the test strategies: Should we try to train the model(set of the importance factors) and test the model with the subset of the training data? What if this takes far more than hours, maybe days to do that? Should we try to reduce the set of importance factors (X_i) and get the convergence(E->0) on the reduced model? Should we try to reduce the training data set(the variety of set of values for X as an input to the algorithm) and keep the original model and get the convergence by any price? Should we be happy with reducing both the model size and the training set? Are we going to worry for over-fitting in the test environment? Given the original data is online data and evolves fast, are we going to be satisfied with fixed data test set or change the input test data frequently? What are the triggers that will make you do so? What else should we do? Drop us a note, all ideas are more than welcome.

7 comments

Clean Code Talks - Global State and Singletons

Friday, November 21, 2008

Miško HeveryVideo

Slides

No comments

My Unified Theory of Bugs

Tuesday, November 18, 2008

Miško Hevery

Logical: Logical bug is the most common and classical "bug." This is your "if"s, "loop"s, and other logic in your code. It is by far the most common kind of bug in an application. (Think: it does the wrong thing)

Wiring: Wiring bug is when two different objects are miswired. For example wiring the first-name to the last-name field. It could also mean that the output of one object is not what the input of the next object expects. (Think: Data gets clobbered in process to where it is needed.)

Rendering: Rendering bug is when the output (typical some UI or a report) does not look right. The key here is that it takes a human to determine what "right" is. (Think: it "looks" wrong)

NOTETypical Application Distribution (without Testability in Mind)

Cost of Finding the BugCost of Fixing the Bug

Logical

Wiring

Rendering

Probability of Occurrence

High

Medium

Low

Difficulty of Discovering

Difficult

Easy

Trivial

Cost of Fixing

High Cost

Medium

Low

How does testability change the distribution?

Clear separation between classes (Testable Seams) --> clear separation between classes makes it less likely that a wiring problem is introduced. Also, less code per class lowers the probability of logical bug.

Dependency Injection --> makes wiring explicit (unlike singletons, globals or service locators).

Clear separation of Logic from Wiring --> by having wiring in a single place it is easier to verify.

We Like Writing Unit-Tests

Unit-tests focus on the logical bugs. Unit tests focus on your "if"s and "loop"s, a Focused unit-test does not directly check the wiring. (and certainly not rendering)

Unit-test are focused on a single CUT (class-under-test). This is important, since you want to make sure that unit-tests will not get in the way of future refactoring. Unit-tests should HELP refactoring not PREVENT refactorings. (Again, when I hear people say that tests prevent refactorings, I know that they have not understood what unit-tests are)

Unit-tests do not directly prove that wiring is OK. They do so only indirectly by forcing you to write more testable code.

Functional tests verify wiring, however there is a trade-off. You "may" have hard time refactoring if you have too many functional test OR, if you mix functional and logical tests.

Managing Your Bugs

7 comments

TotT: Finding Data Races in C++

Thursday, November 13, 2008

multi-threaded codedata raceshard to find and reproduce 4 int bottles_of_beer = 0;
5 void Intern1() { bottles_of_beer++; } // Intern1 forgot to use Mutex.
6 void Intern2() { bottles_of_beer++; } // Intern2 copied from Intern1.
7 int main() {
8 // Folks, bring me one bottle of beer each, please.
9 ClosureThread intern1(NewPermanentCallback(Intern1)),
10 intern2(NewPermanentCallback(Intern2));
11 intern1.SetJoinable(true); intern2.SetJoinable(true);
12 intern1.Start(); intern2.Start();
13 intern1.Join(); intern2.Join();
14 CHECK_EQ(2, bottles_of_beer) << "Who didn't bring me my beer!?";
15 }
find data races in your codeHelgrind$ helgrind path/to/your/program
Possible data race during read of size 4 at 0x5429C8
at 0x400523: Intern2() tott.cc:6
  by 0x400913: _FunctionResultCallback_0_0::Run() ...
  by 0x4026BB: ClosureThread::Run() ...
  ...
  Location 0x5429C8 has never been protected by any lock
  Location 0x5429C8 is 0 bytes inside global var "bottles_of_beer"
  declared at tott.cc:4
HelgrinddeadlocksHelgrindValgrindNo beer was wasted in the making of this TotT.this episode

2 comments

Clean Code Talks - Dependency Injection

Tuesday, November 11, 2008

Miško HeveryVideo

Slides

3 comments

Clean Code Talks - Unit Testing

Wednesday, November 05, 2008

Miško HeveryVideo

Slides

12 comments

Partial Automation: Keeping humans in the loop

Tuesday, November 04, 2008

Posted by Patricia Legaspi, Test Engineering Manager

One of the challenges of automation is achieving complete automation. Ideally, complete or total automation would not require any human intervention or verification yet this is a difficult level to achieve. Investing Engineering time to completely automate tests is expensive and, many times, has diminishing returns. Rather than trying to achieve complete automation, investing in ways to make the most out of the automated test and the human time is time better spent.

Effective test report
Consider an automated UI test... Routinely, automated UI tests result in false negatives due to timing issues, the complexity of the steps taken, and other factors. These have to be investigated which can be time consuming. A thorough report can be created for automated UI tests to present log information, error reports, screen shots of the state of the application when the test failed, and an easy way to re-run the test in question. A human can make use of the information provided and effectively investigate the possible issue. If we can reduce the amount of work that someone has to do to investigate a test failure, the UI tests become more valuable and the human plays a much more important role as a "verifier." Had the report not provided any information for the failed test, the human would have spent, in some cases, hours investigating the issue rather than continuing to automate or run exploratory tests which would be of more value. Is there a way to maximize what people do well, while also maximizing the use of automation?

Applying human intelligence
What about tests that require a human eye? There are many arguments about tests that require a human make a judgment about the appearance of rendered UI objects. Machines are great at running tests that return in a firm pass or fail but for tests that require opinion or judgment, the human has an advantage. We've been experimenting with image comparison tests. Screen shots of a golden version of the application under test are compared to screen shots of the current release candidate to verify that the application continues to render properly and that the UI "looks good". Although image comparison techniques can determine if the screen shots are different, they cannot help determine if the difference is "important". The human comes back into the picture to complement the automated test. To make effective use of a person's time, the test results should be organized and well presented. For image comparison tests, a report which shows the golden and release candidate screen shots side-by-side with clearly highlighted differences or allows you to replace the golden screen shot is key. A tester can quickly navigate the reported differences and determine if they are actual failures or acceptable differences. Frequently, an application's UI will change which will result in expected image differences. If the differences are expected changes, the tester should be able to replace the golden with the new image with the click of a button for future comparison.

Aside from test automation, efficiency can also be improved by streamlining the various components of the process. This includes test environment setup, test data retrieval, and test execution. Release cycles tighten and so should the testing processes. Automating the environment setup can be a major time saver and allow for added time to run in depth tests. Creating continuous builds to run automated tests ahead of time and preparing environments overnight results in added testing time.

Automated tests are rarely bullet proof, they are prone to errors and false failures. In some cases, creating an infrastructure to make test analysis easy for humans, is much more effective than trying to engineer tests to automatically "understand" the UI. We coined this, "Partial Automation." We found that by having a person in the loop dramatically reduces the time spent trying to over engineer the automated tests. Obviously, there are trade-offs to this approach and one size doesn't fit all. We believe that automation does not always need to mean complete automation; we automate as much as you can and consider how human judgment can be used efficiently.

6 comments

Testing Blog

Guide to Writing Testable Code

Online Machine Learning Testing == Extreme Testing

Clean Code Talks - Global State and Singletons

My Unified Theory of Bugs

TotT: Finding Data Races in C++

Clean Code Talks - Dependency Injection

Clean Code Talks - Unit Testing

Partial Automation: Keeping humans in the loop

Labels

Archive

Feed