Google Testing Blog: September 2008

Friday, September 26, 2008

by Miško Hevery

We talked about how it is important to separate the new operators from the application logic. This separation forces your code to have factories which are responsible for wiring your application together. By separating these responsibilities the tests can always wire together a subset of an application with key components replaced for friendlies making testing easier and more focused.
Let's look at a sample factory
class CarFactory {
Car create() {
return new Car(
new EngineCompartment(
new Engine(),

                new Door(new PowerWindow()),

              new Door(new PowerWindow()),

              new PowerSeat(), new PowerSeat()

new ManualTransmission(),
new PowerSteering(),
new Battery()
),
new Cabin(
new Door(new PowerWindow()),
new Door(new PowerWindow()),
new PowerSeat(), new PowerSeat()
),
Arrays.asList(
new Wheel(new Tire(), new Rim()),
new Wheel(new Tire(), new Rim()),
new Wheel(new Tire(), new Rim()),
new Wheel(new Tire(), new Rim())
)
);
}
}newCar(EngineCompartment ec, Cabin c, List ws);
EngineCompartment(Engine e, Transmission t,
Steering s, Battery b);
Cabin(Door driverDoor, Door passengerDoor,
Seat driverSeat, Seat passengerSeat);
Engine(float dissplacement, int pistonCount);
Battery(float voltage);
Door(Window window);

                new Door(new PowerWindow()),

              new Door(new PowerWindow()),

              new PowerSeat(), new PowerSeat()

PowerWindow() implements Window;
PowerSeat() implements Seat;
Wheel(Tire tire, Rim rim);
... Imagine you could just ask for things. Lets start simple and look at the Wheel. The constructor of Wheel needs a Tire and Rim. So when we ask for a Wheel it should be obvious that we want new Wheel(new Tire(), new Rim()). Why do we need to make this explicit in our factory? Lets build a framework from which we can ask for a class and it returns an instance of that class. So in our case if we ask for getInstance(Wheel.class) it returns a new Wheel(new Tire(), new Rim()). Now a framework like this is easy to build since all we need to do is look at the constructor and recursively try to instantiate the objects until all recursive constructors are satisfied.
But things are a bit more complicated than that. What if we ask for Cabin, as in getInstance(Cabin.class)? Well Cabin needs two Doors and two Seats, but Seat is an interface so we have to make a decision: What subclass of Seat should we instantiate? To help our framework make that decision, somewhere we will add a bind method such as bind(Seat.class, PowerSeat.class). Great! Now when we call getInstance(Seat.class) the framework returns new PowerSeat(). Similarly, we will have to call bind(Window.class, PowerWindow.class). Now we can call getInstance(Cabin.class) and the framework will return new Cabin(new Door(new PowerWindow()), new Door(new PowerWindow()), new PowerSeat(), new PowerSeat()).
Notice that closer a class you ask for is to the root (Car in this case), the more work will the framework do for us. So ideally we just want to ask for the root object, Car. Calling getInstance(Car.class) will cause the framework to do all of the work originally in our factory.
As you can see a framework which will call the new operators on your behalf is very useful. This is because you only have to ask for the root object (in our case the Car) and the framework will build the whole object graph on your behalf. This kinds of frameworks are called Automatic Dependency Injection frameworks and there are few of them our there. Namely GUICE, PicoContainer, and Spring.
Since I know most about GUICE, The above example can be rewritten in GUICE like this:
class CarModule extends AbstractModule() {
public void bind() {
bind(Seat.class, PowerSeat.class);
bind(Seat.class, PowerSeat.class);
bind(Transmission.class, ManualTransmission.class);
// maybe use a provider method?(jwolter)
// maybe explain the need for different wheels, so use a Provider;
bind(new TypeLiteral>(){})
.toProvider(new Provider>(){
@Inject Provider wp;
List get() {
return Array.asList(wp.get(), wp.get(),
wp.get(), wp.get());
}
});
}

// or, what I think is simpler and clearer
@Provides
List provideWheels(Provider wp) {
return Array.asList(wp.get(), wp.get()
wp.get(), wp.get());
}
}
// Then somewhere create your application, using the injector only
// once, for the root object.
Injector injector = Guice.createInjector(new CarModule());
Car car = injector.getInstance(Car.class);Root Cause of SingletonsSingletons are Pathological LiarsCalculatorServerModule.java is injected into the provider method for List<Wheel>. This is because our 4 wheeled car needs to have different wheels. If we injected a single wheel, it would be assigned as the same wheel on all 4 positions. Providers, when used without any explicit scopes will return a new instance every time get() is called on them.

2 comments

TotT: Mockin Ur Objectz

Thursday, September 18, 2008

[A light hearted episode this week... but still with a serious message. Enjoy. -Dave]GIMME lol_io LIKE LOLIO

SO IM LIKE PROCESSIN WIT DATAZ OK?
  GIMME EACH BUCKET IN UR DATAZ OK?
    BUCKET OWN FUBARRED?
      N CAN HAS NONE
    NOPE?
      N CAN HAS 1
  KTHXBYE N

IZ __name__ KINDA LIKE “__main__”?
  UR PROCESSIN WIT LOLIO OWN GET_SOME_DATAZ
  BTW, GET_SOME_DATAZ USES UR INTERNETS LOL
GIMME mock_lol_io LIKE LOLIO

BTW, GIMME THING TO TEST
BTW, TEST THE THING NOW KTHX
BTW, SOMETIMES THEY BE CALLIN DIS DEPENDENCY INJECTION ROFL

BTW, YOU CAN UZE MOCKZ N STUF FER DIS LOOK:

IN MAI library GIMME mock_filesystem LIKE LOL_FAKE_FILEYSTEM

BTW, NOW U CAN USE LOL_FAKE_FILESYSTEM TO MAKE FAKE FILEZ IN MEMORY N STUFF
BTW, IS FASTER THAN OPENIN FILEZ ON TEST SERVAR
this episode

7 comments

Presubmit And Performance

Thursday, September 18, 2008

When looking at whether to check a new change in, there are several testing-related questions that can be asked, which at least include:
1. Does the new functionality work?
2. Did this change break existing functionality?
3. What is the performance impact of this change?

We can either answer these questions before we check things in, or after. We found that we can save a lot of time and effort in trying to do detective work and track bad changes down if we do this before check in to the depot. In most teams developers kick off the tests before they check anything in, but in case somebody forgets, or doesn't run all of the relevant tests, a more automated system was desired. So we have what we call presubmit systems at Google to run all of the relevant tests in an automated fashion pre-checkin, without the developer doing anything before the code is actually checked in. The idea behind this is that once the code is finished being written there are minutes or hours before it's checked in while it's being reviewed via the Google code review process (more info on that at Guido's Tech Talk on Mondrian). So we have taken advantage of this time by running tests via the presubmit system that finds all of the tests that are relevant to the change, runs the tests, and reports the results.

Most of the work around presubmit at Google has been looking at functionality (question 1, and 2 above), however in the GFS Team where we are making a lot of performance-related changes we wondered whether a performance presubmit system would be possible. So recently we developed such a system and have begun to start using it. Basically a hook was added to the code review process such that once a developer sends out a change for review, and automated performance test is started via the GFS Performance Testing Tool that we previously described on another Testing Blog post. Once two runs of the performance test are finished, they are compared against a known-good baseline that gets automatically created at the beginning of each week, and a comparison graph is generated and e-mailed to the user submitting the CL. So now we know, prior to checking some code in if it has some unexpected hit to performance. Additionally, now when a developer wants to make a change they think will improve performance they don't need to do anything other than a normal code review which will trigger an automatic performance test to see if they got the performance bump they were hoping for.

Example of a performance presubmit report:

I've seen many companies and projects where performance testing is cumbersome and not run very often until near release date. As you might expect, this is very problematic because once you find a performance bug you have to go back through all of the changes and try to narrow it down, and many times it's simply not that easy. Doing the performance testing early, and often helps narrow things down, but we've found that it's even better to make sure that these performance regressions never creep into the code with pre-check in performance testing. Certainly you should still do a final performance test once this code is frozen, but with the presubmit performance testing system in place you are essentially certain that the performance will be as good as you expect it to be.

7 comments

The Google Maps API Open Source Their Selenium Test Suite

Wednesday, September 17, 2008

Issue 517our blog

5 comments

Where Have all the "new" Operators Gone?

Thursday, September 11, 2008

Miško Hevery(the other title: Your Application has a Wiring Problem)My main() Method Is Better Than YoursHow to Think About the “new” Operatorremoving the new operators from the application logicmain() methodJettysource public Server buildServer() {
Server server = new Server();

SocketConnector socketConnector
= new SocketConnector();
socketConnector.setPort(8080);
server.addConnector(socketConnector);

new ServletBuilder(server)
.addServlet("/calc", new CalculatorServlet(
new Calculator()))
.addServlet("/time", new TimeServlet(
new Provider() {
public Date get() {
return new Date();
}
}));

return server;
}

"So now each class needs a factory, therefore I have twice as many classes!" Heavens No! Notice how our ServerFactory acted as a factory for many different classes. Looking at it I counted 7 or so classes which we instantiated in order to wire up our application. So it is not true that we have one to one correspondence. In theory you only need one Factory per object lifetime. You need one factory for all long-lived objects (your singletons) and one for all request-lifetime objects and so on. Now in practice we further split those by related concepts. (But that is a discussion for a separate blog article.) The important thing to realize is that: yes, you will have few more classes, but it will be no where close to doubling your load.

"If each object asks for its dependencies, than I will have to pass those dependencies through all of the callers. This will make it really hard to add new dependencies to the classes." The myth here is that call-graph and instantiation-graph are one and the same. We looked into this myth in Where have all the Singletons Gone. Notice that the Jetty server calls the TimeServlet which calls the Date. If the constructor of Date or TimeServlet all of a sudden needed a new argument it would not effect any of the callers. The only code which would have to change is factory class above. This is because we have isolated the instantiation/wiring problem into this factory class. So in reality this makes it easier to add dependencies not harder.

herehere public Server buildServer() {
Server server = new Server();

SocketConnector socketConnector
= new SocketConnector();
socketConnector.setPort(8080);
server.addConnector(socketConnector);

DBConnectionPool pool = new DBConnectionPool();
new ServletBuilder(server)
.addServlet("/calc", new CalculatorServlet(
pool,
new Calculator()))
.addServlet("/time", new TimeServlet(
pool,
new Provider() {
public Date get() {
return new Date();
}
}));

return server;
}

6 comments

Test first is fun!

Monday, September 08, 2008

Posted by Philip ZembrodWrite a failing test for each featureThen implement the feature until the test succeeds.a way to higher productivitygetting tests written was indeed one of the less important reasons to write tests first!leads you to think about the interface firstfocus on how the new interface will be usedregard the test as a coded specificationa technical safeguard against growing stale: if it does, the test will probably failTests are no longer a necessary chore, but the place and time where I start to design something new. see my code running the minute it is written and compiled!writing new code for complex systems was always accompanied by fearslowed me downNow, this fear is gone!It will cost me just a few keystrokes to run my finished code, and I will immediately see what it does.It's the same old enthusiasm that I felt more than 20 years ago when I wrote and ran my first programs.

10 comments

My main() Method Is Better Than Yours

Monday, September 08, 2008

Miško Heverysrcpublic static void main(String[] args)
throws Exception {
// Creation Phase
Injector injector = Guice.createInjector(
new CalculatorServerModule(args));
Server server = injector.getInstance(Server.class);
// Run Phase
server.start();
}srcpublic static void main(String[] args)
throws Exception {
// Creation Phase
Server server = new ServerFactory(args)
.createServer();
// Run Phase
server.start();
}hereherehere

Notice how the creation-phase contains the code which builds the object graph of the application. The last line runs the application. The separation is very important. We can test the ServerFactory in isolation. Passing it different arguments and than asserting that the correct object graph got built. But, in order to do that the Factory class should do nothing but object graph construction. The object constructors better do nothing but field assignments. No reading of files, starting of threads, or any other work which would cause problems in unit-test. All we do is simply instantiate some graph of objects. The graph construction is controlled by the command line arguments which we passed into the constructor. So we can test creation-phase in isolation with unit-test. (Same applies for GUICE example)

The last line gets the application running. Here is where you can do all of your fun threads, file IO etc code. However, because the application is build from lots of objects collaborating together it is easy to test each object in isolation. In test I just instantiate the Server and pass in some test doubles in the constructor to mock out the not so interesting/hard to test code.

How to Think About the “new” Operator

No comments

TotT: Data Driven Traps!

Thursday, September 04, 2008

const struct {const char* word; bool is_word;} test_data[] = {
  {"milk", true}, {"centre", false}, {"jklm", false},
};

TEST(IsWordTest, TestEverything) {
  for (int i = 0; i < ARRAYSIZE(test_data); i++)
    EXPECT_EQ(test_data[i].is_word, IsWord(test_data[i].word));
}
faster than linearly Locale LOCALES[] = { Word::US, Word::UK, Word::France, ... };
const struct {const char* word; bool is_word[NUM_LOCALES];} data[] = {
  {"milk", {true, true, false, ...}}, // one bool per language
  {"centre", {false, true, true, ...}},
  {"jklm", {false, false, false, ...}}
};

TEST(IsWordTest, TestEverything) {
  for (int i = 0; i < ARRAYSIZE(data); i++)
    for (int j = 0; j < ARRAYSIZE(LOCALES); j++)
      EXPECT_EQ(data[i].is_word[j], IsWord(data[i].word, LOCALES[i]));
}
keeping the data in the test TEST(IsWordTest, IsWordInMultipleLocales) {
  EXPECT_TRUE(IsWord("milk", Word::UK));
  EXPECT_TRUE(IsWord("milk", Word::US));
  EXPECT_FALSE(IsWord("milk", Word::France));
}

TEST(IsWordTest, IsWordWithNonExistentWord) { // 'jklm' test is not repeated
  EXPECT_FALSE(IsWord("jklm", Word::US)); // as it uses the same code path
}
unmanageablethis episode

8 comments

Testing Blog