Posted by Marc Kaplan, Test Engineering ManagerWhen looking at whether to check a new change in, there are several testing-related questions that can be asked, which at least include: presubmit systems at Google to run all of the relevant tests in an automated fashion pre-checkin, without the developer doing anything before the code is actually checked in. The idea behind this is that once the code is finished being written there are minutes or hours before it's checked in while it's being reviewed via the Google code review process (more info on that at Guido's Tech Talk on Mondrian ). So we have taken advantage of this time by running tests via the presubmit system that finds all of the tests that are relevant to the change, runs the tests, and reports the results. GFS Performance Testing Tool that we previously described on another Testing Blog post. Once two runs of the performance test are finished, they are compared against a known-good baseline that gets automatically created at the beginning of each week, and a comparison graph is generated and e-mailed to the user submitting the CL. So now we know, prior to checking some code in if it has some unexpected hit to performance. Additionally, now when a developer wants to make a change they think will improve performance they don't need to do anything other than a normal code review which will trigger an automatic performance test to see if they got the performance bump they were hoping for. I've seen many companies and projects where performance testing is cumbersome and not run very often until near release date. As you might expect, this is very problematic because once you find a performance bug you have to go back through all of the changes and try to narrow it down, and many times it's simply not that easy. Doing the performance testing early, and often helps narrow things down, but we've found that it's even better to make sure that these performance regressions never creep into the code with pre-check in performance testing. Certainly you should still do a final performance test once this code is frozen, but with the presubmit performance testing system in place you are essentially certain that the performance will be as good as you expect it to be.
"2. Did this change brake existing functionality?"
ReplyDeleteI'm not sure this is quite write...
hi Mike!, did u mean RIGHT?. Cheers, Reine.
ReplyDelete"So we have taken advantage of this time by running tests via the presubmit system that finds all of the tests that are relevant to the change, runs the tests, and reports the results."
ReplyDeleteI'm very curious to know how you guys figure out the range of relevant components. Is it analyzed from the ready-to-check-in code, or something else? The range could be extremely large or difficult to be marked, I think. Could you please shed some light on it?
Thanks in advance.
Is there anyway I can convince the people I work with to use Unit testing? I mean they don't even use version control...
ReplyDeleteHow can I convince them that this is actually a good idea?
Andrew Leer: You don't need to convince all people at work, you just need to convince your boss. I would start easy with the source control thing, though.
ReplyDeleteI agree with Ryan. It would be very interesting to learn how you find "all of the tests that are relevant to the change".
ReplyDeleteI would imagine you use the tests code coverage data to do so. Please enlighten us!
@Ryan @cuqui
ReplyDeletei don't know how they do it.
it could be reverse code coverage database for existing code