By Marko Ivanković, Google Zürich Introduction
Code coverage is a very interesting metric, covered by a large body of research that reaches somewhat contradictory results. Some people think it is an extremely useful metric and that a certain percentage of coverage should be enforced on all code. Some think it is a useful tool to identify areas that need more testing but don’t necessarily trust that covered code is truly well tested. Others yet think that measuring coverage is actively harmful because it provides a false sense of security.
How we measured
Google build system whose parallelism and flexibility allowed us to simply scale our measurements to Google scale. The build system had integrated various language-specific open source coverage measurement tools like Gcov (C++), Emma / JaCoCo (Java) and Coverage.py (Python), and we provided a central system where teams could sign up for coverage measurement.briefly explained in this article ) and display the data visually to both the commit author and the reviewers. We display the data on two levels: color coded lines right next to the color coded diff and a total aggregate number for the entire commit.
numbers is the coverage information. We use light green for covered lines, orange for non-covered lines and white for non-instrumented lines.
before the commit is submitted to the codebase, because this is the time when engineers are most likely to be interested in improving it.
Results
C++
Java
Go
JavaScript
Python
56.6%
61.2%
63.0%
76.9%
84.2%
Future work
job site ?
Don't know if my first comment got through, so I'll try again. Did you try correlating a project's code coverage and the amount of open issues?
ReplyDeleteBranch Coverage is more important than Statement coverage, while measuring the thoroughness of tests.
ReplyDelete+1
DeleteI am curious to find out if you tried to correlate branch coverage (instead of statement) and the presence of defects if it would not result in stronger correlation, but I am guessing yes.
Did you try that? Can you share your findings?
We did. We didn't find any correlation yet. But we are looking into that.
ReplyDeleteOK. Because that sounds like it might be interesting for clarifying some of the myths you mention in the first paragraph. Keep us posted if you make some discoveries there ;)
DeleteYes, knowing how coverage relates to problems encountered by users would be great information to have.
ReplyDeleteWe are working on this. We will publish the results as soon as we have concrete findings. So far we have neither definite positive nor definite negative results for that question.
DeleteHow do you measure JS code coverage?
ReplyDeleteYour first paragraph mentions the controversy around code coverage, but it looks like your results and research don't attempt to address that core issue, e.g. whether code coverage actually leads to fewer errors in the resulting project.
ReplyDeleteWhich system do you use for code reviews? Is it's Gerrit: How did you integrate the coverage metrics into Gerrit?
ReplyDeleteIn some ways not unexpected that coverage is inversely correlated to language type strength
ReplyDeleteAm confused with the below statement in future work section. the code that is being hit by the tests need to be instrumented. why do we need to instrument the tests?
ReplyDelete"In particular, the individual tools we use to measure coverage are not perfect. Large integration tests, end to end tests and UI tests are difficult to instrument, so large parts of code exercised by such tests can be misreported as non-covered. "
Apart from unit and integration testing are you also capturing statement coverage through manual functional test? What toolset you use for Java.
ReplyDeleteHi!
ReplyDeleteDo you have any news about turn open source the central system you developed to measure the test coverage?
How do you measure your functional tests code coverage?
ReplyDelete