Levels of Granularity in Code Coverage

Code coverage granularity is an important aspect of automated tools that measure code coverage metrics for testing, but not all tools provide the same level of insight into code coverage.

To determine the supported level of granularity, ask yourself “What is the smallest unit of code whose coverage status I can determine unambiguously?”For many available tools, the granularity is limited to individual lines. This can be a bigger limitation than it seems at first glance. The key problem is that Java, like most popular programming languages, is a free-form language. A developer may write a complete Java class as one very long line of code. If a coverage tool that offers a line-based coverage granularity reports that this one-liner class is “covered,” what exactly does that mean? Does it mean that every single expression in that line was covered, or is 100% coverage reported if at least one expression in the line was covered?

Admittedly, the example of a Java class that is written as a single line of code is somewhat contrived, and such a programming style would already be a cause for concern. However, line-based granularity can reach its limits for some pretty common idioms. Consider the method in Listing 1.

public class Listing1
{
    static int minOrMax(boolean minimum, int a, int b)
    {
        return minimum? Math.min(a, b):Math.max(a, b);
    }
}

Listing 1: A class demonstrating the limits of line-based coverage granularity

A single test is necessary to partially cover the line containing the return statement, but at least two test cases are needed to cover both alternatives of the conditional expression. Even without using the conditional operator, it is not difficult to create situations where a line of code is only partially covered. Exceptions during program execution can always leave parts of a line without coverage. Line-based coverage granularity is sufficient for most cases—as long as the coverage tool does not report complete coverage for lines that are, in reality, only partially covered. Also, it may be tricky to determine why a particular line is not fully covered and what tests need to be added to achieve full coverage.

Coverage tools that report coverage for individual expressions make it easier to identify missing test cases. Visualization of expression-based coverage granularity is a little more intrusive. Line-based coverage can easily be displayed in a side ruler of the source editor, whereas expression-based coverage requires markers (like coloring or underlining) in the source code itself.

In rare cases, a seemingly atomic expression translates to multiple executable instructions—some of which may not get covered under certain circumstances. Ideally, coverage granularity reaches down to the level of individual instructions. In fact, many coverage tools collect information about whether or not individual instructions are executed. However, the granularity of the collected data may be reduced to expression-based or line-based coverage when reported to the user.

The Reasons for Low Code Coverage

The problem of increasing test coverage can typically be reduced to three scenarios:

• Available tests do not utilize input values that will cause control expressions for conditionals in the code to exercise all branches.
• Control expressions in a tested function depend on values returned by other functions, thus on the state of the program / code when the test is executing.
• Available tests do not use the proper set up for code under test so running a test results in an exception, breaking up the execution flow at the point of the exception.

Depending on the structure of the code, control expressions may be trivial (e.g. one if/else statement per function) or not (nested loop conditionals with control expressions that are returns of function calls intermixed with branch expressions). Based on these scenarios, different techniques for controlling conditions within the test framework are appropriate in each case.

Coverage Isn’t Everything

It’s common to mistakenly think that coverage is the only metric driving quality. Once teams are able to measure coverage, it’s not uncommon for managers to want to increase the number, because “higher coverage is better, right? Eventually the number itself becomes more important than the testing. Therein lies the problem – coverage needs to reflect real, meaningful use of the code, otherwise it’s just noise. Unless certain types and levels are required by an industry standard a product must comply with, code coverage, while important needs a pragmatic approach.

It’s important to realize that the cost of coverage goes up as coverage increases. Increasing coverage means new tests that must be maintained in the future. The need for new tests must be justified in terms of the overall project and quality goals. Creating tests for the sake of increasing coverage alone, introduces costs and technical debt.

By Parasoft

Parasoft’s industry-leading automated software testing tools support the entire software development process, from when the developer writes the first line of code all the way through unit and functional testing, to performance and security testing, leveraging simulated test environments along the way.