What You Need to Know About Code Coverage for Embedded C/C++ Systems

Code coverage measures how much of an application’s source code is tested through various methods, such as unit testing, manual testing, and automated functional testing. Code coverage percentage goals can be subjective. The thoroughness of the coverage in safety-critical systems depends on the application safety integrity level (SIL) metric used in different industries and the design assurance level (DAL) commonly used in avionics. In building safety-critical applications where failure may cause death, regulatory and industry standards require 100% structural code coverage.

This presentation dives into the essential topic of code coverage for embedded C/C++ systems, explaining its importance, different measurement types, and practical application. We explore how to achieve thorough testing, especially in safety-critical environments, and demonstrate tools that simplify the process.

Understanding Code Coverage

Code coverage is a metric that tells you how much of your application’s source code has been executed during testing. It’s a way to answer the question: “Have we tested enough?” By highlighting untested code, it helps uncover potential bugs and identify dead code. The most common way to achieve this is through code instrumentation, where extra code is added to track the execution of statements, decisions, or branches. This instrumentation then logs the execution, allowing for a percentage of code coverage to be calculated and visualized, often with code highlighted in green (executed) or red (not executed).

Key Coverage Criteria

• Statement Coverage: Ensures every statement in the application has been executed.
• Line Coverage: Checks if every line of code has been executed. The value result can differ from statement coverage if multiple statements exist on a single line.
• Branch Coverage (Decision Coverage): Verifies that each possible execution path after a decision point (like an if statement) is taken at least once. This includes ensuring both true and false outcomes for conditions are tested.
• Modified Condition/Decision Coverage (MC/DC): A more rigorous standard, particularly for safety-critical systems. MC/DC requires that each condition within a decision has been shown to independently affect the decision’s outcome. This means testing all possible outcomes for each condition and ensuring that each condition can change the decision’s outcome.
• Multiple Condition Coverage (MCC): Tests all possible combinations of outcomes for conditions within decisions. While thorough, it can lead to a very high number of test cases.

Practical Demonstration Highlights

The presentation included a demonstration showcasing:

• Automated Unit Test Case Generation: Tools can automatically generate numerous unit tests to help achieve coverage goals.
• Coverage Analysis: Visualizing coverage metrics (line, statement, branch, MC/DC) within IDEs like Eclipse, VS Code, or through standalone command-line tools.
• Application Coverage: Using command-line tools to instrument and collect coverage data for standalone applications, which can then be imported into an IDE for analysis.
• Target Hardware Testing: The ability to perform coverage analysis on embedded targets, with optimizations for size and speed, and support for multi-threaded applications.
• Reporting: Generating reports and publishing coverage data to platforms like DTP (Development Testing Platform) for centralized analysis and dashboarding.

Why Specific Coverage Types Matter in Embedded Systems

In the embedded world, C and C++ are frequently used in safety and security-critical systems. Industries like automotive (ISO 26262), avionics (DO-178C), and medical devices (IEC 62304) have strict regulatory requirements and process standards. These standards often assign Safety Integrity Levels (SIL) or Design Assurance Levels (DAL) to software components. Higher levels, indicating greater risk if the software fails, typically mandate more rigorous testing. For instance, SIL 4 in IEC 61508 highly recommends 100% coverage for statements, branches, and MC/DC. The common thread across these standards is the focus on statement, branch, and MC/DC coverage, as these are deemed best practices by industry experts to ensure high-quality, safe, secure, and reliable code.

Methods for Obtaining Code Coverage

Achieving code coverage can be done through various testing methods:

• System Testing: Running existing system test cases against instrumented code can provide a significant portion of coverage with minimal extra effort. However, it often falls short of 100%, especially for defensive code that only executes under fault conditions.
• Unit Testing: Creating specific unit tests can target uncovered code, including defensive code or specific branches, to reach higher coverage goals.
• Manual Testing: While less automated, manual test execution can also contribute to coverage metrics.

Many organizations combine results from these different testing methods to achieve their overall coverage targets. For example, coverage from unit tests can be merged with coverage from system tests.

Integrating Coverage into CI/CD

Code coverage is a vital part of a Continuous Integration/Continuous Delivery (CI/CD) pipeline. Tools can automate the instrumentation, execution, and reporting of coverage data, integrating seamlessly with build systems and CI/CD platforms like Jenkins, GitLab, and Azure DevOps. This provides real-time feedback on code quality and helps manage risks effectively.

Handling Challenges in Code Coverage

• Unreachable Code: Sometimes, due to infinite loops or specific code structures, certain statements might be impossible to reach through normal testing. In such cases, coverage by inspection (visually verifying execution, commonly through debugging) is a recommended approach to satisfy standards.
• Code Bloat: Instrumentation can increase the size of the executable. If this prevents the application from fitting on the target hardware, a common practice is to instrument half the code, collect coverage, then instrument the other half and merge the results.
• Assembly Level Coverage: For high-assurance systems (like DO-178C Level A), coverage at the assembly or object code level is required because compilers can generate code not directly traceable to the source. Tools exist to automate this process.

Ultimately, code coverage is a powerful technique for ensuring the quality and reliability of embedded C/C++ systems, especially in safety-critical domains. By understanding the different coverage criteria and leveraging appropriate tools, development teams can achieve their testing objectives more efficiently.