Structural code coverage is the identification of code that has been executed and logged for the purpose of determining if the system has been adequately tested.
Code coverage expresses the degree to which the application’s source code is exercised by all testing practices, including unit testing, manual testing, automated functional testing, and the like. It also:
As a result, application coverage provides extremely powerful insight into risk. In most organizations, unit testing is the primary vehicle for driving coverage. And while unit testing is a valuable testing practice that also enables several process-oriented benefits, it is also expensive in terms of the expertise and time to create, manage, and maintain the tests.
Code coverage can be leveraged as part of the continuous integration (CI) process, as well as part of the developer desktop workflow. You can also perform advanced analytics on the source code and identify which unit tests need to be rerun based on code changes the developer has performed.
These code coverage tools are especially useful in industries for embedded development of safety- and security-critical applications where software systems cannot fail, or lives will be lost.
The thoroughness of the coverage in safety-critical systems depends on the application safety integrity level (SIL or ASIL) metric used in various industries and the development assurance level (DAL) commonly used in avionics. Thoroughness refers to the structural elements in code. These are typically broken down to the code statement, branch, modified condition decisions and with Parasoft, you can also drill down to a much finer level of coverage granularity, such as the object code or assembly language.
Code coverage is often identified or measured through code instrumentation. This is done by adorning the source code with additional tracking code to determine while it runs whether each statement, branch, and/or modified condition decision (MC/CD) path has been executed.
It’s important to know that code instrumentation causes code bloat. The increase in code size isn’t usually a concern. For embedded targets with memory constraints, the increase may impact the ability to load the code onto your target hardware for testing.
Code coverage percentage goals can be subjective in some cases. In other cases, they’re mandatory. When building applications that are safety-critical where failure may cause death, regulatory and industry standards require 100% structural code coverage.
For applications that aren’t safety-critical, code coverage tends to be left to the development organization to decide. The common comfort level for these types of circumstances is achieving a goal of 80% code coverage.
Code coverage can help measure test completeness and it guarantees that you have inspected every single line of code in your application through execution. Leveraging a combination of testing methods like static analysis, unit testing, system testing, performance testing, and the like, is the best way to ensure code quality.
Fulfill code coverage requirements for functional safety standards like DO-178B/C, ISO 26262, ISO 21434, IEC 62304, IEC 61508, and EN 50128. Code coverage requirements help ensure code safety, security, and reliability by exposing untested code and flushing out defects.
Code defects found in production are the most expensive. Prevent them from slipping through the cracks by highlighting code that has not been tested before you release your application. This can be performed at the developer’s workstation or automated as part of the continuous integration (CI) pipeline.
For the absolute most stringent safety-critical code coverage requirements, such as DO-178C Level A, Parasoft offers code coverage at the assembly level. Save days — even weeks — of labor-intensive, error-prone manual effort with our automated solution.
Thorough testing of embedded safety and security-critical applications requires testing on-target hardware and in the host environment. Parasoft provides code instrumentation for both to capture and report structural code coverage for statement, branch, modified condition decision coverage (MC/DC), and more.
Although you can generally achieve 100% structural code coverage goals through unit testing, save time and effort by aggregating your code coverage results automatically obtained from system testing. And then complete the remaining uncovered code using unit testing.
Deploy Parasoft’s suite of products to conduct structural code coverage of applications written in C, C++, Java, C#, and VB.NET.
Collect coverage from unit testing, system testing, manual testing, as well as all other test execution methods used. Parasoft C/C++test supports a range of coverage metrics (Branch, Statement, MC/DC, etc.), which teams can use in native and cross application development.
Request Demo
Perform static analysis, unit testing, and code coverage to develop high-quality C and C++ code that is robust, safe, secure, and compliant with industry standards.
C/C++test is made for today’s high-velocity Agile DevOps environments. It integrates tightly into your CI/CD pipeline and containerized deployments.
Cut down on labor, costs and expedite time to market by automating compliance verification and validation demands, including reports and audit trail.
Parasoft’s AI/ML solution reviews static analysis findings, then organizes, predicts relevance, and prioritizes code violation findings.
Achieve static code analysis, unit testing, and high code coverage with JUnit to accelerate the delivery of secure and reliable Java applications.
Get a complete view of quality and compliance with aggregated reports and advanced analytics across Parasoft testing solutions.
Collect and monitor code coverage during manual or automated functional testing performed on your Java application. Users can send coverage data and test results to merge and correlate for analysis, which provides insights about how well the application is tested and your test quality.
Request Demo
Achieve static code analysis, unit testing, and high code coverage with JUnit to accelerate the delivery of secure and reliable Java applications.
Jtest integrates tightly into your CI/CD pipeline for real-time static analysis, unit testing, and code coverage testing.
Jtest users can easily generate, augment, and reuse unit tests with AI assistance, reducing costs and time.
Jtest provides code coverage compliance and visibility into what code has been tested and how thoroughly.
Perform static analysis, unit testing, and code coverage to develop high-quality C and C++ code that is robust, safe, secure, and compliant with industry standards.
Reduce the risk of C# or VB.NET development in the Microsoft framework with deep static analysis, security, and coverage for enterprise and embedded applications.
Collect code coverage information for standalone and web applications written in C# and VB.NET. Install the monitored application on the same machine where dotTEST is installed or on another machine.
Request Demo
Reduce the risk of C# or VB.NET development in the Microsoft framework with deep static analysis, security, and coverage for enterprise and embedded applications.
dotTEST helps users improve code quality, identify security vulnerabilities by applying CWE, OWASP and other standards.
dotTEST integrates tightly into your CI/CD pipeline for real-time, static analysis findings of C# and VB.NET codebases.
Teams can view test results on the desktop, web, or in static HTML reports, regardless of where they were executed.
Perform static analysis, unit testing, and code coverage to develop high-quality C and C++ code that is robust, safe, secure, and compliant with industry standards.
Achieve static code analysis, unit testing, and high code coverage with JUnit to accelerate the delivery of secure and reliable Java applications.
Obtaining the amount of code exercised during testing is a powerful metric in order to understand the level of risk remaining in the application. Here are some best practices to follow.
Integrating testing and code coverage into your build system will increase efficiency, promote collaboration, and provide greater visibility across your entire software development lifecycle.
Take advantage of your system tests to obtain code coverage and identify missing tests that may exposed hidden bugs.
Don’t burn time trying to determine all the input values and preconditions required in a test case in order to get coverage on an elusive statement, branch or MC/CD line of code. Use coverage advisor.
Use the various testing methods that you have to perform to expedite your coverage requirements. Aggregate coverage can be obtained from system, integration, unit, and manual testing.
Parasoft solutions can automatically analyze and generate unit test cases to flush out null pointer issues, out of boundary value conditions, divide by zero, and many other types of defects. In addition, these test cases can be enabled to produce code coverage and, in some cases, get you up to 90% structural code coverage.
Code instrumentation causes code bloat and the increase in code size may impact the ability to load the code onto your memory-constrained target hardware for testing. Instrument portions of your code at a time, run your tests, and aggregate your coverage metrics from each run.
If you have to satisfy 100% structural code coverage but have defensive code that will only execute upon the system entering into a fault state that could take weeks, months, or even years (or perhaps never), use your debugger to simulate the conditions needed and perform a visual inspection of the code execution/coverage.
If code is found because it has no pathway to its execution, for security and safety concerns, make sure it is deleted and never gets out into the field.
Parasoft code coverage within the IDE and DTP dashboard reporting and analytics solutions.
To get started in collecting code coverage, become familiar with your coverage requirement needs. Maybe your industry and type of application don’t require any coverage metrics to be obtained, but you want to ensure or improve code quality, so 75% might be your initial goal.
If your industry requires compliance to a functional standard like DO-178C for avionics or ISO 26262 and ISO 21434 for automotive or IEC 62304 for medical, then determine if you need to obtain 100% code coverage or another percentage of code coverage is recommended. Also, know what structural code coverage type you need to satisfy. It could be the function, line, statement, block, call, path, decision, simple condition, MC/DC, object/assembly, or a combination of these.
One of the easiest ways to obtain code coverage is during implementation when engineers are creating unit test cases to test their code. With solutions like Parasoft C/C++test and Jtest,
Be aware that instrumenting the code can cause code bloat and if you are testing on target hardware you may need to do partial instrumentation. Instrumented code may also alter the performance of the execution, so you may want to become acquainted with various optimization features available to you.
Collecting and reporting your code coverage metrics is crucial. There are several methods based on if your application is running on an embedded target, or a fully resourced system or server.
Ultimately, you’ll obtain or merge coverage data to get your full code coverage metrics. Coverage reports can be generated from your IDE, and they can be exported to Parasoft’s reporting and analytics dashboard DTP. In DTP, users can visually inspect the high-risk areas, understand what should be addressed next, and produce compliance and audit reports.
Parasoft’s solution for code coverage provides critical feedback about the completeness and thoroughness of the testing process, by correlating tests and structural code coverage results. You can use these results or metrics for assessing unit, integration, and system-level testing completeness through our support of all important types of code coverage (function, call, line, statement, block, path, decision, simple condition, and MC/DC), including object/assembly coverage.
Coverage results are available directly in the IDE, with convenient views and highlights in the source code editor, as well as in the form of static HTML or pdf reports, and dynamic reports through Parasoft’s centralized reporting dashboard.
Users can monitor applications executed natively on the desktop, cross-platform using simulators, or on real embedded hardware. C/C++test’s coverage module is optimized to minimize the impact on the execution performance and test binary footprint, which makes it suitable for use with high-end, server-based applications, as well as very limited systems based on 16-bit microcontrollers.
When connected with Parasoft’s Process Intelligence Engine, users benefit from test impact analysis. For each and every test performed, including manual, system-level, or UI-based, tests are recorded for not just test/fail and results but also their coverage impact on the codebase.
Each additional test is overlaid on this existing information, creating a complete picture of test success and coverage. As code is changed, the impact is clearly visible on the underlying record, highlighting coverage tests that now fail or code that is now untested. Raising this information in various degrees of detail allows developers and testers to quickly identify what needs to be altered/fixed for the next test run.
With Parasoft, teams can concentrate on code coverage for the areas of active development, instead of the entire codebase, which can be especially problematic when working with legacy codebases. Rather than solely trying to achieve a coverage number for the entire codebase, Parasoft helps you pinpoint the parts of the code that are changing.
Parasoft’s reporting dashboard correlates the data from C/C++test with observed changes in the codebase to focus the development team on achieving higher levels of code coverage for those specific, modified parts of the codebase. With Parasoft, you can minimize the impact of changes by efficiently managing the change itself.
Structural code coverage is the identification of code that has been executed and logged for the purpose of determining if the system has been adequately tested. Code coverage expresses the degree to which the application’s source code is exercised by all testing practices, including unit testing, manual testing, automated functional testing, and so on. This enables more accurate quality and coverage metrics while exposing dead code, untested, or under tested parts of the application. As a result, application coverage provides extremely powerful insight into risk. In most organizations, unit testing is the primary vehicle for driving coverage.
Code coverage should be determined by your coverage requirement needs or goals. Do you need to satisfy a safety functional standard (DO-178C for avionics or ISO 26262 for automotive, or IEC 62304 for medical) that mandates or recommends 100% coverage? Or maybe you just want to improve your code quality, so 60% might be your initial goal. Also, determine what structural code coverage type you need to satisfy. It could be the call, line, statement, block, function, path, decision, simple condition, MC/DC, object/assembly, or a combination of these.
We recommend a minimum of 60% code coverage, which should be easy to obtain. This will give you some comfort but knowing that 40% of the code was not tested may give you some sleepless nights.
Some recommend 70-80% but the best way to determine if you need to increase your coverage percentage is based on the number of defects that have been identified. If it’s a high number, then code quality is an issue, and we’d recommend increasing your code coverage. Know that even with 100% code coverage, it doesn’t guarantee that all defects have been identified. It only provides a reduction in risk.
Yes, obtaining 100% code coverage is possible. You can achieve this through the aggregation of various testing methods and through observation of code execution using your debugger. For example, unit testing can get you very close to your 100% target, but for lines of code that require very specific circumstances to execute, such as defensive code, it may take weeks, months, years, or you may never reach that event. If your coverage tool does not provide a way to reach 100% code coverage, you can use your debugger and, in some cases, edit the call stack and step through the uncovered code by visual inspection.
Knowing that your tests have not left out any untested code is your signal. However, your code coverage analysis goals should be based on any compliance requirement or your organization’s goal to improve code quality. If the requirement is 100% statement coverage, branch coverage, and MC/DC coverage, then that’s your target. When you have tested all of your lines of code and have reached the percentage of coverage the contract calls for, then you are finished. Bear in mind that open source testing is not as accurate or reliable as professionally developed testing tools, so you’ll want to use certified tools, such as those that Parasoft offers.
Absolutely! Parasoft’s test automation solutions have built integration into Azure pipelines and with other tools like GitHub, GitLab, Jira, and others that automate your continuous integration and continuous delivery workflow. The integrations provide flexibility, like the use of the command line API to automate not just code coverage testing but all of your software testing by invoking all your test suites.
