Take a faster, smarter path to AI-driven C/C++ test automation. Discover how >>
Whitepaper
Want a quick overview before you dive in? Start below.
Safety-critical C/C++ teams use unit testing frameworks to ensure correct implementation and adherence to software requirements. Open-source frameworks like GoogleTest offer lightweight integration with complex builds, aligning with industry trends toward open-source usage.
However, adopting GoogleTest for safety-critical standards compliance presents two main challenges: extending GoogleTest with functionalities critical for standards compliance, and navigating the formal tool qualification process required for safety-critical software development.
Safety standards intentionally avoid a rigid definition of a “unit,” as it depends on technology and architecture. From a compliance perspective, a unit is typically the smallest architectural component with a defined interface that can be tested in isolation.
Key standards like ISO 26262 (automotive) and DO-178C (aviation) mandate rigorous unit-level testing, requiring a common set of verification techniques:
To comply, development practices must include a unit testing framework, tools for coverage monitoring and fault injection, and a reporting system that correlates tests with requirements for traceability.
GoogleTest provides rich functionality for defining test cases and supports various test source strategies required by standards. ISO 26262 recommends four methods for deriving test cases: requirements analysis, generation and analysis of equivalence classes, boundary values analysis, and error guessing based on knowledge and experience.
Commercial frameworks typically support all four methods with automation, but often force proprietary APIs that abstract from source code. These APIs hamper test creation for modern C++, making STL containers initialization and lambda inputs exceedingly difficult.
GoogleTest wins with flexibility and modern C++ API for test creation, making complex data structure scenarios simpler and more manageable. Test scenarios scale better with team size and test case volume.
Tool qualification for safety-related GoogleTest applications requires special attention. Commercial frameworks are precertified with qualification packages, but open-source community provides no such support for GoogleTest.
Parasoft’s experience with leading automotive and aerospace companies shows teams strongly prefer GoogleTest’s ease of use and test clarity over basic generation capabilities, especially when using Bazel build systems.
Tool qualification is a formal process mandated by safety standards to ensure tool-related error risk is acceptably low. The open-source community provides no support for GoogleTest qualification.
The qualification process must include tool classification and validation, concluding with appropriate documentation. The documentation package contains guidelines for deployments in critical development (Tool Safety Manual/TSM).
Tool Classification determines qualification rigor based on tool use cases and additional measures. Common practice is tool impact analysis—the probability of the tool introducing an error or failing to detect one.
Unit testing frameworks like GoogleTest don’t generate code for the target system, so there’s no possibility of introducing an error. However, they can fail to detect errors. A malfunctioning assertion may report a failing test as passed, resulting in faulty code deployed to product.
ISO 26262 defines three confidence levels (TCL 1-3), with level 3 imposing highest qualification rigor. DO-178C uses five levels (TQL 1-5). Unit testing frameworks are typically classified as TQL5 for DO-178C compliance.
Tool qualification through validation proves the tool behaves as declared in operational requirements, ensuring tool error probability is acceptably low. Two critical aspects are tool operational requirements and tests validating these requirements.
For GoogleTest qualification, both aspects require significant investment. GoogleTest comes with quality documentation but no formal requirements available for validation. Organizations must develop requirements documentation and validation tests.
Requirements play a critical role in safety processes, precisely specifying what has been validated and can be safely used. Tool documentation cannot replace requirements because they serve different roles.
Parasoft’s experience resulted in a three-phased process:
Requirements were developed based on GoogleTest public documentation, with dedicated experiments conducted when documentation wasn’t exhaustive. Each requirement was linked to API element, source code, and user documentation to facilitate maintenance.
Complete analysis resulted in 1500+ requirements.
Recommendations:
Qualification test cases must derive from requirements for validation purposes. Preexisting GoogleTest test cases are high quality but difficult to use directly in qualification due to granularity mismatches and insufficient negative tests.
Parasoft’s experience shows most validation test cases were developed from scratch.
One key challenge was developing a testing strategy for API call-chains. GoogleTest is a template-based framework offering enormous flexibility. Exhaustive testing of all possible API combinations is impossible. The EXPECT_CALL GoogleTest Mock macro illustrates the challenge—it takes parameters and can be fine-tuned with modifiers, resulting in astronomical test case numbers.
All-Pairs Approach: For long call chains, this methodology ensures each combination of two different modifiers is tested at least once instead of testing all possible combinations.
MC/DC Coverage-Inspired Strategy: Applied to Assertions/Expectations long call chains. The entire call chain was treated as a decision with each modifier as a condition. Testing focused on proving each individual modifier can affect the outcome with other modifiers unchanged.
Both strategies enabled significant reduction in test case numbers for complex API invocations.
The qualification process requires specific documents including operational requirements, confidence level declaration, qualification plan, requirements coverage matrix, and guidelines for safe tool application (Tool Safety Manual/TSM).
Organizations must extract TSM-relevant guidelines from GoogleTest documentation and analyze all known bugs for implications. Enforcing TSM restrictions is challenging—typically manual effort. Parasoft applied static analysis rules to automate enforcement of many TSM recommendations, significantly reducing costs.
The GoogleTest unit testing framework is likely the best option for testing modern C++ applications. Projects utilizing Adaptive AUTOSAR or similar platforms and those based on AI/ML modern C++ libraries select GoogleTest by default. Increasing GoogleTest use reflects a strong industry trend toward open-source software for safety systems development.
Deploying GoogleTest for safety-critical software development is possible. However, significant effort and a skilled team are required to qualify the tool successfully. Organizations must recognize GoogleTest qualification may not be a one-time effort. With advancements in modern C++ and new functionalities added to subsequent releases, development teams may demand adopting newer versions, forcing tool re-qualification.
However, overall gains in software testing efficiency, tool integration effort, ease of testing assets development, and maintenance outweigh the additional compliance-related costs.
Ready to dive deeper?
