Table of Contents
Mutation Testing is a powerful method for finding errors in software
programs. Until now, however, no tool has been able to bring this power to
the everyday programmer. This changes with Insure++, the first tool to bring
Mutation Testing's unique methodology to the error-detection field.
This paper will describe the process of Mutation Testing and how Insure++'s
revolutionary approach benefits the entire software industry.
What is Mutation Testing?
Mutation Testing has been well-known to computer scientists for decades. It
is the process of "re-writing" source code (hence the name
Mutation) in order to flush out ambiguities that may exist in the code.
These ambiguities can cause failures in software if not detected and fixed by
the developer. Because these faults are often very subtle, they can easily
pass conventional testing and debugging only to be passed on to the
customer.
Although powerful, Mutation Testing is complicated and time-consuming to
perform without an automated tool. Lacking a tool to bring its benefits to the
commercial market, Mutation Testing has not spread far outside the research
community. The longtime goal of researchers has been a testing system that is
highly effective, efficient, and convenient to use.(1) This breakthrough has
been achieved with the release of Insure++ v4.0, which incorporates
Mutation Testing automatically into its error-detection technology. Now
software developers and testers everywhere can enjoy the most powerful
error-detection capabilities with the speed and ease-of-use of an automatic
runtime debugging tool.
Classical Approaches To Mutation Testing
Mutation Testing originated as a means of creating more effective test
suites. By creating more effective test suites, the tester can be more
confident that his program is adequately tested.
The classical approach to Mutation Testing begins by creating many versions
of a program. Each of these versions is "mutated" to introduce a
single fault. These "mutant" programs are then run against test
cases with the goal of causing each faulty version to fail. Each time a test
case causes a faulty version to fail, that mutant is considered
"killed" and the test case is saved. When all mutants have been
killed, the saved test cases will comprise the test suite used to test the
original program.
This approach to Mutation Testing is useful for creating test suites, but
is little help for finding bugs in the original program. Insure++ takes a new
approach to Mutation Testing, shifting the focus from the test suite to the
program itself. This change enables Insure++ to find bugs that exist in the
original source code.
How Insure++ Revolutionizes Mutation Testing to Find More
Bugs Automatically
Insure++ takes a different approach to Mutation Testing, creating
"equivalent" (rather than "faulty") mutants. This
technique makes for more effective bug-checking, as any ambiguities that may
exist in the original source code are exposed during mutation. Insure++'s
patented Source Code Instrumentation (SCI) technology (patent
#5,581,696) then provides in-depth debugging information that makes it fast
and easy for the developer to find and fix bugs.
During the checking process, Insure++ parses, analyzes, and converts a
developer's original source code into a new, functionally equivalent source
code. This equivalent code is stored in a temporary file that is passed to the
compiler. (It is important to note that Insure++ does not modify the original
source and no user intervention is required. The process is completely
automatic and invisible to the user.)
Once all of the developer's files have been "mutated" into
equivalent code, they are linked into a final executable, ready for run-time
error detection.
During the error-detection process, the "functionally equivalent"
mutants are run in place of the original source code. Unlike faulty mutants,
these "equivalent mutants" are expected to "pass" instead
of fail. If the original program is correct, then the mutant can not be
killed. If a mutant is killed, it means there is an error in the original
program that must be fixed. Through this method, Insure++ is able to uncover
ambiguities that are undetectable by any other method or tool.
Whenever a bug is found, Insure++ automatically reports the fault and the
line responsible. The inclusion of debugging information is a major difference
between classical Mutation Testing and the Mutation Testing performed by
Insure++. This improvement is made possible through a process called
code insertion in which Insure++ inserts a line of code before and after
each line of mutated code. This extra code enables Insure++ to accurately
check for and report errors found during program execution.
An additional benefit to Insure++'s approach to Mutation Testing is that
users can create as many (or as few) mutants as they wish. Using Insure++'s
flexible suppression mechanism, users can create equivalent mutants for any
code they wish, from a single function to an entire project. Of course, the
more mutants created, the more thorough the error-detection.
This new approach to Mutation Testing lets Insure++ meet the goals of
effectiveness, efficiency, and convenience aspired to by researchers. More
debugging information is provided to the developer, improving effectiveness,
less time and computing is required, increasing efficiency, and for ultimate
convenience, the entire process is made completely automatic.
Benefits of Mutation Testing
Mutation Testing brings a whole new level of error-detection to the
software developer. This powerful method is able to uncover ambiguities in
code previously thought impossible to detect automatically. By incorporating
Mutation Testing into its state-of-the-art error-detection technology,
Insure++ is able to flush out more faults than any other tool.
Software developers and testers using Insure++ will benefit enormously from
Mutation Testing, as more bugs will be reported automatically with Insure++
than with any other tool. Errors that were once frustrating and
time-consuming to find and fix can now be found automatically, and Insure++'s
accurate, detailed error reports make debugging and maintenance fast and
easy.
The customer also benefits from Mutation Testing, as the program he
receives is less buggy and more reliable. This increased confidence will in
turn benefit your company where it matters most-- the bottom line.
(1) Offutt, A.J. A Practical System for Mutation Testing: Help for the
Common Programmer.
|
