15 Seconds to Find a Pointer Error in 500,000+ Lines of Code
March 21, 2013
5 min read
Here’s the story of how 15 seconds of analysis uncovered a crash-causing error that evaded 40 hours of manual inspection
BITTT Enterprises specializes in business processes and provides strategic business solutions for information management. BITTT helps their clients improve internal technology systems, increasing efficiency and productivity for a healthier bottom line.
Timothy W. Okrey, Managing Partner, is in charge of development at BITTT. In fact, he is the mastermind behind the code that BITTT writes. Recently, Okrey was continuing development on an on-going project that had been in the works for a couple of years. The program was in virtual production when it suddenly started crashing. The situation left Okrey completely dumbfounded.
After trying to resolve the problem on his own and hitting brick walls in every direction, Okrey discovered Parasoft’s solution for C/C++ runtime analysis and error detection. Parasoft Insure++ not only helped Okrey resolve the issue at hand, but also enabled him to simultaneously and effectively enhance a dozen separate projects.
Emergence of a Critical Pointer Error
The product that Okrey was developing had been stable and running in a virtual production mode. But the program started failing after a recent build to address a number of enhancements requested by the customer.
BITTT had invested two years on the product, a payroll-related solution designed to help the customer bring down the 60 to 70 man hours invested every week to manually complete payroll for 1000 employees in 14 states. As a result of BITTT’s work, their customer’s payroll was now automated, enabling them to spend less than 12 man hours on it each week. Unfortunately, the show-stopping error that emerged with the latest build caused BITTT’s customer to revert back to their manual payroll process.
Based on 20+ years of development experience, Okrey knows that if you run into a brick wall, it’s time to redo the entire project a different way. Unfortunately, that wasn’t even an option in this situation because “There was no smoking gun or even a traceable error.” Okrey explains further, “This was not new development. Nor did we try to pull parts of code together to make it work. This particular program was written from scratch using a toolkit as the backend for the details.”
The toolkit is one that Okrey started creating in 1993. It allows him to pull working functions into raw source code or use them as a library for any project. The toolkit provides a stable foundation for all of his projects and alleviates the need to rewrite code over and over again. This toolkit has grown to well over 500,000 lines of code, which was written with utmost diligence. Okrey strictly follows the rules of structured programming and is judicious about keeping his code clean. He had never used a third party tool to analyze his code and has never had the need.
Reducing the Length and Cost of Downstream Development Processes
For over a week, Okrey tried to re-engineer different pieces of the class that was causing trouble.
But his attempts at fixing the problem only resulted in changing some of the internals so that the point of failure occurred in a different location. “I spent more than 40 hours going through all of my code with a fine tooth comb and a magnifying glass, like I usually do. I was unable to locate the problem. I could see what was happening; I just could not see why it was happening,” Okrey said.
That’s when his search for help began. He found only a handful of tools that were able to do what he wanted. Of that handful, most of the products merely allowed for static review of code. Parasoft was the only product that also performed dynamic analysis. “Parasoft gives me the ability to analyze my content in the environment where it’s being run as opposed to just looking at the code on paper,” Okrey said.
After getting set up and running, Parasoft ran through the first build of Okrey’s source code—all 500,000+ lines. Within 15 seconds of launching, Parasoft surfaced a stale pointer error. “If I hadn’t found Parasoft, it would have led to very drastic requirements from the client,” Okrey said as he reflected on the rapid return on investment. He added that “To go from a functioning version of the program to a non-functioning version simply due to an upgrade would have led to a reversal of progress and forced financial concessions that I do not want to even consider. It was an ugly situation.”
Increasing Code Quality, Stability, and Compliance
Parasoft enabled Okrey to completely revamp the toolkit source code; specifically, improving string-handling. The improvement spread to other projects. Okrey has dozens of other programs for various clients that use the same backend toolkit, so all of these programs reaped the benefits. Okrey states, “I can’t even begin to tell you all of the programs that are dependent on the backend toolkit. As a result of the improvements Parasoft enabled me to make, they are all that much more stable and compliant.”
Okrey said that the Parasoft solution gave him the ability to implement and enforce his high coding standards. “Parasoft forces you to verify that the standards and practices that are being used are absolutely pristine,” Okrey said. “One of the challenges as a project leader—or managing partner, like myself—is confirming that your team is writing code that meets high standards. Parasoft can help me verify that my team is writing code that meets my standards and allow me to guarantee results. I am really excited about that.”
Getting Value from Parasoft’s Solution
Okrey is pleased with the quality that Parasoft has rapidly ingrained into his application development process. Not only has he been able to rectify a problem for a valued customer, but he has also been able to improve the quality of dozens of programs for other customers.
Okrey says, “I’m very particular about products that I choose to endorse. The majority of software written in the world just doesn’t work the way it’s supposed to work for various reasons. Maybe it’s poorly designed so it runs slow, or system requirements aren’t realistic. The list goes on.
“However, there are a few products that I really like. One of those is a system software product that I’ve come to rely on. I’ve never experienced a GPF with it. Never. When I learned that the provider of that product was a Parasoft customer, that was it. That’s what made me decide to give Parasoft a try and I’m very happy that I did.”