Enterprise Unix Roundup: The Fed-Backed Bug Zapper

By Amy Newman (Send Email)
Posted Mar 10, 2006

Main     In Other News     Elsewhere in the Corral     Tips of the Trade
Coverity's initial results are in. How did the 32 open source projects it looked at fare? Do you know the four references for building efficient, strong firewalls? This week's Tip of the Trade reveals all.

Brian Proffitt
Amy Newman

There is an adage in the open source community, first attributed to Linux kernel-meister Linus Torvalds: "many eyes make shallow bugs." We would be hard-pressed to disagree, because the overall effect of so many developers working with and examining open source code projects has created some mighty fine applications and operating systems.

But defects and bugs don't always leap out and say "here I am!" no matter how many eyes look right at them. Eyes have brains behind them, and brains can become tired, impatient, or just uncaring about finding certain bugs. Still, the bugs may be there, waiting to trip some applications up. What to do? Automated testing.

Back on January 11, it was announced that the Department of Homeland Security was awarding a grant to Stanford University, Symantec, and a company called Coverity to launch a project using Coverity's defect-searching tools on various open source projects and provide those projects with defect information. What was unspoken at the time was the question: Would these open source projects accept the recommended changes based on the findings? Were the changes going to be valid or necessary?

This week, Coverity announced the initial results of its code scans, churning out numbers for 32 open source projects. The numbers themselves are interesting. Using a 3.2-Ghz Linux box, the Coverity tool ran through 17.5 million lines of code in 27 hours. In that time, it found the average defect rate to be .434 defects per 1,000 lines of code for the projects, which included Linux, Apache, Firefox, FreeBSD, and X.

Somewhat tellingly, the average defect density of just the LAMP (Linux, Apache, MySQL, and Perl/PHP) stack was .290. "Somewhat" because Coverity CTO and co-founder Ben Chelf suggested this might indicate that the "more popular projects were seeing less defects." Chelf stopped just short of making an official statistical pronouncement, however.

We, having no PR concerns, will venture to say that the numbers do seem to support that trend. The more popular and older open source projects trended toward lower defect numbers. Of course, we are pundits, not statisticians, so feel free to check the numbers out yourself at the results Web site, where the defect numbers will be updated on a regular basis.

These numbers are all well and good, but what are open source developers supposed to do with them now?

Chelf explained to us that Coverity will be in contact with all of the project maintainers for the examined applications to provide them with the actual defect information. From there, managers can decide what to do with the information. "We want to give the first look to people who own the code," he stated. This hand-off to maintainers is also important because only they will understand the importance of a given set of defects.

Chelf emphasized that Coverity's results are not prioritized in terms of importance. There are no highlighted critical errors in Coverity's data. Instead, errors are classified by type: memory leaks, path errors, and so on. It will be up to project maintainers to decide which bugs get the higher fix priority and who on their teams will fix said bugs.

Early reports from the open source community seem favorable. We thought the reaction of Linus Torvalds, whom we spoke to back in January about Coverity's initial scans of the Linux kernel in 2005, summed up the overall reaction of the community thus far.

When asked if Coverity's 2005 results were being used, Torvalds confirmed some bug fixes had indeed found their way into Linux patches. More importantly, the issue is not of quantity, but of quality.

"Well, Coverity doesn't send patches, but yes, some people go through the Coverity results and send patches based on them. Sometimes the warnings are bogus — automated checking has very definite limits — but just as a statistic, 'coverity' is mentioned 75 times in the commit messages over the last eight months or so," Torvalds stated, adding: "Not all of them were bugs — some of it was to just shut up a bogus warning, but the point being that it does actually end up being useful."

It will be interesting to see how the open source project managers use this information. Some progress has already been seen. On Monday, the number of defects for the gcc compiler went from a total of 140 bugs to 96 bugs as of this Thursday. Small progress like this has also been noted in other projects.

Our sense of perfectionism gets a warm, fuzzy feeling from squashing bugs, big or small, so we hope the progress continues.

>> To Other News
>> Elsewhere in the Corral
>> To Tips of the Trade

Page 1 of 2

Comment and Contribute

Your name/nickname

Your email

(Maximum characters: 1200). You have characters left.