Challenges in Bug Finding

Finding Bugs is the Easy Part

• Simple unsophisticated checkers can find a big class of "silly" bugs
  • e.g. X.org vulnerability (missing parentheses)
  • Variables are not renamed after copy and paste of existing code
• Need a good GUI to sell the bug finder

Finding Source Code to Analyze

• What is the right source code to analyze?
  • The Build System knows because it links all the files
• But what if don't have the code for the Build System?
  • Use logs from gcc and replay them
  • Run Build System in a debugger
• C / C++ Programs hard to build
• Coverity made a product that is able to run without having to rebuild the system that it analyzes => More scalable
• Q: Don't users want to just push "play" and expect it to work?
• A: Yes, that is also the reason people don't like to make build systems.

Compiling the Code

• Not easy
  • There is no C/C++ standard -- different compilers have their own specific variations
  • e.g. Some programmers abuse precompiled headers of the .NET compiler
    • Rationale for precompiled headers -- template expansion can become very large and expensive to recompute each time
    • Some programmers take the treatment of precompiled headers literally:

The compiler treats all code occurring before the .h file as precompiled. It skips to just beyond the #include directive associated with the .h file, uses the code contained in the .pch file, and then compiles all code after filename.

• Very few tools for C & C++
  • EDG front-end
  • Too costly to deal with issues like the build system.

Values of Finding Bugs

Finding bugs is meaningless if they aren't fixed

• Value of finding bugs is in FIXING the bug, not finding the bug.
• Had some customers who hadn't fixed the bug even after finding the bug.
• Problem in process where Coverity fits in Company (ie. Customer) working-cycle
  • Important to know who the end-user is. (ie. Software Dev Organization)
• Requires Blame, Push, Oversight to enforce fixing the bug.
  • Blame: Someone needs to be held accountable if the bug is not fixed.
  • Push: Involves e-mail infrastructure where bugs are reported to developers and managers.
  • Oversight: Managers knows/sees about what's going on with the bugs.
• Static Analysis Tool provides a proactive way to find an impossible problem (ie. finding the bugs) if the user is willing to increase the software quality.
• The user has priority in fixing bugs. Coverity tool needs to be able to find the high priority bugs such that by putting in a little effort, the user can fix important bugs.

Usefulness of Static Analysis Tool

• Users/Customers understand that false positives exist in the tool. Customers are ok with 30~40% of false positives but feel waste of time if > 50%.
• Using heuristics frequently and algorithms reduces false positives.
• Usefulness of the tool means it's easy to use and doesn't require big "ramp-up" time.
• Observation: Bugs are most related to the memory issues.
• The tool tries to find the common-case bugs rather than a few very hard bugs.
• The company (ie. the user) having the product release data and organizational process has a different perspective view on the bug-finding tool
  • Engineer's Logic
    • Version A of tool X finds lots of bugs and some false positives
    • Version B finds more bugs with fewer false positives
    • Version B is better than version A
  • Organizational Logic
    • Version A of tool X produces a metric that we have come to rely on to make decisions
    • Version B produces a different value of that metric for the same code
    • Version B makes the metric unstable

Seth's thoughts/comments on good software

• Three things to build a good software
  • Programmers thinking like a user
  • Programmers thinking like a system architect (ie. thinking about putting pieces together to build the whole and scalability)
  • Programmers having good programming skills
  • The above three don't always come as one
• Complexity Tipping Point
  • Debt (complexity) vs. Equity (good programmers)
  • New features adds to complexity ("debt")
  • Good programmers build equity, but when the code becomes too complex (over the tipping point) no amount of programmers can manage it
• No software is perfect
  • Change is inevitable
• Very Difficult to make a change and add new features in the existing system due to its high complexity
  • High Complexity in Software -> Hard to Change -> Prohibits New Features
• How to make software better with Coverity
  • Reduces the risk of change and helps the system converge faster.
• To-be seen: How can we make programmers write less code?
  • Automatic interface generation from designer (architect) tools (e.g. Rational Rose)
  • Higher-level languages -- requires programmer education
  • Code synthesis from logic programming (in the future?)
• Q & A
  • Q: Why don't we see Coverity Analysis Tool being widely adopted and used by software companies if the tool finds the bugs and improves the quality of software?
  • A: Because the tool is not inexpensive, it might not be available to everyone. Had the tool been free to the public, it would've been spread out immdiately but it would mean that Coverity would have to support to broader range of users. (A joint project with Stanford and the Department of Homeland Security tries to find bugs in open source software and improve their quality using Coverity.)
  • Q: Have you had any customer who was reluctant to use your analysis tool because the customer was not comfortable having third-parties see their code?
  • A: No, because users run the tool by themselves, they are comfortable using it. Even customers like defense contractors who have highly confidential materials used the analysis tool.

Writeup Information

• Author
  • Chang-Seo Park - cpark05 at stanford.edu
  • Sung Chan Yu - yujosh at stanford.edu
• Talk Date
  • May 23, 2007