The missing link between industry and academia

In the final discussion round of SCAM, suddenly things got interesting as we were discussing (industrial) evaluation of tools.

There were a few reasons given by attendees for the lack of industrial evaluation.

An attendee from ABB David C. Shepherd said that they often try academic tools and that most of the time the tools don’t scale, prohibiting real academic evaluations.  Another participant (from industry) named the fact that the tools we built are often not based on industrial requirements. He asserted that it wouldn’t be hard to find industrial studies if our tools had more relation to real world problems. Both are true, I believe.

A final very interesting point was raised, with which I fully agree: In academia, focus lies on completeness: i.e. false negatives are very bad. However, for industrial use, eliminating false positives is way more important, as a few false positives already diminish trust in the tool. And a few true positives found with a tool help, even though large categories are not found. I think this is very true, we shouldn’t focus so much on finding all bugs/clones/smells, but on finding a few useful, real ones very precise. Food for thought!

Update: the ABB attendee has revealed himself 🙂

4 Comments

  1. Pascal Cuoq

    Hello, Felienne.

    The idea that the industry as a whole doesn’t care about false negatives and cannot work with false positives sound to me like a preconceived idea. Some companies with extremely large testing budgets are looking to automate a process that, as currently implemented, practically eliminates bugs at a very high cost. Any automated replacement for bits of these currently manual activities can only be a sound analyzer (*).

    The companies that do not have the budget to pay an engineer to spend time on false positives may be more numerous, but how much to they care about fixing bugs? The money they do not have to spend looking at false positives, they do not have to spend on sophisticated complete analyzers either.

    Sound static analysis is, in its own way, low-hanging fruit.

    (*) http://www.erts2012.org/Site/0P2RUC89/5C-3.pdf

    1. Felienne (Post author)

      Hi Pascal,

      Thanks for your comment. Of course there are scenario’s in industry where soundness is of vital importance. However, I think in general we put too much effort into building academic tools that can do a lot, but at the expense of giving so much warnings that it is too hard for users to see what is useful and what not.

  2. Pingback: The interaction of academia and business

  3. Dr. Evgueni Kolossov

    I am very glad that this topic has got quite wide response. I am responsible for that because it is me who raised it on conference. As a software vendors who has been pioneers in Static Analysis we are in contact with customers on daily basis. I would say that your view on what customers want is very primitive. False positives and false negatives are both in very high importance for the customers. Your view unfortunately based on the very limited information about customers and about Static Analysis. There are different types (levels) of Static Analysis. Let me briefly highlight this types. First – LINT-like, which is inexpensive but with limited analysis capability. This type usually producing high level of both false positives and false negatives. Second – “Bug Catchers”, which is strong on simulation, whole program test verification, allows multi-language support; often part of Swiss Army knife solutions: bundled with testing tools. This type have high level of false negatives rates, not good language usage, portability and preventative analysis. Third – Automatic Code Inspection which strong on four technology types: pattern-based, simulation, metrics, and comprehension. This type facilitates code review and pretest checking with code collaboration, sophisticated suppression management and measurement analysis; producing low false positives and low false negatives rates. Disadvantage is that multi-language support is very difficult and they quite weak on the whole program static test. Unfortunately there is very small number of companies which can do that type of analysis.
    So, you can see that before discuss good and bad requirements from industry, you need to specify what type of static analysis you are talking about. The current trends in industry is to use not a single tool but complete integrated solutions to cover the most of the possible ways to automatically identify the ‘bugs’, ‘defects’, and possible ‘problems’. Deep language analysis is extremely important because it is not possible to simulate and test all possible situations of software behavior under different circumstances. This is why initially static analysis has been used (and still mandatory) for safety-critical environments. Lately other industries started to look at it because this unforeseen situations can lead to the huge money loss (for example RBS software upgrade failure cost bank a billions).
    I have suggested on conference and can repeat that if industry is not always opened to academia then software vendors are looking for collaboration with academia and happy to discuss it any time. This is not pure about sorting out problems these vendors may have but also to discuss potential use of new ideas and suggest some directions for your research.

Comments are closed.