In the final discussion round of SCAM, suddenly things got interesting as we were discussing (industrial) evaluation of tools.
There were a few reasons given by attendees for the lack of industrial evaluation.
attendee from ABB David C. Shepherd said that they often try academic tools and that most of the time the tools don’t scale, prohibiting real academic evaluations. Another participant (from industry) named the fact that the tools we built are often not based on industrial requirements. He asserted that it wouldn’t be hard to find industrial studies if our tools had more relation to real world problems. Both are true, I believe.
A final very interesting point was raised, with which I fully agree: In academia, focus lies on completeness: i.e. false negatives are very bad. However, for industrial use, eliminating false positives is way more important, as a few false positives already diminish trust in the tool. And a few true positives found with a tool help, even though large categories are not found. I think this is very true, we shouldn’t focus so much on finding all bugs/clones/smells, but on finding a few useful, real ones very precise. Food for thought!