Are Mutants a Valid Substitute for Real Faults in Software Testing? – René Just

2014-11-20 14.41.23

What exactly is a mutant? For example, deleting a line of code or changing the value of a variable. With this, a number of mutants, we can compute the mutation detection score. In this paper, Rene is going to analyze the idea of mutants in depth:

1. Do stronger tests detect more mutants?
2. What types of real faults are not coupled to mutants?
3. Is mutant detection correlated with real fault detection?

So first, we need fault, to be able to detect if mutation testing can detect them. The faults were extracted from version control systems. This way, the authors found 357 real reproducible real faults, and mutated the source code in a number of ways: Replace constants, Replace
operators, Modify branch conditions, and Delete statements, plus the associated test suites, with a coverage of about 90%. In addition to those tests, three state of the arts test generators were used: EvoSuite, Randoop and JCrasher. Their average coverage was about 55%.

Results

2014-11-20 14.55.20

This is interesting, because branch and statement coverage was largely unchanged.

2014-11-20 14.56.46

Finally, is mutant detection correlated with real fault detection? Yes, mutant detection is correlated with real fault detection, and stronger than statement coverage has.

In summary: mutation testing works, better than code coverage, but there will always be a class of faults that cannot be found with mutants:

2014-11-20 15.00.43

Preprint: http://homes.cs.washington.edu/~rjust/publ/mutants_real_faults_fse_2014.pdf