Diversity-aware mutation adequacy criterion for improving the fault detection capability of test suites

Abstract

Mutation testing is a fault-based software testing technique which provides an effective testing adequacy criterion in evaluating the fault detection capability of a test suite. Whereas mutation testing has been studied widely over three decades, the mutation adequacy criterion remains essentially the same as it was first proposed in the 1970s: it simply considers whether a mutant (i.e., an artificial fault) is killed (i.e., detected) or not by a given test suite. Such simplicity makes the inherent diversity of mutants wasted: different mutants can be regarded as redundant in terms of their kill information, and it becomes too easy to satisfy the mutation adequacy criterion. This thesis argues that the mutation adequacy criterion is able to be improved, using the same set of mutants, by additionally considering the diversity of mutants based on their kill patterns. This thesis proposes a theoretical framework of mutation testing to formally describe the essence of mutation testing including the mutation adequacy criterion. Based on the formal framework, the thesis proposes a novel mutation adequacy criterion called distinguishing mutation adequacy criterion, which is fully satisfied when each of the considered mutants are distinguished from the other by the set of tests that kill it, thereby encouraging inclusion of more diverse range of tests in the vast space of possible program behaviors. Theoretically, satisfying the distinguishing mutation adequacy criterion guarantees satisfying the traditional mutation adequacy criterion. Empirically, the test suites satisfying the distinguishing mutation adequacy criterion can detect 8.26 times more unseen real-world faults than the test suites satisfying the traditional mutation adequacy criterion on average, at the cost of 3.07 times increase in test suite size.