Evaluating Static Mutant Selection Techniques for Accurate Mutation Score Approximation

Abstract

Mutation analysis is known for its effectiveness in assessing the quality of test suites. However, it is a costly approach, as it generates many mutants even for small programs. Generating, compiling, and executing these mutants is a slow and resource-intensive process. Many mutant selection techniques have been proposed to reduce the number of mutants considered and thus lower the cost of mutation analysis. Yet, the effectiveness of all these techniques has not been systematically compared to understand the advantages of each technique and when they should be used. This work focuses on static mutant selection techniques (i.e., those that do not require executing tests against the mutants to select them) and compares their effectiveness in approximating the mutation score of a test suite. Using a dataset of 15 Java projects of different sizes and application domains and the LittleDarwin mutation tool, we compare the performance of ten state of the art static mutant selection techniques under different settings. Results show that no one technique provides better results than the others in all situations, i.e., across all projects and mutants sampling rates. Still, we found that stratification based selection techniques mostly outperform the other techniques (in up to 129 out of 135 of the studied settings). In particular, stratified sampling based on the source file in which the mutants appear provided the best approximation of the mutation score in nearly half the cases considered in our experiments (up to 68/135). Additionally, we found that the quality of a project’s test suite had a noticeable influence on the selection techniques’ performance. Indeed, for lower quality test suites, the selected mutants performed worse and strongly under-estimated the mutation score.