Fast and accurate incremental feedback for students’ software tests using selective mutation analysis

Abstract

As incorporating software testing into programming assignments becomes routine, educators have begun to assess not only the correctness of students' software, but also the adequacy of their tests. In practice, educators rely on code coverage measures, though its shortcomings are widely known. Mutation analysis is a stronger measure of test adequacy, but it is too costly to be applied beyond the small programs developed in introductory programming courses. We demonstrate how to adapt mutation analysis to provide rapid automated feedback on software tests for complex projects in large programming courses. We study a dataset of 1389 student software projects ranging from trivial to complex. We begin by showing that although the state-of-the-art in mutation analysis is practical for providing rapid feedback on projects in introductory courses, it is prohibitively expensive for the more complex projects in subsequent courses. To reduce this cost, we use a statistical procedure to select a subset of mutation operators that maintains accuracy while minimizing cost. We show that with only 2 operators, costs can be reduced by a factor of 2-3 with negligible loss in accuracy. Finally, we evaluate our approach on open-source software and report that our findings may generalize beyond our educational context.

Beijing Language and Culture University, China

“…By observing if test suites can detect these changes [17,27] one can identify weaknesses of their test suites, i.e., mutants that are not killed. Besides testing, mutation analysis is a tool that can be used to support multiple applications in the field of software engineering [24,25,41,48,63,67,79,81]. Most commonly, mutations are employed as substitutes for real-world bugs [6,13,38,52], to guide fault localization [50,60], test prioritization [68] and program repair [84,85].…”

Hybrid support vector machine and K‐nearest neighbor‐based software testing for educational assistant

“…The comparison results observed for the proposed methodology when compared to the AI‐Powered writing assistant, 11 STCAE, 12 SVM, 26 IATP, 14 Two‐stage hybrid classifier, 24 PBDA, 13 ANN, 28 and two‐stage hybrid classifier are presented in Table 10. The experiments are conducted in the testing dataset.…”

“…However, this methodology suffers from the zero prediction problem and independent predictor value assumption issues that affect the predictability of the model. Kazerouni A.M et al 13 utilized mutation analysis to provide rapid feedback on software tests conducted for intricate projects in a large programming course. Since mutation analysis is an expensive process, to minimize the cost, the authors used a statistical technique to select a subset of mutation operations that increases the accuracy and minimizes the cost.…”

“…Kazerouni et al 13 presented a teaching resource‐sharing platform based on big data for sharing a large number of teaching resources. In this paper, they have used a pulse‐based distance accumulation (PBDA) algorithm, which is an auxiliary probability‐based data allocation algorithm.…”

Optimization of Mutation Testing Challenges to Fixing Faults

“…The cost of creating the mutants and repairing the program is expensive. Researchers have proposed many techniques to reduce their costs [12], such as weak mutation testing [13], selective mutation testing, high mutation testing, mutant relationship redundancy [14],…”