Mutation analysis vs. code coverage in automated assessment of students' testing skills

Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research

Edwards

2013

Software testing is being added to programming courses at many schools, but current assessment techniques for evaluating student-written tests are imperfect. Code coverage measures are typically used in practice, but they have limitations and sometimes overestimate the true quality of tests. Others have proposed using mutation analysis instead, but mutation analysis poses a number of practical obstacles to classroom use. This paper describes a new approach to mutation analysis of studentwritten tests that is more practical for educational use, especially in an automated grading context. This approach combines several techniques to produce a novel solution that addresses the shortcomings raised by more traditional mutation analysis. An evaluation of this approach in the context of both CS1 and CS2 courses illustrates how it differs from code coverage analysis. At the same time, however, the evaluation results also raise questions of concern for CS educators regarding the relative value of more comprehensive assessment of test quality, the value of more open-ended assignments that offer significant design freedom for students, the cost of providing higher-quality reference solutions in order to support better quality assessment, and the cost of supporting assignments that require more intensive testing, such as GUI assignments.

Section: Automated Grading Systems and Their Evaluation Measuresmentioning

confidence: 99%

Section: Mutation Analysis For Assessing Testsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Toward practical mutation analysis for evaluating the quality of student-written software tests

Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research

Edwards

2013

“…As tests written in compiled languages, such as Java, do not compile against a solution that differs in structure from the author's solution, no capability for all-pairs testing is available. Aaltonen et al [1] proposed using mutation analysis to evaluate adequacy of the tests but computational overhead makes it impractical to generate real-time feedback and use in assessment tools.…”

Section: Background and Related Workmentioning

confidence: 99%

Automatically assessing the quality of student-written tests

Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research

2013

Software testing is frequently being added to programming courses at many schools, but current assessment techniques for evaluating student-written software tests are imperfect. Code coverage measures are typically used in practice, but that approach does not assess how much of the expected behavior is checked by the tests and sometimes overestimates the true quality of the tests. Running one student's tests against others' code (known as all-pairs testing) and mutation analysis are better indicators of test quality but both of them posed a number of practical obstacles to classroom use. This proposal describes technical obstacles behind using these two approaches in automated grading. We propose novel and practical solutions to apply all-pairs testing and mutation analysis of student-written tests, especially in an automated grading context. Experimental results of applying our techniques in eight CS1 and CS2 assignments submitted by 147 students show feasibility of our solution. Finally, we discuss our plan to combine the approaches to evaluate tests of assignments having large amounts of design freedom and explain their evaluation plan.

“…The rationale behind code coverage is: the more code executed during testing the higher the chance of finding flaws in them. However, code coverage may falsely indicate test quality as it does not check if the executed code has been tested against expected behavior [1,9]. Moreover a students' solution may be incomplete or incorrect.…”

Section: Introductionmentioning

confidence: 99%

Automated assessment of students' testing skills for improving correctness of their code

Proceedings of the 2013 Companion Publication for Conference on Systems, Programming, &Amp; Applications: Software for Humanity

2013

Although software testing is included as a regular part of many programming courses, current assessment techniques used in automated grading tools for evaluating student-written software tests are imperfect. Code coverage measures are typically used in practice, but that approach does not assess how much of the expected behavior is checked by the tests and sometimes, overestimates the true quality of the tests. Two robust and thorough measures for evaluating student-written tests are running each students' tests against others' solutions(known as all-pairs testing) and injecting artificial bugs to determine if tests can detect them (also known as mutation analysis). Even though they are better indicators of test quality, both of them posed a number of practical obstacles to classroom use. This proposal describes technical obstacles behind using these two approaches in automated grading. We propose novel and practical solutions to apply all-pairs testing and mutation analysis of student-written tests, especially in the context of classroom grading tools. Experimental results of applying our techniques in eight CS1 and CS2 assignments submitted by 147 students show the feasibility of our solution. Finally, we discuss our plan to combine the approaches to evaluate tests of assignments having variable amounts of design freedom and explain their evaluation method.