Running students' software tests against each others' code

Proceedings of the 2014 Conference on Innovation &Amp; Technology in Computer Science Education - ITiCSE '14

2014

Self Cite

While many educators have added software testing practices to their programming assignments, assessing the effectiveness of student-written tests using statement coverage or branch coverage has limitations. While researchers have begun investigating alternative approaches to assessing student-written tests, this paper reports on an investigation of the quality of student written tests in terms of the number of authentic, human-written defects those tests can detect. An experiment was conducted using 101 programs written for a CS2 data structures assignment where students implemented a queue two ways, using both an arraybased and a link-based representation. Students were required to write their own software tests and graded in part on the branch coverage they achieved. Using techniques from prior work, we were able to approximate the number of bugs present in the collection of student solutions, and identify which of these were detected by each student-written test suite. The results indicate that, while students achieved an average branch coverage of 95.4% on their own solutions, their test suites were only able to detect an average of 13.6% of the faults present in the entire program population. Further, there was a high degree of similarity among 90% of the student test suites. Analysis of the suites suggest that students were following naïve, "happy path" testing, writing basic test cases covering mainstream expected behavior rather than writing tests designed to detect hidden bugs. These results suggest that educators should strive to reinforce test design techniques intended to find bugs, rather than simply confirming that features work as expected.

Section: Master Suite Tests Duplicated By Studentscontrasting

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Do student programmers all tend to write the same software tests?

Edwards

Proceedings of the 2014 Conference on Innovation &Amp; Technology in Computer Science Education - ITiCSE '14

2014

Self Cite

“…As a result, these systems cannot evaluate partial or incomplete submissions and gives no credit in those situations. Edwards et al first presented a solution for assessing partial or incomplete Java programs by applying late binding to test cases [11].…”

Section: Automated Grading Systems and Their Evaluation Measuresmentioning

confidence: 99%

Toward practical mutation analysis for evaluating the quality of student-written software tests

Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research

Edwards

2013

Self Cite

Software testing is being added to programming courses at many schools, but current assessment techniques for evaluating student-written tests are imperfect. Code coverage measures are typically used in practice, but they have limitations and sometimes overestimate the true quality of tests. Others have proposed using mutation analysis instead, but mutation analysis poses a number of practical obstacles to classroom use. This paper describes a new approach to mutation analysis of studentwritten tests that is more practical for educational use, especially in an automated grading context. This approach combines several techniques to produce a novel solution that addresses the shortcomings raised by more traditional mutation analysis. An evaluation of this approach in the context of both CS1 and CS2 courses illustrates how it differs from code coverage analysis. At the same time, however, the evaluation results also raise questions of concern for CS educators regarding the relative value of more comprehensive assessment of test quality, the value of more open-ended assignments that offer significant design freedom for students, the cost of providing higher-quality reference solutions in order to support better quality assessment, and the cost of supporting assignments that require more intensive testing, such as GUI assignments.

“…Results of applying our solutions in 8 assignments from CS1 and CS2 courses, encompassing 147 student programs, shows the feasibility of the approach and provides insight into the quality of students' tests. To our knowledge, we are the first to successfully apply all-pairs testing [2] in automated grading. Outcomes on applying mutation analysis are under review.…”

Section: Discussionmentioning

confidence: 99%

Automatically assessing the quality of student-written tests

Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research

2013

Self Cite

Software testing is frequently being added to programming courses at many schools, but current assessment techniques for evaluating student-written software tests are imperfect. Code coverage measures are typically used in practice, but that approach does not assess how much of the expected behavior is checked by the tests and sometimes overestimates the true quality of the tests. Running one student's tests against others' code (known as all-pairs testing) and mutation analysis are better indicators of test quality but both of them posed a number of practical obstacles to classroom use. This proposal describes technical obstacles behind using these two approaches in automated grading. We propose novel and practical solutions to apply all-pairs testing and mutation analysis of student-written tests, especially in an automated grading context. Experimental results of applying our techniques in eight CS1 and CS2 assignments submitted by 147 students show feasibility of our solution. Finally, we discuss our plan to combine the approaches to evaluate tests of assignments having large amounts of design freedom and explain their evaluation plan.