Teaching Software Testing with Automated Feedback

Abstract

Computer science and software engineering courses commonly use automated grading systems to evaluate student programming assignments. These systems provide various types of feedback, such as whether student code passes instructor test cases. The literature contains little data on the association between feedback policies and student learning. This work analyzes the association between different types of feedback and student learning, specifically on the topic of software testing. Our study examines a second-semester computer programming course with a total of 1,556 students over two semesters. The course contained five programming projects where students wrote code according to a specification as well as test cases for their code. Students submitted their code and test cases to an automated grading system. These test cases were evaluated by running them against intentionally buggy instructor solutions. The first semester comprised the control group, while the second semester comprised the experiment group. The two groups received different kinds of feedback on their test cases. The control group was shown whether their tests were free of false positives. In addition to the same feedback as the control group, the experiment group was shown how many intentionally buggy instructor solutions their tests exposed. Our results measured the quality of student test cases for the control and experiment groups. After students in the experiment group completed two projects with additional feedback on their test cases, they completed a final project without the additional feedback. Despite not receiving additional feedback, their test cases were of higher quality, exposing on average 5% more buggy solutions than students from the control group. We found this difference to be statistically significant after controlling for GPA and whether students worked alone or with a partner.

Integrating DevOps to Enhance Student Experience in an Undergraduate Research Project

“…These student perspectives reinforce the findings of Perretta, et al, that automated feedback improves student outcomes compared to manual feedback [2] and Eddy, et al, who found that introducing students to more complex topics such as Continuous Integration also increased their comfort with more basic tools such as Git [9].…”

“…Generally, students appreciate when their coursework is going to be relevant preparation for the industry, especially when those courses use industry tooling to do so [1]. More specifically, students appreciate tooling that provides automated feedback rather than having to wait on a mentor or professor [2]. Such tooling is even more beneficial to students when they are involved in the creation of the test harness itself as it prepares them for the industry where developers are often responsible for writing their own tests [3].…”

Work in Progress: An Interview-Based Retrospective on the Redesign of an Introductory Computing Course

“…Souza et al shows that project-based learning in the context of software engineering education can be valuable [21], which is somewhat supported by our preliminary interview results. Perretta and DeOrio show that providing feedback on student-written testing (particularly the presence of false positives) can improve the quality of student tests [22], and we have several small exercises to do this in SoftDes. Effenberger and Pelánek describe a variety of code quality defects that are common in student code in introductory courses and suggest targeted feedback to help curb these practices [16].…”