Using Spectrum-Based Fault Location and Heatmaps to Express Debugging Suggestions to Student Programmers

Edmison, Bob; Edwards, Stephen H.; Pérez-Quiñones, Manuel A.

doi:10.1145/3013499.3013509

Cited by 6 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Automated testing is commonly applied in the context of programming education for tasks such as assessing student programs to provide feedback after a task has been completed, or during its creation (Shute, 2008). In many text-based programming environments, automated tests have been shown to enable various types of feedback, such as by displaying failed test cases (Edwards and Murali, 2017), suggesting likely misconceptions (Gusukuma et al, 2018), and highlighting erroneous code (Edmison et al, 2017). Offering such immediate, automated feedback has been shown to improve students performance and learning outcomes (Corbett and Anderson, 2001).…”

Section: Automated Testing and Analysis For Scratch Programsmentioning

confidence: 99%

Automated Test Generation for Scratch Programs

Deiner¹,

Patric²,

Schweikl³

et al. 2022

Preprint

View full text Add to dashboard Cite

The importance of programming education has lead to dedicated educational programming environments, where users visually arrange block-based programming constructs that typically control graphical, interactive game-like programs. The SCRATCH programming environment is particularly popular, with more than 70 million registered users at the time of this writing. While the block-based nature of SCRATCH helps learners by preventing syntactical mistakes, there nevertheless remains a need to provide feedback and support in order to implement desired functionality. To support individual learning and classroom settings, this feedback and support should ideally be provided in an automated fashion, which requires tests to enable dynamic program analysis. In prior work we introduced WHISKER, a framework that enables automated testing of SCRATCH programs. However, creating these automated tests for SCRATCH programs is challenging. In this paper, we therefore investigate how to automatically generate WHISKER tests. Generating tests for SCRATCH raises important challenges: First, game-like programs are typically randomised, leading to flaky tests. Second, SCRATCH programs usually consist of animations and interactions with long delays, inhibiting the application of classical test generation approaches. Thus, the new application domain raises the question which test generation technique is best suited to produce high coverage tests. We investigate these questions using an extension of the WHISKER test framework for automated test generation. Evaluation on common programming exercises, a random sample of 1000 SCRATCH user programs, and the 1000 most popular SCRATCH programs demonstrates that our approach enables WHISKER to reliably accelerate test executions, and even though many SCRATCH programs are small and easy to cover, there are many unique challenges for which advanced search-based test generation using many-objective algorithms is needed in order to achieve high coverage.Keywords Search-based testing • Block-based programming • Scratch.

show abstract

Section: Automated Testing and Analysis For Scratch Programsmentioning

confidence: 99%

Automated Test Generation for Scratch Programs

Deiner¹,

Patric²,

Schweikl³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Unlike instructor feedback, automated feedback can be easily propagated to all students having the same mistake, allowing a larger group of students to benefit from receiving feedback. This feedback can take the form of success or failure of test cases [8,14,30], highlighting erroneous code [7,22], or identifying likely misconceptions [11]. Providing such immediate feedback has been shown to engage and motivate students [17], improve their learning outcomes [4], and does not pose social threat to students [23].…”

Section: Related Workmentioning

confidence: 99%

SnapCheck: Automated Testing for Snap ! Programs

Wang

Zhang

Stahlbauer

et al. 2021

Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education v. 1

View full text Add to dashboard Cite

Programming environments such as Snap!, Scratch, and Processing engage learners by allowing them to create programming artifacts such as apps and games, with visual and interactive output. Learning programming with such a media-focused context has been shown to increase retention and success rate. However, assessing these visual, interactive projects requires time and laborious manual effort, and it is therefore difficult to offer automated or real-time feedback to students as they work. In this paper, we introduce SnapCheck, a dynamic testing framework for Snap! that enables instructors to author test cases with Condition-Action templates. The goal of SnapCheck is to allow instructors or researchers to author property-based test cases that can automatically assess students' interactive programs with high accuracy. Our evaluation of SnapCheck on 162 code snapshots from a Pong game assignment in an introductory programming course shows that our automated testing framework achieves at least 98 % accuracy over all rubric items, showing potentials to use SnapCheck for auto-grading and providing formative feedback to students.

show abstract

“…As such it is more geared to expert programmers since evidence in the literature suggests that most student programmers do not yet have debugging tactics to employ and struggle to form hypotheses [68]. When they do develop a hypothesis, they tend to narrowly focus on that single hypothesis rather than exploring different possibilities [17]. This limits their exploration of the problem space and makes the process of diagnosis much more difficult than correcting the fault [21].…”

Section: Debugging Processesmentioning

confidence: 99%

Analysis of a Process for Introductory Debugging

Whalley

Settle

Luxton-Reilly

2021

Proceedings of the 23rd Australasian Computing Education Conference

View full text Add to dashboard Cite

Debugging code is a complex task that requires knowledge about the mechanics of a programming language, the purpose of a given program, and an understanding of how the program achieves the purpose intended. It is generally accepted that prior experience with similar bugs improves the debugging process and that a systematic process is needed to be able to successfully move from the symptoms of a bug to the cause. Students who are learning to program may struggle with one or more aspect of debugging, and anecdotally, spend a lot of their time debugging faulty code.In this paper we analyse student answers to questions designed to focus student attention on the symptoms of a bug and to use those symptoms to generate a hypothesis about the cause of a bug. To ensure students focus on the symptoms rather than the code, we use paper-based exercises that ask students to reflect on various bugs and to hypothesize about the cause. We analyse the students' responses to the questions and find that using our structured process most students are able to generalize from a single failing test case to the likely problem in the code, but they are much less able to identify the appropriate location or an actual fix. CCS CONCEPTS• Social and professional topics → Computing education.

show abstract

Using Spectrum-Based Fault Location and Heatmaps to Express Debugging Suggestions to Student Programmers

Cited by 6 publications

References 22 publications

Automated Test Generation for Scratch Programs

Automated Test Generation for Scratch Programs

SnapCheck: Automated Testing for Snap ! Programs

Analysis of a Process for Introductory Debugging

Contact Info

Product

Resources

About