Automated test generation for Scratch programs

Abstract

The importance of programming education has led to dedicated educational programming environments, where users visually arrange block-based programming constructs that typically control graphical, interactive game-like programs. The Scratch programming environment is particularly popular, with more than 90 million registered users at the time of this writing. While the block-based nature of Scratch helps learners by preventing syntactical mistakes, there nevertheless remains a need to provide feedback and support in order to implement desired functionality. To support individual learning and classroom settings, this feedback and support should ideally be provided in an automated fashion, which requires tests to enable dynamic program analysis. In prior work we introduced Whisker, a framework that enables automated testing of Scratch programs. However, creating these automated tests for Scratch programs is challenging. In this paper, we therefore investigate how to automatically generate Whisker tests. Generating tests for Scratch raises important challenges: First, game-like programs are typically randomised, leading to flaky tests. Second, Scratch programs usually consist of animations and interactions with long delays, inhibiting the application of classical test generation approaches. Thus, the new application domain raises the question of which test generation technique is best suited to produce high coverage tests capable of detecting faulty behaviour. We investigate these questions using an extension of the Whisker test framework for automated test generation. Evaluation on common programming exercises, a random sample of 1000 Scratch user programs, and the 1000 most popular Scratch programs demonstrates that our approach enables Whisker to reliably accelerate test executions, and even though many Scratch programs are small and easy to cover, there are many unique challenges for which advanced search-based test generation using many-objective algorithms is needed in order to achieve high coverage.

Combining Neuroevolution with the Search for Novelty to Improve the Generation of Test Inputs for Games

“…Whisker maps Boolean predicates such as the one shown in the code snippet of Fig. 1 to branch distance measurements based on the underlying program state (e.g., the distance between the current sprite and the closest pixel with the chosen target colour) [10].…”

“…Our case study analyses whether the neuroevolution-based generation of test cases benefits from promoting novel behaviours. To this end, we extend Neatest, which is part of the open-source Whisker testing framework [10] with a novelty score integrated into the test generator as a secondary fitness criterion as explained in Section 3. The effect of adding novelty-rewarding mechanisms to the search is evaluated by comparing the achieved coverage of the default Neatest algorithm (Fitness) against our proposed approach (Fitness+Novelty).…”

“…Each of these networks serves as a test input generator optimised by an evolutionary search algorithm (i.e., neuroevolution) to reach a specific source code statement reliably, regardless of challenging or randomised program behaviour, by generating input actions for the given game dynamically based on the current program state. The search is guided by an objective function that measures the distance of candidate test executions to program statements the search has yet to reach [10].…”

“…Whenever the player touches the orange portal, they are placed into the next level by changing the scene's background image, resetting the player's location to the left and increasing a variable that keeps track of the current level. Since the if-condition serves as a guard for reaching these three statements, an objective function responsible for optimising test cases to reach one of those statements will compute the Euclidean distance between the player and the target location [10]. However, achieving this requires the player to temporarily increase the distance between the player and the portal in order to pass the wall depicted in grey.…”