Using Relational Problems to Teach Property-Based Testing

Wrenn, John; Nelson, Tim; Krishnamurthi, Shriram

doi:10.22152/programming-journal.org/2021/5/9

Cited by 3 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We believe that this work puts subproperty decomposition on sound, scalable, and useful footing-at least for educational purposes. We hope that, if pj21 [40] stimulated interest in teaching pbt, this work will provide a solid footing, techniques, and tooling to ease translating that interest into curricular practice. To that end, we have made our scripts and a full Toposortacle example available in an online appendix.1…”

Section: Discussionmentioning

confidence: 99%

“…Unfortunately, at time of writing, pbt is not commonly covered in undergraduate computer-science education. This lamentable neglect was the topic of prior work [40] in this journal (by a subset of the present authors)-henceforth pj21-which argued that pbt could productively be taught to undergraduates in multiple settings. Teaching effectively requires giving accurate and focused feedback (and if that can be automated, so much the better).…”

Section: Introductionmentioning

confidence: 88%

“…Because of this uncertainty, testing implementations is a vital and pervasive challenge in programming. We have argued [40] in the past that relational problems pose a special challenge in testing and pose an obvious application area for pbt.…”

Section: Terminologymentioning

confidence: 99%

See 2 more Smart Citations

Automated, Targeted Testing of Property-Based Testing Predicates

et al. 2021

Self Cite

View full text Add to dashboard Cite

Context This work is based on property-based testing (pbt). pbt is an increasingly important form of software testing. Furthermore, it serves as a concrete gateway into the abstract area of formal methods. Specifically, we focus on students learning pbt methods. Inquiry How well do students do at pbt? Our goal is to assess the quality of the predicates they write as part of pbt. Prior work introduced the idea of decomposing the predicate's property into a conjunction of independent subproperties. Testing the predicate against each subproperty gives a "semantic" understanding of their performance. Approach The notion of independence of subproperties both seems intuitive and was an important condition in prior work. First, we show that this condition is overly restrictive and might hide valuable information: it both undercounts errors and makes it hard to capture misconceptions. Second, we introduce two forms of automation, one based on pbt tools and the other on sat-solving, to enable testing of student predicates. Third, we compare the output of these automated tools against manually-constructed tests. Fourth, we also measure the performance of those tools. Finally, we re-assess student performance reported in prior work. Knowledge We show the difficulty caused by the independent subproperty requirement. We provide insight into how to use automation effectively to assess pbt predicates. In particular, we discuss the steps we had to take to beat human performance. We also provide insight into how to make the automation work efficiently. Finally, we present a much richer account than prior work of how students did. Grounding Our methods are grounded in mathematical logic. We also make use of well-understood principles of test generation from more formal specifications. This combination ensures the soundness of our work. We use standard methods to measure performance. Importance As both educators and programmers, we believe pbt is a valuable tool for students to learn, and its importance will only grow as more developers appreciate its value. Effective teaching requires a clear understanding of student knowledge and progress. Our methods enable a rich and automated analysis of student performance on pbt that yields insight into their understanding and can capture misconceptions. We therefore expect these results to be valuable to educators. ACM CCS 2012Software and its engineering → Correctness; Software testing and debugging; Social and professional topics → CS1;

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 88%