Jask

Abstract

We present Jask, a system capable of generating questions about a learner's code written in Java. Given Java code as input, Jask provides a set of meaningful questions formulated in terms of the actual code (using its constructs and identifiers) and the corresponding correct answers. We integrated Jask in a web-based system where students submit their code (e.g., from lab exercises), answer questions about it, and obtain immediate formative feedback with the correct answers. An initial study involving 123 distinct introductory programming students providing 2274 answers revealed that questions pertaining to program dynamics tend to register low scores, possibly evidencing fragile comprehension of programming constructs. Participants were surveyed, revealing a positive view towards the usefulness of Jask, especially with respect to consolidating terminology.

Automated Questions About Learners' Own Code Help to Detect Fragile Prerequisite Knowledge

“…In this study, we have confirmed the results of earlier studies [7,8,16]. We found similar levels of incorrect answers to simple QLCs on the second programming course as had previously been found for the first weeks of starting programming.…”

“…We hypothesized that QLCs could reveal weaknesses in prerequisites which again can lead to low grades or even drop-outs. We found that success rates for the different types of QLCs were comparable to those reported in previous studies [7,8,16] even though the students had passed a CS1 in Python. Furthermore, students answering incorrectly about their own program logic had a lower median for course points compared with those that answered correctly.…”

“…Santos et al [16] present a tool that automates QLCs for Java programs and report on a pilot evaluation on CS1. The evaluation asks "How do students perform in QLCs in their lab exercises?"…”

“…Students in the previous study [16] self-reported terminology as the topic they learned the most from answering QLCs. We assume students can reason about program execution without knowing the terminology that well.…”

“…We partially replicate previous studies on automatically generated Questions about Learners' Code (QLCs) [7,8,16] by adapting selected research questions to new context. The previous studies focused on students' program comprehension during the first weeks of their first programming course (CS1).…”