Extending the Hint Factory for the assistance dilemma: A novel, data-driven HelpNeed Predictor for proactive problem-solving help

Abstract

Determining when and whether to provide personalized support is a well-known challenge called the assistance dilemma. A core problem in solving the assistance dilemma is the need to discover when students are unproductive so that the tutor can intervene. Such a task is particularly challenging for open-ended domains, even those that are well-structured with defined principles and goals. In this paper, we present a set of data-driven methods to classify, predict, and prevent unproductive problem-solving steps in the well-structured open-ended domain of logic. This approach leverages and extends the Hint Factory, a set of methods that leverages prior student solution attempts to build data-driven intelligent tutors. We present a HelpNeed classification, that uses prior student data to determine when students are likely to be unproductive and need help learning optimal problem-solving strategies. We present a controlled study to determine the impact of an Adaptive pedagogical policy that provides proactive hints at the start of each step based on the outcomes of our HelpNeed predictor: productive vs. unproductive. Our results show that the students in the Adaptive condition exhibited better training behaviors, with lower help avoidance, and higher help appropriateness (a higher chance of receiving help when it was likely to be needed), as measured using the HelpNeed classifier, when compared to the Control. Furthermore, the results show that the students who received Adaptive hints based on HelpNeed predictions during training significantly outperform their Control peers on the posttest, with the former producing shorter, more optimal solutions in less time. We conclude with suggestions on how these HelpNeed methods could be applied in other well-structured open-ended domains.

Does Knowing When Help Is Needed Improve Subgoal Hint Performance in an Intelligent Data-Driven Logic Tutor?

“…When and whether to provide support to students is a well-known challenge. The HelpNeed model discussed in section 1 (Maniktala and Barnes 2020) (Maniktala et al 2020b) aimed to predict when students need help learning efficient strategies. More details about this model are discussed in section 3.…”

“…The HelpNeed model showed promising results (Maniktala et al 2020b) by providing a way to determine when to provide proactive hints to students. The model has a HelpNeed observation method which uses a step's duration and efficiency to identify whether the student is possibly using suboptimal strategies in that step, where step efficiency is a measure of how much a student's most recent step contributes to an efficient (short) solution.…”

“…We measure the hint justification rate (HJR) defined as the number of hints given (on-demand or proactive) that are correctly justified divided by total number of hints given. We also investigate student help seeking behaviors such as possible help avoidance, possible help abuse, and possible help appropriateness using the HelpNeed observation and predictor as defined by the HelpNeed study (Maniktala et al 2020b). Note that the prefix possible is added to these terms because HelpNeed does not represent ground truth as classified by experts but rather a heuristic.…”

“…A recent study (Maniktala et al 2020b) presented the HelpNeed model which predicts when students need help. It has been shown to improve student performance by predicting when to provide assistance to students and providing proactive next-step hints 1 accordingly in a logic tutor.…”