Autonomously Generating Hints by Inferring Problem Solving Policies

Piech, Chris; Sahami, Mehran; Huang, Jonathan; Guibas, Leonidas J.

doi:10.1145/2724660.2724668

Cited by 190 publications

(218 citation statements)

References 17 publications

Supporting

Mentioning

217

Contrasting

Order By: Relevance

“…The authors in [13,12] provide real-time adaptive hints to coding assignments in the context of computer programming MOOCs. Both approaches are "step-loop" [1] in that they provide adaptive hints regarding the learners' problem solving process.…”

Section: Related Workmentioning

confidence: 99%

Enabling Real-Time Adaptivity in MOOCs with a Personalized Next-Step Recommendation Framework

Pardos

Tang

Davis

et al. 2017

Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale

View full text Add to dashboard Cite

In this paper, we demonstrate a first-of-its-kind adaptive intervention in a MOOC utilizing real-time clickstream data and a novel machine learned model of behavior. We detail how we augmented the edX platform with the capabilities necessary to support this type of intervention which required both tracking learners' behaviors in real-time and dynamically adapting content based on each learner's individual clickstream history. Our chosen pilot intervention was in the category of adaptive pathways and courseware and took the form of a navigational suggestion appearing at the bottom of every non-forum content page in the course. We designed our pilot intervention to help students more efficiently navigate their way through a MOOC by predicting the next page they were likely to spend significant time on and allowing them to jump directly to that page. While interventions which attempt to optimize for learner achievement are candidates for this adaptive framework, behavior prediction has the benefit of not requiring causal assumptions to be made in its suggestions. We present a novel extension of a behavioral model that takes into account students' time spent on pages and forecasts the same. Several approaches to representing time using Recurrent Neural Networks are evaluated and compared to baselines without time, including a basic n-gram model. Finally, we discuss design considerations and handling of edge cases for real-time deployment, including considerations for training a machine learned model on a previous offering of a course for use in a subsequent offering where courseware may have changed. This work opens the door to broad experimentation with adaptivity and serves as a first example of delivering a data-driven personalized learning experience in a MOOC.

show abstract

Section: Related Workmentioning

confidence: 99%

Enabling Real-Time Adaptivity in MOOCs with a Personalized Next-Step Recommendation Framework

Pardos

Tang

Davis

et al. 2017

Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale

View full text Add to dashboard Cite

show abstract

“…Fortunately, the learning sciences provide a wealth of knowledge on pedagogical interventions that likely improve achievement and that can be scaled, tested and implemented online. Examples of interventions that have already proven to scale, as diverse as increasing the motivation to learn, optimizing the exerted effort in learning, and providing proper guidance in the learning process, include mindset interventions (Paunesku, Walton, Romero, Smith, Yeager, & Dweck, 2015), spaced practice (Xiong & Beck, 2014), and personalized feedback (Piech, Huang, Nguyen, Phulsuksombati, Sahami, & Guibas, 2015).…”

Section: Powerful Yet Involved Methods That Can Fulfill the Above Reqmentioning

confidence: 99%

The Role of A/B Tests in the Study of Large-Scale Online Learning

Savi¹,

Williams²,

Maris³

et al. 2017

Preprint

View full text Add to dashboard Cite

Although large-scale online learning increasingly succeeds in attracting learners worldwide, to date it fails to deliver on its promise. We first show the immense popularity of online learning and discuss its (unsatisfactory) effectiveness. We then discuss large-scale online randomized controlled experiments (A/B tests) as a powerful complimentary means to enable the desired leap forward. Although these experiments are widely and intensively used for web page optimization, and are slowly being adopted by the online learning community, their use, benefits, PREPRINT A/B TESTS IN THE STUDY OF ONLINE LEARNING 2 and challenges have only limitedly seeped through to the larger learning community. We summarize existing efforts in employing A/B tests in online learning, argue that such tests should take into account the typical nature of (online) learning, and encourage the use of knowledge

show abstract

“…Methods based on or extended from knowledge tracing usually utilize a computational model of the effect of practice on KCs (i.e. knowledge components, which may include skills, concepts or facts) as the way to individually monitor and infer students' learning performance [29]. Bayesian Knowledge Tracing (BKT) [9] was the most popular approach to model students' learning process, where each learning concept was represented as a binary variable to indicate whether or not the student has mastered the learning concept or not.…”

Section: Student Performance Predictionmentioning

confidence: 99%

Why you should(n't) build your own game engine

Rivela¹

2019

ACM SIGGRAPH 2019 Talks

View full text Add to dashboard Cite

Modeling student learning and further predicting the performance is a well-established task in online learning and is crucial to personalized education by recommending different learning resources to different students based on their needs. Interactive online question pools (e.g., educational game platforms), an important component of online education, have become increasingly popular in recent years. However, most existing work on student performance prediction targets at online learning platforms with a well-structured curriculum, predefined question order and accurate knowledge tags provided by domain experts. It remains unclear how to conduct student performance prediction in interactive online question pools without such well-organized question orders or knowledge tags by experts. In this paper, we propose a novel approach to boost student performance prediction in interactive online question pools by further considering student interaction features and the similarity between questions. Specifically, we introduce new features (e.g., think time, first attempt, and first drag-and-drop) based on student mouse movement trajectories to delineate students' problem-solving details. In addition, heterogeneous information network is applied to integrating students' historical problem-solving information on similar questions, enhancing student performance predictions on a new question. We evaluate the proposed approach on the dataset from a real-world interactive question pool using four typical machine learning models. The result shows that our approach can achieve a much higher accuracy for student performance prediction in interactive online question pools than the traditional way of only using the statistical features (e.g., students' historical question scores) in various models. We further discuss the performance consistency of our approach across different prediction models and question classes, as well as the importance of the proposed interaction features in detail.

show abstract

Autonomously Generating Hints by Inferring Problem Solving Policies

Cited by 190 publications

References 17 publications

Enabling Real-Time Adaptivity in MOOCs with a Personalized Next-Step Recommendation Framework

Enabling Real-Time Adaptivity in MOOCs with a Personalized Next-Step Recommendation Framework

The Role of A/B Tests in the Study of Large-Scale Online Learning

Why you should(n't) build your own game engine

Contact Info

Product

Resources

About