Delaying access to a problem-skipping option increases effortful practice: Application of an A/B test in large-scale online learning

Savi, Alexander Olof; Ruijs, Nienke; Maris, Gunter; Maas, Han L. J. van der

doi:10.1016/j.compedu.2017.12.008

Cited by 14 publications

(19 citation statements)

References 14 publications

(10 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the ability estimates, obtained during 14 weeks of practice, no differences were found between treatment groups. This suggests that the previously established increase in effortful practice (Savi et al, 2018) has no effect on the average ability during a period of 14 weeks. Or in a strict sense, that the absence of an average effect cannot be rejected.…”

Section: Discussionmentioning

confidence: 85%

“…This study builds upon the online field experiment and data described in Savi et al (2018). In the experiment, a problem skipping option was delayed, aimed at increasing effortful practice.…”

Section: Methodsmentioning

confidence: 99%

“…In between both extremes, the distribution is skewed to right, with the great majority of the participants skipping less than a quarter of the problems. Savi et al (2018) showed that the intervention diminishes the use of the problem skipping option.…”

Section: Descriptive Statisticsmentioning

confidence: 99%

See 2 more Smart Citations

Combating Effort Avoidance in Computer Adaptive Practicing: Does a Problem Skipping Restriction Promote Learning?

Savi¹,

C²,

Cornelisz³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Effort is key in learning, evidenced by its omnipresence in both empirical findings and educational theories. At the same time, students are consistently found to avoid effort. In this study, we investigate whether limiting effort avoidance improves learning outcomes, and explore for whom this would be the case. In a large-scale computer adaptive practice system for primary education, over 150,000 participants were distributed across four conditions in which a problem-skipping option was delayed for 0, 3, 6, or 9 seconds. The results show that after a 14 week period, no average treatment effects in learning outcomes can be found between conditions. A substantive typology of students, based on the expected target mechanisms of the intervention, neither shows consistent conditional average treatment effects. Nevertheless, the substantive typology is shown to be meaningful, as the different types—toilers, skippers, and rushers—differ with respect to their learning outcomes. We argue that although the scale of the experiment suggests a precise null finding, the cumulative nature of the effect of problem skipping cautions against generalizing this finding to sustained intervention.

show abstract

Section: Discussionmentioning

confidence: 85%

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Combating Effort Avoidance in Computer Adaptive Practicing: Does a Problem Skipping Restriction Promote Learning?

Savi¹,

C²,

Cornelisz³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…one could use leave-one-out, split-half, holdout, or re-usable holdout cross-validation, to name but a few [20,27,28]). To avoid that cross-validation is tried repeatedly until favourable results are obtained—analogous to p -hacking in behavioural research [29]–one may opt to pre-register the cross-validation procedures; that is determining the procedure beforehand, making a public pre-registration of that procedure, and reporting the findings of that procedure [30].…”

Section: Discussionmentioning

confidence: 99%

Being accurate about accuracy in verbal deception detection

2019

View full text Add to dashboard Cite

Purpose Verbal credibility assessments examine language differences to tell truthful from deceptive statements (e.g., of allegations of child sexual abuse). The dominant approach in psycholegal deception research to date (used in 81% of recent studies that report on accuracy) to estimate the accuracy of a method is to find the optimal statistical separation between lies and truths in a single dataset. However, this method lacks safeguards against accuracy overestimation. Method & Results A simulation study and empirical data show that this procedure produces overoptimistic accuracy rates that, especially for small sample size studies typical of this field, yield misleading conclusions up to the point that a non-diagnostic tool can be shown to be a valid one. Cross-validation is an easy remedy to this problem. Conclusions We caution psycholegal researchers to be more accurate about accuracy and propose guidelines for calculating and reporting accuracy rates.

show abstract

“…The governing method was evaluated in an A/B test [25,26] running for two full weeks [27]. Students that were or had been actively practicing in the Math Garden were randomly assigned to the control variant A (default system) or treated with the governing method (variant B).…”

Section: Online Experimentsmentioning

confidence: 99%

Governing games Adaptive game selection in the Math Garden

2020

View full text Add to dashboard Cite

Utilizing online digital educational content has become the norm when teaching young students. A variety of adaptive educational practice sys- tems is readily available and allows students to practice various domains, on a preferred difficulty and pace. However, due to the intensification of the teaching profession and the possibilities of practicing from home, students might be left unsupervised, and as a result do not practice domains that are most important. This study proposes a solution to govern these students, i.e., provide comput- erized data driven supervision that guides students in practicing domains most important with no intervention of a teacher. Through an experiment involving 13 578 participants, a new governing method was tested and found to have positive effects on both engagement and learning, with almost no changes to the visual interface needed. Governing seems a promising technique in general, and was effectively tested and introduced in Math Garden.

show abstract

Delaying access to a problem-skipping option increases effortful practice: Application of an A/B test in large-scale online learning

Cited by 14 publications

References 14 publications

Combating Effort Avoidance in Computer Adaptive Practicing: Does a Problem Skipping Restriction Promote Learning?

Combating Effort Avoidance in Computer Adaptive Practicing: Does a Problem Skipping Restriction Promote Learning?

Being accurate about accuracy in verbal deception detection

Governing games Adaptive game selection in the Math Garden

Contact Info

Product

Resources

About