Institutional strategies related to test-taking behavior in low stakes assessment

Schüttpelz-Brauns, Katrin; Hecht, Martin; Hardt, Katinka; Karay, Yassin; Zupanic, Michaela; Kämmer, Juliane E.

doi:10.1007/s10459-019-09928-y

Cited by 12 publications

(13 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We imposed two outlier criteria on the remaining 2971 students to filter out participants with low test‐taking effort because high variations in test‐taking effort can impair the evaluation of formative assessments . First, we excluded 436 students (14.7%) who answered substantially fewer questions than their peers with medium or high levels of confidence (ie, for whom the number of questions answered [excluding guesses] was more than 1 standard deviation [SD] below the respective mean for students of that semester).…”

Section: Resultsmentioning

confidence: 99%

Self‐monitoring accuracy does not increase throughout undergraduate medical education

2020

Self Cite

View full text Add to dashboard Cite

Context Accurate self‐assessment of one's performace on a moment‐by‐moment basis (ie, accurate self‐monitoring) is vital for the self‐regulation of practising physicians and indeed for the effective regulation of self‐directed learning during medical education. However, little is currently known about the functioning of self‐monitoring and its co‐development with medical knowledge across medical education. This study is the first to simultaneously investigate a number of relevant aspects and measures that have so far been studied separately: different measures of self‐monitoring for a broad area of medical knowledge across 10 different performance levels. Methods This study assessed the self‐monitoring accuracy of medical students (n = 3145) across 10 semesters. Data collected during the administration of the formative Berlin Progress Test Medicine (PTM) were analysed. The PTM comprises 200 multiple‐choice questions covering all major medical disciplines and organ systems. A self‐report indicator (ie, confidence) and two behavioural indicators of self‐monitoring accuracy (ie, response time and the likelihood of changing an initial answer to a correct rather than an incorrect item) were examined for their development over semesters. Results Analyses of more than 390 000 observations (of approximately 250 students per semester) showed that confidence was higher for correctly than for incorrectly answered items and that 86% of items answered with high confidence were indeed correct. Response time and the likelihood of the initial answer being changed were higher when the initial answer was incorrect than when it was correct. Contrary to expectations, no differences in self‐monitoring accuracy were observed across semesters. Conclusions Convergent evidence from different measures of self‐monitoring suggests that medical students self‐monitor their knowledge on a question‐by‐question basis well, although not perfectly, and to the same degree as has been found in studies outside medicine. Despite large differences in performance, no variations in self‐monitoring across semesters (with the exception of the first semester) were observed.

show abstract

Section: Resultsmentioning

confidence: 99%

Self‐monitoring accuracy does not increase throughout undergraduate medical education

2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…We used a pseudonymized dataset with the shared questions of PT36 and PT41, PT37 and PT42, and PT38 and PT43. ese datasets contained the answers of participants to each question as well as the semester to which they belong, the pseudonymized faculty where they study, and whether their participation in the test is considered "serious" or not; participants classified as nonserious are excluded from the calculation of comparison groups since the validity of results would otherwise be jeopardized [10].…”

Section: Participantsmentioning

confidence: 99%

Effects of COVID-19 Pandemic on Progress Test Performance in German-Speaking Countries

Sehy

Atanet

Sieg

et al. 2022

Education Research International

View full text Add to dashboard Cite

Background. The COVID-19 pandemic has been the source of many challenges for medical students worldwide. The authors examined short-term effects on the knowledge gain of medical students in German-speaking countries. Methods. The development of the knowledge gain of medical students during the pandemic was measured by comparing the outcomes of shared questions within Berlin Progress Test (PT) pairs. The PT is a formative test of 200 multiple choice questions at the graduate level, which provides feedback to students on knowledge and knowledge gain during their course of study. It is provided to about 11,000 students in Germany and Austria around the beginning of each semester. We analyzed three successive test pairs: PT36-PT41 (both conducted before the pandemic), PT37-PT42 (PT37 took place before the pandemic; PT42 was conducted from April 2020 onwards), and PT38-PT43 (PT38 was administered before the pandemic; PT43 started in November 2020). The authors used mixed-effect regression models and compared the absolute variations in the percentage of correct answers per subject. Results. The most recent test of each PT pair showed a higher mean score compared to the previous test in the same pair (PT36-PT41 : 2.53 (95% CI: 1.31–3.75), PT37-PT42 : 3.72 (2.57–4.88), and PT38-PT43 : 5.66 (4.63–6.69)). Analogously, an increase in the share of correct answers was observed for most medical disciplines, with Epidemiology showing the most remarkable upsurge. Conclusions. Overall, PT performance improved during the pandemic, which we take as an indication that the sudden shift to online learning did not have a negative effect on the knowledge gain of students. We consider that these results may be helpful in advancing innovative approaches to medical education.

show abstract

“…and PT42, and PT38 and PT43. These datasets contained the answers of participants to each question as well as the semester to which they belong, the pseudonymized faculty where they study and whether their participation in the test is considered as 'serious' or not; participants classified as non-serious are excluded from the calculation of comparison groups since the validity of results would otherwise be jeopardized 8 .…”

Section: Participantsmentioning

confidence: 99%

Effects of COVID-19 Pandemic on Progress Test performance in German speaking countries. 

Berlin

Sehy

Atanet

et al. 2022

Preprint

View full text Add to dashboard Cite

Background The COVID-19 pandemic has been the source of many challenges for medical students worldwide. The authors examined short-term effects on the knowledge gain of students in German speaking countries. Method The development of the knowledge gain of medical students during the pandemic was measured comparing the outcomes of shared questions within Berlin Progress Test (PT) pairs. The PT is a formative test of 200 multiple choice questions at graduate level, which provides feedback to students on knowledge and knowledge gain during their course of study. It is provided to about 11,000 students in Germany and Austria around the beginning of each semester. We analysed three successive test pairs: PT36-PT41 (both conducted before the pandemic), PT37-PT42 (PT37 took place before the pandemic; PT42 was conducted from April 2020 onwards) and PT38-PT43 (PT38 was administered before the pandemic; PT43 started in November 2020). The authors used mixed-effect regression models and compared the absolute variations in the percentage of correct answers per subject. Results The most recent test of each PT-pair showed a higher mean score compared to the previous test in the same pair (PT36-PT41: 2.53 (95% CI: 1.31-3.75), PT37-PT42: 3.72 (2.57-4.88), PT38-PT43: 5.66 (4.63-6.69)). Analogously, an increase in the share of correct answers was observed for most medical disciplines, with Epidemiology showing the most remarkable upsurge. Conclusion: Overall, PT performance improved during the pandemic, which we take as an indication that the sudden shift to online learning did not have a negative effect on the knowledge gain of students. We consider that these results may be helpful in advancing innovative approaches to medical education

show abstract

Institutional strategies related to test-taking behavior in low stakes assessment

Cited by 12 publications

References 43 publications

Self‐monitoring accuracy does not increase throughout undergraduate medical education

Self‐monitoring accuracy does not increase throughout undergraduate medical education

Effects of COVID-19 Pandemic on Progress Test Performance in German-Speaking Countries

Effects of COVID-19 Pandemic on Progress Test performance in German speaking countries.

Contact Info

Product

Resources

About