Identifying low test-taking effort during low-stakes tests with the new Test-taking Effort Short Scale (TESS) – development and psychometrics

Abstract

BackgroundLow-stakes tests are becoming increasingly important in international assessments of educational progress, and the validity of these results is essential especially as these results are often used for benchmarking. Test scores in these tests not only mirror students’ ability but also depend on their test-taking effort. One way to obtain more valid scores from participating samples is to identify test-takers with low test-taking effort and to exclude them from further analyses. Self-assessment is a convenient and quick way of measuring test-taking effort. We present the newly developed Test-taking Effort Short Scale (TESS), which comprises three items measuring attainment value/intrinsic value, utility value, and perceived benefits, respectively.MethodsIn a multicenter validation study with N = 1837 medical students sitting a low-stakes progress test we analyzed item and test statistics including construct and external validity.ResultsTESS showed very good psychometric properties. We propose an approach using stanine norms to determine a cutoff value for identifying participants with low test-taking effort.ConclusionWith just three items, TESS is shorter than most established self-assessment scales; it is thus suited for administration after low-stakes progress testing. However, further studies are necessary to establish its suitability for routine usage in assessment outside progress testing.Electronic supplementary materialThe online version of this article (10.1186/s12909-018-1196-0) contains supplementary material, which is available to authorized users.

Self‐monitoring accuracy does not increase throughout undergraduate medical education

“…We imposed two outlier criteria on the remaining 2971 students to filter out participants with low test‐taking effort because high variations in test‐taking effort can impair the evaluation of formative assessments . First, we excluded 436 students (14.7%) who answered substantially fewer questions than their peers with medium or high levels of confidence (ie, for whom the number of questions answered [excluding guesses] was more than 1 standard deviation [SD] below the respective mean for students of that semester).…”

Measuring Digital Literacy during the COVID‐19 Pandemic: Experiences with Remote Assessment in Hong Kong

“…As a low‐stakes test, a potential threat to the DL assessment was low engagement, which can result in higher rates of rapid guessing or skips (Schüttpelz‐Brauns et al., 2018; Wise et al., 2009). Furthermore, during the testing period, schools gradually resumed onsite classes after a long period of school suspension.…”

Comparison of the evaluation of formative assessment at two medical faculties with different conditions of undergraduate training, assessment and feedback

“…On the one hand, those students who chose the “don't know” option for all questions or skipped all questions when filling out the test in winter term 2015/16 were identified as non-serious, since even in the first semester at least two questions can be answered. On the other hand, the test effort was determined by means of the Test-Effort Short Scale (TESS) [ 23 ]. TESS consists of three five-stage Likert items with the gradations 1 to 5, which ask for the performance value (“I would like to achieve the best possible result on the BPT”), the utility value (“I find the BPT useful”) and the perceived costs (“The BPT is a valuable part of my undergraduate training”).…”