Certainty-Based Marking on Multiple-Choice Items: Psychometrics Meets Decision Theory

Bančov

2022

IJERPH

Certainty-Based Marking (CBM) involves asking students not only the answer to an objective question, but also how certain they are that their answer is correct. In a mixed method design employing an embedded approach with a quasi-experimental design, we have examined the use of CBM during a 5-week Gynaecology and Obstetrics course. The study was conducted as a non-mandatory revision exam with two additional questionnaires on Moodle. Majority of students perceive CBM as fair (78%) and useful (94%). Most students would immediately want CBM to be used for revision exams, but more practice would be needed for CBM to be used in graded exams. The lowest self-evaluation of knowledge was mostly seen by worst (less than 70% Accuracy) and best achievers (more than 90% Accuracy); the worst achievers probably have knowledge gaps, and the best achievers probably correctly guessed at least one question. Our findings conclude that CBM does not discriminate any learner type (p = 0.932) and does not change the general distribution of the exam scores, since there is no significant differences between Certainty-Based Score (M = 80.4%, SD = 10.4%) and Accuracy (M = 79.8%, SD = 11.1%); t(176) = 0.8327, p = 0.4061. These findings are widely applicable, as learner type study models are used extensively in education. In the future, larger samples should be studied and the implementation of CBM on question types other than MCQ should be investigated.

Section: Reliability Of Cbm In Terms Of Learner Types and Students' S...mentioning

confidence: 82%

Section: Introductionmentioning

confidence: 99%

The Reliability and Medical Students’ Appreciation of Certainty-Based Marking

Bančov

2022

IJERPH

“…Velocity v 0 holds on the function of mass m, given by the equation v 0 (m) = 1−m, which k equals 0 (i.e., perfect player). Based on the interpretation of solving hardness m to the player's ability k, the subjective velocity of the player (or individual) with ability level k is denoted mathematically as (16).…”

Section: E Objectivity and Subjectivity Using Law Of Conservationmentioning

confidence: 99%

“…Moreover, there are different types of scoring methods in order to improve their psychometric properties and increase both reliability and validity as cited in [14], [15], and [16]. Tests should also be viewed as circumstances in which people generate various uncertain responses.…”

mentioning

confidence: 99%

Objectivity and Subjectivity in Variation of Multiple Choice Questions: Linking the Theoretical Concepts Using Motion in Mind

2023

Multiple-choice questions (MCQs) have been considerably used for assessing the individual's performance in various contexts. The optimal number of options in MCQs is a debatable issue, followed by contradictions and discussions, and is needed for a firm conclusion from empirical and hypothetical findings. This study aims to link theoretical concepts, including challenge-based gamification, zone of proximal development, and prospect theory, and generate insight into educational assessment using motionin-mind measures. Classical test theory was used to determine reliability and validity. Variations of MCQs experimented: the number of options, settings, and scoring methods. The experimental data was gathered from human and AI simulations and measured using motion-in-mind. It was found that increasing the number of options in the MCQ makes the test more challenging, explaining an increase of mass in mind m. The findings also revealed that time pressure provides competitiveness while scaffolding provides support. In addition, the hybrid system demonstrates the balance of education and entertainment. Finally, the results addressed the general discussion and analogical interpretation in the education context based on physicsin-mind values. These findings can be promising for analyzing the balance between competitiveness and entertainment while enabling the learning process in the practical assessment. INDEX TERMSEducational assessment, Motion-in-mind, Multiple-choice questions, Challenge-based gamification, Number of options, Zone of proximal development, Prospect theory I. INTRODUCTION

J Educational Measurement

“…As an example of how CRIs have been used, Caleon and Subramaniam (2010a) assessed students' misconceptions when answering science test items by using CRIs measured on a six‐point rating scale with 0 indicating a complete guess and 5 indicating absolute confidence in the selected answer. CRIs can also be reported on a continuous scale (Wu et al., 2021) where students indicate their certainty in terms of probability (i.e., 0% for a complete guess and 100% for absolute confidence). The rationale behind considering the CRI in addition to the actual item response is that the combination of response accuracy and self‐reported confidence can reveal student knowledge statuses.…”

mentioning

confidence: 99%

A Factor Mixture Model for Item Responses and Certainty of Response Indices to Identify Student Knowledge Profiles

Chen

Andersson

Zhu

2022

The certainty of response index (CRI) measures respondents' confidence level when answering an item. In conjunction with the answers to the items, previous studies have used descriptive statistics and arbitrary thresholds to identify student knowledge profiles with the CRIs. Whereas this approach overlooked the measurement error of the observed item responses and indices, we address this by proposing a factor mixture model that integrates a latent class model to detect student subgroups and a measurement model to control for student ability and confidence level. Applying the model to 773 seventh graders' responses to an algebra test, where some items were related to new material that had not been taught in class, we found two subgroups: (1) students who had high confidence in answering items involving the new material; and (2) students who had low confidence in answering items involving the new material but higher general self‐confidence than the first group. We regressed the posterior probability of the group membership on gender, prior achievement, and preview behavior and found preview behavior a significant factor associated with the membership. Finally, we discussed the implications of the current study for teaching practices and future research.