Effects of Score Feedback on Test-Taker Behavior in Self-Adapted Testing

Arieli‐Attali, Meirav; Budescu, David V.

doi:10.1080/00273171.2015.1120178

Cited by 3 publications

(17 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper we conduct a secondary analysis of the data from Arieli-Attali (2016). The original study evaluated how the goal orientation conditions affected test takers' item difficulty choices, as well as the influence of different feedback conditions that will not be considered here.…”

Section: The Current Studymentioning

confidence: 99%

“…In this study we tap into motivational and metacognitive aspects of task performance via modeling process data. We are analyzing data from a previous study (Arieli-Attali, 2016) that applied the goal-orientation manipulation in a self-adapted test, while collecting also confidence ratings. Self-adapted testing is designed to allow test takers to choose the level of the difficulty of the items they receive.…”

Section: Introductionmentioning

confidence: 99%

“…Self-adapted testing is designed to allow test takers to choose the level of the difficulty of the items they receive. In her study, Arieli-Attali (2016) instructed participants in one condition to perform at their best on the test, with incentive of a reward; participants in the second condition were instructed to use the self-adapted test as a learning tool for a test the following day. Main findings showed that participants in the learning goal condition chose overall more difficult items (about half a level on average out of seven possible levels) compared to the performance goal condition, after controlling for pre-test performance, manifested both in the start of the test (the first choice) and the mean choices across all items.…”

Section: Introductionmentioning

confidence: 99%

“…Self-adapted tests are designed to allow test takers to choose the level of difficulty of the items they receive (Rocklin and O'Donnell, 1987; Wise et al, 1992; Hontangas et al, 2004; Arieli-Attali, 2016). Such tests provide both product data—which items were answered correctly—as well as process data—what difficulty levels were chosen across time.…”

Section: Introductionmentioning

confidence: 99%

“…Since process data can be complicated due to interdependencies in time, our traditional psychometric models may not necessarily fit, and we need to look for additional ways to analyze such data. In this study, we draw process data from a study on self-adapted test under different goal conditions (Arieli-Attali, 2016) and use hidden Markov models to learn about test takers' choice making behavior. Self-adapted test is designed to allow test takers to choose the level of difficulty of the items they receive.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Understanding Test Takers' Choices in a Self-Adapted Test: A Hidden Markov Modeling of Process Data

Arieli‐Attali

Simmering

2019

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

With the rise of more interactive assessments, such as simulation- and game-based assessment, process data are available to learn about students' cognitive processes as well as motivational aspects. Since process data can be complicated due to interdependencies in time, our traditional psychometric models may not necessarily fit, and we need to look for additional ways to analyze such data. In this study, we draw process data from a study on self-adapted test under different goal conditions (Arieli-Attali, 2016) and use hidden Markov models to learn about test takers' choice making behavior. Self-adapted test is designed to allow test takers to choose the level of difficulty of the items they receive. The data includes test results from two conditions of goal orientation (performance goal and learning goal), as well as confidence ratings on each question. We show that using HMM we can learn about transition probabilities from one state to another as dependent on the goal orientation, the accumulated score and accumulated confidence, and the interactions therein. The implications of such insights are discussed.

show abstract

Section: The Current Studymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Understanding Test Takers' Choices in a Self-Adapted Test: A Hidden Markov Modeling of Process Data

Arieli‐Attali

Simmering

2019

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

show abstract

An item response approach to calibration of confidence judgments.

2020

Self Cite

View full text Add to dashboard Cite

Two consistent findings from the study of the fit between judgment of performance and actual performance are general overconfidence and the hard-easy effect, with overconfidence being higher with more difficult stimuli. These findings are based on aggregated analyses of confidence and accuracy, despite the fact that confidence judgments are individual and are provided at the item level. Furthermore, an important characteristic of item performance judgments that is ignored by traditional analyses is that the objective difficulty of any item can be estimated before it is administered to a person. We argue that traditional analyses confound possible bias in subjective estimates of the difficulty of items (i.e., confidence judgments) with variations in objective difficulty of items. We propose a multilevel approach to the analysis of confidence judgments, whereby the probability of a correct response is modeled as a function of both objective difficulty and subjectively judged difficulty. In this model, the intercept represents the possible overall bias (over-or underconfidence) in subjective difficulty judgments, after controlling for objective difficulty as well as variations across persons and items. In effect we are proposing a new, more nuanced, standard for defining calibration and identifying distinct patterns of miscalibration. We demonstrate the confounding effects of conventional aggregated analysis through synthetic examples and apply the proposed approach to the analysis of empirical data. Conventional analyses replicated the overall overconfidence and the hard-easy effect, but the item response modeling results failed to identify an overall bias in confidence judgments or a test difficulty effect.

show abstract

Design and implementation of an AI‐enabled visual report tool as formative assessment to promote learning achievement and self‐regulated learning: An experimental study

Liao,

Zhang,

Wang

et al. 2024

Brit J Educational Tech

View full text Add to dashboard Cite

Formative assessment is essential for improving teaching and learning, and AI and visualization techniques provide great potential for its design and delivery. Using NLP, cognitive diagnostic and visualization techniques designed to analyse and present students' monthly exam data, we developed an AI‐enabled visual report tool comprising six modules and conducted an empirical study of its effectiveness in a high school biology classroom. A total of 125 students in a ninth‐grade biology course were assigned to a treatment group (n = 63) receiving AI‐enabled visual reports as the intervention and a control group (n = 62) receiving overall oral feedback from the teacher. We present the main statistical results of the within‐subjects design and the between‐subjects design respectively, to better capture the main findings. Repeated measures ANOVA revealed a significant interaction effect of intervention and time on learning achievement, and the paired‐sample Wilcoxon test indicated that the treatment group had experienced increasing learning anxiety (Cohen's d = 0.203, p = 0.046) and self‐efficacy (Cohen's d = 1.793, p = 0.000) over time. Moreover, we conducted a series of non‐parametric tests to compare the effects of AI‐enabled visual reports and teacher feedback, but found no significant differences except for an increased self‐efficacy (Cohen's d = 0.312, p = 0.046). Additionally, we had the students in the treatment group rate their favourable modules in the AI‐enabled visual report and provide evaluative feedback. The study results provide important insights into the design and implementation of effective formative assessment supported by artificial AI and visualization techniques. Practitioner notesWhat is already known about this topic Formative assessment is essential for improving teaching and learning. Traditional formative assessment tools lack accurate data‐oriented assessment and usability. AI and visualization techniques have great potential for formative assessment. What this paper adds This study designs and implements an AI‐enabled visual report tool that generates data‐driven, user‐friendly reports. The AI‐enabled visual report can not only enhance students' learning achievement and self‐regulated learning over time but also increase their test anxiety. The AI‐enabled visual report has a comparable effect with teacher feedback but leads to increased self‐efficacy. Implications for practice and/or policy We recommend using the AI‐enabled visual report in large‐size classes for its overall positive effects on both learning achievement and self‐regulated learning. We recommend using the AI‐enabled visual report over teacher feedback for its capacity to enhance students' self‐efficacy. We recommend prioritizing the modules of Performance Ranking, Personal Mastery and Knowledge Alert when designing the AI‐enabled visual report.

show abstract

Effects of Score Feedback on Test-Taker Behavior in Self-Adapted Testing

Cited by 3 publications

References 2 publications

Understanding Test Takers' Choices in a Self-Adapted Test: A Hidden Markov Modeling of Process Data

Understanding Test Takers' Choices in a Self-Adapted Test: A Hidden Markov Modeling of Process Data

An item response approach to calibration of confidence judgments.

Design and implementation of an AI‐enabled visual report tool as formative assessment to promote learning achievement and self‐regulated learning: An experimental study

Contact Info

Product

Resources

About