Decision consistency and accuracy indices for the bifactor and testlet response theory models

LaFond, Lee James

doi:10.17077/etd.ytivc04x

Cited by 4 publications

(4 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Classification consistency and accuracy have also been discussed in the literature under the UIRT framework (Huynh, ; W. Lee, ; Schulz, Kolen, & Nicewander, ). Knupp () and LaFond () employed SS‐MIRT and BF‐MIRT, respectively, to examine classification indices. Kim and Lee () considered classification consistency and accuracy for mixed‐format exams under the three IRT frameworks.…”

Section: Psychometric Properties Of Composite Raw and Scale Scoresmentioning

confidence: 99%

IRT Approaches to Modeling Scores on Mixed‐Format Tests

Lee

Kim

Choi

et al. 2019

J Educational Measurement

View full text Add to dashboard Cite

This article considers psychometric properties of composite raw scores and transformed scale scores on mixed‐format tests that consist of a mixture of multiple‐choice and free‐response items. Test scores on several mixed‐format tests are evaluated with respect to conditional and overall standard errors of measurement, score reliability, and classification consistency and accuracy under three item response theory (IRT) frameworks: unidimensional IRT (UIRT), simple structure multidimensional IRT (SS‐MIRT), and bifactor multidimensional IRT (BF‐MIRT) models. Illustrative examples are presented using data from three mixed‐format exams with various levels of format effects. In general, the two MIRT models produced similar results, while the UIRT model resulted in consistently lower estimates of reliability and classification consistency/accuracy indices compared to the MIRT models.

show abstract

Section: Psychometric Properties Of Composite Raw and Scale Scoresmentioning

confidence: 99%

IRT Approaches to Modeling Scores on Mixed‐Format Tests

Lee

Kim

Choi

et al. 2019

J Educational Measurement

View full text Add to dashboard Cite

show abstract

“…According to Lee ( 2010 ), if the test is believed to be truly multidimensional, then a multidimensional IRT model could be used to estimate the CC indices under Lee method. Some studies have already considered the estimation of CC indices for multidimensional tests (e.g., Yao, 2013 ; LaFond, 2014 ; Wang et al, 2016 ) and BMIRT software (Yao, 2003 ) can be used to estimate the multidimensional CC indices. Future research is encouraged to further investigate the performance of the CC indices for multidimensional latent ability.…”

Section: Discussion and Limitationsmentioning

confidence: 99%

Using Procedure Based on Item Response Theory to Evaluate Classification Consistency Indices in the Practice of Large-Scale Assessment

Zhang

Chen

et al. 2017

Front. Psychol.

View full text Add to dashboard Cite

In spite of the growing interest in the methods of evaluating the classification consistency (CC) indices, only few researches are available in the field of applying these methods in the practice of large-scale educational assessment. In addition, only few studies considered the influence of practical factors, for example, the examinee ability distribution, the cut score location and the score scale, on the performance of CC indices. Using the newly developed Lee's procedure based on the item response theory (IRT), the main purpose of this study is to investigate the performance of CC indices when practical factors are taken into consideration. A simulation study and an empirical study were conducted under comprehensive conditions. Results suggested that with negatively skewed distribution, the CC indices were larger than with other distributions. Interactions occurred among ability distribution, cut score location, and score scale. Consequently, Lee's IRT procedure is reliable to be used in the field of large-scale educational assessment, and when reporting the indices, it should be treated with caution as testing conditions may vary a lot.

show abstract

“…Decision accuracy was operationalized years ago as a measure of the correctness of a series of decisions as judged by experts (Taylor, 1975). Many researchers have since operationalized decision accuracy similarly (LaFond, 2014). Measures of decision accuracy by degrees have also been developed (Friedman et al, 1999).…”

Section: Decision Accuracy Uncertainty and Advicementioning

confidence: 99%

Theory of advice as an information object targeted at an unmade decision

Flynn

2019

View full text Add to dashboard Cite

Purpose The purpose of this paper is to advance an understanding of the concept of advice and its relationship to documents, information and knowledge. Design/methodology/approach A conceptual analysis of a sample of 48 relevant advice studies and two books, directly informed by documentation and information theories, was conducted to find out how researchers have approached advice conceptually since 1940. Further gains in understanding advice came from analyzing its relationship with environmental uncertainty. Findings Researchers have studied advice in the context of human-human, machine-machine and information and communication technology-intermediated interactions. Advice has been conceptualized and categorized in many different ways. Over time, conceptualizations of advice have broadened and become more general. In this light, it is theorized that advice is as an information object targeted at an unmade decision. This conceptualization of advice permits situated and momentary advice documents. A newly developed content-based framework of advice leads to an advice typology with four content-based categories of best possible advice: correct answers, probabilities, possibilities, and acknowledgments of the unknown. Research limitations/implications The refined advice theory, content-based advice framework and related typology of advice contributed here are small steps toward improved clarity about the nature of advice. These findings are limited in their focus to advice theory and advice categorization. Practical implications Scholars, practitioners and information system developers may reconsider advice theory and make use of the content-based framework and related advice typology in their work. These contributions will help advice-givers and the developers of advice-giving information systems and advice networks to provide better advice. Originality/value This paper fills a need for a clear and straightforward overall conceptualization of advice that accounts for advice documents and is informed by how advice has been previously conceptualized in multiple scientific fields.

show abstract

Decision consistency and accuracy indices for the bifactor and testlet response theory models

Cited by 4 publications

References 41 publications

IRT Approaches to Modeling Scores on Mixed‐Format Tests

IRT Approaches to Modeling Scores on Mixed‐Format Tests

Using Procedure Based on Item Response Theory to Evaluate Classification Consistency Indices in the Practice of Large-Scale Assessment

Theory of advice as an information object targeted at an unmade decision

Contact Info

Product

Resources

About