Item-Score Reliability in Empirical-Data Sets and Its Relationship With Other Item Indices

Zijlmans, Eva A. O.; Tijmstra, Jesper; Ark, L. Andries van der; Sijtsma, Klaas

doi:10.1177/0013164417728358

Cited by 71 publications

(54 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In line with previous findings, scale length, number of response categories, and selection rates also had an effect on the outcome variables (e.g., Crişan et al, 2017 ; Zijlmans et al, 2018 ). The item scalability coefficient is equivalent to a normed item-rest correlation, which, in turn, is used as an index of item-score reliability (e.g., Zijlmans et al, 2018 ). Therefore, it is not surprising that overall scale reliability decreased as the item scalability coefficients decreased.…”

Section: Discussionsupporting

confidence: 90%

On the Practical Consequences of Misfit in Mokken Scaling

Crisan

Tendeiro

Meijer

2020

Applied Psychological Measurement

View full text Add to dashboard Cite

Mokken scale analysis is a popular method to evaluate the psychometric quality of clinical and personality questionnaires and their individual items. Although many empirical papers report on the extent to which sets of items form Mokken scales, there is less attention for the effect of violations of commonly used rules of thumb. In this study, the authors investigated the practical consequences of retaining or removing items with psychometric properties that do not comply with these rules of thumb. Using simulated data, they concluded that items with low scalability had some influence on the reliability of test scores, person ordering and selection, and criterion-related validity estimates. Removing the misfitting items from the scale had, in general, a small effect on the outcomes. Although important outcome variables were fairly robust against scale violations in some conditions, authors conclude that researchers should not rely exclusively on algorithms allowing automatic selection of items. In particular, content validity must be taken into account to build sensible psychometric instruments.

show abstract

Section: Discussionsupporting

confidence: 90%

On the Practical Consequences of Misfit in Mokken Scaling

Crisan

Tendeiro

Meijer

2020

Applied Psychological Measurement

View full text Add to dashboard Cite

show abstract

“…In line with previous findings, scale length, number of response categories, and selection rates also had an effect on the outcome variables (e.g., Crișan et al, 2017;Zijlmans, Tijmstra, van der Ark, & Sijtsma, 2018). The item scalability coefficient is equivalent to a normed item-rest correlation, which, in turn, is used as an index of item-score reliability (e.g., Zijlmans et al, 2018). Therefore, it is not surprising that overall scale reliability decreased as the item scalability coefficients decreased.…”

Section: Discussionsupporting

confidence: 85%

On the Practical Consequences of Misfit in Mokken Scaling

Crișan¹,

Tendeiro²,

Meijer³

2019

Preprint

View full text Add to dashboard Cite

Mokken scale analysis is a popular method to evaluate the psychometric quality of clinical and personality questionnaires and their individual items. Although many empirical papers report on the extent to which sets of items form Mokken scales, there is less attention for the effect of violations of commonly used rules of thumb. In this study we investigated the practical consequences of retaining or removing items with psychometric properties that do not comply with these rules-of-thumb. Using simulated data, we concluded that items with low scalability had some influence on the reliability of test scores, person ordering and selection, and criterion-related validity estimates. Removing the misfitting items from the scale had, in general, a small effect on the outcomes. Although important outcome variables were fairly robust against scale violations in some conditions, we conclude that researchers should not rely exclusively on algorithms allowing automatic selection of items. In particular, content validity must be taken into account in order to build sensible psychometric instruments.

show abstract

“…Item-rest correlation of 11 raters who provided masculinity ratings and 10 raters who provided femininity ratings was below the cut-off of r = .20 (Zijlmans et al, 2018); hence, their ratings were excluded from the analyses. Intraclass correlation coefficient analysis revealed good inter-rater agreement (r = .84, p < .001) and high internal consistency (Cronbach's a = .90) of the femininity ratings of the female composite faces.…”

Section: Resultsmentioning

confidence: 99%

“…Results AQ scores and masculinity/femininity ratings Item-rest correlation analyses were used to identify any raters whose ratings correlated poorly with the rest of the raters. All correlations were above the recommended cut-off of r = .20 (Zijlmans, Tijmstra, van der Ark, & Sijtsma, 2018), and none was excluded from subsequent analyses. Intraclass correlation coefficient analysis revealed excellent interrater agreement in the femininity ratings of the female faces (r = .97, p < .001) and high internal consistency (Cronbach's a = .98).…”

Section: Statistical Analysesmentioning

confidence: 99%

Sex‐specific variation in facial masculinity/femininity associated with autistic traits in the general population

Tan

Maybery

Ewing

et al. 2019

British J of Psychology

View full text Add to dashboard Cite

Reports linking prenatal testosterone exposure to autistic traits and to a masculinized face structure have motivated research investigating whether autism is associated with facial masculinization. This association has been reported with greater consistency for females than for males, in studies comparing groups with high and low levels of autistic traits. In the present study, we conducted two experiments to examine facial masculinity/femininity in 151 neurotypical adults selected for either low, mid‐range, or high levels of autistic traits. In the first experiment, their three‐dimensional facial photographs were subjectively rated by 41 raters for masculinity/femininity and were objectively analysed. In the second experiment, we generated 6‐face composite images, which were rated by another 36 raters. Across both experiments, findings were consistent for ratings of photographs and composite images. For females, a linear relationship was observed where femininity ratings decreased as a function of higher levels of autistic traits. For males, we found a U‐shaped function where males with mid‐range levels of traits were rated lowest on masculinity. Objective facial analyses revealed that higher levels of autistic traits were associated with less feminine facial structures in females and less masculine structures in males. These results suggest sex‐specific relationships between autistic traits and facial masculinity/femininity.

show abstract

Item-Score Reliability in Empirical-Data Sets and Its Relationship With Other Item Indices

Cited by 71 publications

References 57 publications

On the Practical Consequences of Misfit in Mokken Scaling

On the Practical Consequences of Misfit in Mokken Scaling

On the Practical Consequences of Misfit in Mokken Scaling

Sex‐specific variation in facial masculinity/femininity associated with autistic traits in the general population

Contact Info

Product

Resources

About