A Q₃-Based Permutation Test for Assessing Local Independence

Abstract

Item response theory (IRT) is a powerful statistical methodology used in the analysis of psychological and educational assessments. IRT rests on three fundamental assumptions about the data, including local independence, which means that after accounting for the latent trait(s) being measured, the item responses are independent of one another. Traditionally, this assumption is assessed using Yen's statistic. However, does not have a known sampling distribution, and thus, it is typically used in a descriptive fashion, such that values larger than an arbitrary cut-value (e.g., 0.2) indicate the presence of local dependence. The current study introduces a formal test of the null hypothesis that for a given item pair is 0, based on permutation test methodology. A small simulation study carried out to assess the Type I error and power rates of the permutation test found that this new statistic maintains good Type I error control, while also yielding power for detecting local dependence at a rate higher than that associated with the use of the 0.2 cut-value.

Critical Values for Yen’s Q3: Identification of Local Dependence in the Rasch Model Using Residual Correlations

“…Within the parameter ranges that were tested, any residual correlation .0.2 above the average correlation would appear to indicate LD, and any residual correlation of independent items at a value .0.3 above the average would seem unlikely. Finch and Jeffers (2016) proposed a permutation test for LD based on the Q 3 and found it to have good Type I error control, while also yielding more power for detecting LD than the use of the 0.2 cut-value. Bootstrapping and determining critical values for the Q 3 is one option, but using one of the statistics with known null distribution listed in the introduction is a better option.…”

Automatic Generation of Number Series Reasoning Items of High Difficulty

“…Items generated from the same item model were used as testlets. Average Q 3 -values within and between testlets were 0.128 and 0.028, respectively; as these were both smaller than the critical value of 0.2 (Chen and Thissen, 1997; Makransky et al, 2014; Finch and Jeffers, 2016), we concluded that there was negligible dependence between items from the same item models and between items from different item models. Therefore, we were convinced that the assumption of the local independence of items was not violated.…”

A computerized adaptive testing advancing the measurement of subjective well-being

“…A Q 3 statistic was proposed by Yen (1984) to detect local independence of IRT. If the Q 3 value of an item was larger than an arbitrary cut value, it meant the item had local dependence (Finch & Jeffers, 2016). As Flens et al.…”

“…A Q 3 statistic was proposed by Yen (1984) to detect local independence of IRT. If the Q 3 value of an item was larger than an arbitrary cut value, it meant the item had local dependence (Finch & Jeffers, 2016). As Flens et al (2016) pointed out that the value of Q 3 above 0.36 represents a moderate deviation or dependence, therefore in this article, items with a Q 3 value larger than 0.36 were removed from the item bank to ensure all remaining items met the IRT assumption of local independence.…”