Methodologies for Investigating Item- and Test-Level Measurement Equivalence in International Large-Scale Assessments

Oliveri, María Elena; Olson, Brent F.; Ercikan, Kadriye; Zumbo, Bruno D.

doi:10.1080/15305058.2011.617475

Cited by 25 publications

(22 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous research demonstrated considerable measurement incomparability between countries in international assessments (Ercikan, Roth, & Asil, in press;Kankaraš & Moores, 2013;Oliveri, Ercikan, & Zumbo, 2013;Oliveri et al, 2012). This incomparability existed even between countries administering tests in the same language (Ercikan & McCreith, 2002;Ercikan et al, in press;Roth et al, 2013) and between language groups within countries (Ercikan et al, 2014;Kankaraš & Moores, 2013;Oliveri et al, 2012).…”

Section: Differential Item Functioning Analysesmentioning

confidence: 89%

“…This incomparability existed even between countries administering tests in the same language (Ercikan & McCreith, 2002;Ercikan et al, in press;Roth et al, 2013) and between language groups within countries (Ercikan et al, 2014;Kankaraš & Moores, 2013;Oliveri et al, 2012). It is important to identify whether item scores are comparable across groups since, if item scores are not comparable, the creation of a single scale score intended to represent all groups is not appropriate.…”

Section: Differential Item Functioning Analysesmentioning

confidence: 93%

See 1 more Smart Citation

Reading Proficiency and Comparability of Mathematics and Science Scores for Students From English and Non-English Backgrounds: An International Perspective

Ercikan

Chen

Lyons-Thomas

et al. 2014

International Journal of Testing

Self Cite

View full text Add to dashboard Cite

The purpose of this research is to examine the comparability of mathematics and science scores for students from English language backgrounds (ELB) and nonEnglish language backgrounds (NELB). We examine the relationship between English reading proficiency and performance on mathematics and science assessments in Australia, Canada, the United Kingdom, and the United States. The findings indicate a strong relationship with reading proficiency accounting for up to 43% of the variance in mathematics and up to 79% in science. In all comparisons, ELB students either outperformed NELB students or performed at the same level. However, when statistical adjustments were made for reading proficiency, in both mathematics and science, the score gap between the groups became statistically non-significant in three out of the four countries. These findings point to differences in score meaning in mathematics and science assessments and limitations in comparing performances of ELB and NELB.Correspondence should be sent to Kadriye Ercikan, 2125 Main Mall, ECPS,

show abstract

Section: Differential Item Functioning Analysesmentioning

confidence: 89%

Section: Differential Item Functioning Analysesmentioning

confidence: 93%

Reading Proficiency and Comparability of Mathematics and Science Scores for Students From English and Non-English Backgrounds: An International Perspective

Ercikan

Chen

Lyons-Thomas

et al. 2014

International Journal of Testing

Self Cite

View full text Add to dashboard Cite

show abstract

“…Both techniques require large samples, but even larger samples are generally needed for IRT analyses. It was mentioned earlier that IRT methods can estimate both DIF at the item level and DTF at the level of cumulative responses over items (e.g., Oliveri et al, 2012). I am aware of no similar capability in MGCFA.…”

Section: Structural Equation Modelling Techniques For Analysing Test mentioning

confidence: 97%

“…There are also IRT-based methods that estimate the degree of differential test functioning (DTF), or whether total scores based on cumulative responses over a set of items relate to the underlying (latent) dimension in the same over different populations. Oliveri, Olson, Ercikan, and Zumbo (2012) described the application of parametric and nonparametric forms of IRT and the technique of ordinal logistic regression to simultaneously estimate DIF and DTF over samples of English-versus French-speaking students who completed an objective measure of problem solving. They found that differential functioning at the item level was not generally detected by analysis at the test level.…”

Section: Standard Statistical Techniques For Analysing Test Biasmentioning

confidence: 99%

Assessing statistical aspects of test fairness with structural equation modelling

Kline

2013

Educational Research and Evaluation

View full text Add to dashboard Cite

Test fairness and test bias are not synonymous concepts. Test bias refers to statistical evidence that the psychometrics or interpretation of test scores depend on group membership, such as gender or race, when such differences are not expected. A test that is grossly biased may be judged to be unfair, but test fairness concerns the broader, more subjective evaluation of assessment outcomes from perspectives of social justice. Thus, the determination of test fairness is not solely a matter of statistics, but statistical evidence is important when evaluating test fairness. This work introduces the use of the structural equation modelling technique of multiplegroup confirmatory factor analysis (MGCFA) to evaluate hypotheses of measurement invariance, or whether a set of observed variables measures the same factors with the same precision over different populations. An example of testing for measurement invariance with MGCFA in an actual, downloadable data set is also demonstrated.

show abstract

“…The psychometric property that typically must hold for scores to be comparable is known as measurement invariance (Meredith 1993), absence of differential item functioning (Hambleton et al 1991;Mellenbergh 1994;Swaminathan & Rogers 1990), or lack of bias (Lord 1980). Regardless of the term used, the literature on scale score equivalence in large-scale achievement tests has received considerable attention (e.g., Ercikan 2002;Hambleton 2002;Oliveri et al 2012). Many of these investigations have focused on pairwise comparisons of countries Oliveri 2012), the latter of which uses both empirical and simulated data.…”

Section: Introductionmentioning

confidence: 99%

Detecting differential item functioning using generalized logistic regression in the context of large-scale assessments

Svetina

Rutkowski

2014

Large-scale Assess Educ

View full text Add to dashboard Cite

Background: When studying student performance across different countries or cultures, an important aspect for comparisons is that of score comparability. In other words, it is imperative that the latent variable (i.e., construct of interest) is understood and measured equivalently across all participating groups or countries, if our inferences regarding performance can be regarded as valid. Relatively fewer studies examined an item-level approach to measurement equivalence, particularly in settings where a large number of groups is included. Methods: This simulation study examines item-level differential item functioning (DIF) in the context of international large-scale assessment (ILSA) using a generalized logistic regression approach. Manipulated factors included the number of groups (10 or 20), magnitude of DIF, percent of DIF items, the nature of DIF, as well as the percent of affected groups with DIF. Results: Results suggested that the number of groups did not have an effect of the performance of the method (high power and low Type I error rates); however, other factors had impacted the accuracy. Specifically, Type I error rates were inflated in non-DIF conditions, while they were very conservative in all of the DIF conditions. Power was generally high, in particular in conditions where DIF magnitude was large, with one exception -in conditions where DIF was introduced in difficulty parameters and the percent of DIF items was 60. Conclusions: Our findings presented a mixed picture with respect to the performance of the generalized logistic regression method in the context of large number of groups with large sample sizes. In the presence of DIF, the method was successful in distinguishing between DIF and non-DIF, as evidenced by low Type I error and high power rates. On the other hand, however, in the absence of DIF, the method yielded increased Type I errors.

show abstract

Methodologies for Investigating Item- and Test-Level Measurement Equivalence in International Large-Scale Assessments

Cited by 25 publications

References 24 publications

Reading Proficiency and Comparability of Mathematics and Science Scores for Students From English and Non-English Backgrounds: An International Perspective

Reading Proficiency and Comparability of Mathematics and Science Scores for Students From English and Non-English Backgrounds: An International Perspective

Assessing statistical aspects of test fairness with structural equation modelling

Detecting differential item functioning using generalized logistic regression in the context of large-scale assessments

Contact Info

Product

Resources

About