Examining testlet effects in the TestDaF listening section: A testlet response theory modeling approach

Abstract

Testlets are subsets of test items that are based on the same stimulus and are administered together. Tests that contain testlets are in widespread use in language testing, but they also share a fundamental problem: Items within a testlet are locally dependent with possibly adverse consequences for test score interpretation and use. Building on testlet response theory (Wainer, Bradlow, & Wang, 2007), the listening section of the Test of German as a Foreign Language (TestDaF) was analyzed to determine whether, and to which extent, testlet effects were present. Three listening passages (i.e., three testlets) with 8, 10, and 7 items, respectively, were analyzed using a two-parameter logistic testlet response model. The data came from two live exams administered in April 2010 (N = 2859) and November 2010 (N = 2214). Results indicated moderate effects for one testlet, and small effects for the other two testlets. As compared to a standard IRT analysis, neglecting these testlet effects led to an overestimation of test reliability and an underestimation of the standard error of ability estimates. Item difficulty and item discrimination estimates remained largely unaffected. Implications for the analysis and evaluation of testlet-based tests are discussed.

Examination of Different Item Response Theory Models on Tests Composed of Testlets

“…In his study conducted with the PISA results, DeMars (2006) made ability predictions using a two-factor model, testlet model, independent item model and multiple categorized item model, and estimated the correlations between the ability predictions obtained from different models to be close to 1.00. Eckes (2014), Min and He (2014) and Bradlow et al (1999) also stated that the ability parameters were similar in their studies.…”

“…However, as a result of the use of testlets based on a common stimulant, the assumption of local independence is disrupted. Many studies in literature have shown that the disruption of local independence causes false predictions in the item and ability parameters obtained by standard IRT models (Ackerman, 1987;Chang & Wang, 2010;Eckes, 2014;Ip, 2000;Marais & Andrich, 2008;Monseur, Baye, Lafontaine, & Quittre, 2011;Reese, 1995;Wainer, 1995). In the present study, testlets were focused upon and data sets were analyzed with different IRT based models, with different sets of data from the items in the tenth booklet of PISA 2012, which measured mathematical literacy.…”

“…In their study, Chang and Wang (2010) compared the standard errors of the ability parameters obtained from the standard IRT model and TRT model over the real data set, and similar to the results of this research, they found that when the local independency assumption was not ensured among the items, the standard error of the ability parameter was predicted to be lower. Another study was conducted by Eckes (2014), who expressed in his study that in analysis results made with standard IRT models where the testlet effect was ignored, standard errors were predicted lower than actual. Based on the results of these research studies, having lower values of error averages of ability parameters obtained from UIRT when compared to the other two models can be due to the existence of local dependency among the items.…”

For a Greater Good: Bias Analysis in Writing Assessment

“…Writing assessment involves more than raters in the process. Factors as examinees and rating scales have also been scrutinized (Eckes, 2005(Eckes, , 2008(Eckes, , 2009(Eckes, , 2012(Eckes, , 2013a(Eckes, , 2013bGoodwin, 2016;Saeidi et al, 2013;Weigle, 1998;Wolfe, 2004, among others). Schaefer (2008) thinks highly of MFRM because "it has shown great promise in the area of performance assessment and rating scale validation because it can analyze sources of variation in test scores beside item difficulty and person ability" (p. 466).…”

Test review: TestDaF

“…A second area of research emphasis has been on the investigation of item and test quality with an eye towards assuring comparability in terms of difficulty, fairness for diverse examinee populations, and construct coverage (e.g., Arras, Müller-Karabil, & Zimmerman, 2013;Eckes, 2008aEckes, , 2013Eckes & Grotjahn, 2006b). Findings from these studies point clearly to the effectiveness of careful item and test development procedures, on the one hand, but perhaps most interestingly they have highlighted the contribution of cutting-edge analytic approaches to quality assurance for operational assessments (e.g., testlet response theory models for determining local item dependencies; see Eckes, 2014). Research in a variety of other areas has also been conducted, and new directions in research are currently being explored by g.a.s.t., all pointing to substantial efforts at maintaining and enhancing the test's quality.…”