Modeling Instructional Sensitivity Using a Longitudinal Multilevel Differential Item Functioning Approach

Instruktionssensitivität von Tests und Items

“…Clauser et al, 1996;Linn und Harnisch, 1981). DIF-Untersuchungen zur Instruktionssensitivität trugen maßgeblich dazu bei, die Verletzung von Messinvarianzannahmen als eine notwendige Voraussetzung für Instruktionssensitivität anzusehen (Naumann et al, 2014). Aus psychometrischer Sicht ist DIF jedoch keine Voraussetzung für Instruktionssensitivität (Naumann et al, 2017).…”

A Psychometric Framework for the Evaluation of Instructional Sensitivity

“…Recently, Naumann and colleagues (2014) However, recent studies have shown that these items statistics do not lead to consistent results when applied to the same set of data (Li, Ruiz-Primo, & Wills, 2012;Naumann, Hochweber, & Hartig, 2014). Additionally, the connection to instructional sensitivity of the test and to instructional measures has not been widely investigated for the item statistics described above (Polikoff, 2010).…”

“…Analogously, the between-time-points facet should be examined when studies focus on students' progress, ultimately leading to the differences-between-groups-and-time points perspective if both facets are of interest. With respect to the different sources of variance addressed in the evaluation of instructional sensitivity, a single statistic might not be sufficient to thoroughly describe the sensitivity of a test or a single item, because a single indicator cannot express all of the variability in item responses (Naumann et al, 2014). That is, differences between time points and differences between groups need to be expressed with appropriate items or test statistics.…”

“…To date, empirical studies rarely have made their perspective on instructional sensitivity explicit. We believe this is a serious oversight, as on the one hand, results may diverge substantially when they are based on different approaches rooted in different perspectives and therefore based on different sources of variance (Li et al, 2012;Naumann et al, 2014), and on the other hand, it remains unclear whether the facet of instructional sensitivity under investigation has fitted the intended use and interpretation of test scores (Kane, 2013 In summary, a thorough evaluation of an instrument's (instructional) sensitivity requires evaluation of instructional sensitivity on multiple levels of analysis and from multiple perspectives'. If the analysis takes into account only one perspective, judgment of instructional sensitivity may be somewhat incomplete.…”

“…In contrast, data requirements for single-perspective investigations are comparably modest since either longitudinal data or large sample sizes on the classroom-level are needed, and hence these approaches might be easier to implement. For example, PPDI may be calculated based on classical item difficulty p using standard software (e.g., SPSS), while the LMLDIF model requires Bayesian estimation (see Naumann et al, 2014, for details).…”