Iota Reliability Concept of the Second Generation

Abstract

Appendix C -Global Indices of Model Fit in Simulation Study IIAppendix D -Global Indices of Model Fit in Simulation Study III included measures. For example, they argue that Alpha penalizes improved coding. That is, if raters correct errors, the values for Alpha can decrease (Zhao et al., 2013, p. 457). Furthermore, cases exist where rater agreement is nearly 100%, while the Alpha values are about 0, indicating the absence of reliability. Based on their findings, Zhao et al. (2013, p. 475) recommend developing and trying new reliability measures. Feng and Zhao (2016, p. 146) suggest not to base new approaches on classical test theory, but on item response theory.In classical test theory, reliability is characterized with measures such as Cronbach's Alpha. These measures produce a single numeric value for a complete scale. Item response theory however, is more detailed. With the help of the test information curve, the reliability of a scale can be investigated for different characteristics of that scale (e.g., Ayala, 2009, pp. 27-33; Baker & Kim, 2017, pp. 96-98). Furthermore, some models of item response theory such as Rasch models offer the opportunity to investigate if a scale produces bias for different groups of individuals. That is, they allow to investigate if an instrument functions similarly for different groups of people (subgroup invariance) (e.g., Baker & Kim, 2017, pp. 38-42).Berding et al. ( 2022) transferred the idea of item response theory to content analysis by suggesting the Iota Reliability Concept. This concept provides several measures for characterizing the reliability of every single category of a coding scheme. In addition, the concept is able to produce insights into how errors in one category influence the data representing other categories, and how data may be biased for different groups of individuals. In the first study, the Iota Concept showed promising statistical properties such as high values for recovering the true reliability of a category, independence from the number of raters, the number of categories, the underlying distribution of categories, and the sample size. The Iota Concept showed a comparable ability as Krippendorff's Alpha to predict the deviation between the true and estimated relationship for two ordinal variables.