Creating Test Score Bands for Assessments Involving Ratings using a Generalizability Theory Approach to Reliability Estimation

Are Consensus Ratings of Functional Job Analysis Scales More Reliable than Ratings Made by Independent Raters?

“…The reliability of the FJA scale ratings across the ten scales were obtained by conducting a Generalizability (G), then a Decision (D) study of the ratings (Shavelson & Webb, 1991). A G-study is a psychometric investigation that evaluates the relative contributions of various sources of error in a given measurement procedure (see Scherbaum, Dickson, Larson, Bellenger & Yusko, 2018 for an overview of generalizability theory). A D-study applies the results of the G-study to optimize the procedure for specific uses.…”

Comparison of G and Phi coefficients estimated in generalizability theory with real cases

“…For example, in a D study, the G and Phi coefficients are estimated after at least three items have been added to a measurement tool, but to what extent these coefficients are sensitive to the item difficulty index (pj) of the added items remains unknown, and whether the items are easy or difficult also remains undefined. In the literature, there are many studies in which items related to various measurement and evaluation practices have been considered a source of variability and reliability studies based on G Theory (Choi & Wilson, 2018;Çakıcı Eser & Gelbal, 2013;Deliceoğlu & Çıkrıkçı Demirtaşlı, 2012;Demir, 2016;Doğan & Anadol, 2017;Doğan & Bıkmaz Bilgen, 2017;Güler, 2011;Güler et al, 2014;Gülle et al, 2018;Hathcoat & Penn, 2012;Hill et al, 2012;Scherbaum et al, 2018;Solano-Flores & Li, 2013;Yılmaz Nalbantoğlu & Gelbal, 2011). Furthermore, in some of these studies (Doğan & Anadol, 2017;Scherbaum et al, 2018;Yılmaz Nalbantoğlu & Gelbal, 2011) comparisons were also made regarding the use of crossed and nested research designs within the scope of G theory.…”

“…In the literature, there are many studies in which items related to various measurement and evaluation practices have been considered a source of variability and reliability studies based on G Theory (Choi & Wilson, 2018;Çakıcı Eser & Gelbal, 2013;Deliceoğlu & Çıkrıkçı Demirtaşlı, 2012;Demir, 2016;Doğan & Anadol, 2017;Doğan & Bıkmaz Bilgen, 2017;Güler, 2011;Güler et al, 2014;Gülle et al, 2018;Hathcoat & Penn, 2012;Hill et al, 2012;Scherbaum et al, 2018;Solano-Flores & Li, 2013;Yılmaz Nalbantoğlu & Gelbal, 2011). Furthermore, in some of these studies (Doğan & Anadol, 2017;Scherbaum et al, 2018;Yılmaz Nalbantoğlu & Gelbal, 2011) comparisons were also made regarding the use of crossed and nested research designs within the scope of G theory. In other studies (Doğan & Bıkmaz Bilgen, 2017;Güler et al, 2014;Gülle et al, 2018;Hathcoat & Penn, 2012;Solano-Flores & Li, 2013) it was observed whether the reliability of performance-based measures could be examined through G theory.…”