Generalizability Theory

Brennan, Robert L.

doi:10.1111/j.1745-3984.2003.tb01098.x

Cited by 189 publications

(380 citation statements)

References 1 publication

Supporting

Mentioning

362

Contrasting

Unclassified

Order By: Relevance

“…We included only appropriate generalizations in our analyses 8 . In generalizability theory, when aiming to "generalize across" conditions of measurement, such conditions are treated as sources of unreliable variance (Brennan, 2001;Cronbach et al, 1972;Putka & Hoffman, 2014). In Table 3, for example, when scores aggregated to the dimension-level are generalized across both assessors and exercises, both assessor-and exercise-related sources of variance are specified as contributing to unreliable variance.…”

Section: Discussionmentioning

confidence: 99%

“…There are many possible reasons for model misspecification and thus confounding: these include omission, data unavailability, and design factors that prevent the isolation of particular effects (Brennan, 2001;Putka & Hoffman, 2013). For example, if the assessor-to-participant ratio in an AC is not >1:1, then it is impossible to isolate assessor-related effects.…”

Section: Do Confounds Limit the Interpretability Of Research On Ac Ramentioning

confidence: 99%

“…Moreover, depending on sample size and model complexity, the computational demands associated with REML estimators can be nontrivial or even insurmountable (Brennan, 2001). …”

Section: Do Confounds Limit the Interpretability Of Research On Ac Ramentioning

confidence: 99%

“…Bayesian methods overcome the potentially problematic negative or fenced variance components that can occur with REML estimation, particularly when the number of levels specific to particular effects is small (e.g. when only 3 exercises are used in an AC, see Brennan, 2001). Through the use of weakly informative priors, Bayesian methods can even address the issue of estimates being too close to 0 (Chung, Rabe-Hesketh, Dorie, Gelman, & Liu, 2013).…”

Section: Do Confounds Limit the Interpretability Of Research On Ac Ramentioning

confidence: 99%

“…To assuage concerns about the number of dimensions and exercises in our study relative to the other examples listed in Table 5, we applied a decision study from generalizability theory methodology, which is akin to applying the Spearman-Brown prediction formula for multifaceted measures (Brennan, 2001;Shavelson & Webb, 1991). Using a decision study, we extrapolated from our observed reliability outcomes based on three exercises and six dimensions to an alternative design based on five exercises and 13 dimensions: the largest number of exercises and dimensions used in the studies listed in Table 5 (i.e., in the Bowler & Woehr, 2009, study).…”

Section: Limitations and Future Directionsmentioning

confidence: 99%

See 4 more Smart Citations

Everything that you have ever been told about assessment center ratings is confounded.

Jackson

Michaelides

Dewberry

et al. 2016

Journal of Applied Psychology

View full text Add to dashboard Cite

Despite a substantial research literature on the influence of dimensions and exercises in assessment centers (ACs), the relative impact of these two sources of variance continues to raise uncertainties because of confounding. With confounded effects, it is not possible to establish the degree to which any one effect, including those related to exercises and dimensions, influences AC ratings. In the current study (N = 698) we used Bayesian generalizability theory to unconfound all of the possible effects contributing to variance in AC ratings. Our results show that ≤ 1.11% of the variance in AC ratings was directly attributable to behavioral dimensions, suggesting that dimension-related effects have no practical impact on the reliability of ACs.Even when taking aggregation level into consideration, effects related to general performance and exercises accounted for almost all of the reliable variance in AC ratings. The implications of these findings for recent dimension-and exercise-based perspectives on ACs are discussed. When, in the context of selection, appraisal, and development, behavioral criteria are used to evaluate individuals, it is essential that these criteria are measured reliably.Unsurprisingly, therefore, the measurement properties of assessment center (AC) ratings have come under close scrutiny in the applied psychology literature. In ACs, the behavior of jobholders or candidates is sampled across several work-related situations (exercises, e.g., a role play exercise, group discussion, presentation) and is typically assessed by trained assessors in terms of pre-defined behavioral dimensions (e.g., communication skills, teamwork, planning and organizing). As a result of their multifaceted measurement properties, incorporating dimensions, exercises, and assessors, ACs provide a rich source of information about the extent to which work-related behavioral criteria can be reliably measured in a job-relevant setting.Historically, researchers have questioned the extent to which behavioral dimensions are measured reliably in ACs, and have implied that researchers should utilize an exercise-oriented approach to scoring ACs

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Do Confounds Limit the Interpretability Of Research On Ac Ramentioning

confidence: 99%

“…Moreover, depending on sample size and model complexity, the computational demands associated with REML estimators can be nontrivial or even insurmountable (Brennan, 2001). …”

Section: Do Confounds Limit the Interpretability Of Research On Ac Ramentioning

confidence: 99%

Section: Do Confounds Limit the Interpretability Of Research On Ac Ramentioning

confidence: 99%

Section: Limitations and Future Directionsmentioning

confidence: 99%

See 3 more Smart Citations

Everything that you have ever been told about assessment center ratings is confounded.

Jackson

Michaelides

Dewberry

et al. 2016

Journal of Applied Psychology

View full text Add to dashboard Cite

show abstract

Inter‐rater reliability of the PCL‐R total and factor scores among psychopathic sex offenders: are personality features more prone to disagreement than behavioral features?

Edens

Boccaccini

Johnson

2010

Behavioral Sci & The Law

View full text Add to dashboard Cite

Despite considerable support for the inter-rater reliability of the Hare Psychopathy Checklist--Revised (PCL-R; Hare, 1991, 2003) in research contexts, there is increasing concern that scores from this instrument may be considerably less stable across examiners in applied contexts, particularly when scoring is based on separate interviews. The present study examines archival data from a sample of imprisoned sex offenders (n = 20) who obtained relatively high PCL-R total scores (> or =25) and were administered this instrument on a second occasion by a different examiner. Intraclass correlations for the total and Factor 2 score were lower than those generally reported in research studies. Of greater concern, Factor 1 scores were only negligibly related to each other (ICC(A,1) = .16). Correcting for potential range restriction among these high scoring individuals resulted in total and Factor 2 score measures of agreement that were somewhat more consistent with published research, but Factor 1 continued to display exceedingly poor agreement across examiners.

show abstract

Monitoring Faculty Consultant Performance in the Advanced Placement English Literature and Composition Program With a Many‐faceted Rasch Model

Engelhard

Myford

2003

ETS Research Report Series

View full text Add to dashboard Cite

The purpose of this study was to examine, describe, evaluate, and compare the rating behavior of faculty consultants who scored essays written for the Advanced Placement English Literature and Composition (AP® ELC) Exam. Data from the 1999 AP ELC Exam were analyzed using FACETS (Linacre, 1998) and SAS. The faculty consultants were not all interchangeable in terms of the level of severity they exercised. If students' ratings had been adjusted for severity differences, the AP grades of about 30 percent of the students would have been different from the one they received. Almost all the differences were one grade or less. Adjusting ratings for faculty consultant severity differences would not impact some student subgroups more than others.

show abstract

Generalizability Theory

Cited by 189 publications

References 1 publication

Everything that you have ever been told about assessment center ratings is confounded.

Everything that you have ever been told about assessment center ratings is confounded.

Inter‐rater reliability of the PCL‐R total and factor scores among psychopathic sex offenders: are personality features more prone to disagreement than behavioral features?

Monitoring Faculty Consultant Performance in the Advanced Placement English Literature and Composition Program With a Many‐faceted Rasch Model

Contact Info

Product

Resources

About