Theory of Generalizability: A Liberalization of Reliability Theory†

Cronbach, Lee J.; Rajaratnam, Nageswari; Gleser, Goldine C.

doi:10.1111/j.2044-8317.1963.tb00206.x

Cited by 670 publications

(397 citation statements)

References 18 publications

Supporting

Mentioning

364

Contrasting

Unclassified

Order By: Relevance

“…This means that any aggregate score (e.g., based on dimensions) that is derived from an AC will reflect the constituent effects (i.e., sample dependencies, the general performance of the participants, dimensions, exercises, assessors, indicator items, and respective interaction terms) making up that score. If, when researchers attempt to model variance in AC ratings, any of these effects are not acknowledged, then that model is ultimately misspecified and, as a result, confounded (Cronbach, Gleser, Nanda, & Rajaratnam, 1972;Cronbach, Rajaratnam, & Gleser, 1963). Depending on the extent of the confounding, its presence generally renders hazardous the interpretation of results (Herold & Fields, 2004).…”

Section: Do Confounds Limit the Interpretability Of Research On Ac Ramentioning

confidence: 99%

Everything that you have ever been told about assessment center ratings is confounded.

Jackson

Michaelides

Dewberry

et al. 2016

Journal of Applied Psychology

View full text Add to dashboard Cite

Despite a substantial research literature on the influence of dimensions and exercises in assessment centers (ACs), the relative impact of these two sources of variance continues to raise uncertainties because of confounding. With confounded effects, it is not possible to establish the degree to which any one effect, including those related to exercises and dimensions, influences AC ratings. In the current study (N = 698) we used Bayesian generalizability theory to unconfound all of the possible effects contributing to variance in AC ratings. Our results show that ≤ 1.11% of the variance in AC ratings was directly attributable to behavioral dimensions, suggesting that dimension-related effects have no practical impact on the reliability of ACs.Even when taking aggregation level into consideration, effects related to general performance and exercises accounted for almost all of the reliable variance in AC ratings. The implications of these findings for recent dimension-and exercise-based perspectives on ACs are discussed. When, in the context of selection, appraisal, and development, behavioral criteria are used to evaluate individuals, it is essential that these criteria are measured reliably.Unsurprisingly, therefore, the measurement properties of assessment center (AC) ratings have come under close scrutiny in the applied psychology literature. In ACs, the behavior of jobholders or candidates is sampled across several work-related situations (exercises, e.g., a role play exercise, group discussion, presentation) and is typically assessed by trained assessors in terms of pre-defined behavioral dimensions (e.g., communication skills, teamwork, planning and organizing). As a result of their multifaceted measurement properties, incorporating dimensions, exercises, and assessors, ACs provide a rich source of information about the extent to which work-related behavioral criteria can be reliably measured in a job-relevant setting.Historically, researchers have questioned the extent to which behavioral dimensions are measured reliably in ACs, and have implied that researchers should utilize an exercise-oriented approach to scoring ACs

show abstract

Section: Do Confounds Limit the Interpretability Of Research On Ac Ramentioning

confidence: 99%

Everything that you have ever been told about assessment center ratings is confounded.

Jackson

Michaelides

Dewberry

et al. 2016

Journal of Applied Psychology

View full text Add to dashboard Cite

show abstract

“…Comparable numbers of mother-infant dyads across the three groups were included in the reliability sample to ensure that all groups were represented equally. Generalizability coefficients, which reflect the ratio of participant score variance over the sum of the participant score variance plus error variance (Cronbach, Gleser, & Rajaratnam, 1963) …”

Section: Methodsmentioning

confidence: 99%

Predicting cognitive-language and social growth curves from early maternal behaviors in children at varying degrees of biological risk.

Landry¹,

Smith²,

Miller-Loncar³

et al. 1997

Developmental Psychology

348

296

View full text Add to dashboard Cite

Growth modeling was used to examine the relation of early parenting behaviors (averaged across 6 and 12 months) with rates of change in children's cognitive-language and social response and initiating skills assessed at 6, 12, 24, and 40 months. Groups of full-term (n = 112) and very low birth weight children, divided into medically low (n = 114) and high risk (HR; n = 73), were included to evaluate whether children who vary in their rate of development are influenced in different ways by early parenting styles. Parenting behaviors that were sensitive to children's focus of interest and did not highly control or restrict their behaviors predicted greater increases and faster rates of cognitive-language and social development, with relations stronger for the HR versus the other two groups. These maternal behaviors may provide the support all infants need to establish an optimal early foundation for later development and the specific support HR children need to learn in spite of early attentional and organizational problems. Across the past several decades, studies have examined aspects of early parenting that promote children's competencies, such as cooperation, self-assertion, initiative, and independence in problem solving (e.g.

show abstract

“…Because the immediate stimulus context is the same for all judges, it is difficult to determine how susceptible these scales are to contextual effects. Within the framework of generalizability theory (Cronbach et al, 1963), the pertinent universe of generalization is one that includes different judges and different contexts. As demonstrated in our studies, a reliability coefficient generated within a constant context will overestimate the reliability of the measure across contexts.…”

Section: Discussionmentioning

confidence: 99%

“…There are many senses of reliability that have been described in the psychometric literature (for a discussion of the multiple levels at which reliability may be assessed, see Cronbach, Gleser, & Rajaratnam, 1963). One sense of reliability is captured in terms of the correlation between repeated assessments of the same individuals.…”

Section: Implications For Reliability Of Judgmentmentioning

confidence: 99%

Reducing the dependence of clinical judgment on the immediate context: Effects of number of categories and type of anchors.

Wedell¹,

Parducci²,

Lane³

1990

Journal of Personality and Social Psychology

View full text Add to dashboard Cite

Two experiments explored methods for standardizing ratings of the psychopathology of clinical case histories. In both experiments, the same case histories were rated as more pathological when mostly mild rather than severe cases were presented as the immediate context. Psychometric analyses demonstrated that this type of contextual effect is a potentially important source of unreliability in clinical judgment. In Experiment 1, increasing the number of points in the rating scale from 3 to either 7 or 100 significantly reduced the effects of the immediate context. Ratings were parsimoniously modeled by Parducci's (1983) range-frequency theory. In Experiment 2, providing verbal anchors in the form of either detailed DSM descriptions for each rating category or sample case histories for the two end-categories increased the reliability of the ratings by reducing the effects of the immediate contexts; however, these reductions occurred only when the ranges of the immediate contexts had been severely restricted. According to the range-frequency analysis, verbal anchors served to equate the endpoints of the subjective range for the different contextual conditions. Comparison with previous research suggests that the anchors also reduced the effects of the sequential position in which clinical cases appear. We therefore recommend that studies of the reliability of behavioral assessment techniques take into account the effects of differences in context.Although reliability does not imply validity, it is a necessary prerequisite. Unfortunately, clinical assessment in psychology is notoriously unreliable. Eysenck, Wakefield, and Friedman (1983) have documented the lack of interrater reliability. Goldberg (1968Goldberg ( , 1970 concluded that clinical judgments tend to be unreliable and only minimally related to the degree of experience of the person making the assessment. Arnoff (1954) reported an actual decrease in reliability with greater clinical experience.One generally accepted method for improving clinical prediction has been the use of statistical models. In particular, there is overwhelming evidence for the superiority of simple linear models over clinicians in combining information to form a diagnosis (Dawes, 1979;Dawes&Corrigan, 1974;Einhorn, 1972;Goldberg, 1968Goldberg, , 1970Meehl, 1954). But as Sawyer (1966) has pointed out, combining information is only the second half of the problem, the first half being the problem of measurement. Although there are a large number of objective measurement devices currently in use in clinical psychology, it is doubtful that the clinician will be supplanted in the near future as a primary source of information used in prediction. Therefore, development of greater accuracy of clinical judgments is of fundamenPreparation of this article was carried out while Douglas H.

show abstract

Theory of Generalizability: A Liberalization of Reliability Theory†

Cited by 670 publications

References 18 publications

Everything that you have ever been told about assessment center ratings is confounded.

Everything that you have ever been told about assessment center ratings is confounded.

Predicting cognitive-language and social growth curves from early maternal behaviors in children at varying degrees of biological risk.

Reducing the dependence of clinical judgment on the immediate context: Effects of number of categories and type of anchors.

Contact Info

Product

Resources

About