What sources contribute to variance in observer ratings? Using generalizability theory to assess construct validity of psychological measures

Abstract

We illustrate the utility of generalizability theory (GT) as a conceptual framework that encourages psychological researchers to address this question and as a flexible set of analytic tools that can provide answers to inform both substantive theory and measurement practice. To illustrate these capabilities, we analyze observer ratings of 27 caregiver-child dyads, focusing on the importance of situational (contextual) factors as sources of variance in observer ratings of caregiver-child behaviors. Cross-situational consistency was relatively low for the categories of behavior analyzed, indicating that dyads vary greatly in their interactional patterns from one situation to the next, so that it is difficult to predict behavioral frequencies in one context from behaviors observed in a different context. Our findings suggest that single-situation behavioral measures may have limited generalizability, either to behavior in other contexts or as measures of global interaction tendencies. We discuss the implications of these findings for research and measurement design in developmental psychology.

Teachers’ Assessment of Physical Aggression With the Preschool Behavior Questionnaire: A Multitrait-Multimethod Evaluation of Convergent and Discriminant Validity

“…Teachers' judgments were rooted in many daily observations and interactions with the children across various situations and school contexts. In contrast, observer ratings were restricted to occurrences of children's antisocial behavior in a specific small-group peer situation when the teacher was not present as well as to a limited time frame (Ladd & Profilet, 1996;Lakes & Hoyt, 2008). Notwithstanding the divergence, the results do suggest that teacher ratings of physical aggression are to some extent related to actual physical aggression in a specific context, as has been found in other observational studies (McEvoy, Estrem, Rodriguez, & Olson, 2003;McNeilly-Choque et al, 1996;Ostrov & Keating, 2004).…”

Teacher and observer views on student–teacher relationships: Convergence across kindergarten and relations with student engagement

“…This lack of stability may be due to the fact that observer ratings are only based on a relatively small sample of teacher-child interactions. Observer ratings may still include substantial occasion-specific information, which may be due to inconsistency of the observed teacher-child interactions over time, the nature of the situation in which the interactions occur, or a combination of both (Lakes & Hoyt, 2008). 8 Despite this occasion-specific variance, observations at different time points had sufficient in common throughout the kindergarten school year to have significant factor loadings on their intended latent construct.…”

“…Fourth, observers' scores for behavioral engagement were only based on limited samples of child behavior. Although interrater reliability was excellent and the expected associations with teacher ratings of classroom participation were found, the use of more extensive observation periods may increase the validity of this measurement (Lakes & Hoyt, 2008).…”

Here today, gone tomorrow? Revisiting the stability of teachers' achievement goals

“…One important methodological implication is that researchers should be clear about the aspects they want to generalize with their measures: When measuring a construct we are often not interested in single performances regarding this construct but in the general construct-related values of persons (Cronbach, Gleser, Nanda, & Rajaratnam, 1972;Lakes & Hoyt, 2008;Shavelson & Webb, 1991). If a measure is largely influenced by occasion characteristics, only statements regarding this specific measurement point can be made.…”