Test Validation for Scientific Understanding: Two Demonstrations of an Approach to Studying Predictor‐criterion Linkages

Abstract

This paper argues that a construct-oriented approach to test validation is likely to enhance scientific understanding of our predictor measures, performance criteria, and links between them. In particular, examining relationships between relatively homogeneous predictors and criteria tapping specific performance areas operationalizes earlier conceptual statements made by Guion and Dunnette about test validation for scientific understanding. Two demonstrations are offered to show how measures of predictor constructs have predictably different patterns of correlations with different criteria. In a study of Navy recruiters (N = 267), individual personality scales had significantly different relationships with three different rating criteria; in a second study, with Army enlisted soldiers (N = 8,642), cognitive ability and personality construct measures also showed predictable patterns of correlations, with rating criteria measuring three different performance areas. The paper discusses scientific and practical implications of this construct-oriented approach to test validation. For many years, researchers have advocated the use of construct validation strategies in personnel selection research (e.g., Cronbach, 1949;Guion, 1976). In particular, it has been argued that emphasis should be placed on identifying homogeneous criterion factors, specifying individual differences important for predicting performance in each of these criterion areas, and then investigating the linkages between measures of these multiple constructs (Dunnette, 1963). The rationale behind this approach is that

Capturing Interpersonal Performance in a Latent Performance Model

“…This suggests that separating the technical and interpersonal aspects of task performance could expose different patterns of validity coefficients for interpersonal job-specific proficiency and technical job-specific proficiency, revealing richer and more accurate information about the relations between predictors and these criteria. Evidence from Pulakos et al (1988) supports this. As indicated in the introduction, Pulakos et al factor analyzed ratings of 16 dimensions of performance for Navy recruiters and found a three-factor solution was the most interpretable: selling skills, organizing skills, and human relations skills.…”

“…According to Campbell et al, job-specific task proficiency refers to the behaviors that are central to a job and differentiate one job from another such as driving a bus, directing air traffic, and doing word processing. Although none of the examples offered by Campbell et al are interpersonal in nature, it seems clear given the findings of Distefano and Pryer (1986) and Pulakos et al (1988) that the core behaviors of a job can be interpersonal in nature (e.g., interacting therapeutically with a patient or selling).…”

“…Pulakos, Borman, and Hough (1988), for example, factor analyzed pooled peer and supervisor ratings of 16 dimensions of performance for Navy recruiters and found a three-factor solution was the most interpretable: selling skills, organizing skills, and human relations skills. Both selling skills and human relations skills represent task-related dimensions of performance that are interpersonal in nature.…”

Distinguishing personality and biodata items using confirmatory factor analysis of multitrait-multimethod matrices

“…The use of construct validity strategies in personnel research has increased rapidly in recent years (e.g., Binning & Barrett, 1989;Borman, Rosse, & Abrahams, 1980;Landy, 1986;Pulakos, Borman, & Hough, 1988;Shore, Shore, & Thornton, 1992;Vance, MacCallum, Coovert, & Hedge, 1988). This theoretical and conceptual enrichment is important from both the practical aspect of the need to show job relatedness of selection devices and the theoretical aspect of explaining why certain selection devices are predictive of job related behaviors, or more predictive than other measures.…”

The Validity of Individual Psychological Assessments for Entry-Level Police and Firefighter Positions

“…An employee performance form was designed by the consulting firm specifically for this study, and contained seven items derived from the same competency model that served as the framework for the assessment report. Structuring the performance measure to align with the assessment structure emphasizes the prediction of those aspects of performance that are most relevant to the purpose of the assessment (Pulakos Borman, & Hough, 1988) rather than prediction of overall job performance. All ratings were on a five-point scale.…”