Temporal Interval and the Estimation of the Reliability of Work Performance Data

Abstract

The stability of estimates of test-retest reliability was studied in a situation in which the interval separating observation periods was permitted to vary. Weekly production data were collected from 24 women sewing-machine operators, and 16 women in folding and packaging jobs. Correlations between performance periods were assembled into arrays each containing weeks separated by a fixed interval, K. The variability of the distribution of coefficients, Sr, was determined for each value of K, and the regression of S r on K was found to fit the general form Sr = K/( a + bK) for each group. Most of the increase in Sr occurred between the interval K = 1 to 25, and subsequent analyses yielded no significant relationship between the presence of aberrant reliability estimates and the time separating observation periods. Both magnitude of the reliability estimate and the accuracy with which the estimate is made vary as a function of the temporal dimension of the test-retest design.

The Past, Present, and Future of Dynamic Performance Research

“…While psychological research often insists on a highly reliable measure of job performance (or for that matter, any criterion) assessed at a point in time, scant attention was paid to whether the criterion had reliability from one time-period to the next. Consequently, a body of research emerged examining the reliability of performance ratings at various time lags (e.g., Ghiselli, 1956;Prien, 1966;Rambo, Chomiak, & Price, 1983;Rambo, Chomiak, & Rountree, 1987;Rothe, 1946Rothe, , 1978Rothe & Nye, 1958, 1959, 1961.…”

“…Specifically, most studies give the issue very little consideration, instead simply use list-wise deletion to address the missing data problem. That is, previous studies examining employee performance levels over time have most frequently eliminated from the sample those employees who leave the job before the full length of data collection (e.g., Henry & Hulin, 1987;Ployhart & Hakel, 1998;Rambo et al, 1983Rambo et al, , 1987Rothe, 1978). When interested in only predicting data in the final wave of a study (e.g., predicting performance in wave six of a six-wave study), then list-wise deletion does perform as well as other missing data techniques (Newman, 2003).…”

The implications of linking the dynamic performance and turnover literatures.

“…Essentially, the issue of removing those who leave over the span of a longitudinal job performance study is one of missing data. Most studies examining dynamic performance require that all participants be present for the time periods of the study (e.g., Deadrick et al, 1997; Deadrick & Madigan, 1990; Ghiselli & Haire, 1960; Henry & Hulin, 1987; Hofmann et al, 1993; Rambo, Chomiak, & Price, 1983; Rambo, Chomiak & Rountree, 1987; Rothe, 1947). Those who left during the time frame of these studies were listwise deleted and thus removed from all of the analyses.…”

Dynamic Criteria Revisited: A Longitudinal Study of Performance Stability and Predictive Validity

“…Moreover, performance over time might not be equally predictable from a given selection procedure, and selection researchers would need to determine the appropriate time to collect criterion data in a predictive validation study (Guion & Gibson, 1988). Findings of instability also raise questions about the usefulness of test-retest correlations as estimates of criterion reliability (Rambo, Chomiak, & Rountree, 1987) and the generalizability of predictive validities over time (Hulin et al, 1990).…”