No abstract

Summary: This paper presents a meta-analysis of published job analysis interrater reliability data in order to predict the expected levels of interrater reliability within specific combinations of moderators, such as rater source, experience of the rater, and type of job descriptive information. The overall mean interrater reliability of 91 reliability coefficients reported in the literature was .59. The results of experienced professionals (job analysts) showed the highest reliability coefficients (.76). The method of data collection (job contact versus job description) only affected the results of experienced job analysts. For this group higher interrater reliability coefficients were obtained for analyses based on job contact (.87) than for those based on job descriptions (.71). For other rater categories (e.g., students, organization members) neither the method of data collection nor training had a significant effect on the interrater reliability. Analyses based on scales with defined levels resulted in significantly higher interrater reliability coefficients than analyses based on scales with undefined levels. Behavior and job worth dimensions were rated more reliable (.62 and .60, respectively) than attributes and tasks (.49 and .29, respectively). Furthermore, the results indicated that if nonprofessional raters are used (e.g., incumbents or students), at least two to four raters are required to obtain a reliability coefficient of .80. These findings have implications for research and practice.

In the present research two studies are used to investigate the relation between g loading of tests and practice (test-retest) and coaching (active teaching) effects. The data on practice do not support the hypothesis that the higher a test's g loading, the less susceptible it is to preparation, but the data on coaching support the hypothesis. There is evidence that practice and coaching reduce the g-loadedness of a collection of tests. The implications of these results for predictive validity, practical usability of the tests, the relevance of traditional intelligence taxonomies, and for future research are discussed.

No abstract