Empirical Item Keying Versus a Rational Approach to Analyzing a Psychological Climate Questionnaire

Abstract

The present study compared two approaches to scoring a Psychological Climate Questionnaire— an empirical keying of items using item analysis and a rational approach which focused on identifying the underlying constructs measured by the questionnaire. The approaches were compared with respect to prediction of performance criteria and the ability to ascertain the theoretical underpinnings of the instrument. Questionnaires were completed by 398 male firemen in a large metropolitan area. Results demonstrated that a summation of item scores to reflect carefully designed a priori constructs did not necessarily result in weaker prediction of performance criteria. Similarities of the present study with previous work are discussed. Three popular approaches to scoring a set of questionnaire items (e.g., personality inventories, biographical inventories, attitude surveys) for which there is no single correct response have been evident in the psychological literature (cf. Sparks, 1969;Nunnally, 1967). These three approaches have been labeled the external (or empirical) approaclr, the internal approach, and the intuitive approach (see Goldberg, 1972, for an extensive review of these approaches).The empirical approach places initial emphasis on the prediction of an external criterion, typically using an empirical keying procedure based on item-criterion relationships. Understanding of underlying relationships or constructs accounting for the prediction has received secondary or no emphasis. This approach has been thought to maximize prediction of an external criterion (Guion, 1965). It has been criticized, however, because a capitalization on unique item variance often results in criterionspecific scoring keys with low internal consistency if the criterion is heterogeneous (Goldberg, 1972;Guion, 1965;Nunnally, 1967). The resulting lack of internal consistency often hinders determination of the theoretical composition or construct validity of the scoring keys (Baehr & Williams, 1967; Matteson et al., 1969;Nunnally, 1967). Thus, little understanding of conceptual relationships between the items and the criterion may be conveyed by this approach.The internal approaclr uses the internal structure of the questionnaire items as the sole source of information to determine where and how items are to be used in the construction of item composites. Typically, this has been accomplished through the use of factor-analytic procedures or a strategy directed toward maximization of internal consistency (e.g., Loevinger, Gleser, & Dubois, 1953).

A Meta-Analytical Multilevel Reliability Generalization of Situational Judgment Tests (SJTs)

“…With respect to measurement error, the latter approach seems most promising to provide superior levels of internal consistency because it intends to provide homogeneous subscale scores. Additionally, it could be hypothesized that the empirical scoring approach leads to decreased alpha coefficients, as this effect is often reported in the biodata literature (Hornick, James, & Jones, 1977).…”

An Exploratory Study of the Jones and James Organisational Climate Scales

“…For health managers, (James, Hartman, Stebbins and Jones, 1977) the sixth factor appeared to represent a finer breakdown of the Challenge, Importance and Variety dimension. The sixth factor for firemen (Hornick, James and Jones, 1977) reflected Mutual Trust, with loadings from confidence and trust in both subordinates and in the leader. Jones and Butler (1980), also using Navy enlisted personnel, replicated the 5 main component solutions with coefficients of congruence (Mulaik, 1972) of (.74), (.86), (.93), (.82) and (.91) respectively.…”

Perceived Job Characteristics and Job Satisfaction: An Examination of Reciprocal Causation

“…The job perception variables included the three dimensions of job challenge, job autonomy,and job importance. Items representing these dimensions were based on prior research on job/ task perceptions (Dunham, 1976;Hackman and Lawler, 197 1;Oldham, 1975, 1976;Sirns, Szilagyi, and Keller, 1975;Turner and Lawrence, 1965), and climate research (Hornick et al, 1977;James et al, 1977;Jones and James, 1979). Job challenge measured the extent to which the job was seen as providing the individual with opportunities to employ his/her skills and abilities, to work on challenging and creative problems, and to engage in a wide range of job behaviors.…”

“…Individuals are also considered to be predisposed toward higher levels of satisfaction when they have comparatively lower levels of education (X12). The reason here is that individuals with more education generally tend to require more inducements for satisfaction, have more options in the job market and thus need not develop high identification with a particular job, and may not feel that their talents are being employed effectively (Hornick et al, 1977;Seybolt, 1976;Steers, 1977;Weaver, 1978). In the present context, age (XI,) was again employed as a surrogate variable, which in effect connoted a greater probability for experiencing satisfying activities as well as a greater propensity to be committed to a particular job and organization.…”