An experiment designed to test the validity of a rating technique.

Abstract

The validity of ratings of behavior, provided that two conditions arc met, has been fairly generally accepted. These conditions are that there be several competent judges, 1 and that they have ample opportunity to observe the behavior being rated. For the past two summers the writer has been in a situation where both of these conditions were adequately met. Data were gathered of such a nature that ratings could be compared with more objective measures of the same behaviors; and enough raters were available so that a study of the uniformity of the ratings could be made.The subjects for each of the studies here reported were thirty problem boys who had been sent for study and treatment to a summer camp maintained through the cooperation of Western Reserve University and the Child Guidance Clinic of Cleveland, Ohio. The boys remained for a period of five weeks where they were under the constant observation of a psychiatrist and of six or more counselors trained in psychology and mental hygiene.The first summer, in connection with another experiment, the attempt was made to validate certain "objective" measures of problem boy behavior by comparing them with ratings on the same behavior. The results were such as to make the experimenter conclude that the ratings were themselves being validated. The evidence, however, was rather meager, and it was decided to gather fuller data concerning the same problem. IIDuring the first summer a daily record was kept of "specifically remembered incidents" included under twenty-six different behaviors. 2 1 Note particularly Rugg, H. O.: Is the Rating of Human Character Possible?

Fact and Fiction in Implicit Personality Theory

“…The implications of the present study are somewhat puzzling in this respect because it suggests that IPT reflects mamly the structure of language but does not distort the correlations among retrospective ratings The reason is that the correlations among act frequencies are themselves predetermined by the meaning relationships among the behavior-descnptive terms Thus the present study favors an optimistic view with respect to the accuracy of personality impressions Imphcit personality theory quite accurately reflects the intercorrelations among act frequencies recorded on-lme Accordingly, as far as our trait-attnbutions are shaped by IPT (Newcomb, 1931, Passmi & Norman, 1966, their intercorrelations seem not to be very much distorted However, there are also other facts about behavior that may not be predicted from meaning relationships, for example, different base rates for distinct types of behavior Retrospective judges seem to be highly sensitive to base rates but they do not incorporate this knowledge into conditional probabihty estimates of an IPT-type In this respect, Mirels's (1976) findings were corroborated m the present study, thus demonstrating illusory aspects of IPT The finding, however, that judges are very poor m accurately estimating conditional probabilities need not bother personologists very much Whereas act frequency and trait ratings are very common in personality research, and the correlations calculated from these data are of crucial importance for personality theory (Shweder, 1975), no study comes to our minds where any important personological assumption would have relied upon conditional probability estimates In the correlational sense, where the accuracy of IPT is crucial for personality research, IPT seems to be highly accurate With respect to conditional probability estimates, however, which seem to be senously flawed, the importance of IPT for the accuracy of personality impressions may be regarded as negligible…”

Relationships between Perceptions of Conducting Effectiveness and Ensemble Performance

“…Rosenberg and Jones (1972) state, "these expectations about others generally consist of; (a) the categories we employ to describe the range of abilities, attitudes, interests, physical features, traits and behaviors that we perceive in others; and (b) the beliefs we hold concerning which of these perceived characteristics tend to go together and which do not" (p. 373). These previously held generalizations are superimposed on individuals to determine whether or not they fit the stereotypical category (Allport, 1958;Bruner & Tagiuri, 1954;Cronbach, 1955;Guilford, 1954;Hastorf, Richardson, & Dombusch, 1958;Newcomb, 1931;Tagiuri, 1958).…”

Differential contributions of linguistic factors to memory-based ratings: Systematizing the systematic distortion hypothesis.

“…Characteristically, these terms do not stand in a directly definable relation to empirical realities of ongoing action, and their application is mediated through "family resemblances" (cf. Wittgeni The systematic distortion hypothesis incorporates elements of longstanding problems in the area of rating methods, for example, Thorndike's (1920) "halo effect" and Newcomb's (1929, 1931 postulate of a "logical error." As a recent review of these studies seems to indicate (Cooper, 1980), the detection and elimination of these difficulties, particularly the former, are problems that have yet to achieve a satisfactory solution.…”