Interrater agreement and reliability for the Rorschach have recently come under increasing scrutiny. This is the second report examining methods of Comprehensive System reliability using principles derived from observational methodology and applied behavioral analysis. This study examined a previous nonpatient sample of 20 protocols (N = 412 responses) and also examined a new clinical sample of 20 protocols (N = 374 responses) diagnosed with Research Diagnostic Criteria. Reliability was analyzed at multiple levels of Comprehensive System data, including response-level individual codes and coding decisions and ratios, percentages, and derivations from the Structural Summary. With a number of exceptions, most Comprehensive System codes, coding decisions, and summary scores yield acceptable, and in many instances excellent, levels of reliability. Limitations arising from the nature of Rorschach data and Comprehensive System coding criteria are discussed.

Although the Rorschach test has demonstrated significant refinements in reliability, validity, and statistical power as a result of the procedural standardization and scoring innovations introduced by Exner's Comprehensive System, the issue of Rorschach interrater reliability remains unexplored. This article examines the psychometric foundations of Rorschach interrater reliability and applies notions from applied behavioral analysis to the treatment of Rorschach data. We empirically compare 3 methods of quantifying interrater agreement, their accuracy in estimating interrater agreement, and efficiency in reducing error in Rorschach research. Results indicate that the magnitude of differences between methods of quantifying interrater agreement and the associated reductions of error are significant. We propose a standard method for quantifying interrater agreement in Rorschach research.

The Rorschach Inkblot Test has been the source of long-standing controversies as to its nature and its psychometric properties. Consistent with behavioral science research in general, the concept of statistical power has been entirely ignored by Rorschach researchers. The concept of power is introduced and discussed, and a power survey of the Rorschach literature published between 1975 and 1991 in the Journal of Personality Assessment, Journal of Consulting and Clinical Psychology, Journal of Abnormal Psychology, Journal of Clinical Psychology, Journal of Personality, Psychological Bulletin, American Journal of Psychiatry, and Journal of Personality and Social Psychology was undertaken. Power was calculated for 2,300 statistical tests in 158 journal articles. Power to detect small, medium, and large effect sizes was .13, .56, and .85, respectively. Similar to the findings in other power surveys conducted on behavioral science research, we concluded that Rorschach research is underpowered to detect the differences under investigation. This undoubtedly contributes to the inconsistency of research findings which has been a source of controversy and criticism over the decades. It appears that research conducted according to the Comprehensive System for the Rorschach is more powerful. Recommendations are offered for improving power and strengthening the design sensitivity of Rorschach research, including increasing sample sizes, use of parametric statistics, reduction of error variance, more accurate reporting of findings, and editorial policies reflecting concern about the magnitude of relationships beyond an exclusive focus on levels of statistical significance.

No abstract