Does interrater (dis)agreement on Psychopathy Checklist scores in sexually violent predator trials suggest partisan allegiance in forensic evaluations?

Abstract

Many studies reveal strong interrater agreement for Hare's Psychopathy Checklist-Revised (PCL-R) when used by trained raters in research contexts. However, no systematic research has examined agreement between PCL-R scores from independent clinicians who are retained by opposing sides in adversarial legal proceedings. We reviewed all 43 sexual-offender civil-commitment trials in one state and identified 23 cases in which opposing evaluators reported PCL-R total scores for the same individual. Differences between scores from opposing evaluators were usually in a direction that supported the party who retained their services. These score differences were greater in size than would be expected based on the instrument's standard error of measurement or the rater agreement values reported in previous PCL-R research. The intraclass correlation for absolute agreement for the PCL-R Total score from a single rater (ICC 1,A = .39) was well below levels of agreement observed for the PCL-R in research contexts, and below published test-retest values for the PCL-R. Results raise concerns about the potential for a forensic evaluator's "partisan allegiance" to influence PCL-R scores in adversarial proceedings.

Forensic psychologists’ perceptions of bias and potential correction strategies in forensic mental health evaluations.

“…(e.g., DeMatteo, et al, 2014;Murrie et al, 2008;Murrie, Boccaccini, Turner, Meeks, Woods, & Tussey, 2009) and in Canada (Lloyd, Clark, & Forth, 2010). Beyond these studies, surveys of judges and attorneys consistently reveal their concern about bias among experts as well (e.g., Krafka, Dunn, Johnson, Cecil, & Miletich, 2002;Shuman, Whitaker, & Champagne, 1994).…”

Psychopathy: Assessment and Forensic Implications

“…In a research context, misuse of these instruments will have few negative consequences for the person tested. However, when the scores are used in clinical and criminal justice contexts 42 It is important when conducting an assessment to use all information available to provide a complete picture of the person. In each case, the PCL-R must be used properly and in accordance with the highest ethical and professional standards.…”

Inter‐rater reliability of the PCL‐R total and factor scores among psychopathic sex offenders: are personality features more prone to disagreement than behavioral features?

“…A recent review of rater agreement findings from more than 120 PCL-R studies found that about half provided little or no information concerning whether independent interviews were used for the cases used to check rater agreement (Rufino et al, 2009). In only a handful of studies was it clear that rater agreement statistics were based on cases in which both evaluators conducted an independent interview (see, e.g., Levenson, 2004;Murrie, Boccaccini, Johnson, & Janke, 2008;Rutherford, Cacciola, Alterman, McKay, & Cook, 1999;Tyrer et al, 2005). In several of these studies, ICC values were notably lower than those reported in the PCL-R manual.…”

“…More recently, Murrie et al (2008) compared rates of agreement between PCL-R scores provided by experts called by opposing sides (termed ''petitioners'' and ''respondents'') in Texas Sexually Violent Predator (SVP) civil commitment cases. In 23 cases, an expert retained by the respondent also administered the PCL-R. For the total score, the single-evaluator ICC 1 for absolute agreement across these cases was .39, well below the high levels of agreement observed for the PCL-R in most research contexts.…”