Interrater reliability of Violence Risk Appraisal Guide scores provided in Canadian criminal proceedings.

Abstract

Published research suggests that most violence risk assessment tools have relatively high levels of interrater reliability, but recent evidence of inconsistent scores among forensic examiners in adversarial settings raises concerns about the "field reliability" of such measures. This study specifically examined the reliability of Violence Risk Appraisal Guide (VRAG) scores in Canadian criminal cases identified in the legal database, LexisNexis. Over 250 reported cases were located that made mention of the VRAG, with 42 of these cases containing 2 or more scores that could be submitted to interrater reliability analyses. Overall, scores were skewed toward higher risk categories. The intraclass correlation (ICCA1) was .66, with pairs of forensic examiners placing defendants into the same VRAG risk "bin" in 68% of the cases. For categorical risk statements (i.e., low, moderate, high), examiners provided converging assessment results in most instances (86%). In terms of potential predictors of rater disagreement, there was no evidence for adversarial allegiance in our sample. Rater disagreement in the scoring of 1 VRAG item (Psychopathy Checklist-Revised; Hare, 2003), however, strongly predicted rater disagreement in the scoring of the VRAG (r = .58). (PsycINFO Database Record

Prediction of violent crime on discharge from secure psychiatric hospitals: A clinical prediction rule (FoVOx)

“…Tools like FoVOx show similar or better performance to other tools, but are easier, quicker, and free to use whilst at the same time being scalable, fully transparent, and less subjective. Additionally, while measures of interrater reliability of structured clinical judgement tools are generally high in research settings [45] , this may not be the case when used in adversarial settings [46] .…”

An update and expansion on the role of the Violence Risk Appraisal Guide and Historical Clinical Risk Management‐20 in United States case law

“…This was not possible with the current dataset, as only four decisions included actual raw scores and only one case included scores from more than one evaluator. The differences in available data are probably due to the restricted time frame of the current study (compared with Edens et al, ) and the jurisdictional differences (Edens et al reviewed Canadian case law only). Nonetheless, Edens et al () reported lower than expected intraclass correlations, with rater disagreement strongly predicted by a single VRAG item, the offender's PCL‐R score.…”

“…The differences in available data are probably due to the restricted time frame of the current study (compared with Edens et al, ) and the jurisdictional differences (Edens et al reviewed Canadian case law only). Nonetheless, Edens et al () reported lower than expected intraclass correlations, with rater disagreement strongly predicted by a single VRAG item, the offender's PCL‐R score. This item was altered in a revised version of the measure (Rice, Harris, & Lang, ) and, although it is unlikely that this revision would impact the application and communication of the VRAG, more research examining score reliability across setting and evaluators is warranted.…”

“…The actuarial measure was mainly introduced by the prosecution as a measure of violence risk and rarely challenged. Other case law reviews have coded the specific VRAG scores reported in appellate decisions and examined the reliability of these scores across legal question, retaining side, and evaluator training (e.g., Edens et al, ). This was not possible with the current dataset, as only four decisions included actual raw scores and only one case included scores from more than one evaluator.…”

“…In this survey of over 2000 mental health clinicians from 44 countries, researchers found that 48.34% of participants reported using the HCR-20 over their lifetime, and 31.33% reported using the measure within the previous year. The VRAG enjoyed somewhat less popularity with survey respondents, with only 8.24% reporting 1 Other research has questioned the reliability of VRAG scores, due primarily to the inclusion of Psychopathy Checklist-Revised (Hare, 2003) scores, which was included in the original VRAG scoring system (Edens, Penson, Ruchensky, Cox, & Smith, 2016). The most recent version of the VRAG (VRAG-R; Rice, Harris, & Lang, 2013) has eliminated the inclusion of the full PCL-R score in favor of one specific PCL-R facet (facet 4).…”