Estimating the Accuracy of Jury Verdicts

Abstract

Average accuracy of jury verdicts for a set of cases can be studied empirically and systematically even when the correct verdict cannot be known. The key is to obtain a second rating of the verdict, for example, the judge's, as in the recent study of criminal cases in the United States by the National Center for State Courts (NCSC). That study, like the famous Kalven‐Zeisel study, showed only modest judge‐jury agreement. Simple estimates of jury accuracy can be developed from the judge‐jury agreement rate; the judge's verdict is not taken as the gold standard. Although the estimates of accuracy are subject to error, under plausible conditions they tend to overestimate the average accuracy of jury verdicts. The jury verdict was estimated to be accurate in no more than 87 percent of the NCSC cases (which, however, should not be regarded as a representative sample with respect to jury accuracy). More refined estimates, including false conviction and false acquittal rates, are developed with models using stronger assumptions. For example, the conditional probability that the jury incorrectly convicts given that the defendant truly was not guilty (a “Type I error”) was estimated at 0.25, with an estimated standard error (s.e.) of 0.07, the conditional probability that a jury incorrectly acquits given that the defendant truly was guilty (“Type II error”) was estimated at 0.14 (s.e. 0.03), and the difference was estimated at 0.12 (s.e. 0.08). The estimated number of defendants in the NCSC cases who truly are not guilty but are convicted does seem to be smaller than the number who truly are guilty but are acquitted. The conditional probability of a wrongful conviction, given that the defendant was convicted, is estimated at 0.10 (s.e. 0.03).

Judge‐Jury Agreement in Criminal Cases: The First Three Years of the Korean Jury System

“…Partially consistent with the findings in Spencer (), the results of the study showed that juries appeared to be more accurate than judges when the true verdict (as inferred by statistical methods) was not guilty, as indicated by greater values of relative to for Models 1 and 2. In turn, judges appeared to be more accurate than juries when the true verdict was guilty, as judges showed perfect accuracy by convicting the defendants; the corresponding statistic, , was 1.000 for all models, while ranged from 0.880 to 0.973.…”

“…Interestingly, Type II error rates for judges () were invariably 0.000 for both models. These results suggest that while judges were making highly accurate decisions when the true verdict was guilty, they were more likely than juries to commit Type I errors when evidentiary strength was considered along with the judges' and juries' verdicts in the model, which replicates the results in Spencer ().…”

“…The statistical analyses were guided by previous studies (Eisenberg et al. ; Spencer ). Results indicated that judges and juries agreed on the verdict 91.4 percent of the time (70.3 percent for conviction and 21.1 percent for acquittal).…”

The role of corroborative evidence in child sexual abuse evaluations

“…The central problem is that, in real judgements about allegations of CSA by legal fact fi nders, there is rarely any criterion that can be used to judge the accuracy of fact fi nders' judgements, other than the evidence that was already known to the fact fi nders at the time they made their judgements. There are a number of interesting studies that have used indirect methods to provide rough estimates of the accuracy of verdicts rendered by judges and juries (Arkes & Mellers, 2002;Spencer, 2007); these studies suggest that jury verdicts in criminal cases may be signifi cantly less accurate than is commonly believed. A study of nine real-world criminal CSA cases in Sweden suggests that legal fact fi nders' judgements about the validity of CSA allegations are biased and unreliable when they are made on the basis of psychosocial evidence (Lindblad & Lainpelto, 2008).…”

Identifiability of Models for Multiple Diagnostic Testing in the Absence of a Gold Standard

“…In addition to their use in diagnosing disease, such models are used in the social sciences. See for example Gastwirth and Sinclair (1998), and Spencer (2007).…”