Probabilistic reporting in criminal cases in the United States: A baseline study

Abstract

Over a decade ago, the National Research Council (NRC) Report, Strengthening Forensic Science in the United States, highlighted the lack of standards with regard to reporting of evidence in forensic science: [M]any terms are used by forensic examiners in reports and in court testimony to describe findings, conclusions, and the degrees of association between evidentiary material (e.g., hairs, fingerprints, fibers) and particular people or objects. Such terms include but are not limited to " match," " consistent with," " identical," " similar in all respects tested," and " cannot be excluded as the source of."…Yet the forensic science disciplines have not reached agreement or consensus on the precise meaning of any of these terms. Although some disciplines have developed vocabulary and scales to be used in reporting results, they have not become standard practice [1]. In subsequent years, working groups in a number of forensic disciplines promulgated guidelines in an effort to standardized reporting among practitioners [2-7]. As the above quotatione illustrates, many forensic disciplines historically reported in what can reasonably be characterized as a " non-statistical" manner, using verbal formulations that were categorical rather than continuous, reflecting certainty rather than uncertainty (e.g., [8,9]). The standards issued following the NRC report often continued to promote these non-statistical forms of forensic evidence reporting, albeit it in a more systematized manner. At the same time, recent years have seen increasing efforts to promote the application of statistics to forensic evidence in the United States (e.g., [8,9]). Within this burgeoning field of forensic statistics, a vigorous scholarly debate has developed over how forensic scientists should report results. Although all forensic statisticians urge that results should be reported in a statistically defensible manner, they differ regarding the precise tools and frameworks for such reporting. For example, some forensic statisticians have promoted the likelihood ratio framework as an all-encompassing solution to forensic reporting, applicable to all, or nearly all, situations (e.g., [10,11]). Yet, even among those who advocate for likelihood ratios, debates remain over myriad technical issues in formulating those ratios. Some forensic statisticians argue that, in order to adequately address underlying model assumptions and sampling uncertainty, forensic reports should be expressed in terms of a range of plausible likelihood ratios rather than a single likelihood ratio (e.g., [8,9]). Likewise, there have been strong debates over what we might call " reporting standards": documents that instruct forensic practitioners how they should report results. But what if practitioners do not actually report in the way that disciplinary standards say they should? In contrast to the robust scholarship on how forensic scientists should report, less attention has been paid to how they do report. Accordingly, our study focuses on the following key research questio...

Surveying practicing firearm examiners

“…Cole and Barno [ 15 ] examined expert reports and transcripts regarding friction ridge prints (91 reports), firearms and toolmarks (48), questioned documents (52), and shoeprint comparisons (381 reports from CTS), obtained from a variety of sources (e.g., a search of Westlaw for expert materials), in the United States. Of the 48 firearm and toolmark reports, 28 (58%) reported an identification, 14 (29%) reported an inconclusive, and 6 (13%) reported an exclusion.…”

“…All of the firearm and toolmark examiner reports followed the AFTE range of conclusions (i.e., identification, inconclusive, or exclusion). Cole and Barno [ 15 ] concluded that they overall “found relatively few probabilistic reports. The probabilistic reports that we did find were almost entirely 'Inconclusive' reports which, by their very nature, were always coded as probabilistic (p. 412).”…”

Advancing a paradigm shift in evaluation of forensic evidence: The rise of forensic data science

“…Curran [ 1 ] lamented that only 13% of laboratories surveyed used the likelihood-ratio framework for glass evidence , but this may be one of the highest rates of adoption after DNA. In many other branches of forensic science the rate of adoption of the likelihood-ratio framework by practitioners is near zero (Bali et al [ 48 ]; Cole & Barno [ 49 ]).…”

Latent print quality in blind proficiency testing: Using quality metrics to examine laboratory performance

“…If a latent print is searched in AFIS, examiners subsequently make one of three conclusions depending on the AFIS candidate list outcome 4 and further examination. Examiners may conclude that there is No Association, meaning the latent print does not appear to correspond to any print on the AFIS candidate list.…”

“…For instance, inconclusive determinations on low quality prints are likely to be more "accurate" than inconclusive determinations for higher quality prints. The ambiguity about how best to evaluate these inconclusive decisions also speaks to the importance of shifting away from categorical outcomes generally and towards reporting conclusions probabilistically, though the practice is not currently the norm in latent print comparisons [4].…”