Shifting decision thresholds can undermine the probative value and legal utility of forensic pattern-matching evidence

Abstract

Forensic pattern analysis requires examiners to compare the patterns of items such as fingerprints or tool marks to assess whether they have a common source. This article uses signal detection theory to model examiners’ reported conclusions (e.g., identification, inconclusive, or exclusion), focusing on the connection between the examiner’s decision threshold and the probative value of the forensic evidence. It uses a Bayesian network model to explore how shifts in decision thresholds may affect rates and ratios of true and false convictions in a hypothetical legal system. It demonstrates that small shifts in decision thresholds, which may arise from contextual bias, can dramatically affect the value of forensic pattern-matching evidence and its utility in the legal system.

Implicit Blackstone ratios in decisions made by firearm and toolmark examiners.

“…While some inconsistency may be expected in practical contexts, an order of magnitude difference is unacceptable. To the extent that the data from Guyll et al’s (2023) non-test-blind study faithfully reflect how examiners conduct casework (Koehler, 2013), a questionable assumption (Thompson, 2023), the findings here reaffirm concerns that firearm examiners following the Association of Firearm and Toolmark Examiners Range of Conclusions fail to apply consistent standards (Baldwin et al, 2023).…”

Unpacking Workplace Stress and Forensic Expert Decision-Making: From Theory to Practice

“…In turn, this combination of factors (decisional, internal, and external) could affect examiners' available cognitive resources and their decision thresholds in forensic pattern-matching tasks (e.g., comparing cartridge cases, handwriting samples, etc) or forensic recognition tasks (e.g., recognizing bloodstain patterns, examining/triaging items from crime scenes, etc.). As Thompson (2023) recently explained, even seemingly minor shifts in examiners' decision thresholds can have a significant impact on their performance: "small shifts in forensic examiners' decision thresholds can dramatically affect their error rates and the probative value of their evidence, which can in turn affect the accuracy of the legal system. For example, small reductions in the threshold for identification, which might plausibly arise from an examiner's exposure to taskirrelevant information [which may include workplace stressors], can dramatically increase the risk of convicting an innocent person."…”

“…For that reason, differences in examiners' dispositional aversions to ambiguity and risk-which could be amplified under stressful conditions-may predict how often they reach inconclusive decisions. This is a critical point because, again, even very small fluctuations in an examiner's decision threshold can dramatically affect their performance (i.e., decision accuracy; Thompson, 2023).…”

Science, evidence, law, and justice

“…A persistent counterargument is that the additional information afforded by context contributes to more accurate decisions ( 23 – 25 ). Employing a Bayesian network model of the forensic decision process, Bill Thompson’s research article in this collection examines the impact of varying decision thresholds on probabilities of true and false convictions ( 26 ). Thompson’s thoughtful analysis proves that use of lower decision thresholds, induced by task-irrelevant information, can markedly increase—not decrease—the risk of convicting an innocent person.…”

“…They also offer means to identify and precisely assess specific factors that influence examiner performance. Tools for rigorous predictive modeling—Bayesian inference, multivariate regression, and neural networks—have also entered the field with much promise, as illustrated, for example, by Thompson’s research article in this collection ( 26 ), highlighted above. These powerful tools also pose considerable risk, as Brandon Garrett and Cynthia Rudin argue in their essay for this collection (see below) ( 36 ).…”