Signal detection theory fails to account for real-world consequences of inconclusive decisions

Sinha, Maneka; Gutierrez, Richard E

doi:10.1093/lpr/mgad001

Cited by 6 publications

(8 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One study found that the same examiners examining the same prints on two different occasions reach different conclusions about 10% of the time ( 28 ). There is also marked variation from study to study, and across disciplines, in the percentage of comparisons that pattern-matching examiners deem to be inconclusive ( 29 ). Hence, as noted earlier, it is possible, perhaps even likely, that forensic examiners’ decision thresholds in casework will differ from their thresholds when participating in black-box studies.…”

Section: Discussionmentioning

confidence: 99%

Shifting decision thresholds can undermine the probative value and legal utility of forensic pattern-matching evidence

Thompson

2023

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Forensic pattern analysis requires examiners to compare the patterns of items such as fingerprints or tool marks to assess whether they have a common source. This article uses signal detection theory to model examiners’ reported conclusions (e.g., identification, inconclusive, or exclusion), focusing on the connection between the examiner’s decision threshold and the probative value of the forensic evidence. It uses a Bayesian network model to explore how shifts in decision thresholds may affect rates and ratios of true and false convictions in a hypothetical legal system. It demonstrates that small shifts in decision thresholds, which may arise from contextual bias, can dramatically affect the value of forensic pattern-matching evidence and its utility in the legal system.

show abstract

Section: Discussionmentioning

confidence: 99%

Shifting decision thresholds can undermine the probative value and legal utility of forensic pattern-matching evidence

Thompson

2023

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…In doing so, they concealed from outside observers their knowledge that the reference sample did not match the evidence sample. Taken together with our signal detection analysis, there appears to be mounting evidence that there exists a subsample of biased examiners whose reports do not reflect their actual beliefs (Sinha & Gutierrez, 2022).…”

Section: Examiners Know That They Are Calling Nonmatches Inconclusivementioning

confidence: 68%

Telling us less than what they know: Expert inconclusive reports conceal exculpatory evidence in forensic cartridge-case comparisons.

Smith,

Wells

2024

Journal of Applied Research in Memory and Cognition

View full text Add to dashboard Cite

Cartridge-case comparison experts are tasked with determining whether a reference sample associated with the suspect originated from the same source as the evidence sample found at the crime scene. Same-source reports have strong inculpatory value and different-source reports have strong exculpatory value. Surprisingly, inconclusive reports have strong exculpatory value as they occur much more frequently for actual nonmatches than for actual matches. We used a signal detection model to test our hypothesis that some experts report inconclusive when they detect a nonmatch. Consistent with this hypothesis, we found that even when examiners were able to perfectly discriminate matches from nonmatches, they rendered inconclusive reports on 32% of nonmatch trials. Experts are biased to avoid rendering different-source reports, which conceals exculpatory information that innocent persons desperately need to establish their innocence. We argue that this biased examiner might result from an adversarial allegiance bias combined with a flawed response scale.

show abstract

“…This debate has not always favored the field of firearms examination. Scholars have highlighted a host of flaws they have viewed as undercutting the validity of even the best designed accuracy studies, including (but not limited to), potentially strategic and erroneous deployment of inconclusive responses by participants, declared rather than blind administration, inadequate/ unrepresentative sampling of participants and comparison items, high rates of participant attrition, and de minimus data transparency [8,9,[20][21][22][23][24][25][26]. And a growing number of courts have excluded identification conclusions and evidence regarding the comparison of "individual" characteristics outright [27][28][29][30][31][32].…”

Section: Introductionmentioning

confidence: 99%

The false promise of firearms examination validation studies: Lay controls, simplistic comparisons, and the failure to soundly measure misidentification rates

Gutierrez,

Prokesch

2024

Journal of Forensic Sciences

View full text Add to dashboard Cite

Several studies have recently attempted to estimate practitioner accuracy when comparing fired ammunition. But whether this research has included sufficiently challenging comparisons dependent upon expertise for accurate conclusions regarding source remains largely unexplored in the literature. Control groups of lay people comprise one means of vetting this question, of assessing whether comparison samples were at least challenging enough to distinguish between experts and novices. This article therefore utilizes such a group, specifically 82 attorneys, as a post hoc control and juxtaposes their performance on a comparison set of cartridge case images from one commonly cited study (Duez et al. in J Forensic Sci. 2018;63:1069–1084) with that of the original participant pool of professionals. Despite lacking the kind of formalized training and experience common to the latter, our lay participants displayed an ability, generally, to distinguish between cartridge cases fired by the same versus different guns in the 327 comparisons they performed. And while their accuracy rates lagged substantially behind those of the original participant pool of professionals on same‐source comparisons, their performance on different‐source comparisons was essentially indistinguishable from that of trained examiners. This indicates that although the study we vetted may provide useful information about professional accuracy when performing same‐source comparisons, it has little to offer in terms of measuring examiners' ability to distinguish between cartridge cases fired by different guns. If similar issues pervade other accuracy studies, then there is little reason to rely on the false‐positive rates they have generated.

show abstract

Signal detection theory fails to account for real-world consequences of inconclusive decisions

Cited by 6 publications

References 11 publications

Shifting decision thresholds can undermine the probative value and legal utility of forensic pattern-matching evidence

Shifting decision thresholds can undermine the probative value and legal utility of forensic pattern-matching evidence

Telling us less than what they know: Expert inconclusive reports conceal exculpatory evidence in forensic cartridge-case comparisons.

The false promise of firearms examination validation studies: Lay controls, simplistic comparisons, and the failure to soundly measure misidentification rates

Contact Info

Product

Resources

About