Accuracy of comparison decisions by forensic firearms examiners

Abstract

This black box study assessed the performance of forensic firearms examiners in the United States. It involved three different types of firearms and 173 volunteers who performed a total of 8640 comparisons of both bullets and cartridge cases. The overall false-positive error rate was estimated as 0.656% and 0.933% for bullets and cartridge cases, respectively, while the rate of false negatives was estimated as 2.87% and 1.87% for bullets and cartridge cases, respectively. The majority of errors were made by a limited number of examiners. Because chi-square tests of independence strongly suggest that error probabilities are not the same for each examiner, these are maximum-likelihood estimates based on the beta-binomial probability model and do not depend on an assumption of equal examiner-specific error rates. Corresponding 95% confidence intervals are (0.305%, 1.42%) and (0.548%, 1.57%) for false positives for bullets and cartridge cases, respectively, and (1.89%, 4.26%) and (1.16%, 2.99%) for false negatives for bullets and cartridge cases, respectively. The results of this study are consistent with prior studies, despite its comprehensive design and challenging specimens.

Authors' response to Gutierrez et al Commentary on

“…It is not possible in any field to guarantee that a selected individual practitioner is proficient but extant data indicate that most firearms examiners are proficient, based on the convenience samples we must rely on. Our study, together with numerous other studies involving different examiners and test materials, shows that the majority of examiners can perform firearms comparisons with high accuracy [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19]. We reported that most errors were committed by a minority of examiners: 6 examiners account for ~30% of all errors and 13 account for ~50% of errors.…”

“…We thank the correspondents for reviewing instances of poor comparison performance that we reported for a fraction of the firearms examiners participating in our accuracy study [1].…”

Commentary on: Monson KL, Smith ED, Peters EM. Accuracy of comparison decisions by forensic firearms examiners. J Forensic Sci. 2022;68(1):86–100. https://doi.org/10.1111/1556‐4029.15152.

“…However justified Monson et al may be in concluding that their results "add to the ever increasing body of empirical data that firearms examiners conduct comparisons with a high level of accuracy" [1]-and it may be that their conclusion cannot stand [4,6,20]-the data on poor performers in their study call into doubt whether opinions reached by firearms examiners can or should ever be admitted as evidence in any given criminal proceeding. The rules of evidence in both federal and state courts across the United States require judges, not only to assess the admissibility of particular methods, but also to evaluate, on a case-by-case basis, whether a given witness is "qualified" to testify about an accepted method because their knowledge or skills exceed those of everyday people, thereby allowing them to "assist the trier of fact" by accurately formulating opinions that would be beyond the capacity of lay jurors [14][15][16][17][18][19].…”

“…The authors of "Accuracy of comparison decisions by forensic firearms examiners" forthrightly note in their paper that "[e]rrors tended to be concentrated within a relatively small number of examiners" [1]. But if their results are, as those authors suggest, to "offer additional resources to the courts as they weigh the admissibility and value of firearms testimony," then further exploration and discussion of the raw data concerning poor-performing examiners is warranted.…”

“…This commentary must again use the caveating language of "at least" because of confusing discrepancies in the way that the authors of Monson et al report demographic data on accreditation. In the study under comment, those authors report that 18 of the total participants (including two who committed false positives) answered in the negative when asked whether they were employed by an accredited lab [1]. But this figure appears to contradict earlier claims by the same authors that "[p]articipation was limited to fully qualified examiners who were employed in the firearms section of an accredited public crime laboratory" [24], as well as data supplied in an early pre-publication of the study indicating that 170 of the 173 participants answered "yes" when asked "Is your laboratory accredited in firearms examination?"…”