Repeatability and reproducibility of comparison decisions by firearms examiners

Abstract

In a comprehensive study to assess various aspects of the performance of qualified forensic firearms examiners, volunteer examiners compared both bullets and cartridge cases fired from three different types of firearms. They rendered opinions on each comparison according to the Association of Firearm & Tool Mark Examiners (AFTE) Range of Conclusions, as Identification, Inconclusive (A, B, or C), Elimination, or Unsuitable. In this part of the study, comparison sets used previously to characterize the overall accuracy of examiners were blindly resubmitted to examiners to assess the repeatability (105 examiners; 5700 comparisons of bullets and cartridge cases) and reproducibility (191 examiners of bullets, 193 of cartridge cases; 5790 comparisons) of firearms examinations. Data gathered using the prevailing AFTE Range were also recategorized into two hypothetical scoring systems. Consistently positive differences between observed agreement and expected agreement indicate that the repeatability and reproducibility of examiners exceed chance agreement. When averaged over bullets and cartridge cases, the repeatability of comparison decisions (involving all five levels of the AFTE Range) was 78.3% for known matches and 64.5% for known nonmatches. Similarly averaged reproducibility was 67.3%% for known matches and 36.5% for known nonmatches. For both repeatability and reproducibility, many of the observed disagreements were between a definitive and inconclusive category. Examiner decisions are reliable and trustworthy in the sense that identifications are unlikely when examiners are comparing non‐matching items, and eliminations are unlikely when they are comparing matching items.

The most consistent finding in forensic science is inconsistency

“…Specifically, in firearms for example "when averaged over bullets and cartridge cases, the repeatability of comparison decisions (involving all levels of the AFTE Range) was 78.3% for known matches and 64.5% for known nonmatches. Similarly averaged reproducibility was 67.3% for known matches and 36.5% for known nonmatches" [14].…”

Authors' response

“…We appreciate the opportunity to respond to comments on our report on the repeatability and reproducibility of conclusions by firearms examiners [1]. Although disparaged as "hardly reassuring" by Dorfman and Valliant (DV) [2], our reported overall repeatability of 78.3% for known matches and 64.5% for known nonmatches, averaged over bullets and cartridge cases, compared with rates reported in other studies involving subjective comparisons by firearms and other forensic examiners, and to those in nonforensic fields [1,[3][4][5].…”

Commentary on: Monson KL, Smith ED, Peters EM. Repeatability and reproducibility of comparison decisions by firearms examiners. J Forensic Sci. 2023;68(5):1721–40. https://doi.org/10.1111/1556‐4029.15318

“…AL concluded there was ample evidence for consistency. Monson et al [1] ("MSP") expand on the AL results, with the same major conclusion: "…plots of observed vs. expected proportion of agreement are evidence of the consistency that can be expected from examiners in evaluating cartridge case and bullet sets." ( [1], p. 17).…”

“…The direct measures reported by MSP of repeatability, when averaged over bullets and cartridge cases using all five levels of the AFTE range, were 78.3% for known matches and 64.5% for known nonmatches ( [1], p. 1). In other words, examiners' two evaluations of the same materials were different in over 20% of pairs for matches and over 35% of pairs for nonmatches.…”

“…The astute guessing model that DV uses to assess values of kappa does not fit the AL data. "Examiners did not know the ratio of matching vs. non-matching specimens…which 'astute guessing' would require…but even had they known…expected agreements always exceed 'astute guessing'" ([1], p. 17)Response: (i) The text of DV makes clear that the astute guessing model was introduced to gain some clarity on the meaning in general of Expected Agreement and Cohen's kappa; there is no claim that examiners were acting in accord with the model. (ii)…”