Forensic Footwear Reliability: Part I—Participant Demographics and Examiner Agreement*

Abstract

In order to assess the extent of agreement between forensic footwear examiners in the United States, a reliability study was performed by West Virginia University between February 2017 and August 2018. Over the span of 19 months, 70 examiners each performed 12 comparisons and reported a total of 840 conclusions. For each comparison, participants were queried on a number of factors in order to determine the degree to which different types of features were identified, evaluated, and weighted, before arriving at a final decision regarding the strength of the association or disassociation between questioned and test impressions. Preliminary results from this study are divided into a series of three summaries. This manuscript (Part I) describes participant demographics as well as community agreement in both feature identification/annotation, and final reporting. Results indicate considerable variation in feature identification/annotation (as low as 66.5% agreement), but higher consistency in the reporting of overall conclusions. For mated pairs, this agreement was 79.7% AE 14.1% (median of 85.7% and a 90% confidence interval between 75.9% and 83.2%). For nonmated pairs, the equivalent overall agreement was 89.8% AE 6.69% (median of 91.4% and a 90% confidence interval between 87.4% and 92.1%). These estimates of agreement are further compared with previous published findings, and collectively, the work extends the body of knowledge concerning reliability in forensic footwear comparisons and conclusions.

Forensic Footwear Reliability: Part II—Range of Conclusions, Accuracy, and Consensus*

“…In Part I (1), community agreement was defined using the IQR. In comparison to the accepted range of conclusions, Fig.…”

“…Interestingly, the remainder of the conclusions were dissociative in nature, and in fact the exclusion decisions are considered statistical outliers (as was the single response of insufficient detail). Unfortunately, of the 15 examiners that reported inaccurately, only one marked features (see annotation map from the Part I summary [1]) which makes it difficult to determine the reason for the disassociations.…”

“…Participant demographics are fully characterized in Part I (1). In total, 115 forensic footwear examiners were recruited, and 77 submitted results (resulting in a 67% response rate).…”

“…The details of each simulated case are also fully described in Part I (1), but for convenience, Table 1 repeats limited information concerning the shoes, substrates, media, and processing techniques used to create case materials. Note that across all seven cases, participants were required to perform 12 pairwise comparisons and that each case consisted of 1200 PPI digital and print imagery, comprised of a single questioned impression, one‐two exemplars (outsoles), and two Handiprint replicate exemplars per known shoe.…”

“…The goals of this project were several‐fold, including an evaluation of accuracy, consensus, and reproducibility in feature identification, feature evaluation, and overall conclusions regarding source attribution. In terms of organization, initial results are dispersed in three manuscripts; Part I (1) described the reproducibility in feature identification, feature evaluation, and generalized community agreement, while Part II (this summary) reports on examiner accuracy and reproducibility (or consensus) of conclusions. Within the context of this study, accuracy is defined according to the President’s Council of Advisors on Science and Technology (PCAST) 2016 report, or the known probability (or frequency) at which “an examiner obtains correct results both (1) for samples from the same source (true positives) and (2) samples from different sources (true negatives)” (2).…”