A blind quality control (QC) program was successfully developed and implemented in the Toxicology, Seized Drugs, Firearms, Latent Prints (Processing and Comparison), Forensic Biology, and Multimedia (Digital and Audio/Video) sections at the Houston Forensic Science Center (HFSC). The program was put into practice based on recommendations set forth in the 2009 National Academy of Sciences report and is conducted in addition to accreditation required annual proficiency tests. The blind QC program allows HFSC to test its entire quality management system and provides a real-time assessment of the laboratory's proficiency. To ensure the blind QC cases mimicked real casework, the workflow for each forensic discipline and their evidence submission processes were assessed prior to implementation. Samples are created and submitted by the HFSC Quality Division to whom the expected answer is known. Results from 2015 to 2018 show that of the 973 blind samples submitted, 901 were completed, and only 51 were discovered by analysts as being blind QC cases. Implementation data suggests that this type of program can be employed at other forensic laboratories.

Open proficiency tests meet accreditation requirements and measure examiner competence but may not represent actual casework. In December 2015, the Houston Forensic Science Center began a blind quality control program in firearms examination. Mock cases are created to mimic routine casework so that examiners are unaware they are being tested. Once the blind case is assigned to an examiner, the evidence undergoes microscopic examination and comparison to determine whether the fired evidence submitted was fired in the same firearm. Fifty‐one firearms blind cases resulting in 570 analysis and comparison determinations were reported between December 2015 and June 2021. No unsatisfactory results were obtained; however, 40.3% of comparisons in which the ground truth was either elimination or identification resulted in inconclusive conclusions. Due to the quality of some of the evidence submitted, inconclusive results were not unexpected. A ground truth of elimination and comparison result of inconclusive was observed at a rate of 74%, while a ground truth of identification and comparison result of inconclusive was observed at a rate of 31%. Bullets (61.8%) were the main contributors to inconclusive conclusions; variables such as the assigned examiners, training program, examiner experience, and the intended complexity of the case did not significantly contribute to the results. The program demonstrates that the quality management system and firearms section procedures can obtain accurate and reliable results and provides examiners added confidence in court. Additionally, the program can be tailored to target specific research questions and provide opportunities for collaboration with other laboratories and researchers.

Declared proficiency tests are limited in their use for testing the performance of the entire system, because analysts are aware that they are being tested. A blind quality control (BQC) is intended to appear as a real case to the analyst to remove any intentional or subconscious bias. A BQC program allows a real-time assessment of the laboratory’s policies and procedures and monitors reliability of casework. In September 2015, the Houston Forensic Science Center (HFSC) began a BQC program in blood alcohol analysis. Between September 2015 and July 2018, HFSC submitted 317 blind cases: 89 negative samples and 228 positive samples at five target concentrations (0.08, 0.15, 0.16, 0.20 and 0.25 g/100 mL; theoretical targets). These blood samples were analyzed by a headspace gas chromatograph interfaced with dual-flame ionization detectors (HS-GC-FID). All negative samples produced `no ethanol detected’ results. The mean (range) of reported blood alcohol concentrations (BACs) for the aforementioned target concentrations was 0.075 (0.073–0.078), 0.144 (0.140–0.148), 0.157 (0.155–0.160), 0.195 (0.192–0.200) and 0.249 (0.242–0.258) g/100 mL, respectively. The average BAC percent differences from the target for the positive blind cases ranged from −0.4 to −6.3%, within our uncertainty of measurement (8.95–9.18%). The rate of alcohol evaporation/degradation was determined negligible. A multiple linear regression analysis was performed to compare the % difference in BAC among five target concentrations, eight analysts, three HS-GC-FID instruments and two pipettes. The variables other than target concentrations showed no significant difference (P > 0.2). While the 0.08 g/100 mL target showed a significantly larger % difference than higher target concentrations (0.15–0.25 g/100 mL), the % differences among the higher targets were not concentration-dependent. Despite difficulties like gaining buy-in from stakeholders and mimicking evidence samples, the implementation of a BQC program has improved processes, shown methods are reliable and added confidence to staff’s testimony in court.

Calls for blind proficiency testing in forensic science disciplines intensified following the 2009 National Academy of Sciences report and were echoed in the 2016 report by the President's Council of Advisors on Science and Technology. Both practitioners and scholars have noted that "open" proficiency tests, in which analysts know they are being tested, allow for test-taking behavior that is not representative of behavior in routine casework. This study reports the outcomes of one laboratory's blind quality control (BQC) program. Specifically, we describe results from approximately 2.5 years of blind cases in the latent print section (N = 376 latent prints submitted as part of 144 cases). We also used a widely available quality metrics software (LQMetrics) to explore relationships between objective print quality and case outcomes. Results revealed that nearly all BQC prints (92.0%) were of sufficient quality to enter into AFIS. When prints had a source present in AFIS, 41.7% of print searches resulted in a candidate list containing the true source. Examiners committed no false positive errors but other types of errors were more common. Average print quality was in the midpoint of the range (53.4 on a 0-to-100 scale), though prints were evenly distributed across the Good, Bad, and Ugly categories. Quality metrics were significantly associated with sufficiency determinations, examiner conclusions, and examiner accuracy. Implications for blind testing and the use of quality metrics in routine casework as well as proficiency testing are discussed.

No abstract