Testing for Calibration Discrepancy of Reported Likelihood Ratios in Forensic Science

Hannig, Jan; Iyer, Hari

doi:10.1111/rssa.12747

Cited by 7 publications

(5 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The calibration of LR systems is a means to assess the performance of LR systems and avoid underestimating or overestimating the strength of the evidence. In a more general context, researchers and practitioners continue discussing the need for calibrating likelihood ratios and evaluating the optimal approaches for calibration (Hannig & Iyer, 2022; Malmborg & Nordgaard, 2022; Ramos et al, 2013; Ramos et al, 2021; Vergeer et al, 2021; Vergeer, Alberink, et al, 2020). Malmborg et al described SAILR, an open‐source software designed to calculate forensic likelihood ratios from probability distributions of reference data.…”

Section: The Current State Of Methodologies For Forensic Glass Examin...mentioning

confidence: 99%

Forensic glass examinations—A review focused on elemental spectrochemical analysis

Trejos

2022

WIREs Forensic Science

View full text Add to dashboard Cite

Glass is a trace material commonly found at crime scenes that can provide valuable information at early investigative stages and during a trial. The forensic analysis of glass has steadily evolved since the 1970s with numerous technological advances in spectroscopy and spectrometry. This article presents an advanced review of the recent literature concerning the forensic examination of glass evidence and discusses the current state and future opportunities of the discipline. The review focuses on established elemental spectrochemical techniques, including Inductively Coupled Plasma methods (ICP), laser‐ablation, and x‐ray Fluorescence (XRF) methods, and newer applications using Laser Induced Breakdown Spectroscopy (LIBS), Raman Spectroscopy, and accelerator and reactor nuclear techniques. A brief overview of the transfer, persistence, interpretation, and analytical schemes implemented in the field is revised to provide context to the discussion of the capabilities and limitations of the current and emerging technology. This article is categorized under: Forensic Chemistry and Trace Evidence > Trace Evidence

show abstract

Section: The Current State Of Methodologies For Forensic Glass Examin...mentioning

confidence: 99%

Forensic glass examinations—A review focused on elemental spectrochemical analysis

Trejos

2022

WIREs Forensic Science

View full text Add to dashboard Cite

show abstract

“…Fiducial calibration discrepancy plots provide information for regions of LR values (intervals) on whether evidence is overstated and/ or understated. Furthermore, they provide a magnitude and direction of discrepancy between reported LRs and the empirically supported values for well-calibrated LRs, and provide an approach for obtaining confidence bands for the calibration discrepancy [31].…”

Section: Fiducial Calibration Discrepancy Plotsmentioning

confidence: 99%

“…Fiducial calibration discrepancy plots and PAV plots make this data dependence very explicit by only showing performance for the log 10 LR ranges in which LR values exist for both H p -true and H d -true scenarios [30, 31, 32]. Beyond these limits, there are insufficient data to evaluate calibration performance [31]. In theory, ECE plots and calibration tables indicate calibration across the full range of LRs [27, 29], yet these metrics are still limited by dataset size.…”

Section: Introductionmentioning

confidence: 99%

“…Different metrics have been proposed to assess calibration, which vary in use across different forensic disciplines [24, 25, 26]. Examples are Tippett plots [27], Turing tests or non-contributor performance tests [28], calibration tables [18, 29], Empirical Cross-Entropy (ECE) plots [27, 30, 31], log likelihood ratio cost (Cllr and Cllr cal ) [30, 32], pool adjacent violators (PAV) plots, devPav [30] and fiducial calibration discrepancy plots [31]. However, except for Tippett plots and calibration tables, very few studies have reported on these metrics for PG systems [26].…”

Section: Introductionmentioning

confidence: 99%

“…For example, to ascertain that the proportion of LRs equal to a 10 6 under H d is around 1/10 6 , we need around a million H d -true LRs. Fiducial calibration discrepancy plots and PAV plots make this data dependence very explicit by only showing performance for the log 10 LR ranges in which LR values exist for both H p -true and H d -true scenarios [30, 31, 32]. Beyond these limits, there are insufficient data to evaluate calibration performance [31].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

‘Low’ LRs obtained from DNA mixtures: On calibration and discrimination performance of probabilistic genotyping software

McCarthy-Allen,

Bleka,

Ypma

et al. 2024

Preprint

View full text Add to dashboard Cite

The validity of a probabilistic genotyping (PG) system is typically demonstrated by following international guidelines for the developmental and internal validation of PG software. These guidelines mainly focus on discriminatory power. Very few studies have reported with metrics that depend on calibration of likelihood ratio (LR) systems. In this study, discriminatory power as well as various calibration metrics, such as Empirical Cross-Entropy (ECE) plots, pool adjacent violator (PAV) plots, log likelihood ratio cost (Cllr and Cllrcal), fiducial calibration discrepancy plots, and Turing’ expectation were examined using the publicly-available PROVEDIt dataset. The aim was to gain deeper insight into the performance of a variety of PG software in the ‘lower’ LR ranges (∼LR 1-10,000), with focus on DNAStatistX and EuroForMix which use maximum likelihood estimation (MLE). This may be a driving force for the end users to reconsider current LR thresholds for reporting. In previous studies, overstated ‘low’ LRs were observed for these PG software. However, applying (arbitrarily) high LR thresholds for reporting wastes relevant evidential value. This study demonstrates, based on calibration performance, that previously reported LR thresholds can be lowered or even discarded. Considering LRs >1, there was no evidence for miscalibration performance above LR ∼1,000 when using Fst 0.01. Below this LR value, miscalibration was observed. Calibration performance generally improved with the use of Fst 0.03, but the extent of this was dependent on the dataset: results ranged from miscalibration up to LR ∼100 to no evidence of miscalibration alike PG software using different methods to model peak height, HMC and STRmix.This study demonstrates that practitioners using MLE-based models should be careful when low LR ranges are reported, though applying arbitrarily high LR thresholds is discouraged. This study also highlights various calibration metrics that are useful in understanding the performance of a PG system.HighlightsDiscriminatory power and calibration performance of PG software are evaluated.The utility of various calibration metrics are explored in ‘low’ LR ranges.Focus was on DNAStatistX and EuroForMix software using the MLE method.Calibration performance was dependent on Fst value and dataset size.Results suggest reconsideration of lower LR thresholds and cautious reporting of ‘low’ LRs.

show abstract

Fine alignment, flexible fusion: A novel framework of multi-model score fusion in face identification

Xie,

Li,

Wang

et al. 2024

Forensic Science International

View full text Add to dashboard Cite

Testing for Calibration Discrepancy of Reported Likelihood Ratios in Forensic Science

Abstract: The use of likelihood ratios for quantifying the strength of forensic evidence in criminal cases is gaining widespread acceptance in many forensic disciplines.

Cited by 7 publications

References 55 publications

Forensic glass examinations—A review focused on elemental spectrochemical analysis

Forensic glass examinations—A review focused on elemental spectrochemical analysis

‘Low’ LRs obtained from DNA mixtures: On calibration and discrimination performance of probabilistic genotyping software

Fine alignment, flexible fusion: A novel framework of multi-model score fusion in face identification

Contact Info

Product

Resources

About