Dataset shift assessment measures in monitoring predictive models

Becker, Aneta; Becker, Jarosław

doi:10.1016/j.procs.2021.09.112

Cited by 8 publications

(3 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It will seem obvious that the evaluation of the logarithm in Eq. ( 5) is impossible or meaningless when either f 0,i = 0 or f 1,i = 0, which is what occurs when a bin or category is un-populated in either the original or new population, respectively, as also noted by [2]. In practice, this may happen fairly often, depending on the use case for the PSI.…”

Section: The Unstable Population Indicatormentioning

confidence: 98%

See 1 more Smart Citation

Measuring data drift with the unstable population indicator1

Haas,

Sibbald

2024

View full text Add to dashboard Cite

Measuring data drift is essential in machine learning applications where model scoring (evaluation) is done on data samples that differ from those used in training. The Kullback-Leibler divergence is a common measure of shifted probability distributions, for which discretized versions are invented to deal with binned or categorical data. We present the Unstable Population Indicator, a robust, flexible and numerically stable, discretized implementation of Jeffrey’s divergence, along with an implementation in a Python package that can deal with continuous, discrete, ordinal and nominal data in a variety of popular data types. We show the numerical and statistical properties in controlled experiments. It is not advised to employ a common cut-off to distinguish stable from unstable populations, but rather to let that cut-off depend on the use case.

show abstract

Section: The Unstable Population Indicatormentioning

confidence: 98%

“…This requires a quantitative measure of data drift (also known as data shift or concept drift) between populations. This drift can, and if possible should be measured on both predictor and target values, as noted by [2,14]. Such a measure should be well-behaved, i.e.…”

Section: R Haas and L Sibbald / The Unstable Population Indicatormentioning

confidence: 99%

Measuring data drift with the unstable population indicator1

Haas,

Sibbald

2024

View full text Add to dashboard Cite

show abstract

“…They particularly recommended the multivariate PAI (MPAI) over the univariate version that considers only one explanatory variable at a time. Becker and Becker (2021) provided further examples where the PSI and PAI produce different results; however, they only considered the univariate version.…”

Section: Introductionmentioning

confidence: 99%

Investigating Causes of Model Instability: Properties of the Prediction Accuracy Index

Taplin

2023

Risks

View full text Add to dashboard Cite

The Prediction Accuracy Index (PAI) monitors stability, defined as whether the predictive power of a model has deteriorated due to a change in the distribution of the explanatory variables since its development. This paper shows how the PAI is related to the Mahalanobis distance, an established statistic for examining high leverage observations in data. This relationship is used to explore properties of the PAI, including tools for how the PAI can be decomposed into effects due to (a) individual observations/cases, (b) individual variables, and (c) shifts in the mean of variables. Not only are these tools useful for practitioners to help determine why models fail stability, but they also have implications for auditors and regulators. In particular, reasons why models containing econometric variables may fail stability are explored and suggestions to improve model development are discussed.

show abstract