Some suggestions for measuring predictive performance

Abstract

The performance of a prediction or measurement model is often evaluated by computing the correlation coefficient and/or the regression of predictions on true (reference) values. These provide, however, only a poor description of predictive performance. The mean square prediction error (precision) and the mean prediction error (bias) provide better descriptions of predictive performance. These quantities are easily computed, and can be used to compare prediction methods to absolute standards or to one another. The measures, however, are unreliable when the reference method is imprecise. The use of these measures is discussed and illustrated.

A study of the feasibility and accuracy of pharmacokinetically guided etoposide dosing in children

“…The daily target AUC for each patient was defined by the equation: where K = [5'Cr]EDTA elimination rate constant and peak concentration is the measured day-I end of infusion etoposide concentration (Lowis et al, 1993 The dose administered on day 3 was calculated such that the total dose after 3 days was identical to that specified by the protocol. For each patient, the difference between the day-2 target and measured AUC was calculated, and the mean error (ME) and root mean squared error (RMSE) used as measurements of bias and precision respectively (Sheiner and Beal, 1981). The measured AUC on day 2 was calculated by fitting a compartmental model to the etoposide concentration-time data, using software kindly supplied by D'Argenio and Schumitzky (1979), as described previously (Lowis et al, 1993).…”

Individual and population pharmacokinetic compartment analysis: a graphic procedure for quantification of predictive performance

“…Quantification of the predictive performance by the fraction of Predicted/Observed drug concentration ratios within specified ranges is not sensitive to outliers in contrast to evaluation of precision by RMSE% and bias by MPE%17,18. The squaring process used by Sheiner and Beal5 may also produce non-normal distribution of errors with possible bias in percentage prediction error, which might violate the assumptions of standard parametric statistical procedures17,18.…”

“…Original data from previously published pharmacokinetic compartment analyses after intravenous, oral, and epidural administration8–10 were used for estimating the predictive performance according to the proposed graphical method and by the method of Sheiner and Beal5. Digitized data, obtained from published scatter plots of observed vs predicted drug concentrations from population pharmacokinetic studies using the NPEM algorithm and NONMEM computer program and Bayesian forecasting procedures, were also included in the present study11–13.…”

“…Precision expressed as percentage root mean square prediction error (RMSE%) and bias, expressed as percentage mean prediction error (MPE%), were calculated as outlined by Sheiner and Beal5 using the Microsoft Excel spreadsheet program (Microsoft Excel 97, SR1, Microsoft Corporation, Redmond, WA).…”

“…The percentage root mean square prediction error (RMSE%) has been suggested as a measure of precision and the percentage mean prediction error (MPE%) as a measure of bias in pharmacokinetic analysis5. Neither RMSE% nor MPE% are, however, indicators for time- or concentration-dependent inaccuracies.…”