Evaluation of different tests based on observations for external model evaluation of population analyses

Abstract

To evaluate by simulation the statistical properties of normalized prediction distribution errors (NPDE), prediction discrepancies (pd), standardized prediction errors (SPE), numerical predictive check (NPC) and decorrelated NPC (NPC(dec)) for the external evaluation of a population pharmacokinetic analysis, and to illustrate the use of NPDE for the evaluation of covariate models. We assume that a model M(B) has been built using a building dataset B, and that a separate validation dataset, V is available. Our null hypothesis H(0) is that the data in V can be described by M(B). We use several methods to test this hypothesis: NPDE, pd, SPE, NPC and NPC(dec). First, we evaluated by simulation the type I error under H(0) of different tests applied to the four methods. We also propose and evaluate a single global test combining normality, mean and variance tests applied to NPDE, pd and SPE. We perform tests on NPC and NPC(dec), after a decorrelation. M(B) was a one compartment model with first order absorption (without covariate), previously developed from two phase II and one phase III studies of the antidiabetic drug, gliclazide. We simulated 500 external datasets according to the design of a phase III study. Second, we investigated the application of NPDE to covariate models. We propose two approaches: the first approach uses correlation tests or mean comparisons to test the relationship between NPDE and covariates; the second evaluates NPDE split by category for discrete covariates or quantiles for continuous covariates. We generated several validation datasets under H(0) and under alternative assumptions with a model without covariate, with one continuous covariate (weight), or one categorical covariate (sex). We calculated the powers of the different tests using simulations, where the covariates of the phase III study were used. The simulations under H(0) show a high type I error for the different tests applied to SPE and an increased type I error for pd. The different tests present a type I error close to 5% for the global test appied to NPDE. We find a type I error higher than 5% for the test applied to classical NPC but this test becomes close to 5% for NPC(dec). For covariate models, when model and validation dataset are consistent, type I error of the tests are close to 5% for both effects. When validation datasets and models are not consistent, the tests detect the correlation between NPDE and the covariate. We recommend to use NPDE over SPE for external model evaluation, since they do not depend on an approximation of the model and have good statistical properties. NPDE represent a better approach than NPC, since in order to perform tests on NPC, a decorrelation step must be applied before. NPDE, in this illustration, is also a good tool to evaluate model with or without covariates.

Association between Vancomycin Trough Concentration and Area under the Concentration-Time Curve in Neonates

“…A total of 2,000 bootstrap data sets were generated from the original data set by repeated sampling with replacement, and the final pharmacokinetic model was used to estimate the model parameters for each data set. In addition, the final pharmacokinetic model was assessed using an internal evaluation procedure by computing the normalized prediction distribution errors (NPDE) of 5,000 simulated data sets compared to those of the observed data set (16,17).…”

A New Exact Test for the Evaluation of Population Pharmacokinetic and/or Pharmacodynamic Models Using Random Projections

“…For WRES and CWRES, residuals may not be homoscedastic and centred on zero as a consequence of the approximation of the model. This approximation can be very crude for WRES, which makes WRES a very poor diagnostic tool, as has also been reported by previous studies (9,11,14). By construction, PWRES are expected to be homoscedastic and centred on zero; however, the non-normality of the data and their dependency within subjects might create artificial patterns (e.g.…”

“…The problem is that no classical test can be applied due to the data dependency within subjects, e.g. a major increase in type I errors (around 13%) was reported for the exact binomial test applied to VPCs (14).…”

“…This point has been rightly discussed by the authors (16), but the consequences on model evaluation have not been thoroughly investigated. Therefore, although NPDE represent a major improvement over weighted residuals, both in theory and in practice (11,14,16), they might not be an optimal tool for the objective assessment of population PK/PD models.…”

Development of a Complex Parent-Metabolite Joint Population Pharmacokinetic Model

“…For the internal model evaluation before the addition of the genetic effect, we performed visual predictive check plots where the 90% confidence intervals around the 5th, 50th, and 95th prediction percentiles from 250 simulated datasets were overlaid to the 5th, 50th, and 95th percentiles of the observed data binned using the theoretical sampling times (27). Then after the addition of the genetic effect, we computed the normalized prediction distribution errors (npde), i.e., the observation percentiles within the empirical distribution obtained from the model simulations, decorrelated and normalized using the inverse function of the normal cumulative density function (28).…”