Exact sequential test for clinical trials and post‐market drug and vaccine safety surveillance with Poisson and binary data

Silva, Ivair R.; Maro, Judith C.; Kulldorff, Martin

doi:10.1002/sim.9094

Cited by 7 publications

(11 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the second approach, to adjust for multiple testing when testing the same hypothesis sequentially we apply maximum sequential probability ratio testing (MaxSPRT) to all methods by computing the log likelihood ratio (LLR) as well as a critical value for the observed power and alpha of 0.05 ( Kulldorff et al, 2011 ). Critical values were computed using the “Sequential” R package version 3.3.1, using the Poisson model for the historical comparator design, and the binomial model for all other designs ( Silva and Kulldorff, 2021 ).…”

Section: Methodsmentioning

confidence: 99%

Vaccine Safety Surveillance Using Routinely Collected Healthcare Data—An Empirical Evaluation of Epidemiological Designs

et al. 2022

View full text Add to dashboard Cite

Background: Routinely collected healthcare data such as administrative claims and electronic health records (EHR) can complement clinical trials and spontaneous reports to detect previously unknown risks of vaccines, but uncertainty remains about the behavior of alternative epidemiologic designs to detect and declare a true risk early.Methods: Using three claims and one EHR database, we evaluate several variants of the case-control, comparative cohort, historical comparator, and self-controlled designs against historical vaccinations using real negative control outcomes (outcomes with no evidence to suggest that they could be caused by the vaccines) and simulated positive control outcomes.Results: Most methods show large type 1 error, often identifying false positive signals. The cohort method appears either positively or negatively biased, depending on the choice of comparator index date. Empirical calibration using effect-size estimates for negative control outcomes can bring type 1 error closer to nominal, often at the cost of increasing type 2 error. After calibration, the self-controlled case series (SCCS) design most rapidly detects small true effect sizes, while the historical comparator performs well for strong effects.Conclusion: When applying any method for vaccine safety surveillance we recommend considering the potential for systematic error, especially due to confounding, which for many designs appears to be substantial. Adjusting for age and sex alone is likely not sufficient to address differences between vaccinated and unvaccinated, and for the cohort method the choice of index date is important for the comparability of the groups. Analysis of negative control outcomes allows both quantification of the systematic error and, if desired, subsequent empirical calibration to restore type 1 error to its nominal value. In order to detect weaker signals, one may have to accept a higher type 1 error.

show abstract

Section: Methodsmentioning

confidence: 99%

Vaccine Safety Surveillance Using Routinely Collected Healthcare Data—An Empirical Evaluation of Epidemiological Designs

et al. 2022

View full text Add to dashboard Cite

show abstract

“…With the tuning parametrization above, the optimal alpha spending was calculated with the function of the R Sequential package. 20 We provide the details of the results in Table 2. The first column shows the number of tests.…”

Section: Illustrative Applicationmentioning

confidence: 99%

“…In the sixth column, C V shows the signaling threshold given in the scale of the number of events associated with the optimal alpha spending, which in turn were run using the function of the R Sequential package. 20…”

Section: Illustrative Applicationmentioning

confidence: 99%

“…The “Variable Bernoulli probabilities” section states the theorem for optimal group sequential analysis under variable Bernoulli probabilities. Using the R Sequential package, 20 the “Illustrative application” section presents an illustrative application using the real data of the 2009–2010 pandemic H1N1 inactivated influenza vaccine. The “Last comments” section has the concluding remarks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Bounded-width confidence interval following optimal sequential analysis of adverse events with binary data

Silva

Zhuang

2022

Stat Methods Med Res

Self Cite

View full text Add to dashboard Cite

In sequential testing with binary data, sample size and time to detect a signal are the key performance measures to optimize. While the former should be optimized in Phase III clinical trials, minimizing the latter is of major importance in post-market drug and vaccine safety surveillance of adverse events. The precision of the relative risk estimator on termination of the analysis is a meaningful design criterion as well. This paper presents a linear programming framework to find the optimal alpha spending that minimizes expected time to signal, or expected sample size as needed. The solution enables (a) to bound the width of the confidence interval following the end of the analysis, (b) designs with outer signaling thresholds and inner non-signaling thresholds, and (c) sequential designs with variable Bernoulli probabilities. To illustrate, we use real data on the monitoring of adverse events following the H1N1 vaccination. The numerical results are obtained using the R Sequential package.

show abstract

“…In this study, we evaluated whether the serial strategy (first applying a sensitive method and then a specific method) is useful for signal detection. Epidemiologists commonly use this method for exposure and outcome classification [10] [11]. A "gold standard" serial testing method would have 100% sensitivity and specificity [12].…”

Section: Introductionmentioning

confidence: 99%

Best of intent, worst of both worlds: why sequentially combining epidemiological designs does not improve signal detection in vaccine safety surveillance

Arshad

Schuemie

Minty

et al. 2022

Preprint

View full text Add to dashboard Cite

Background Vaccine safety surveillance commonly includes a serial testing approach with a sensitive method for 'signal generation' and specific method for 'signal validation.' Whether serially combining epidemiological designs improves both sensitivity and specificity is unknown. Methods We assessed the overall performance of serial testing using three administrative claims and one electronic health record database. We compared Type I and II errors before and after empirical calibration for historical comparator, SCCS, and the serial combination of those designs against six vaccine exposure groups with 93 negative control and 279 imputed positive control outcomes. Results Historical comparator mostly had lower Type II error than SCCS. SCCS had lower Type I error than the historical comparator. Before empirical calibration, serial combination increased specificity and decreased sensitivity. Type II errors mostly exceeded 50%. After empirical calibration, Type I errors returned to nominal; sensitivity was lowest when the methods were combined. Conclusion We recommend against the serial approach in vaccine safety surveillance. While serial combination produced fewer false positive signals compared to the most specific method, it generated more false negative signals compared to the most sensitive method. Using the noisy historical comparator in front of SCCS deteriorated overall performance in evaluating safety signals.

show abstract

Exact sequential test for clinical trials and post‐market drug and vaccine safety surveillance with Poisson and binary data

Cited by 7 publications

References 34 publications

Vaccine Safety Surveillance Using Routinely Collected Healthcare Data—An Empirical Evaluation of Epidemiological Designs

Vaccine Safety Surveillance Using Routinely Collected Healthcare Data—An Empirical Evaluation of Epidemiological Designs

Bounded-width confidence interval following optimal sequential analysis of adverse events with binary data

Best of intent, worst of both worlds: why sequentially combining epidemiological designs does not improve signal detection in vaccine safety surveillance

Contact Info

Product

Resources

About