Vaccine safety surveillance using routinely collected healthcare data – An empirical evaluation of epidemiological designs

Abstract

Background: Routinely collected healthcare data such as administrative claims and electronic health records (EHR) can complement clinical trials and spontaneous reports when ensuring the safety of vaccines, but uncertainty remains about what epidemiological design to use. Methods: Using 3 claims and 1 EHR database, we evaluate several variants of the case-control, comparative cohort, historical comparator, and self-controlled designs against historical vaccinations with real negative control outcomes (outcomes with no evidence to suggest that they could be caused by the vaccines) and simulated positive controls. Results: Most methods show large type 1 error, often identifying false positive signals. The cohort method appears either positively or negatively biased, depending on the choice of comparator index date. Empirical calibration using effect-size estimates for negative control outcomes can restore type 1 error to close to nominal, often at the cost of increasing type 2 error. After calibration, the self-controlled case series (SCCS) design shows the shortest time to detection for small true effect sizes, while the historical comparator performs well for strong effects. Conclusions: When applying any method for vaccine safety surveillance we recommend considering the potential for systematic error, especially due to confounding, which for many designs appears to be substantial. Adjusting for age and sex alone is likely not sufficient to address the differences between vaccinated and unvaccinated, and for the cohort method the choice of index date plays an important role in the comparability of the groups Inclusion of negative control outcomes allows both quantification of the systematic error and, if so desired, subsequent empirical calibration to restore type 1 error to its nominal value. In order to detect weaker signals, one may have to accept a higher type 1 error.

Best of intent, worst of both worlds: why sequentially combining epidemiological designs does not improve signal detection in vaccine safety surveillance

“…For example, Li et al recently found that historical comparisons had Type II errors ranging between 0% to 10% and Type I errors above 30% [18]. Age and sex adjustment reduced Type I error, had little impact on Type II error, and improved precision in some cases [16]. Historical rate comparison was, overall, good at identifying true safety signals [16].…”

“…Age and sex adjustment reduced Type I error, had little impact on Type II error, and improved precision in some cases [16]. Historical rate comparison was, overall, good at identifying true safety signals [16]. Based on Li et al's results, we evaluated the age and sex adjusted variant for historical comparator, using a time-at-risk of 1-28 days after the historic visit, for both first and second doses.…”

“…To assess the hypothesis that serial testing is favorable, we chose one epidemiological design suspected to be highly sensitive but not highly specific (historical comparator) [16]. Historical comparator compares the observed incidence of adverse events following immunization (AEFI) with the expected incidence of AEFI, estimated from an unexposed patient population, often before or "historical" to vaccine introduction.…”

“…Specifically, we evaluated performance metrics when the historical comparator design (shown elsewhere to be relatively sensitive) [15] was followed by SCCS (shown elsewhere to be relatively specific) [16], using negative and imputed positive control outcomes in patients receiving historic vaccines across four national administrative claims and EHR databases [17].…”

“…It remains unknown whether and how using a serial strategy in vaccine surveillance based on available data improves the overall sensitivity and specificity of signal detection and validation, and therefore generates fewer false positive and false negative population-level signals. Specifically, we evaluated performance metrics when the historical comparator design (shown elsewhere to be relatively sensitive) [15] was followed by SCCS (shown elsewhere to be relatively specific) [16], using negative and imputed positive control outcomes in patients receiving historic vaccines across four national administrative claims and EHR databases [17].…”