Good Signal Detection Practices: Evidence from IMI PROTECT

Abstract

Over a period of 5 years, the Innovative Medicines Initiative PROTECT (Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium) project has addressed key research questions relevant to the science of safety signal detection. The results of studies conducted into quantitative signal detection in spontaneous reporting, clinical trial and electronic health records databases are summarised and 39 recommendations have been formulated, many based on comparative analyses across a range of databases (e.g. regulatory, pharmaceutical company). The recommendations point to pragmatic steps that those working in the pharmacovigilance community can take to improve signal detection practices, whether in a national or international agency or in a pharmaceutical company setting. PROTECT has also pointed to areas of potentially fruitful future research and some areas where further effort is likely to yield less.

An evaluation of statistical approaches to postmarketing surveillance

“…To address this concern, the European Innovative Medicines Initiative (IMI) PROTECT project recently completed an evaluation of several of these methods (ie, empirical Bayesian geometric mean [EBGM], Bayesian confidence propagation neural network [BCPNN], proportional reporting ratios [PRR], reporting odds ratios [ROR], urn) and made recommendations for their use in clinical studies, longitudinal observational data, and ICSR databases . This investigation identified major challenges related to the thresholds used to detect signals.…”

“…That is, “The EudraVigilance Expert Working Group noted that thresholds commonly used to detect signals in spontaneous data are a trade‐off between two conflicting goals: either generating too many false positive signals if the threshold is too low or missing true signals if this threshold is too high. Given this trade‐off, it is important to identify and calibrate methods to strike a reasonable balance between these two parameters.” Although this evaluation was very extensive, it did not include a more recently developed approach using a likelihood ratio test (LRT)‐based method and the FDR‐based extensions of the aforementioned methods. In addition, comparisons among these methods are based on using data from different databases.…”

Signal detection on a patient cohort: A disproportionality analysis of the ANRS CO22 HEPATHER cohort to identify associations between direct acting antivirals and adverse events in patients with hepatitis C virus chronic infection

“…To address this issue, spontaneous pharmacovigilance reporting systems have been set up to record incidental serious or unexpected AEs, and numerous analytical methods have been developed to link these events with current or past medication intake . Analyses of healthcare databases with thousands of patients exposed to a drug, eventually compared with thousands of unexposed patients, also allow the identification of associations between a drug and an AE, ie, “a signal.” Among various signal detection methods, disproportionality (DP) methods are commonly applied to spontaneous reporting systems or healthcare databases . However, in some of these systems, and particularly in healthcare databases, the timeliness of data collection is a strong limitation when one wants rapid identification of a signal for a drug recently released on the market.…”

A prediction model‐based algorithm for computer‐assisted database screening of adverse drug reactions in the Netherlands

“…Lareb has criteria in place for assessors to determine which reports should be discussed at the weekly scientific meeting. However, because multiple assessors are involved in this process and the selection of reports for the weekly scientific is prone to some level of subjectivity, a computer‐assisted database screening tool is in place as an additional approach to reduce the risk for missing potential signals . The screening tool is even more important for ADRs reported by marketing authorization holders (MAHs) that may be indicative of potential signals, as these are not assessed on a case‐by‐case basis at Lareb.…”

“…The computer‐assisted database screening tool used in the Netherlands relies on the number of reports of drug‐ADR associations and disproportionality based on the reporting odds ratio (ROR). With the disproportionality analyses, the observed rate of a drug and ADR together is compared with an expected value based on their relative frequencies reported individually in the spontaneous reporting database . In the approach applied at our centre, the lower limit of the 2‐sided 95% confidence interval (CI) is used combined with a number of at least 3 reports per association.…”