In designing signal detection systems, careful consideration should be given to the criteria that are used to define an SDR. The choice of disproportionality statistic does not appreciably affect the achievable range of signal detection performance and so this can primarily be based on ease of implementation, interpretation and minimisation of computing resources. The changes in sensitivity and precision obtainable by replacing one algorithm with another are predictable. However, the absolute performance of a method is specific to the database and is best assessed directly on that database. New methods may be required to gain appreciable improvements.

Over a period of 5 years, the Innovative Medicines Initiative PROTECT (Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium) project has addressed key research questions relevant to the science of safety signal detection. The results of studies conducted into quantitative signal detection in spontaneous reporting, clinical trial and electronic health records databases are summarised and 39 recommendations have been formulated, many based on comparative analyses across a range of databases (e.g. regulatory, pharmaceutical company). The recommendations point to pragmatic steps that those working in the pharmacovigilance community can take to improve signal detection practices, whether in a national or international agency or in a pharmaceutical company setting. PROTECT has also pointed to areas of potentially fruitful future research and some areas where further effort is likely to yield less.

Introduction Signal validation in pharmacovigilance is the process of evaluating data to decide whether evidence is sufficient to justify further assessment of a detected signal. During the signal validation process, safety experts in our organization are required to review signals of disproportionate reporting (SDRs) and classify them into one of six predefined categories. Objective This experiment explored the extent to which predictive machine learning (ML) models can support the decision making of safety experts by accurately identifying the most appropriate predefined signal validation category. Methods We extracted cumulative data for six medicinal products, consisting of historic SDR validations and Individual Case Safety Reports, from the company’s safety database for training and testing of the ML model. We implemented a decision tree-based supervised multiclass classifier model termed Gradient Boosted Trees followed by a SHapley Additive exPlanations (SHAP) analysis to mitigate the “black box” effect of the ensemble model by identifying the key predicting features in the model. Following a retrospective analysis, a prospective experiment was conducted to test the model accuracy and user acceptance in a real-life setting. Results The prediction accuracy of our ML model ranged from 83 to 86% over 3 months for the six medicinal products. The applicability of the model was confirmed by the company’s safety experts. Additionally, the systematic predictions provided valuable information to the safety experts and assisted them in reviewing the SDRs efficiently and consistently. Conclusions This experiment demonstrated that it is possible to train a multiclass classification model to accurately predict signal validation categories for SDRs. More importantly, the transparency of the predictions provided by the SHAP analysis led to high acceptance by the safety experts.

IntroductionThe goal of signal detection in pharmacovigilance (PV) is to detect unknown causal associations between medicines and unexpected events. Statistical methods serve to detect signals and supplement traditional PV methods. Statistical signal detection (SSD) requires decisions about various settings that influence the quality and efficiency of SSD, as shown in several studies. To our knowledge, the effects of SSD periodicity and resignalling criteria on the quality and workload of routine SSD have not been published before.ObjectiveTo analyse the effects of different periodicities and resignalling criteria on signal detection quality and signal validation workload, and to test the impact of changing the signal threshold for number of cases.MethodsWe calculated signals of disproportionate reporting (SDRs) using thresholds of number of cases (N) ≥3, proportional reporting ratio ≥2 and Chi2 ≥ 4. We retrospectively simulated recurrent SDR calculation and validation with varying periodicity (quarterly vs. monthly), resignalling criteria, and N ≥ 3 vs. N ≥ 5.ResultsChanging the periodicity from quarterly to monthly increased the workload by 46.6 % (0 % signal loss). More restrictive resignalling criteria reduced the workload between 36.3 % (0 % signal loss) and 74.1 % (50 % signal loss). For N ≥ 3, the most efficient monthly SSD resignalling criterion reduced the workload by 36.3 % and detected all true signals earlier than quarterly SSD. N ≥ 5 reduced the workload between 13.8 and 21.4 % (0 % signal loss).ConclusionsIn real-life PV practice, signal detection and validation are recurrent periodic activities. Some true signals are only discovered upon resignalling. Our results demonstrate resignalling criteria with high signal detection quality and high efficiency. We found potential earlier detection of true signals using monthly SSD. Additional studies about resignalling should be performed to complement our findings.