Comparison of Statistical Signal Detection Methods Within and Across Spontaneous Reporting Databases

Abstract

In designing signal detection systems, careful consideration should be given to the criteria that are used to define an SDR. The choice of disproportionality statistic does not appreciably affect the achievable range of signal detection performance and so this can primarily be based on ease of implementation, interpretation and minimisation of computing resources. The changes in sensitivity and precision obtainable by replacing one algorithm with another are predictable. However, the absolute performance of a method is specific to the database and is best assessed directly on that database. New methods may be required to gain appreciable improvements.

What Is the Plural of a ‘Yellow’ Anecdote?

“…However, in practice, the PRR and ROR give similar results [7], and in the evaluation by Candore et al [5], they had virtually identical properties. The PRR is slightly more easily understood by non-statisticians and can be calculated in cases where no estimate of the ROR is obtainable, but where this is so, the data are so sparse than useful conclusions based on the PRR are not easy to obtain.…”

“…A previous paper [5] showed that SDRs typically have a positive predictive value (referred to as 'precision' in that paper) of less than 30 %, and that the different statistical methods can be tuned to give similar results by the choice of the cut-off point to define an SDR. The PRR is simple, but this does not mean that it should be used generally in signal detection, nor will it uniquely be associated with the RR estimated in more rigorous studies.…”

Is the Yellow Card Road Going in the Right Direction?

“…Both the Bayesian methods (and shrinkage applied to the reporting odds ratio) tested here shrink the strength of a signal based on small numbers of observations towards a null (no signal) value, while the frequentist methods have cut-off values that take small observed numbers into account, both directly by not counting as a signal disproportionate values based on small numbers and indirectly through the use of a confidence interval or the essentially equivalent Chi-squared value. Figure 6 in the study by Candore et al [5] clearly shows that varying the cut-off criteria can result in almost any value of precision and sensitivity that can be obtained by any method but using just a single method (proportional reporting ratio in this case). There will be a penalty in some instances in terms of having a longer delay in first detecting an SDR with higher thresholds, but the converse is true in that varying the criteria can obtain earlier detection if desired, as shown in Fig.…”

“…The paper by Candore et al [5] is an evaluation of five methods, different, to some degree, in principle, with multiple cut-off criteria to define a 'signal', in seven different databases. The databases included specific companies, a specific country, and two international databases.…”

“…There will be a penalty in some instances in terms of having a longer delay in first detecting an SDR with higher thresholds, but the converse is true in that varying the criteria can obtain earlier detection if desired, as shown in Fig. 7 of the study by Candore et al [5].…”