Empirical bayes screening for multi-item associations

Abstract

This paper considers the franlework of the so-called "market basket problem", in which a database of transactions is mined for the occurrence of unusually frequent item sets. h~ our case, "unusually frequent" involves estimates of the frequency of each item set divided by a baseline frequency computed as if items occurred independently. The focus is on obtaining reliable estimates of this measure of interestingness for all item sets, even item sets with relatively low frequencies. For example, in a medical database of patient histories, unusual item sets including the item "patient death" (or other serious adverse event) might hopefully be flagged with as few as 5 or 10 occurrences of" the item set, it being unacceptable to require that item sets occur in as many as 0.1% of millions of patient reports before the data mining algorithm detects a signal. Similar considerations apply in fraud detection applications.Thus we abandon the requirement that interesting item sets must contain a relatively large fixed minimal support, and adopt a criterion based on the results of fitting an empirical Bayes model to the item set counts. The model allows us to define a 95% Bayesian lower confidence limit for the "interestingness" measure of every item set, whereupon the item sets can be ranked according to their empirical Bayes confidence limits. For item sets of size J > 2, we also distinguish between muhi-item associations that can be explained by the observed J(J-l)12 pairwise associations, and item sets that are significantly more frequent than their pairwise associations would suggest. Such item sets can uncover complex or synergistic mechanisms generating multi-item associations. This methodology has been applied within the U.S. Food and Drug Administration (FDA) to databases of adverse drug reaction reports and within AT&T to customer international calling histories. We also present graphical techniques for exploring and understanding the modeling results.

arules- A Computational Environment for Mining Association Rules and Frequent Item Sets

“…For example, it would be very useful to interface algorithms which use statistical measures to find "interesting" itemsets (which are not necessarily frequent itemsets as used in an association rule context). Such algorithms include implementations of the χ 2 -test based algorithm by Silverstein, Brin, and Motwani (1998) or the baseline frequency approach by DuMouchel and Pregibon (2001).…”

Quantitative signal detection for vaccines: Effects of stratification, background and masking on GlaxoSmithKline’s spontaneous reports database

“…MGPS is an innovative data-mining algorithm developed by William DuMouchel which introduces the concept of refining the relative reporting ratio calculation by using Bayesian statistics. [19][20][21] The basic assumption behind MGPS is that each observed count (N; drug/vaccine-event combination) is taken from a Poisson distribution with unknown mean (μ), with an interest center on the ratio λ which equals μ divided by E (the expected count estimated by assuming that the count of all reports for the specific drug/vaccine and the count of all reports for the specific event are independent; Table 1). The essential Bayesian contribution supposes that each λ is drawn from a common prior assumed to be a mixture of two g distributions.…”

“…The EB05 is a conservative measure that is supposed to minimize false positive signals. 20 Drug-event combinations with an EB05 ≥2.0 are frequently considered signals for drug-adverse events pairs based on the empiric threshold described by Szarfman et al 14 In addition, results of a simulation study 22 provided empirical evidence that using this threshold provides a degree of conservatism in exchange for the false signal rate in the data.…”

Checkpoint evolution for volatile correlation computing

“…Traditional association-rule mining algorithms (Agrawal et al 1993;Bayardo and Agrawal 1999) are found to be yielding many spurious patterns (Brin et al 1997;Xiong et al 2006Xiong et al , 2008. As a result, in recent years, many statistical correlation measures, such as χ 2 statistics (Brin et al 1997;DuMouchel and Pregibon 2001;Jermaine 2001Jermaine , 2003), Pearson's correlation coefficients (Xiong et al 2004(Xiong et al , 2006Zhou and Xiong 2008), rank-based correlation coefficients (Melucci 2007;Yilmaz et al 2008), and mutual information (Ke et al 2006(Ke et al , 2007 have been considered in the setting of large-scale association analysis.…”

“…Brin et al 1997;Jermaine 2001;DuMouchel and Pregibon 2001;Jermaine 2003;Ilyas et al 2004;Xiong et al 2004Xiong et al , 2006, researchers and practitioners are still facing increasing challenges to measure associations among data objects produced by emerging data-intensive applications, particularly when the data are dynamic and analytical results need to be continually updated. Indeed, with such large and growing data sets, research efforts are needed to develop a dynamic solution for volatile correlation computing.…”