Novel Statistical Tools for Management of Public Databases Facilitate Community-Wide Replicability and Control of False Discovery

Abstract

Issues of publication bias, lack of replicability and false discovery have long plagued the genetics community. Proper utilization of public and shared data resources presents an opportunity to ameliorate these problems. We present an approach to public database management that we term Quality Preserving Database (QPD). It enables perpetual use of the database for testing statistical hypotheses while controlling false discovery and avoiding publication bias on the one hand, and maintaining testing power on the other hand. We demonstrate it on a use case of a replication server for GWAS findings, underlining its practical utility. We argue that a shift to using QPD in managing current and future biological databases will significantly enhance the community's ability to make efficient and statistically sound use of the available data resources.The prevalence and importance of public databases and shared data resources in the genetics and molecular biology community have been constantly increasing, and this trend is expected to continue and accelerate with the growth in genetic data generation. For example, publicly available GWAS data sources like WTCCC (Wellcome Trust Case Control Consortium, 2010) and dbGaP (Mailman et al., 2007) have had a major contribution in genetic research well beyond the findings in the original studies which collected the data. An important class of data sharing efforts is the formation of scientific consortia for collection and analysis of genetic data, ranging in size from several collaborating scientists to entire research communities (Rich et al., 2007). The coming years are expected to see the formation of multi-purpose public genetic databases of unprecedented size and utility (Couzin-Frankel, 2012).

Online rules for control of false discovery rate and false discovery exceedance

“…Statistical databases. Concerned with the above issues and the importance of data sharing in the genetics community, [RAN14] proposed an approach to public database management, called Quality Preserving Database (QPD). A QPD makes a shared data resource amenable to perpetual use for hypothesis testing while controlling FWER and maintaining statistical power of the tests.…”

Noisy Hypotheses in the Age of Discovery Science

“…There are some initiatives in this direction [22]. Rosset et al [25], very recently, proposed a control via access to a large database. Note that this proposal requires the collection of new samples so that the database is continually enriched and FDR control remains possible.…”

Inferential Challenges for Real-world Evidence in the Era of Routinely Collected Health Data

“…Thus, there is an extremely large number of hypotheses that have been tested already and are yet-to-be tested by the community of researchers. 7 With hypotheses of this nature, community-wide multiplicity issues arise, which if unaddressed may lead to a high proportion of false discoveries. 7 Even though researchers do in fact account for withinstudy multiple testing, 6 multiplicity resulting from the number of hypotheses tested within the same database but in independent analyses is rarely accounted for.…”

“…7 With hypotheses of this nature, community-wide multiplicity issues arise, which if unaddressed may lead to a high proportion of false discoveries. 7 Even though researchers do in fact account for withinstudy multiple testing, 6 multiplicity resulting from the number of hypotheses tested within the same database but in independent analyses is rarely accounted for. The totality of tests performed by other researchers using the same database is typically ignored and each hypothesis is examined in isolation from previous hypotheses that are not part of the particular analysis.…”