On the reproducibility of science: unique identification of research resources in the biomedical literature

Abstract

Scientific reproducibility has been at the forefront of many news stories and there exist numerous initiatives to help address this problem. We posit that a contributor is simply a lack of specificity that is required to enable adequate research reproducibility. In particular, the inability to uniquely identify research resources, such as antibodies and model organisms, makes it difficult or impossible to reproduce experiments even where the science is otherwise sound. In order to better understand the magnitude of this problem, we designed an experiment to ascertain the “identifiability” of research resources in the biomedical literature. We evaluated recent journal articles in the fields of Neuroscience, Developmental Biology, Immunology, Cell and Molecular Biology and General Biology, selected randomly based on a diversity of impact factors for the journals, publishers, and experimental method reporting guidelines. We attempted to uniquely identify model organisms (mouse, rat, zebrafish, worm, fly and yeast), antibodies, knockdown reagents (morpholinos or RNAi), constructs, and cell lines. Specific criteria were developed to determine if a resource was uniquely identifiable, and included examining relevant repositories (such as model organism databases, and the Antibody Registry), as well as vendor sites. The results of this experiment show that 54% of resources are not uniquely identifiable in publications, regardless of domain, journal impact factor, or reporting requirements. For example, in many cases the organism strain in which the experiment was performed or antibody that was used could not be identified. Our results show that identifiability is a serious problem for reproducibility. Based on these results, we provide recommendations to authors, reviewers, journal editors, vendors, and publishers. Scientific efficiency and reproducibility depend upon a research-wide improvement of this substantial problem in science today.

Reproducibility: Standardize antibodies used in research

“…What is more, only 44% of publications provide enough information -for instance, on the supplierfor researchers to be able to purchase the same antibody 7 . The quality of the documentation that accompanies batches (such as on functionality in different assays) is enormously variable 2 ; even when it is provided, it may not correspond to the batch supplied 8 .…”

The Resource Identification Initiative: A cultural shift in publishing

“…However, as has long been bemoaned by database curators and documented recently by Vasilevsky and colleagues, these resources are not very well identified in the scientific literature 11 . Vasilevsky and colleagues found that researchers did not include sufficient detail for most key research resources to allow someone to identify conclusively an antibody, a genetically modified animal, or cell lines, for example.…”

Bridging Translation by Improving Preclinical Study Design in AKI

“…It is estimated that 51%-89% of published research cannot be reproduced in other laboratories. [18][19][20][21][22] This problem has been highlighted by a number of recent, high-profile commentaries from the pharmaceutical industry, 18,19,23 funding agencies, including the National Institutes of Health, 24,25 scientific journals, 26,27 veterinarians, 28 academics, and biostatisticians. [29][30][31][32][33][34][35][36] This is of particular importance when animal models are used to advance the development of clinical therapeutics.…”