Understanding of researcher behavior is required to improve data reliability

Abstract

Background A lack of data reproducibility (“reproducibility crisis”) has been extensively debated across many academic disciplines. Results Although a reproducibility crisis is widely perceived, conclusive data on the scale of the problem and the underlying reasons are largely lacking. The debate is primarily focused on methodological issues. However, examples such as the use of misidentified cell lines illustrate that the availability of reliable methods does not guarantee good practice. Moreover, research is often characterized by a lack of established methods. Despite the crucial importance of researcher conduct, research and conclusive data on the determinants of researcher behavior are widely missing. Conclusion Meta-research that establishes an understanding of the factors that determine researcher behavior is urgently needed. This knowledge can then be used to implement and iteratively improve measures that incentivize researchers to apply the highest standards, resulting in high-quality data.

Effect size, sample size and power of forced swim test assays in mice: Guidelines for investigators to optimize reproducibility

“…We suggest that investigators are not able to perceive intuitively whether or not a given sample size is adequate for a given experiment, and this contributes to a mindset that is skeptical of theoretical or statistical arguments. Apart from other educational and institutional reforms [ 2 , 3 , 10 , 11 , 13 , 20 , 22 , 38 , 44 ], altering the real-life behavior of scientists in planning their experiments may require developing tools that allow them to actively visualize the inter-relationships among effect size, sample size, statistical power, and replicability in a direct and intuitive manner.…”

Effect size, sample size and power of forced swim test assays in mice: Guidelines for investigators to optimize reproducibility

“…We suggest that investigators are not able to perceive intuitively whether or not a given sample size is adequate for a given experiment, and this contributes to a mindset that is skeptical of theoretical or statistical arguments. Apart from other educational and institutional reforms [2, 3, 10, 11, 13, 19, 21, 37, 41], altering the real-life behavior of scientists in planning their experiments may require developing tools that allow them to actively visualize the inter-relationships among effect size, sample size, statistical power, and replicability in a direct and intuitive manner.…”

Data variability in standardised cell culture experiments

“…In total, 52,585 compounds were tested in the NCI60 resulting in 2.8 x 10 6 compound/cell line combinations. Two or more (up to 2,286) experiments were carried out for 11,841 compounds and 594,450 compound/cell line combinations ( Figure 1B, Extended Data Table 1; Extended Data Table 2). More than 100 experiments in at least one cell line were performed for 18 compounds and more than 1,000 experiments for two compounds (Extended Data Table 3).…”

“…In the life sciences, there is a crisis narrative and a perception of a lack of data reproducibility ("reproducibility crisis" or "replication crisis") [2,[5][6][7][8][9][10]. However, the actual scale of the crisis remains unclear and evidence is largely anecdotal [2].…”