When null hypothesis significance testing is unsuitable for research: a reassessment

Szűcs, Dénes; Ioannidis, John P. A.

doi:10.1101/095570

Cited by 71 publications

(108 citation statements)

References 168 publications

(255 reference statements)

Supporting

Mentioning

107

Contrasting

Order By: Relevance

“…The ASA statement recommended: 'No single index should substitute for scientific reasoning' (10) -a viewpoint shared by other prominent investigators (45,46). In particular, Ioannidis and colleagues recently stressed monocultural training of biomedical scientists in statistical null-hypothesis testing as one reason behind frequent misuses of statistical methods (47).…”

Section: Discussionmentioning

confidence: 99%

Prediction and inference diverge in biomedicine: Simulations and real-world data

Bzdok¹,

Engemann²,

Grisel³

et al. 2018

Preprint

View full text Add to dashboard Cite

In the 20 th century many advances in biological knowledge and evidence-based medicine were supported by p-values and accompanying methods. In the beginning 21 st century, ambitions towards precision medicine put a premium on detailed predictions for single individuals. The shift causes tension between traditional methods used to infer statistically significant group differences and burgeoning machine-learning tools suited to forecast an individual's future. This comparison applies the linear model for identifying significant contributing variables and for finding the most predictive variable sets. In systematic data simulations and common medical datasets, we explored how statistical inference and pattern recognition can agree and diverge. Across analysis scenarios, even small predictive performances typically coincided with finding underlying significant statistical relationships. However, even statistically strong findings with very low p-values shed little light on their value for achieving accurate prediction in the same dataset. More complete understanding of different ways to define 'important' associations is a prerequisite for reproducible research findings that can serve to personalize clinical care. Keywords: scientific discovery | data science | variable importance | intelligent algorithms | reproducibility'Change your statistical philosophy and all of a sudden different things become important'

show abstract

Section: Discussionmentioning

confidence: 99%

Prediction and inference diverge in biomedicine: Simulations and real-world data

Bzdok¹,

Engemann²,

Grisel³

et al. 2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Historically, this type of deductive reasoning has often drawn on null hypothesis significance testing (NHST). The framework however is sometimes ill-suited and frequently misunderstood [17][18][19]. As an alternative to NHST, one may draw formal inference by means of false discovery rate (FDR), Bayesian posterior inference, or other tools [1, ch.…”

Section: Exploration Inference Prediction: a Typology Of Different mentioning

confidence: 99%

Exploration, Inference, and Prediction in Neuroscience and Biomedicine

Bzdok

Ioannidis

2019

Trends in Neurosciences

168

132

View full text Add to dashboard Cite

The last decades saw dramatic progress in brain research. These advances were often buttressed by probing single variables to make circumscribed discoveries, typically through null hypothesis significance testing. New ways for generating massive data fueled tension between the traditional methodology, used to infer statistically relevant effects in carefully-chosen variables, and pattern-learning algorithms, used to identify predictive signatures by searching through abundant information. In this article, we detail the antagonistic philosophies behind two quantitative approaches: certifying robust effects in understandable variables, and evaluating how accurately a built model can forecast future outcomes. We discourage choosing analysis tools via categories like 'statistics' or 'machine learning'. Rather, to establish reproducible knowledge about the brain, we advocate prioritizing tools in view of the core motivation of each quantitative analysis: aiming towards mechanistic insight, or optimizing predictive accuracy. The emergence of richer datasets alters everyday data-analysis practicesThere is a burgeoning controversy in neuroscience on what data analysis should be about.Similar to many other biomedical disciplines, there are differing perspectives among researchers, clinicians, and regulators about the best approaches to make sense of these unprecedented data resources. Traditional statistical approaches, such as null hypothesis significance testing, were introduced in a time of data scarcity and have been revisited, revised, or even urged to be abandoned. Currently, a growing literature advertises predictive pattern-learning algorithms hailed to provide some traction on the data deluge [2,3]. Such modeling tools for prediction are increasingly discussed in particular fields of neuroscience [for some excellent sources see: 4, 5-9].Ensuing friction is aggravated by the incongruent historical trajectories of mainstream statistics and emerging pattern-learning algorithms -the former long centered on significance testing procedures to obtain p-values, the latter with a stronger heritage in computer science [10][11][12]. We argue here that the endeavor of sorting each analysis tool into categories like 'statistics' or 'machine-learning' is futile [13,14].Take for instance ordinary linear regression, as it is routinely applied by many neuroscientists. The same tool and its underlying mathematical prosthetics can be used to achieve three diverging goals [15, pp. 82-83, 16, ch. 4.12]: a) exploration, to get a first broad impression of the dependencies between a set of measured variables in the data at hand, b) inference, to discern which particular input variables contribute to the target variable beyond chance level, and c) prediction, to enable statements about how well target variables can be guessed based on data measured in the future.Confusion can arise because it is the motivation for using linear regression that differs between these scenarios. The mathematical mechanics underlying model parameter fitting a...

show abstract

“…In contrast to conventional model fitting, Bayesian analysis allows for simple calculations of credible intervals (referred to as confidence intervals in the results) on derived parameters (e.g., parameters that are mathematical functions of fitted coefficients), offers simple construction of realistic hierarchical models (Gelman et al, 2013) and avoids a number of problems with conventional p-values derived from null hypothesis significance testing (Szucs and Ioannidis, 2017). Another advantage of Bayesian analysis is that it determines the probability that the model coefficients take on a particular value or range of values (McMillan and Cannon, 2019).…”

Section: Discussionmentioning

confidence: 99%

Optimizing auditory brainstem response acquisition using interleaved frequencies

Buran

Elkins

Kempton

et al. 2019

Preprint

View full text Add to dashboard Cite

Auditory brainstem responses (ABRs) require averaging responses to hundreds or thousands of repetitions of a stimulus (e.g., tone pip) to obtain a measurable evoked response at the scalp. Fast repetition rates lead to changes in ABR amplitude and latency due to adaptation. To minimize the effect of adaptation, stimulus rates are sometimes as low as 10 to 13.3/s, requiring long acquisition times. The trade-off between reducing acquisition time and minimizing the effect of adaptation on ABR responses is an especially important consideration for studies of cochlear synaptopathy which use the amplitude of short latency responses (wave 1) to assess auditory nerve survival. It has been proposed that adaptation during ABR acquisition can be reduced by interleaving tones at different frequencies, rather than testing each frequency serially. With careful ordering of frequencies and levels in the stimulus train, adaptation in the auditory nerve can be minimized, thereby permitting an increase in the rate at which tone bursts are presented. However, widespread adoption of this paradigm has been hindered by lack of available software. Here, we develop and validate an interleaved stimulus paradigm to optimize the rate of ABR measurement while minimizing adaptation. We implement this method in an open-source data acquisition software tool, which permits either serial or interleaved ABR measurements. The software library, psiexperiment, is compatible with widely-used ABR hardware. Consistent with previous studies, careful design of an interleaved stimulus train can reduce ABR acquisition time by more than half, with minimal effect on ABR thresholds and wave 1 latency, while improving measures of wave 1 amplitude.Keywords ABR · auditory brainstem response · ABR optimization · wave amplitude · tone burst 1 Introduction Auditory brainstem responses (ABRs) are an essential tool for assessing peripheral auditory function in research animals and diagnosing auditory dysfunction

show abstract

When null hypothesis significance testing is unsuitable for research: a reassessment

Cited by 71 publications

References 168 publications

Prediction and inference diverge in biomedicine: Simulations and real-world data

Prediction and inference diverge in biomedicine: Simulations and real-world data

Exploration, Inference, and Prediction in Neuroscience and Biomedicine

Optimizing auditory brainstem response acquisition using interleaved frequencies

Contact Info

Product

Resources

About