Do we know what we test and do we test what we want to know?

Abstract

Null hypothesis testing (NHT) is the most commonly used tool in empirical psychological research even though it has several known limitations. It is argued that since the hypotheses evaluated with NHT do not reflect the research-question or theory of the researchers, conclusions from NHT must be formulated with great modesty, that is, they cannot be stated in a confirmative way. Since confirmation or theory evaluation is, however, what researchers often aim for, we present an alternative approach that is based on the specification of explicit, informative statistical hypotheses. The statistical approach for the evaluation of these hypotheses is a Bayesian model-selection procedure. A non-technical explanation of the Bayesian approach is provided and it will be shown that results obtained with this method give more direct answers to the questions asked and are easier to interpret. An additional advantage of the offered possibility to formulate and evaluate informative hypotheses is that it stimulates researchers to more carefully think through and specify their expectations.

Bayesian benefits with JASP

“…When the data are consistent with the restriction, the more daring hypothesis should be rewarded, receiving a bonus for parsimony (e.g. Jefferys & Berger, 1992;Klugkist, van Wesel, & Bullens, 2011;Lee & Wagenmakers, 2013, Chapter 7;Myung & Pitt, 1997;Vanpaemel, 2010). The prior distribution directly affects the model predictions (i.e.…”

Relationship between social anxiety and perceived trustworthiness

“…However, for the present purposes, it is not the overall probability of one single H 0 that is of interest (see also Cohen, 1994). Instead, a test of the different models provides a more comprehensive and elaborate answer of how likely or unlikely a specific response pattern is for a given dataset (Klugkist, van Wesel, & Bullens, 2011). Accordingly, the most appropriate analytic strategy is to test the four response patterns by converting them into informative hypotheses (Hoijtink, 2012), which implement the specific constraints on the parameters of interest.…”

Foreign language reading and spelling in gifted students with dyslexia in secondary education

“…The BF can also be interpreted as a measure of effect size (i.e., BF 1–3 = small, BF 3–10 = medium, BF > 10 = large; Kass & Raftery, 1995). Competing hypotheses can be compared based on the posterior model probability (PMP), representing the relative support for a specific hypothesis within a set of hypotheses (Klugkist et al, 2011). In general, a difference between PMPs that is smaller than .05 cannot be interpreted and results in equal support for both models.…”