Promoting Good Statistical Practices: Some Suggestions

Abstract

Researchers want to answer three basic questions: (a) Is an observed effect real or should it be attributed to chance? (b) If the effect is real, how large is it? and (c) Is the effect large enough to be useful? The first question concerning whether chance is a viable explanation for an observed effect is usually addressed with a null hypothesis significance test. A null hypothesis significance test tells us the probability of obtaining the effect or a more extreme effect if the null hypothesis is true. A significance test does not tell us how large the effect is or whether the effect is important or useful. Unfortunately, all too often the primary focus of research is on rejecting a null hypothesis and obtaining a small p value. The focus should be on what the data tell us about the phenomenon under investigation. This is not a new idea. Critics of significance testing have been saying it for years. For example, Frank Yates (1951), a contemporary of Ronald Fisher, observed that the use of the null hypothesis significance test has caused scientific research workers to pay undue attention to the results of the tests of significance they perform on their data, and too little to the estimates of the magnitude of the effects they are investigating. . . . The emphasis on tests of significance, and the consideration of the results of each experiment in isolation, have had the unfortunate consequence that scientific workers have often regarded the execution of a test of significance on an experiment as the ultimate objective. (pp. 32-33) The view that an emphasis on null hypothesis significance tests detracts researchers from the main business of science-interpreting the outcome of research, theory development, and so on-is shared by many contemporary

Fair Statistical Communication in HCI

“…p-values capture what is traditionally termed statistical significance, while effect sizes capture practical significance (Kirk, 2001). For example, the effect of pill 1 can be said to exhibit a high statistical significance, but only a moderate practical significance compared to others.…”

A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.

“…NHST addresses whether observed effects or relations stand out above sampling error by test statistic and its p-value, though it is not as useful for estimating the magnitude of these effects (Chow, 1996). The p-values estimate the probability of sample results deviating as much or more than do the actual sample results from those specified by the null hypothesis (Cohen, 1994;Kirk, 2001). The final outcome of NHST is the decision to reject or fail to reject null hypotheses; it meets the needs of research questions that do require a dichotomous answer.…”

Importancia del tamaño del efecto en el análisis de datos de investigación en psicología

“…Sobre el análisis de resultados y la interpretación que se hace de estos, Kirk (2001) plantea que a través de sus estudios los investigadores deben responderse a las siguientes tres preguntas: a) ¿existe un efecto real o los resultados deberían atribuirse al azar?, b) si el efecto es real, ¿qué tan grande es?, y c) ¿es el efecto lo suficientemente grande para considerarse importante o útil? De estas preguntas, el método del NHST solo puede responder la primera de ellas, e incluso esta respuesta sería muy limitada y no consistente con esta interpretación.…”

“…Esto, incluso, a pesar de que es ampliamente aceptado que el NHST es comúnmente mal usado y mal inter-pretado (Fidler, 2002). Esta situación se desarrolla entre expresiones de reforma que plantean la eliminación del uso del NHST (Schmidt, 1996), así como de hacer incorporación complementaria y obligatoria de los tamaños del efecto e intervalos de confianza (Kirk, 2001;Thompson, 1999a), esto para dar un mejor sentido e interpretación a los resultados.…”