The widespread misinterpretation of<i>p</i>-values as error probabilities

Hubbard, Raymond

doi:10.1080/02664763.2011.567245

“…1 By doing so, the p value is interpreted as an indicator for the strength of evidence against H 0 (or for H 1 ). This interpretation would be incorrect from a Neyman-Pearson perspective (Gigerenzer et al, 2004;Hubbard, 2011), but valid from a Fisherian perspective (Royall, 1997), which reflects the confusion in the literature about what p values are and what they are not.…”

Section: The Nhst Procedures With a Priori Power Analysis And Some Of mentioning

confidence: 99%

Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.

Schönbrodt

¹

,

Wagenmakers

²

,

Zehetleitner

³

et al. 2017

Psychological Methods

View full text Add to dashboard Cite

Unplanned optional stopping rules have been criticized for inflating Type I error rates under the null hypothesis significance testing (NHST) paradigm. Despite these criticisms this research practice is not uncommon, probably as it appeals to researcher's intuition to collect more data in order to push an indecisive result into a decisive region. In this contribution we investigate the properties of a procedure for Bayesian hypothesis testing that allows optional stopping with unlimited multiple testing, even after each participant. In this procedure, which we call Sequential Bayes Factors (SBF), Bayes factors are computed until an a priori defined level of evidence is reached. This allows flexible sampling plans and is not dependent upon correct e↵ect size guesses in an a priori power analysis. We investigated the long-term rate of misleading evidence, the average expected sample sizes, and the biasedness of e↵ect size estimates when an SBF design is applied to a test of mean di↵erences between two groups. Compared to optimal NHST, the SBF design typically needs 50% to 70% smaller samples to reach a conclusion about the presence of an e↵ect, while having the same or lower long-term rate of wrong inference.

show abstract

“…Result may be explained by the fact that Yates´s correction is treated as a continuity correction; consequently, it produces more conservative results. For smaller samples in particular the probability of rejecting the hypothesis H 0 is even greater when compared to other sample sizes; in fact, larger samples tend to produce smaller significance probabilities (Hubbard, 2011).…”

Section: Resultsmentioning

confidence: 99%

<b>Principal components in the discrimination of outliers: A study in simulation sample data corrected by Pearson's and Yates´s chi-square distance

Veloso

¹

,

Cirillo

²

2016

View full text Add to dashboard Cite

ABSTRACT. Current study employs Monte Carlo simulation in the building of a significance test to indicate the principal components that best discriminate against outliers. Different sample sizes were generated by multivariate normal distribution with different numbers of variables and correlation structures. Corrections by chi-square distance of Pearson´s and Yates's were provided for each sample size. Pearson´s correlation test showed the best performance. By increasing the number of variables, significance probabilities in favor of hypothesis H 0 were reduced. So that the proposed method could be illustrated, a multivariate time series was applied with regard to sales volume rates in the state of Minas Gerais, obtained in different market segments.Keywords: contaminated samples, Monte Carlo, significance test, p-value.Componentes principais na discriminação de outliers: estudo de simulação em dados amostrais corrigidos pelas distâncias qui-quadrado de Pearson's and Yates.RESUMO. Este trabalho tem por objetivo realizar um estudo, utilizando simulação Monte Carlo na construção de um teste de significância para indicar os componentes principais que melhor discriminam as discrepâncias. Neste contexto, diferentes tamanhos amostrais foram gerados pela distribuição normal multivariada com diferentes números de variáveis e estruturas de correlação. Para cada tamanho amostral, procedeu-se com as correções dadas pela distância qui-quadrado de Pearson e Yates. Concluiu-se ao considerar a correção de Pearson o teste apresentou melhor desempenho, entretanto, aumentando o número de variáveis as probabilidades de significância a favor a hipótese H 0 foram reduzidas. Por fim, para ilustrar a metodologia proposta realizou-se uma aplicação em uma série temporal multivariada referente a índices de volumes de vendas do estado de Minas Gerais obtidos em diferentes segmentos de mercados.

show abstract

“…This interpretation would be incorrect from a Neyman-Pearson perspective (Gigerenzer et al, 2004;Hubbard, 2011), but valid from a Fisherian perspective (Royall, 1997), which reflects the confusion in the literature about what p values are and what they are not.…”

Section: The Nhst Procedures With a Priori Power Analysis And Some Of mentioning

confidence: 93%

Sequential Hypothesis Testing with Bayes Factors: Efficiently Testing Mean Differences

Schönbrodt

¹

,

Wagenmakers

²

,

Zehetleitner

³

et al. 2015

View full text Add to dashboard Cite

Unplanned optional stopping rules have been criticized for inflating Type I error rates under the null hypothesis significance testing (NHST) paradigm. Despite these criticisms this research practice is not uncommon, probably as it appeals to researcher's intuition to collect more data in order to push an indecisive result into a decisive region. In this contribution we investigate the properties of a procedure for Bayesian hypothesis testing that allows optional stopping with unlimited multiple testing, even after each participant. In this procedure, which we call Sequential Bayes Factors (SBF), Bayes factors are computed until an a priori defined level of evidence is reached. This allows flexible sampling plans and is not dependent upon correct e↵ect size guesses in an a priori power analysis. We investigated the long-term rate of misleading evidence, the average expected sample sizes, and the biasedness of e↵ect size estimates when an SBF design is applied to a test of mean di↵erences between two groups. Compared to optimal NHST, the SBF design typically needs 50% to 70% smaller samples to reach a conclusion about the presence of an e↵ect, while having the same or lower long-term rate of wrong inference.

show abstract

The widespread misinterpretation ofp-values as error probabilities

Cited by 30 publications

References 26 publications

Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.

Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.

<b>Principal components in the discrimination of outliers: A study in simulation sample data corrected by Pearson's and Yates´s chi-square distance

Sequential Hypothesis Testing with Bayes Factors: Efficiently Testing Mean Differences

Contact Info

Product

Resources

About