Blending Bayesian and Classical Tools to Define Optimal Sample-Size-Dependent Significance Levels

Abstract

This article argues that researchers do not need to completely abandon the p-value, the best-known significance index, but should instead stop using significance levels that do not depend on sample sizes. A testing procedure is developed using a mixture of frequentist and Bayesian tools, with a significance level that is a function of sample size, obtained from a generalized form of the Neyman-Pearson Lemma that minimizes a linear combination of α, the probability of rejecting a true null hypothesis, and β, the probability of failing to reject a false null, instead of fixing α and minimizing β. The resulting hypothesis tests do not violate the Likelihood Principle and do not require any constraints on the dimensionalities of the sample space and parameter space. The procedure includes an ordering of the entire sample space and uses predictive probability (density) functions, allowing for testing of both simple and compound hypotheses. Accessible examples are presented to highlight specific characteristics of the new tests.

Moving to a World Beyond “p < 0.05”

“…They argue that this method has improved performance relative to use of dichotomized p-values alone. Gannon, Pereira, and Polpo (2019) have developed a testing procedure combining frequentist and Bayesian tools to provide a significance level that is a function of sample size.…”

Adaptive significance levels in linear regression models

“…Neste trabalho apresentamos um método para encontrar um valor de corte para a evidência no FBST minimizando a combinação linear das probabilidades ponderadas de erro tipo I e tipo II para um determinado tamanho de amostra e também para uma certa dimensão do espaço paramétrico. Além disso, comparamos nossa metodologia com os resultados obtidos usando o teste proposto por Pereira et al (2017) e Gannon et al (2019) que apresenta o P -value como medida de evidência e inclui um nível de significância adaptativo. Para isso,é considerado o cenário de modelos de regressão linear sob a abordagem Bayesiana.…”

“…Following the same line of work, Pereira et al (2017) and Gannon et al (2019) present a new hypothesis-testing procedure formulated from the ideas developed in previous works (Irony and Pereira, 1995;Montoya-Delgado et al, 2001;Pereira, 1985;Pereira and Wechsler, 1993) and using a mixture of frequentist and Bayesian tools. This procedure introduces the P -value as a decision-making evidence measure and also includes an adaptive significance level, i.e., a significance level that is function of sample size.…”

“…Such an adaptive significance level is obtained from the generalized form of the Neyman-Pearson Lemma where the linear combination of the type-I and type-II error probabilities is minimized. According to Gannon et al (2019), the resulting hypothesis tests do not violate the Likelihood Principle and do not require any constraints on the dimensionalities of the sample space and parameter space.…”

“…The main goal of our work is to determine how small the Bayesian evidence in the FBST should be in order to reject the null hypothesis. Therefore, taking into account the concepts in DeGroot (1986) and Pereira (1985) associated with optimal hypothesis tests, as well as the conclusions of Oliveira (2014) about the relationship between the significance levels and the sample size, and finally, considering the ideas developed recently by Pereira et al (2017) and Gannon et al (2019) related to adaptive significance levels, we present a method to find a cutoff point for the e-value by minimizing a linear combination of the averaged type-I and type-II error probabilities for a given sample size and also for a given dimensionality of the parameter space. For that purpose, the scenario of linear regression models under the Bayesian approach is considered.…”