Hypothesis Tests for Bernoulli Experiments: Ordering the Sample Space by Bayes Factors and Using Adaptive Significance Levels for Decisions

Abstract

Abstract:The main objective of this paper is to find the relation between the adaptive significance level presented here and the sample size. We statisticians know of the inconsistency, or paradox, in the current classical tests of significance that are based on p-value statistics that are compared to the canonical significance levels (10%, 5%, and 1%): "Raise the sample to reject the null hypothesis" is the recommendation of some ill-advised scientists! This paper will show that it is possible to eliminate this problem of significance tests. We present here the beginning of a larger research project. The intention is to extend its use to more complex applications such as survival analysis, reliability tests, and other areas. The main tools used here are the Bayes factor and the extended Neyman-Pearson Lemma.

Blending Bayesian and Classical Tools to Define Optimal Sample-Size-Dependent Significance Levels

“…The decisions made for binomials are the same as those for negative binomials with the same (x, y). This behavior is much more general than this specific example, and in fact it is proven in Pereira et al (2017) that the new tests are compliant with the Likelihood Principle for any discrete sample space.…”

“…A few examples are H 1 : θ = θ 0 ; H 2 : θ = θ 0 ; H 3 : θ ≤ θ 0 ; H 4 : θ > θ 0 ; H 5 : θ ∈ [θ 1 , θ 2 ]; and H 6 : (θ < θ 1 ) ∪ (θ > θ 2 ); etc. The predictive densities for several such sets of hypotheses are presented in Pereira et al (2017).…”

Adaptive significance levels in linear regression models

“…Neste trabalho apresentamos um método para encontrar um valor de corte para a evidência no FBST minimizando a combinação linear das probabilidades ponderadas de erro tipo I e tipo II para um determinado tamanho de amostra e também para uma certa dimensão do espaço paramétrico. Além disso, comparamos nossa metodologia com os resultados obtidos usando o teste proposto por Pereira et al (2017) e Gannon et al (2019) que apresenta o P -value como medida de evidência e inclui um nível de significância adaptativo. Para isso,é considerado o cenário de modelos de regressão linear sob a abordagem Bayesiana.…”

“…Following the same line of work, Pereira et al (2017) and Gannon et al (2019) present a new hypothesis-testing procedure formulated from the ideas developed in previous works (Irony and Pereira, 1995;Montoya-Delgado et al, 2001;Pereira, 1985;Pereira and Wechsler, 1993) and using a mixture of frequentist and Bayesian tools. This procedure introduces the P -value as a decision-making evidence measure and also includes an adaptive significance level, i.e., a significance level that is function of sample size.…”

“…The main goal of our work is to determine how small the Bayesian evidence in the FBST should be in order to reject the null hypothesis. Therefore, taking into account the concepts in DeGroot (1986) and Pereira (1985) associated with optimal hypothesis tests, as well as the conclusions of Oliveira (2014) about the relationship between the significance levels and the sample size, and finally, considering the ideas developed recently by Pereira et al (2017) and Gannon et al (2019) related to adaptive significance levels, we present a method to find a cutoff point for the e-value by minimizing a linear combination of the averaged type-I and type-II error probabilities for a given sample size and also for a given dimensionality of the parameter space. For that purpose, the scenario of linear regression models under the Bayesian approach is considered.…”