Truncated sequential Monte Carlo test with exact power

Silva, Ivair Ramos; Assunção, Renato M.

doi:10.1214/16-bjps339

Cited by 8 publications

(5 citation statements)

References 20 publications

(30 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further properties and extensions of the permutation and Monte Carlo tests, including their sequential versions, are derived by Silva and Assunção ( 2018 ).…”

Section: Methodsmentioning

confidence: 99%

Kronecker delta method for testing independence between two vectors in high-dimension

2021

View full text Add to dashboard Cite

Conventional methods for testing independence between two Gaussian vectors require sample sizes greater than the number of variables in each vector. Therefore, adjustments are needed for the high-dimensional situation, where the sample size is smaller than the number of variables in at least one of the compared vectors. It is critical to emphasize that the methods available in the literature are unable to control the Type I error probability under the nominal level. This fact is evidenced through an intensive simulation study presented in this paper. To cover this lack, we introduce a valid randomized test based on the Kronecker delta covariance matrices estimator. As an empirical application, based on a sample of companies listed on the stock exchange of Brazil, we test the independence between returns of stocks of different sectors in the COVID-19 pandemic context.

show abstract

“…Further properties and extensions of the permutation and Monte Carlo tests, including their sequential versions, are derived by Silva and Assunção ( 2018 ).…”

Section: Methodsmentioning

confidence: 99%

Kronecker delta method for testing independence between two vectors in high-dimension

2021

View full text Add to dashboard Cite

show abstract

“…In this section, we compute the resampling risk for several truncated procedures as a function of p and thus demonstrate that they do not bound the resampling risk uniformly. We consider truncated versions of CSM and SIMCTEST, which we denote by tCSM and tSIMCTEST, as well as the algorithms in Besag and Clifford (1991); Davidson and MacKinnon (2000); Fay et al (2007); Silva and Assunção (2018). The maximum number of samples is set to 13000 to roughly fit in with the maximum number of steps of Davidson and MacKinnon (2000).…”

Section: Comparison With Other Truncated Sequential Monte Carlo Proce...mentioning

confidence: 99%

“…The maximum number of samples is set to 13000 to roughly fit in with the maximum number of steps of Davidson and MacKinnon (2000). The methods of Besag and Clifford (1991); Davidson and MacKinnon (2000); Fay et al (2007); Silva and Assunção (2018) have tuning parameters, which we choose as follows. As suggested in Besag and Clifford (1991), the recommended parameter h controlling the number of exceedances is set to 10.…”

Section: Comparison With Other Truncated Sequential Monte Carlo Proce...mentioning

confidence: 99%

“…Procedures for Monte Carlo testing can be classified as open-ended and truncated procedures (Silva and Assunção, 2013). A truncated approach (Davidson and MacKinnon, 2000;Silva and Assunção, 2018;Besag and Clifford, 1991;Silva and Assunção, 2013) maximum number of Monte Carlo simulations in advance and forces a decision before or at the end of all simulations. Open-ended procedures (e.g., Gandy (2009)) do not have a stopping rule or an upper bound on the number of steps.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A simple method for implementing Monte Carlo tests

Ding

Hahn

2019

Comput Stat

View full text Add to dashboard Cite

We consider a statistical test whose p-value can only be approximated using Monte Carlo simulations. We are interested in deciding whether the p-value for an observed data set lies above or below a given threshold such as 5%. We want to ensure that the resampling risk, the probability of the (Monte Carlo) decision being different from the true decision, is uniformly bounded. This article introduces a simple open-ended method with this property, the confidence sequence method (CSM). We compare our approach to another algorithm, SIMCTEST, which also guarantees an (asymptotic) uniform bound on the resampling risk, as well as to other Monte Carlo procedures without a uniform bound. CSM is free of tuning parameters and conservative. It has the same theoretical guarantee as SIMCTEST and, in many settings, similar stopping boundaries. As it is much simpler than other methods, CSM is a useful method for practical applications.

show abstract

“…Although testing procedures such as the Bonferroni correction require exact knowledge of the p-value underlying each statistical test, p-values are usually not available analytically in practice and thus have to be approximated using Monte Carlo methods, for instance, bootstrap or permutation tests (Gandy and Hahn, 2014Silva and Assunção, 2018). In the context of such Monte Carlo tests, the (analytical) p-value refers to the one obtained by integrating over the theoretical bootstrap distribution (in case of bootstrap tests) or by exhaustively generating all permutations (in case of permutation tests).…”

Section: Introductionmentioning

confidence: 99%

Optimal allocation of Monte Carlo simulations to multiple hypothesis tests

Hahn

2019

Stat Comput

View full text Add to dashboard Cite

Multiple hypothesis tests are often carried out in practice using p-value estimates obtained with bootstrap or permutation tests since the analytical p-values underlying all hypotheses are usually unknown. This article considers the allocation of a pre-specified total number of Monte Carlo simulations K ∈ N (i.e., permutations or draws from a bootstrap distribution) to a given number of m ∈ N hypotheses in order to approximate their p-values p ∈ [0, 1] m in an optimal way, in the sense that the allocation minimises the total expected number of misclassified hypotheses. A misclassification occurs if a decision on a single hypothesis, obtained with an approximated p-value, differs from the one obtained if its p-value was known analytically. The contribution of this article is threefold: Under the assumption that p is known and K ∈ R, and using a normal approximation of the Binomial distribution, the optimal real-valued allocation of K simulations to m hypotheses is derived when correcting for multiplicity with the Bonferroni correction, both when computing the p-value estimates with or without a pseudo-count. Computational subtleties arising in the former case will be discussed. Second, with the help of an algorithm based on simulated annealing, empirical evidence is given that the optimal integer allocation is likely of the same form as the optimal real-valued allocation, and that both seem to coincide asympotically. Third, an empirical study on simulated and real data demonstrates that a recently proposed sampling algorithm based on Thompson sampling asympotically mimics the optimal (real-valued) allocation when the p-values are unknown and thus estimated at runtime.

show abstract

Truncated sequential Monte Carlo test with exact power

Cited by 8 publications

References 20 publications

Kronecker delta method for testing independence between two vectors in high-dimension

Kronecker delta method for testing independence between two vectors in high-dimension

A simple method for implementing Monte Carlo tests

Optimal allocation of Monte Carlo simulations to multiple hypothesis tests

Contact Info

Product

Resources

About