Formalized Data Snooping Based on Generalized Error Rates

Abstract

Formalized data snooping based on generalized error rates Romano, J P; Shaikh, A M; Wolf, M Romano, J P; Shaikh, A M; Wolf, M (2008) Formalized data snooping based on generalized error rates Abstract It is common in econometric applications that several hypothesis tests are carried out simultaneously. The problem then becomes how to decide which hypotheses to reject, accounting for the multitude of tests. The classical approach is to control the familywise error rate (FWE) which is the probability of one or more false rejections. But when the number of hypotheses under consideration is large, control of the FWE can become too demanding. As a result, the number of false hypotheses rejected may be small or even zero. This suggests replacing control of the FWE by a more liberal measure. To this end, we review a number of recent proposals from the statistical literature. We briefly discuss how these procedures apply to the general problem of model selection. A simulation study and two empirical applications illustrate the methods. Formalized Data Snooping Based on Generalized Error Rates Michael Wolf Institute for Empirical Research in Economics University of Zurich CH-8006 Zurich SwitzerlandAbstract It is common in econometric applications that several hypothesis tests are carried out simultaneously. The problem then becomes how to decide which hypotheses to reject, accounting for the multitude of tests. The classical approach is to control the familywise error rate (FWE) which is the probability of one or more false rejections. But when the number of hypotheses under consideration is large, control of the FWE can become too demanding. As a result, the number of false hypotheses rejected may be small or even zero. This suggests replacing control of the FWE by a more liberal measure. To this end, we review a number of recent proposals from the statistical literature. We briefly discuss how these procedures apply to the general problem of model selection. A simulation study and two empirical applications illustrate the methods.KEY WORDS: Data snooping, false discovery proportion, false discovery rate, generalized familywise error rate, model selection, multiple testing, stepwise methods.JEL CLASSIFICATION NOS: C12, C14, C52.ACKNOWLEDGMENTS: We thank three anonymous referees for helpful comments that have led to an improved presentation of the paper.

Multiple Forecast Model Evaluation

“…Hence, even if FDR is controlled for at a given level, its realization can be quite different. Most of the procedures explained in this Chapter can be modified to control for FDP (for details, see Romano, Shaikh and Wolf, 2008b).…”

“…Suppose that one is concerned with Pr(at least l false discoveries), with l > 1. Then, methods controlling for l-FWER have been developed, based on modified Bonferroni and Holm procedures (for a review of these modifications, see Romano, Shaikh and Wolf, 2008b).…”

Multiple tests for the performance of different investment strategies

“…Those inaccuracies occur despite the fact that a broad literature is concerned with the derivation of appropriate multiple-testing procedures. For a nice overview of existing methods, see Romano et al (2008).…”

“…Additionally, we would like to identify all non-trivial strategies which are outperformed by naive diversification. The last question is a typical application of multiple-testing procedures (Romano et al, 2008).…”

Fund-of-Funds Construction by Statistical Multiple Testing Methods

“…However, simulation studies show that for practically relevant sample sizes T , the finite-sample control of the FWE is very satisfactory; see Romano and Wolf (2005) and Romano et al (2008).…”