Likelihood, Replicability and Robbins' Confidence Sequences

Pace, Luigi; Salvan, Alessandra

doi:10.1111/insr.12355

Cited by 10 publications

(11 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(Molenberghs et al, 2014) and references therein. As such, the time-robust minimax setting nicely fits in recent work promoting always-valid confidence intervals (Howard et al, 2018;Pace and Salvan, 2019) and testing safe under optional continuation (Grünwald, de Heide and Koolen, 2019) as a generic, more robust replacement of traditional testing and confidence.…”

mentioning

confidence: 76%

Minimax rates without the fixed sample size assumption

Kirichenko,

Grünwald

2020

Preprint

View full text Add to dashboard Cite

We generalize the notion of minimax convergence rate. In contrast to the standard definition, we do not assume that the sample size is fixed in advance. Allowing for varying sample size results in time-robust minimax rates and estimators. These can be either strongly adversarial, based on the worst-case over all sample sizes, or weakly adversarial, based on the worst-case over all stopping times. We show that standard and time-robust rates usually differ by at most a logarithmic factor, and that for some (and we conjecture for all) exponential families, they differ by exactly an iterated logarithmic factor. In many situations, time-robust rates are arguably more natural to consider. For example, they allow us to simultaneously obtain strong model selection consistency and optimal estimation rates, thus avoiding the "AIC-BIC dilemma".1. Introduction. Minimax rates are an essential tool for evaluation and comparison of estimators in a wide variety of applications. Classic references on the topic include, among many others, Tsybakov (2009), Wasserman (2006) and Van der Vaart (1998). For a fixed sample size n, the standard minimax rate is computed by first taking the supremum of the expected loss over all parameters (distributions) in the model for each estimator, and then minimizing this value over all possible estimators. Here, we consider a natural extension of this setting in which data comes in sequentially and one does not know n in advance: instead of considering n ≥ 1 fixed, we include it in the worst-case analysis.At first it may seem that such time-robustness trivializes the problem: a naive approach would be to take n as a parameter just like the distribution and compute the supremum of the expected loss over all sample sizes and all distributions in the model. In most cases the supremum would then be trivially attained for sample size one, since the precision of an estimator tends to get better with the increase of the sample size. Therefore, another approach has to be taken. We manage to give meaningful definitions by rewriting the standard definition in terms of a ratio. The precise new definitions, given in Section 2.3, come in two forms: weakly adversarial, in which we take the sup (worst-case) over all stopping times; and strongly adversarial, in which we take the sup over all sample sizes. In general, the weakly adversarial minimax rate cannot be larger than the strongly adversarial one. The weakly adversarial setting corresponds to what has recently been called the always valid (sometimes also "anytime-valid") setting for confidence intervals and testing (Howard et al., 2018): at any point in time n, Nature can decide whether or not to stop generating data and present the data so far for analysis, using a rule that can take into account both past data and the true distribution. This can be seen as a form of minimax analysis under 'optional stopping'. Note however that in the standard interpretation (e.g. in the Bayesian literature) of optional stopping, stopping rules are assumed independent of the...

show abstract

mentioning

confidence: 76%

Minimax rates without the fixed sample size assumption

Kirichenko,

Grünwald

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Wagenmakers for any prior of θ under the alternative H 1 . Remarkably, the universal bound is also valid under sequential analyses with optional stopping as soon as a Bayes factor smaller than k is obtained (Robbins (1970); Pace and Salvan (2020)). In contrast, frequentist tests and confidence sets typically have to be adjusted for sequential analyses to guarantee appropriate error rates, and the theory and applicability can become quite involved.…”

Section: Error Control Via the Universal Boundmentioning

confidence: 95%

Evidential Calibration of Confidence Intervals

Pawel¹,

Ly²,

Wagenmakers³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This function satisfies the criteria of Proposition 35. Hence the process M t := s≤t f (X s ) is an S-martingale and we also have (17). Consider the p-value (p t ) given by p t := inf s≤t 1/M s .…”

Section: Necessary and Sufficient Conditions For Sequential Tests (Pr...mentioning

confidence: 99%

“…Once more, unfortunately, a naive confidence interval based on the central limit theorem or the bootstrap does not satisfy the desired property. As before, these 'standard' constructions instead satisfy the above property only at fixed data-independent times t. To satisfy property (2), one could employ 'confidence sequences' proposed by Robbins and collaborators like Darling, Siegmund, and Lai [3,18,14], and regaining interest in recent years [17,11]; this is a central topic of this paper and we return to it later. Recently, another set of highly interrelated ideas has been put forward under a variety of names by authors such as Shafer, Vovk, Grünwald, and their collaborators: test martingales [20] (since the test statistics are sometimes martingales), e-values or sequential e-values [23,24] (e for expectation), betting scores [19] (since they have roots in gambling), or safe e-values [9] (safe under optional stopping).…”

Section: Introductionmentioning

confidence: 99%

Admissible anytime-valid sequential inference must rely on nonnegative martingales

Ramdas¹,

Ruf²,

Larsson³

et al. 2020

Preprint

View full text Add to dashboard Cite

Wald's anytime-valid p-values and Robbins' confidence sequences enable sequential inference for composite and nonparametric classes of distributions at arbitrary stopping times, as do more recent proposals involving Vovk's 'e-values' or Shafer's 'betting scores'. Examining the literature, one finds that at the heart of all these (quite different) approaches has been the identification of composite nonnegative (super)martingales. Thus, informally, nonnegative (super)martingales are known to be sufficient for valid sequential inference. Our central contribution is to show that martingales are also universal-all admissible constructions of (composite) anytime p-values, confidence sequences, or e-values must necessarily utilize nonnegative martingales (or so-called max-martingales in the case of p-values). Sufficient conditions for composite admissibility are also provided. Our proofs utilize a plethora of modern mathematical tools for composite testing and estimation problems: maxmartingales, Snell envelopes, and new Doob-Lévy martingales make appearances in previously unencountered ways. Informally, if one wishes to perform anytime-valid sequential inference, then any existing approach can be recovered or dominated using martingales. We provide several sophisticated examples, with special focus on the nonparametric problem of testing if a distribution is symmetric, where our new constructions render past methods inadmissible.

show abstract

Likelihood, Replicability and Robbins' Confidence Sequences

Cited by 10 publications

References 30 publications

Minimax rates without the fixed sample size assumption

Minimax rates without the fixed sample size assumption

Evidential Calibration of Confidence Intervals

Admissible anytime-valid sequential inference must rely on nonnegative martingales

Contact Info

Product

Resources

About