Monte Carlo estimation of minimax regret with an application to MDL model selection

Abstract

Abstract-Minimum description length (MDL) model selection, in its modern NML formulation, involves a model complexity term which is equivalent to minimax/maximin regret. When the data are discrete-valued, the complexity term is a logarithm of a sum of maximized likelihoods over all possible data-sets. Because the sum has an exponential number of terms, its evaluation is in many cases intractable. In the continuous case, the sum is replaced by an integral for which a closed form is available in only a few cases. We present an approach based on Monte Carlo sampling, which works for all model classes, and gives strongly consistent estimators of the minimax regret. The estimates convergence almost surely to the correct value with increasing number of iterations. For the important class of Markov models, one of the presented estimators is particularly efficient: in empirical experiments, accuracy that is sufficient for model selection is usually achieved already on the first iteration, even for long sequences.

Selecting amongst multinomial models: An apologia for normalized maximum likelihood

“…One way to sidestep this problem is to approximate the NML penalties using Monte Carlo integration methods (Evans & Swartz, 2000), as done by Kellen (2011, 2015) when comparing different models of recognition memory (see also Roos, 2008).…”

Minimum description length revisited

“…If r grows slower than n or not at all, the leading term tends to the classical form (29), where the leading term is k 2 log n. In practice, the approximation (30) is applicable for a wide range of r/n ratios. Roos [2008] and Zou and Roos [2017] studied the third term in the expansion of Eq. (29), namely the Fisher information integral, under Markov chains and Bayesian networks using Monte Carlo sampling techniques.…”

On Model Selection, Bayesian Networks, and the Fisher Information Integral

“…However, instead of resorting to factorized NML variants, where no numerical guarantees about the approximation error are known, we estimate NML by Monte Carlo sampling in the same fashion as in [13]. The obtained estimates can be shown to be consistent as the number of simulated samples is increased.…”

“…We need to consider other approximate methods such as the Monte Carlo sampling method introduced in [13]. Based on the law of large numbers, the sample average is guaranteed to converge to the mean if the sampling size is large.…”

“…We need to choose a sampling distribution q that is similar to the target distribution. Following [13], we use the sampling distribution by drawing each set of the parameters independently from the Dirichlet distribution Dir( 2 ), which results in the Krichevsky-Trofimov universal model (K-T model) [8]. It has been proved that the K-T model is asymptotically equivalent to NML as long as the parameters are not on the boundary.…”