Optimal learning for sequential sampling with non-parametric beliefs

We consider sequential decision problems in which we adaptively choose one of finitely many alternatives and observe a stochastic reward. We offer a new perspective of interpreting Bayesian ranking and selection problems as adaptive stochastic multi-set maximization problems and derive the first finite-time bound of the knowledge-gradient policy for adaptive submodular objective functions. In addition, we introduce the concept of prior-optimality and provide another insight into the performance of the knowledge gradient policy based on the submodular assumption on the value of information. We demonstrate submodularity for the two-alternative case and provide other conditions for more general problems, bringing out the issue and importance of submodularity in learning problems. Empirical experiments are conducted to further illustrate the finite time behavior of the knowledge gradient policy.

show abstract

“…The knowledge gradient policy can handle the presence of a variety of belief models such as (generalized) linear [30,36] or nonparametric [29,5].…”

Section: Knowledge Gradientmentioning

confidence: 99%

Finite-Time Analysis for the Knowledge-Gradient Policy

Wang

Powell

2018

SIAM J. Control Optim.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The knowledge gradient policy can handle a variety of belief models such as linear [22] or nonparametric [22,19,4].…”

Section: Definition 42 the Knowledge Gradient (Kg) Policy Is Definementioning

confidence: 99%

Nested-Batch-Mode Learning and Stochastic Optimization with An Application to Sequential MultiStage Testing in Materials Science

Wang¹,

Reyes²,

Brown³

et al. 2015

SIAM J. Sci. Comput.

Self Cite

View full text Add to dashboard Cite

Abstract. We consider the nested-batch decision problem where we need to make a first stage choice (e.g. the size of a nanoparticle) after which we then need to run a series of experiments in batch selecting several second stage choices (e.g. testing different densities of the nanoparticle). Since these experiments are time consuming and expensive, we propose to estimate the value of information from the choice of the first stage decision (the size), to help guide the scientist in the selection of the next batch of experiments to run. The batch experiments are designed assuming that we maximize the value of information for an entire batch. The value of information, known as the Knowledge Gradient, requires calculating the expected maximum of a function. Since the calculation of the expected maximum is computationally intractable, we propose a Monte Carlo-based approach to address this hurdle in the context of both the batch and nested-batch problems. We empirically demonstrate the effectiveness of our approach on the material design problem of maximizing output current of a photoactive device, where it is competitive with a fully sequential optimal learning strategy and significantly outperforms pure exploration, pure exploitation and ϵ-greedy strategies with regard to the opportunity cost metric (8.1).

show abstract

“…Negoescu et al [2011] introduces the use of a parametric belief model, making it possible to solve problems with thousands of alternatives. For nonparametric beliefs, Mes et al [2011] proposes a hierarchical aggregation technique using the common features shared by alternatives to learn about many alternatives from even a single measurement, while Barut and Powell [2013] estimates the belief function using kernel regression and aggregation of kernels. However, all the methods above assume low dimensional belief models, where the number of features is relatively small.…”

Section: Literaturementioning

confidence: 99%

The Knowledge Gradient Policy Using A Sparse Additive Belief Model

Li,

Liu,

Powell

2015

Preprint

View full text Add to dashboard Cite

We propose a sequential learning policy for noisy discrete global optimization and ranking and selection (R&S) problems with high dimensional sparse belief functions, where there are hundreds or even thousands of features, but only a small portion of these features contain explanatory power. We aim to identify the sparsity pattern and select the best alternative before the finite budget is exhausted. We derive a knowledge gradient policy for sparse linear models (KGSpLin) with group Lasso penalty. This policy is a unique and novel hybrid of Bayesian R&S with frequentist learning. Particularly, our method naturally combines B-spline basis expansion and generalizes to the nonparametric additive model (KGSpAM) and functional ANOVA model. Theoretically, we provide the estimation error bounds of the posterior mean estimate and the functional estimate. Controlled experiments show that the algorithm efficiently learns the correct set of nonzero parameters even when the model is imbedded with hundreds of dummy parameters. Also it outperforms the knowledge gradient for a linear model.

show abstract

Optimal learning for sequential sampling with non-parametric beliefs

Cited by 13 publications

References 27 publications

Finite-Time Analysis for the Knowledge-Gradient Policy

Finite-Time Analysis for the Knowledge-Gradient Policy

Nested-Batch-Mode Learning and Stochastic Optimization with An Application to Sequential MultiStage Testing in Materials Science

The Knowledge Gradient Policy Using A Sparse Additive Belief Model

Contact Info

Product

Resources

About