Knows what it knows: a framework for self-aware learning

Li, Lihong; Littman, Michael L.; Walsh, Thomas J.; Strehl, Alexander L.

doi:10.1007/s10994-010-5225-4

Cited by 111 publications

(144 citation statements)

References 32 publications

Supporting

Mentioning

142

Contrasting

Order By: Relevance

“…Furthermore, if sup (x,a,t) |R t (x, a)| ≤ R, then for any T , Regret(T ) ≤ 2RKH 2 . Notice that this result can also be derived based on the KWIK online regression with deterministic linear functions (see [17]). …”

Section: Learning With a Coherent Hypothesis Classmentioning

confidence: 65%

Efficient Reinforcement Learning in Deterministic Systems with Value Function Generalization

Wen

Roy

2017

Mathematics of OR

View full text Add to dashboard Cite

We consider the problem of reinforcement learning over episodes of a finite-horizon deterministic system and as a solution propose optimistic constraint propagation (OCP), an algorithm designed to synthesize efficient exploration and value function generalization. We establish that when the true value function Q * lies within a known hypothesis class Q, OCP selects optimal actions over all but at most dimE [Q] episodes, where dimE denotes the eluder dimension. We establish further efficiency and asymptotic performance guarantees that apply even if Q * does not lie in Q, for the special case where Q is the span of pre-specified indicator functions over disjoint sets. We also discuss the computational complexity of OCP and present computational results involving two illustrative examples.Key words : Reinforcement Learning, Efficient Exploration, Value Function Generalization, Approximate Dynamic Programming 1. Introduction A growing body of work on efficient reinforcement learning provides algorithms with guarantees on sample and computational efficiency (see, e.g., [13,6,2,30,4,9] and references therein). This literature highlights the point that an effective exploration scheme is critical to the design of any efficient reinforcement learning algorithm. In particular, popular exploration schemes such as -greedy, Boltzmann, and knowledge gradient (see [27]) can require learning times that grow exponentially in the number of states and/or the planning horizon (see [38,29]).The aforementioned literature focusses on tabula rasa learning; that is, algorithms aim to learn with little or no prior knowledge about transition probabilities and rewards. Such algorithms require learning times that grow at least linearly with the number of states. Despite the valuable insights that have been generated through their design and analysis, these algorithms are of limited practical import because state spaces in most contexts of practical interest are enormous. There is a need for algorithms that generalize from past experience in order to learn how to make effective decisions in reasonable time.There has been much work on reinforcement learning algorithms that generalize (see, e.g., [5,31,32,24] and references therein). Most of these algorithms do not come with statistical or computational efficiency guarantees, though there are a few noteworthy exceptions, which we now discuss. A number of results treat policy-based algorithms (see [10,3] and references therein), in which the goal is to select high-performers among a pre-specified collection of policies as learning progresses. Though interesting results have been produced in this line of work, each entails quite restrictive assumptions or does not make strong guarantees. Another body of work focuses on model-based algorithms. An algorithm proposed by Kearns and Koller [12] fits a factored model to observed data and makes decisions based on the fitted model. The authors establish a sample complexity bound that is polynomial in the number of model parameters rather than the numb...

show abstract

Section: Learning With a Coherent Hypothesis Classmentioning

confidence: 65%

Efficient Reinforcement Learning in Deterministic Systems with Value Function Generalization

Wen

Roy

2017

Mathematics of OR

View full text Add to dashboard Cite

show abstract

“…A first case generalizes the first example for the case where each arm provides an object among k classes. For this we can model each arm as a multinomial distribution and we know that we can learn the task by making a number of queries bounded by [37]: B( , δ) = n 8 2 ln 2n δ . A more complex example is when each model is itself a reinforcement learning problem.…”

Section: A Submodular Costsmentioning

confidence: 99%

“…A more complex example is when each model is itself a reinforcement learning problem. The learning curve has been shown to be polynomial ( [18], [37]). This means that the expected accuracy of the algorithm is always increasing with the number of samples.…”

Section: A Submodular Costsmentioning

confidence: 99%

The strategic student approach for life-long exploration and learning

Lopes

Oudeyer

2012

2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL)

View full text Add to dashboard Cite

Abstract-This article introduces the strategic student metaphor: a student has to learn a number of topics (or tasks) to maximize its mean score, and has to choose strategically how to allocate its time among the topics and/or which learning method to use for a given topic. We show that under which conditions a strategy where time allocation or learning method is chosen from the easier to the more complex topic is optimal. Then, we show an algorithm, based on multi-armed bandit techniques, that allows empirical online evaluation of learning progress and approximates the optimal solution under more general conditions. Finally, we show that the strategic student problem formulation allows to view in a common framework many previous approaches to active and developmental learning.

show abstract

“…This knowledge provides a useful mechanism for efficient exploration. Motivated by this observation, the "Knows What It Knows" (or KWIK) framework is proposed to capture the essence of quantifying estimation uncertainty (Li et al, 2011).…”

Section: Knows What It Knowsmentioning

confidence: 99%

Sample Complexity Bounds of Exploration

2012

Adaptation, Learning, and Optimization

Self Cite

View full text Add to dashboard Cite

Efficient exploration is widely recognized as a fundamental challenge inherent in reinforcement learning. Algorithms that explore efficiently converge faster to near-optimal policies. While heuristics techniques are popular in practice, they lack formal guarantees and may not work well in general. This chapter studies algorithms with polynomial sample complexity of exploration, both model-based and model-free ones, in a unified manner. These so-called PAC-MDP algorithms behave near-optimally except in a "small" number of steps with high probability. A new learning model known as KWIK is used to unify most existing model-based PAC-MDP algorithms for various subclasses of Markov decision processes. We also compare the sample-complexity framework to alternatives for formalizing exploration efficiency such as regret minimization and Bayes optimal solutions.

show abstract

Knows what it knows: a framework for self-aware learning

Cited by 111 publications

References 32 publications

Efficient Reinforcement Learning in Deterministic Systems with Value Function Generalization

Efficient Reinforcement Learning in Deterministic Systems with Value Function Generalization

The strategic student approach for life-long exploration and learning

Sample Complexity Bounds of Exploration

Contact Info

Product

Resources

About