State Abstraction as Compression in Apprenticeship Learning

Abel, David; Arumugam, Dilip; Asadi, Kavosh; Jinnai, Yuu; Littman, Michael L.; Wong, Lawson L. S.

doi:10.1609/aaai.v33i01.33013134

Cited by 47 publications

(68 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our recently completed work extends the above state abstraction theory to an information theoretic framework (Abel et al 2019). We draw a parallel between compression, as understood in Information Theory, and state abstraction, as studied in RL, to offer the first formalism and analysis of the trade-off between compression and performance made by state abstraction.…”

Section: Current Workmentioning

confidence: 99%

A Theory of State Abstraction for Reinforcement Learning

Abel¹

2019

AAAI

Self Cite

View full text Add to dashboard Cite

Reinforcement learning presents a challenging problem: agents must generalize experiences, efficiently explore the world, and learn from feedback that is delayed and often sparse, all while making use of a limited computational budget. Abstraction is essential to all of these endeavors. Through abstraction, agents can form concise models of both their surroundings and behavior, supporting effective decision making in diverse and complex environments. To this end, the goal of my doctoral research is to characterize the role abstraction plays in reinforcement learning, with a focus on state abstraction. I offer three desiderata articulating what it means for a state abstraction to be useful, and introduce classes of state abstractions that provide a partial path toward satisfying these desiderata. Collectively, I develop theory for state abstractions that can 1) preserve near-optimal behavior, 2) be learned and computed efficiently, and 3) can lower the time or data needed to make effective decisions. I close by discussing extensions of these results to an information theoretic paradigm of abstraction, and an extension to hierarchical abstraction that enjoys the same desirable properties.

show abstract

Section: Current Workmentioning

confidence: 99%

A Theory of State Abstraction for Reinforcement Learning

Abel¹

2019

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…Influence-based abstraction is a form of state abstraction, which has a long tradition in AI planning and learning (e.g., Sacerdoti, 1974;Knoblock, 1993;McCallum, 1993;Dearden & Boutilier, 1997;Hoey, St-Aubin, Hu, & Boutilier, 1999;Givan, Leach, & Dean, 2000;Boutilier, Dearden, & Goldszmidt, 2000;Ravindran & Barto, 2003;Jong & Stone, 2005;Konidaris & Barto, 2009;Kaelbling & Lozano-Perez, 2012;Hostetler, Fern, & Dietterich, 2014;Anand, Noothigattu, Mausam, & Singla, 2016;Bai, Srivastava, & Russell, 2016;Abel, Arumugam, Asadi, Jinnai, Littman, & Wong, 2019) . Other types of abstraction (Mahadevan, 2010) are temporal abstractions, such as options and macro-actions (Sutton, Precup, & Singh, 1999;Theocharous & Kaelbling, 2004;Amato, Konidaris, Kaelbling, & How, 2019;Machado, Bellemare, & Bowling, 2017), and functional abstraction, which tries to identify appropriate basis functions (Keller, Mannor, & Precup, 2006;Parr, Painter-Wakefield, Li, & Littman, 2007;Mahadevan & Maggioni, 2007;Petrik, 2007), including the huge body of recent work on deep RL (Schmidhuber, 1991;Mnih et al, 2015;François-Lavet, Henderson, Islam, Bellemare, & Pineau, 2018).…”

Section: Other Forms Of Abstractionmentioning

confidence: 99%

A Sufficient Statistic for Influence in Structured Multiagent Environments

Oliehoek

Witwicki

Kaelbling

2021

jair

View full text Add to dashboard Cite

Making decisions in complex environments is a key challenge in artificial intelligence (AI). Situations involving multiple decision makers are particularly complex, leading to computational intractability of principled solution methods. A body of work in AI has tried to mitigate this problem by trying to distill interaction to its essence: how does the policy of one agent influence another agent? If we can find more compact representations of such influence, this can help us deal with the complexity, for instance by searching the space of influences rather than the space of policies. However, so far these notions of influence have been restricted in their applicability to special cases of interaction. In this paper we formalize influence-based abstraction (IBA), which facilitates the elimination of latent state factors without any loss in value, for a very general class of problems described as factored partially observable stochastic games (fPOSGs). On the one hand, this generalizes existing descriptions of influence, and thus can serve as the foundation for improvements in scalability and other insights in decision making in complex multiagent settings. On the other hand, since the presence of other agents can be seen as a generalization of single agent settings, our formulation of IBA also provides a sufficient statistic for decision making under abstraction for a single agent. We also give a detailed discussion of the relations to such previous works, identifying new insights and interpretations of these approaches. In these ways, this paper deepens our understanding of abstraction in a wide range of sequential decision making settings, providing the basis for new approaches and algorithms for a large class of problems.

show abstract

“…When the cumulative reward function expectations generated by all the strategies are not greater than the cumulative reward function expectations generated by the expert strategy, the reward function of RL will be the reward function learned from the expert data. Apprenticeship learning [30,31] is a type of IRL, which sets the prior basis function as the reward function. This ensures that the optimal strategy obtained from the reward function is near the expert strategy using the given expert data.…”

Section: Inverse Reinforcement Learning (Irl)mentioning

confidence: 99%

Visual Navigation Using Inverse Reinforcement Learning and an Extreme Learning Machine

Fang

Wang

2021

Electronics

View full text Add to dashboard Cite

In this paper, we focus on the challenges of training efficiency, the designation of reward functions, and generalization in reinforcement learning for visual navigation and propose a regularized extreme learning machine-based inverse reinforcement learning approach (RELM-IRL) to improve the navigation performance. Our contributions are mainly three-fold: First, a framework combining extreme learning machine with inverse reinforcement learning is presented. This framework can improve the sample efficiency and obtain the reward function directly from the image information observed by the agent and improve the generation for the new target and the new environment. Second, the extreme learning machine is regularized by multi-response sparse regression and the leave-one-out method, which can further improve the generalization ability. Simulation experiments in the AI-THOR environment showed that the proposed approach outperformed previous end-to-end approaches, thus, demonstrating the effectiveness and efficiency of our approach.

show abstract

State Abstraction as Compression in Apprenticeship Learning

Cited by 47 publications

References 37 publications

A Theory of State Abstraction for Reinforcement Learning

A Theory of State Abstraction for Reinforcement Learning

A Sufficient Statistic for Influence in Structured Multiagent Environments

Visual Navigation Using Inverse Reinforcement Learning and an Extreme Learning Machine

Contact Info

Product

Resources

About