On the hardness of finding symmetries in Markov decision processes

Narayanamurthy, Shravan; Ravindran, Balaraman

doi:10.1145/1390156.1390243

Cited by 11 publications

(5 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, we can conclude that lifting an LP is beneficial regardless of whether the problem is sparse or dense, thus one might view symmetry as a dimension orthogonal to sparsity. Remarkably, the results follow closely what has been achieved with MDP-specific symmetry finding and model minimization approaches [47,48,49].…”

Section: Lifted Linear Programming For Solving Markov Decision Processessupporting

confidence: 85%

Relational linear programming

Kersting

Mladenov

Tokmakov

2017

Artificial Intelligence

View full text Add to dashboard Cite

We propose relational linear programming, a simple framework for combining linear programs (LPs) and logic programs. A relational linear program (RLP) is a declarative LP template defining the objective and the constraints through the logical concepts of objects, relations, and quantified variables. This allows one to express the LP objective and constraints relationally for a varying number of individuals and relations among them without enumerating them. Together with a logical knowledge base, effectively a logic program consisting of logical facts and rules, it induces a ground LP. This ground LP is solved using lifted linear programming. That is, symmetries within the ground LP are employed to reduce its dimensionality, if possible, and the reduced program is solved using any off-the-shelf LP solver. In contrast to mainstream LP template languages such as AMPL, which features a mixture of declarative and imperative programming styles, RLP's relational nature allows a more intuitive representation of optimization problems, in particular over relational domains. We illustrate this empirically by experiments on approximate inference in Markov logic networks using LP relaxations, on solving Markov decision processes, and on collective inference using LP

show abstract

Section: Lifted Linear Programming For Solving Markov Decision Processessupporting

confidence: 85%

Relational linear programming

Kersting

Mladenov

Tokmakov

2017

Artificial Intelligence

View full text Add to dashboard Cite

show abstract

“…a priori knowledge of symmetry, as we do; see Ravindran and Barto (2001) and Narayanamurthy and Ravindran (2008).…”

Section: Introductionmentioning

confidence: 56%

Exploiting Symmetry in High-Dimensional Dynamic Programming

Kahou¹,

Fernández-Villaverde²,

Perla³

et al. 2021

SSRN Journal

View full text Add to dashboard Cite

“…Zinkevich and Balch (2001) formalized the concept of symmetry in MDPs and proved that if such consolidation of symmetrical state–actions is performed accurately, then the optimal function and the optimal policy are not altered. However, automatically identifying symmetries is computationally complex (Narayanamurthy & Ravindran, 2008), especially when the symmetry is only assumed . For example, in the Pursuit domain, one may consider the 90°, 180° and 270° transpositions of the state around its center (along with the direction of the action) as being similar (see Figure 1 (b)).…”

Section: The Sass Approachmentioning

confidence: 99%

Leveraging human knowledge in tabular reinforcement learning: a study of human subjects

Rosenfeld

Cohen²,

Taylor³

et al. 2018

The Knowledge Engineering Review

View full text Add to dashboard Cite

Reinforcement Learning (RL) can be extremely effective in solving complex, real-world problems. However, injecting human knowledge into an RL agent may require extensive effort and expertise on the human designer's part. To date, human factors are generally not considered in the development and evaluation of possible RL approaches. In this article, we set out to investigate how different methods for injecting human knowledge are applied, in practice, by human designers of varying levels of knowledge and skill. We perform the first empirical evaluation of several methods, including a newly proposed method named SASS which is based on the notion of similarities in the agent's state-action space. Through this human study, consisting of 51 human participants, we shed new light on the human factors that play a key role in RL. We find that the classical reward shaping technique seems to be the most natural method for most designers, both expert and non-expert, to speed up RL. However, we further find that our proposed method SASS can be effectively and efficiently combined with reward shaping, and provides a beneficial alternative to using only a single speedup method with minimal human designer effort overhead.

show abstract

On the hardness of finding symmetries in Markov decision processes

Cited by 11 publications

References 7 publications

Relational linear programming

Relational linear programming

Exploiting Symmetry in High-Dimensional Dynamic Programming

Leveraging human knowledge in tabular reinforcement learning: a study of human subjects

Contact Info

Product

Resources

About