Automatic construction and evaluation of macro-actions in reinforcement learning

Farahani, Marzieh Davoodabadi; Mozayani, Nasser

doi:10.1016/j.asoc.2019.105574

Cited by 5 publications

(10 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For a skill to be obtained autonomously, the components of the triplet ⟨I, , ⟩ are needed. Therefore, there are many HRL studies that focus on detecting subgoals and/or defining initiation sets (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019;Machado et al 2017;McGovern and Barto 2001;Stolle and Precup 2002;Şimşek and Barto 2004a;Ghafoorian et al 2013;Daniel et al 2016;Cockcroft et al 2020). Graph theoretic approaches are widely used for subgoal detection (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019;Machado et al 2017).…”

Section: Related Workmentioning

confidence: 99%

“…Therefore, there are many HRL studies that focus on detecting subgoals and/or defining initiation sets (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019;Machado et al 2017;McGovern and Barto 2001;Stolle and Precup 2002;Şimşek and Barto 2004a;Ghafoorian et al 2013;Daniel et al 2016;Cockcroft et al 2020). Graph theoretic approaches are widely used for subgoal detection (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019;Machado et al 2017). Some of these studies apply different community detection algorithms to identify initiation sets besides detecting subgoals (Xu et al 2018;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Farahani and Mozayani 2019).…”

Section: Related Workmentioning

confidence: 99%

“…Graph theoretic approaches are widely used for subgoal detection (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019;Machado et al 2017). Some of these studies apply different community detection algorithms to identify initiation sets besides detecting subgoals (Xu et al 2018;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Farahani and Mozayani 2019). For example, Label Propagation (LP) (Raghavan et al 2007), an agglomerative clustering approach, is used to detect communities in Davoodabadi and Beigy (2011a), Farahani and Mozayani (2019).…”

Section: Related Workmentioning

confidence: 99%

“…Some of these studies apply different community detection algorithms to identify initiation sets besides detecting subgoals (Xu et al 2018;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Farahani and Mozayani 2019). For example, Label Propagation (LP) (Raghavan et al 2007), an agglomerative clustering approach, is used to detect communities in Davoodabadi and Beigy (2011a), Farahani and Mozayani (2019). Oversegmentation, which is the problem that forms the motivation of this study, is also encountered in Davoodabadi and Beigy (2011a), Farahani and Mozayani (2019).…”

Section: Related Workmentioning

confidence: 99%

“…Thus, subgoals are also detected in a dynamic manner (Xu et al 2018). In RL, environments may be effectively modeled using graphs (Xu et al 2018;Simsek and Barreto 2008;Davoodabadi and Beigy 2011a, b;Shoeleh and Asadpour 2017;Kazemitabar et al 2018;Farahani and Mozayani 2019). The connection points of subenvironments may be good candidates for subgoals by their nature.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Autonomous acquisition of arbitrarily complex skills using locality based graph theoretic features: a syntactic approach to hierarchical reinforcement learning

et al. 2023

View full text Add to dashboard Cite

With the growing state/action space, learning a satisfactory policy for regular Reinforcement Learning (RL) algorithms such as flat Q-learning becomes quickly infeasible. One possible solution to handle such cases is to employ hierarchical RL (HRL). In this work, we present two methods to autonomously construct (1) skills (ASKA) and (2) arbitrarily elaborate superskills or complexes through defining an arbitrary number of hierarchies in HRL (ASKAC) over a graph-based iteratively-growing environment model. We employ dynamic community detection (DCD) in detecting subgoals since DCD considers local changes only over the partially growing graphs and lowers the time complexity of the subgoal detection where groups of environment states (i.e., subenvironments) are modeled by communities from the graph theory. DCD's drawback is oversegmentation where it mispartitions a subenvironment further into smaller components. To maintain the robustness of ASKAC against DCD's possible oversegmentation we introduce the concept of skill coupling. Skill coupling does not only robustly solve the oversegmentation issue, but it also improves HRL by building up more elaborate complexes (i.e., skill compositions) obtained at an arbitrary number of hierarchies and reduces the number of decisions leading to the goal employing these complexes. In addition to the experiments that investigate the effect of parameters, proposed methods are experimentally evaluated in grid world and taxi driver benchmark environments.

show abstract