Enhancing Text-based Reinforcement Learning Agents with Commonsense Knowledge

Murugesan, Keerthiram; Atzeni, Mattia; Shukla, Pushkar; Sachan, Mrinmaya; Kapanipathi, Pavan; Talamadupula, Kartik

doi:10.48550/arxiv.2005.00811

Cited by 9 publications

(13 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These results can also be seen as an RL-centric agent-based validation of similar results shown in the broader NLP literature (Kapanipathi et al 2020). We refer the reader to (Murugesan et al 2020) on further discussion on this topic.…”

Section: D2 Discussionsupporting

confidence: 76%

Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines

Murugesan¹,

Atzeni²,

Kapanipathi³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Text-based games have emerged as an important test-bed for Reinforcement Learning (RL) research, requiring RL agents to combine grounded language understanding with sequential decision making. In this paper, we examine the problem of infusing RL agents with commonsense knowledge. Such knowledge would allow agents to efficiently act in the world by pruning out implausible actions, and to perform lookahead planning to determine how current actions might affect future world states. We design a new text-based gaming environment called TextWorld Commonsense (TWC) for training and evaluating RL agents with a specific kind of commonsense knowledge about objects, their attributes, and affordances. We also introduce several baseline RL agents which track the sequential context and dynamically retrieve the relevant commonsense knowledge from ConceptNet. We show that agents which incorporate commonsense knowledge in TWC perform better, while acting more efficiently. We conduct user-studies to estimate human performance on TWC and show that there is ample room for future improvement.

show abstract

Section: D2 Discussionsupporting

confidence: 76%

Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines

Murugesan¹,

Atzeni²,

Kapanipathi³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…To the north a narrow path winds through the trees. You are carrying: a small leaflet Prev act: go north Valid acts: go north, go east, go west, drop leaflet about one's surroundings while navigating novel environments, either through rules [3,2], questionanswering [5], or transformer-based extraction [1,21]. This lifted representation helps agents remember aspects of the world that become unobservable as the agent navigates the environment.…”

Section: North Of Housementioning

confidence: 99%

“…In textual environments, the traditional state representations of choice have been raw text encodings via recurrent neural networks [24,14,13] but have since shifted towards transformer [34] and knowledge graph-based representations [3,1]. Knowledge graphs have been shown to be aid in the challenges of: (1) knowledge representation [3,1], enabling neuro-symbolic reasoning approaches over graph-based state representations [29]; (2) combinatorial state-action spaces [2,1]; and (3) incorporating external knowledge sources for commonsense reasoning [3,21,22,9]. Two of these works are perhaps closest in spirit to ours.…”

Section: Related Workmentioning

confidence: 99%

“…Prior work on text-based game playing agents repeatedly demonstrated that providing agents with structured memory in the form of knowledge graphs (sets of s, r, o tuples such that s is a subject, r is a relation, and o is an object) is critical in enabling them to operate in and model these worlds [3,21,1,2]-aiding in both the challenges mentioned. These works all rely on extracting information…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning Knowledge Graph-based World Models of Textual Environments

Ammanabrolu¹,

Riedl²

2021

Preprint

View full text Add to dashboard Cite

World models improve a learning agent's ability to efficiently operate in interactive and situated environments. This work focuses on the task of building world models of text-based game environments. Text-based games, or interactive narratives, are reinforcement learning environments in which agents perceive and interact with the world using textual natural language. These environments contain long, multi-step puzzles or quests woven through a world that is filled with hundreds of characters, locations, and objects. Our world model learns to simultaneously:(1) predict changes in the world caused by an agent's actions when representing the world as a knowledge graph; and (2) generate the set of contextually relevant natural language actions required to operate in the world. We frame this task as a Set of Sequences generation problem by exploiting the inherent structure of knowledge graphs and actions and introduce both a transformer-based multi-task architecture and a loss function to train it. A zero-shot ablation study on neverbefore-seen textual worlds shows that our methodology significantly outperforms existing textual world modeling techniques as well as the importance of each of our contributions.Preprint. Under review.

show abstract

“…While KGs have been leveraged to handle partial observability [3,4,58], reduce action space [3,4], and improve generalizability [1,5], few of the existing works addresses its potential for reasoning. Recently, Murugesan et al [41] tried to introduce commonsense reasoning for playing synthetic games. They extracted sub-graphs from ConceptNet [44], which is a large-scale external knowledge base with millions of edges and nodes.…”

Section: Related Workmentioning

confidence: 99%

Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Games

Xu¹,

Fang²,

Chen³

et al. 2020

Preprint

View full text Add to dashboard Cite

We study reinforcement learning (RL) for text-based games, which are interactive simulations in the context of natural language. While different methods have been developed to represent the environment information and language actions, existing RL agents are not empowered with any reasoning capabilities to deal with textual games. In this work, we aim to conduct explicit reasoning with knowledge graphs for decision making, so that the actions of an agent are generated and supported by an interpretable inference procedure. We propose a stacked hierarchical attention mechanism to construct an explicit representation of the reasoning process by exploiting the structure of the knowledge graph. We extensively evaluate our method on a number of man-made benchmark games, and the experimental results demonstrate that our method performs better than existing text-based agents.

show abstract

Enhancing Text-based Reinforcement Learning Agents with Commonsense Knowledge

Cited by 9 publications

References 18 publications

Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines

Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines

Learning Knowledge Graph-based World Models of Textual Environments

Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Games

Contact Info

Product

Resources

About