Interactive Machine Comprehension with Information Seeking Agents

Yuan, Xingdi; Fu, Jie; Côté, Marc-Alexandre; Tay, Yi; Pal, Christopher; Trischler, Adam

doi:10.18653/v1/2020.acl-main.211

Cited by 10 publications

(4 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Leveraging the web for traditional NLP tasks. Several papers have explored the use of the web for information extraction [34] and retrieval [1], question answering [57,25], dialog [45], and training language models on webtext [2]. These approaches primarily use web search engines as a knowledge retriever for gathering additional evidence for the task at hand.…”

Section: Related Workmentioning

confidence: 99%

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

Yao¹,

Chen²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Existing benchmarks for grounding language in interactive environments either lack real-world linguistic elements, or prove difficult to scale up due to substantial human involvement in the collection of data or feedback signals. To bridge this gap, we develop WebShop -a simulated e-commerce website environment with 1.18 million real-world products and 12, 087 crowd-sourced text instructions. Given a text instruction specifying a product requirement, an agent needs to navigate multiple types of webpages and issue diverse actions to find, customize, and purchase an item. WebShop provides several challenges for language grounding including understanding compositional instructions, query (re-)formulation, comprehending and acting on noisy text in webpages, and performing strategic exploration. We collect over 1, 600 human demonstrations for the task, and train and evaluate a diverse range of agents using reinforcement learning, imitation learning, and pre-trained image and language models. Our best model achieves a task success rate of 29%, which outperforms rule-based heuristics (9.6%) but is far lower than human expert performance (59%). We also analyze agent and human trajectories and ablate various model components to provide insights for developing future agents with stronger language understanding and decision making abilities. Finally, we show that agents trained on WebShop exhibit non-trivial sim-to-real transfer when evaluated on amazon.com and eaby.com, indicating the potential value of WebShop in developing practical web-based agents that can operate in the wild. * Equal contribution. Project site with code, data, and demos: https://webshop-pnlp.github.io.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

Yao¹,

Chen²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…While behavior evolves with interfaces, users keep parsing results fast and frugally, attending to just a few items. From a similar angle, Yuan et al [2020] offer promising findings on training QA agents with RL for templatized information-gathering and answering actions. Most of the work in language-related RL is otherwise centered on synthetic navigation/arcade environments [Hu et al, 2019].…”

Section: Related Workmentioning

confidence: 99%

Boosting Search Engines with Interactive Agents

Adolphs¹,

Boerschinger²,

Buck³

et al. 2021

Preprint

View full text Add to dashboard Cite

Can machines learn to use a search engine as an interactive tool for finding information? That would have far reaching consequences for making the world's knowledge more accessible. This paper presents first steps in designing agents that learn meta-strategies for contextual query refinements. Our approach uses machine reading to guide the selection of refinement terms from aggregated search results. Agents are then empowered with simple but effective search operators to exert fine-grained and transparent control over queries and search results. We develop a novel way of generating synthetic search sessions, which leverages the power of transformer-based generative language models through (self-)supervised learning. We also present a reinforcement learning agent with dynamically constrained actions that can learn interactive search strategies completely from scratch. In both cases, we obtain significant improvements over one-shot search with a strong information retrieval baseline. Finally, we provide an in-depth analysis of the learned search policies. * Work carried out in part during internships at Google.

show abstract

“…However, these datasets contain multiple choice questions, and the answer choices provide hints as to what information may be needed. Yuan et al (2020) explore this as well using a POMDP in which the context in existing QA datasets is hidden from the model until it explicitly searches for it.…”

Section: Open-domain Question Answeringmentioning

confidence: 99%

IIRC: A Dataset of Incomplete Information Reading Comprehension Questions

Ferguson¹,

Gardner²,

Hajishirzi³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Humans often have to read multiple documents to address their information needs. However, most existing reading comprehension (RC) tasks only focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system's performance at identifying a potential lack of sufficient information and locating sources for that information. To fill this gap, we present a dataset, IIRC, with more than 13K questions over paragraphs from English Wikipedia that provide only partial information to answer them, with the missing information occurring in one or more linked documents. The questions were written by crowd workers who did not have access to any of the linked documents, leading to questions that have little lexical overlap with the contexts where the answers appear. This process also gave many questions without answers, and those that require discrete reasoning, increasing the difficulty of the task. We follow recent modeling work on various reading comprehension datasets to construct a baseline model for this dataset, finding that it achieves 31.1% F1 on this task, while estimated human performance is 88.4%. The dataset, code for the baseline system, and a leaderboard can be found at https://allennlp.org/iirc. * Work done as an intern at the Allen Institute for AI.

show abstract

Interactive Machine Comprehension with Information Seeking Agents

Cited by 10 publications

References 59 publications

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

Boosting Search Engines with Interactive Agents

IIRC: A Dataset of Incomplete Information Reading Comprehension Questions

Contact Info

Product

Resources

About