Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior

Kojima, Noriyuki; Suhr, Alane; Artzi, Yoav

doi:10.1162/tacl_a_00428

Cited by 8 publications

(11 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Beyond the output itself, there are a few concerns over the generation of the models of ChatGPT. Because the mechanisms and intricacies of these algorithms have been extensively detailed in other resources, 26-31 we will not detail these here. However, we will expand on the process of supervised learning in construction of this model.…”

Section: Discussionmentioning

confidence: 99%

“…15 The mean DISCERN score across 40 ChatGPT responses was 44.2 ± 7.4, indicating that the material was overall of "fair" quality. Three responses (7.5%) were found to be of "poor" quality (DISCERN score [27][28][29][30][31][32][33][34][35][36][37][38], 34 responses (68.0%) were found to be of "fair" quality (DISCERN score 39-50), and 3 responses (7.5%) were found to be of "good" quality (DISCERN score 51-62). There were no responses found to be of "excellent" or "very poor" quality.…”

Section: Response Qualitymentioning

confidence: 99%

See 1 more Smart Citation

Exploring the Intersection of Artificial Intelligence and Neurosurgery: Let us be Cautious With ChatGPT

Mishra¹,

Begley²,

Chen³

et al. 2023

Neurosurgery

View full text Add to dashboard Cite

BACKGROUND AND OBJECTIVES: ChatGPT is a novel natural language processing artificial intelligence (AI) module where users enter any question or command and receive a single text response within seconds. As AI becomes more accessible, patients may begin to use it as a resource for medical information and advice. This is the first study to assess the neurosurgical information that is provided by ChatGPT. METHODS: ChatGPT was accessed in January 2023, and prompts were created requesting treatment information for 40 common neurosurgical conditions. Quantitative characteristics were collected, and four independent reviewers evaluated the responses using the DISCERN tool. Prompts were compared against the American Association of Neurological Surgeons (AANS) “For Patients” webpages. RESULTS: ChatGPT returned text organized in paragraph and bullet-point lists. ChatGPT responses were shorter (mean 270.1 ± 41.9 words; AANS webpage 1634.5 ± 891.3 words) but more difficult to read (mean Flesch-Kincaid score 32.4 ± 6.7; AANS webpage 37.1 ± 7.0). ChatGPT output was found to be of “fair” quality (mean DISCERN score 44.2 ± 4.1) and significantly inferior to the “good” overall quality of the AANS patient website (57.7 ± 4.4). ChatGPT was poor in providing references/resources and describing treatment risks. ChatGPT provided 177 references, of which 68.9% were inaccurate and 33.9% were completely falsified. CONCLUSION: ChatGPT is an adaptive resource for neurosurgical information but has shortcomings that limit the quality of its responses, including poor readability, lack of references, and failure to fully describe treatment options. Hence, patients and providers should remain wary of the provided content. As ChatGPT or other AI search algorithms continue to improve, they may become a reliable alternative for medical information.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Response Qualitymentioning

confidence: 99%

Exploring the Intersection of Artificial Intelligence and Neurosurgery: Let us be Cautious With ChatGPT

Mishra¹,

Begley²,

Chen³

et al. 2023

Neurosurgery

View full text Add to dashboard Cite

show abstract

“…For example, if the language generated by a speaker, although correct, is difficult to understand, this calls for unnecessary interpretation effort from the other agent. To measure whether pragmatic agents enable efficient communication, evaluations can use metrics of communicative cost (Walker et al, 1997) such as time to task completion, utterance length and complexity (Effenberger et al, 2021), measures such as lexical entrainment (Clark and Wilkes-Gibbs, 1986;Parent and Eskenazi, 2010;Hawkins et al, 2020), and quality ratings (Kojima et al, 2021).…”

Section: Evaluating Pragmatic Modelsmentioning

confidence: 99%

Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches

Fried,

Tomlin,

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

People rely heavily on context to enrich meaning beyond what is literally said, enabling concise but effective communication. To interact successfully and naturally with people, LLMs and other user-facing NLP systems will require similar skills in pragmatics: relying on various types of context-from shared linguistic goals and conventions, to the visual and embodied world-to use language effectively. We survey existing grounded settings and pragmatic modeling approaches and analyze how the task goals, environmental contexts, and communicative affordances in each work enrich linguistic meaning. We present recommendations for future grounded task design to naturally elicit pragmatic phenomena, and suggest directions that focus on a broader range of communicative contexts and affordances.Paul Bloom. 2002. How children learn the meanings of words. MIT press.

show abstract

“…A followup Kojima et al ( 2021) is especially relevant; in that work they show how multiple rounds of learning can continue to improve the language generation capabilities of a "leader" model. In addition to the embodied agents and players, our work shares with Kojima et al (2021) multiple rounds of data collection and the use of player feedback after "execution" to label examples. However, the key difference is that in Kojima et al (2021) the agent is a single ML model, whereas in this work, we aim to show that credit can be assigned to different components in a modular system, the data for the component can be annotated, and the component re-trained without any engineer intervention.…”

Section: Related Workmentioning

confidence: 99%

Many Episode Learning in a Modular Embodied Agent via End-to-End Interaction

Sun¹,

Carlson²,

Qian³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this work we give a case study of an embodied machine-learning (ML) powered agent that improves itself via interactions with crowd-workers. The agent consists of a set of modules, some of which are learned, and others heuristic. While the agent is not "end-to-end" in the ML sense, end-toend interaction is a vital part of the agent's learning mechanism. We describe how the design of the agent works together with the design of multiple annotation interfaces to allow crowd-workers to assign credit to module errors from end-to-end interactions, and to label data for individual modules. Over multiple automated human-agent interaction, credit assignment, data annotation, and model re-training and re-deployment, rounds we demonstrate agent improvement.

show abstract

Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior

Cited by 8 publications

References 35 publications

Exploring the Intersection of Artificial Intelligence and Neurosurgery: Let us be Cautious With ChatGPT

Exploring the Intersection of Artificial Intelligence and Neurosurgery: Let us be Cautious With ChatGPT

Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches

Many Episode Learning in a Modular Embodied Agent via End-to-End Interaction

Contact Info

Product

Resources

About