In-BoXBART: Get Instructions into Biomedical Multi-Task Learning

Parmar, Mihir; Mishra, Swaroop; Purohit, Mirali; Luo, Ming; Mohammad, Murad; Baral, Chitta

doi:10.18653/v1/2022.findings-naacl.10

Cited by 16 publications

(17 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Design strategies for improving code generation from LLMs. LLMs are capable of learning and generating instructions, and breaking down or decomposing a large task into smaller subtasks has consistently been found to improve the performance of LLMs [26,37,75,83,84,115,119,121,122]. Jayagopal et al [42] examine the usability of program synthesizers, including GitHub Copilot, by novices, finding problems arising from the difficulty of task decomposition, and recommending that designers offer scaffolding for decomposition.…”

Section: Natural Language Programmingmentioning

confidence: 99%

“What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models

Liu¹,

Sarkar²,

Negreanu³

et al. 2023

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Code-generating large language models map natural language to code. However, only a small portion of the infinite space of naturalistic utterances is effective at guiding code generation. For non-expert end-user programmers, learning this is the challenge of abstraction matching. We examine this challenge in the specific context of data analysis in spreadsheets, in a system that maps the user's natural language query to Python code using the Codex generator, executes the code, and shows the result. We propose grounded abstraction matching, which bridges the abstraction gap by translating the code back into a systematic and predictable naturalistic utterance. In a between-subjects, think-aloud study (n=24), we compare grounded abstraction matching to an ungrounded alternative based on previously established query framing principles. We find that the grounded approach improves end-users' understanding of the scope and capabilities of the code-generating model, and the kind of language needed to use it effectively. CCS CONCEPTS• Human-centered computing → Natural language interfaces; Interactive systems and tools; Empirical studies in HCI .

show abstract

Section: Natural Language Programmingmentioning

confidence: 99%

“What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models

Liu¹,

Sarkar²,

Negreanu³

et al. 2023

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…Instruction-based zero-shot learning Instruction-based zero-shot learning is an innovative approach that leverages natural language instructions and definitions to enable neural models to solve a variety of tasks [50,6,55,18,89,44,45,49]. By providing a human-readable prompt, this method enables easier and more efficient specification of the learning task by utilizing knowledge about the task without data.…”

Section: B3 Detalied Related Workmentioning

confidence: 99%

GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning

Zhao

Liu²,

et al. 2023

Preprint

View full text Add to dashboard Cite

Molecule property prediction has gained significant attention in recent years. The main bottleneck is the label insufficiency caused by expensive lab experiments. In order to alleviate this issue and to better leverage textual knowledge for tasks, this study investigates the feasibility of employing natural language instructions to accomplish molecule-related tasks in a zero-shot setting. We discover that existing molecule-text models perform poorly in this setting due to inadequate treatment of instructions and limited capacity for graphs. To overcome these issues, we propose GIMLET, which unifies language models for both graph and text data. By adopting generalized position embedding, our model is extended to encode both graph structures and instruction text without additional graph encoding modules. GIMLET also decouples encoding of the graph from tasks instructions in the attention mechanism, enhancing the generalization of graph features across novel tasks. We construct a dataset consisting of more than two thousand molecule tasks with corresponding instructions derived from task descriptions. We pretrain GIMLET on the molecule tasks along with instructions, enabling the model to transfer effectively to a broad range of tasks. Experimental results demonstrate that GIMLET significantly outperforms molecule-text baselines in instruction-based zero-shot learning, even achieving closed results to supervised GNN models on tasks such as toxcast and muv.

show abstract

“…Followed by this, FLAN [13] has been proposed which uses instructions to achieve generalization across unseen tasks. Recently, Parmar et al [14] proposed instruction learning for biomedical multitask. Along with that, Mishra et al [25] shows reframing instructional prompts can boost both few-shot and zero-shot model performance.…”

Section: Instruction Learningmentioning

confidence: 99%

“…This dataset also serves to evaluate the generalization of a model, a well-known issue that many language models have failed even though they outperform humans in many popular benchmarks [10,11]. Recently, instruction-learning [12,13,14] have improved model's performance to unseen tasks. Inspired by this, we leverage instruction-tuning to build a model and verify whether instruction learning also show stronger generalization on BioTabQA.…”

Section: Introductionmentioning

confidence: 99%

BioTABQA: Instruction Learning for Biomedical Table Question Answering

Luo¹,

Saxena²,

Mishra³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Table Question Answering (TQA) is an important but under-explored task. Most of the existing QA datasets are in unstructured text format and only few of them use tables as the context. To the best of our knowledge, none of TQA datasets exist in the biomedical domain where tables are frequently used to present information. In this paper, we first curate a table question answering dataset, BioTabQA, using 22 templates and the context from a biomedical textbook on differential diagnosis. BioTabQA can not only be used to teach a model how to answer questions from tables but also evaluate how a model generalizes to unseen questions, an important scenario for biomedical applications. To achieve the generalization evaluation, we divide the templates into 17 training and 5 cross-task evaluations. Then, we develop two baselines using single and multi-tasks learning on BioTabQA. Furthermore, we explore instructional learning, a recent technique showing impressive generalizing performance. Experimental results show that our instruction-tuned model outperforms single and multi task baselines on an average by ∼ 23% and ∼ 6% across various evaluation settings, and more importantly, instruction-tuned model outperforms baselines by ∼ 5% on cross-tasks.

show abstract

In-BoXBART: Get Instructions into Biomedical Multi-Task Learning

Cited by 16 publications

References 26 publications

“What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models

“What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models

GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning

BioTABQA: Instruction Learning for Biomedical Table Question Answering

Contact Info

Product

Resources

About