Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

Süzgün, Mirac; Scales, Nathan; Schärli, Nathanael; Gehrmann, Sebastian; Tay, Yi; Chung, Hyung Won; Chowdhery, Aakanksha; Le, Quoc V.; Chi, Ed; Zhou, Denny; Lee, Jason

doi:10.48550/arxiv.2210.09261

Cited by 29 publications

(51 citation statements)

References 13 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We show that a model trained on this collection outperforms other public collections on all tested evaluation benchmarks, including the original Flan 2021 (Wei et al, 2021), T0++ (Sanh et al, 2021), Super-Natural Instructions (Wang et al, 2022c), and the concurrent work on OPT-IML (Iyer et al, 2022). As shown in Figure 1, this includes a 4.2%+ and 8.5% improvements on the MMLU (Hendrycks et al, 2020) and BIG-Bench Hard (Suzgun et al, 2022) evaluation benchmarks, for equally sized models.…”

Section: Introductionmentioning

confidence: 81%

“…We finetune on the prefix language model adapted T5-LM (Lester et al, 2021), using the XL (3B) size for all models, unless otherwise stated. We evaluate on (a) a suite of 8 "Held-In" tasks represented within the 1800+ training task collection (4 question answering and 4 natural language inference validation sets), (b) Chain-of-Thought (CoT) tasks (5 validation sets), and (c) the MMLU (Hendrycks et al, 2020) and BBH (Suzgun et al, 2022) benchmarks as our set of "Held-Out" tasks, as they are not included as part of Flan 2022 finetuning. The Massivley Multitask Language Understanding benchmark (MMLU) broadly tests reasoning and knowledge capacity across 57 tasks in the sciences, social sciences, humanities, business, health, among other subjects.…”

Section: Methodsmentioning

confidence: 99%

“…Figure 1: Comparing public instruction tuning collections on Held-In, Chain-of-Thought, and Held-out evaluation suites, such as BigBench Hard (Suzgun et al, 2022) and MMLU (Hendrycks et al, 2020). All models except OPT-IML-Max (175B) are T5-XL with 3B parameters.…”

Section: T5-xl Flan 2022mentioning

confidence: 99%

See 2 more Smart Citations

The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

Longpre¹,

Hou²,

Vu³

et al. 2023

Preprint

View full text Add to dashboard Cite

We study the design decisions of publicly available instruction tuning methods, and break down the development of Flan 2022 . Through careful ablation studies on the Flan Collection of tasks and methods, we tease apart the effect of design decisions which enable Flan-T5 to outperform prior work by 3-17%+ across evaluation settings. We find task balancing and enrichment techniques are overlooked but critical to effective instruction tuning, and in particular, training with mixed prompt settings (zero-shot, few-shot, and chain-of-thought) actually yields stronger (2%+) performance in all settings. In further experiments, we show Flan-T5 requires less finetuning to converge higher and faster than T5 on single downstream tasks-motivating instruction-tuned models as more computationally-efficient starting checkpoints for new tasks. Finally, to accelerate research on instruction tuning, we make the Flan 2022 collection of datasets, templates, and methods publicly available. 1

show abstract

Section: Introductionmentioning

confidence: 81%

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

Longpre¹,

Hou²,

Vu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…When it was introduced, CoT prompting demonstrated benefits for math and commonsense reasoning. Since then, Suzgun et al (2022) report that CoT prompting gives substantial improvements for a hard subset of the BIG-Bench tasks (Srivastava et al, 2022). 6 This makes it a promising prompt for our proposed task of reasoning about implications of negation.…”

Section: C1 Fully Finetunedmentioning

confidence: 99%

CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation

Ravichander¹,

Gardner²,

Marasović³

2022

Preprint

View full text Add to dashboard Cite

The full power of human language-based communication cannot be realized without negation. All human languages have some form of negation. Despite this, negation remains a challenging phenomenon for current natural language understanding systems. To facilitate the future development of models that can process negation effectively, we present CONDAQA, the first English reading comprehension dataset which requires reasoning about the implications of negated statements in paragraphs. We collect paragraphs with diverse negation cues, then have crowdworkers ask questions about the implications of the negated statement in the passage. We also have workers make three kinds of edits to the passage-paraphrasing the negated statement, changing the scope of the negation, and reversing the negation-resulting in clusters of question-answer pairs that are difficult for models to answer with spurious shortcuts. CONDAQA features 14,182 question-answer pairs with over 200 unique negation cues and is challenging for current state-of-the-art models. The best performing model on CONDAQA (UNIFIEDQA-V2-3B) achieves only 42% on our consistency metric, well below human performance which is 81%. We release our dataset, along with fully-finetuned, few-shot, and zero-shot evaluations, to facilitate the development of future NLP methods that work on negated language. * Work undertaken while Abhilasha Ravichander and Ana Marasović were at the Allen Institute for AI.1 COntrastively-annotated Negation DAtaset of Question-Answer pairs

show abstract

“…To assess SELF-ICL's effectiveness on challenging, unexpected tasks for which existing demonstrations are hard to come by, we perform evaluation on a suite of 23 tasks from BIG-Bench Hard (BBH) (Suzgun et al, 2022). In a head-tohead comparison, experimental results show 16-1-6 (win-tie-lose) for SELF-ICL versus standard zeroshot on the 23 tasks.…”

Section: Introductionmentioning

confidence: 99%

A study of adults' orientations to learning at the national open university of Taiwan

Chen¹

View full text Add to dashboard Cite

Lifelogging has gained more attention due to its wide applications, such as personalized recommendations or memory assistance. The issues of collecting and extracting personal life events have emerged. People often share their life experiences with others through conversations. However, extracting life events from conversations is rarely explored. In this paper, we present Life Event Dialog, a dataset containing fine-grained life event annotations on conversational data. In addition, we initiate a novel conversational life event extraction task and differentiate the task from the public event extraction or the life event extraction from other sources like microblogs. We explore three information extraction (IE) frameworks to address the conversational life event extraction task: OpenIE, relation extraction, and event extraction. A comprehensive empirical analysis of the three baselines is established. The results suggest that the current event extraction model still struggles with extracting life events from human daily conversations. Our proposed life event dialog dataset and in-depth analysis of IE frameworks will facilitate future research on life event extraction from conversations.

show abstract

Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

Cited by 29 publications

References 13 publications

The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation

A study of adults' orientations to learning at the national open university of Taiwan

Contact Info

Product

Resources

About