Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning

Abstract

When pre-trained contextualized embeddingbased models developed for unstructured data are adapted for structured tabular data, they perform admirably. However, recent probing studies show that these models use spurious correlations, and often predict inference labels by focusing on false evidence or ignoring it altogether. To study this issue, we introduce the task of Trustworthy Tabular Reasoning, where a model needs to extract evidence to be used for reasoning, in addition to predicting the label. As a case study, we propose a twostage sequential prediction approach, which includes an evidence extraction and an inference stage. First, we crowdsource evidence row labels and develop several unsupervised and supervised evidence extraction strategies for INFOTABS, a tabular NLI benchmark. Our evidence extraction strategy outperforms earlier baselines. On the downstream tabular inference task, using only the automatically extracted evidence as the premise, our approach outperforms prior benchmarks.

A new era of AI‐assisted journalism at Bloomberg

“…Neural models in NLP can be especially vulnerable to adversarial attacks, and it is important to have solid mechanisms for mitigating such attacks, while maintaining good task performance. Based on the intuition that automatically generated adversarial inputs can be undone by learning to manipulate the textual input instead of retraining the classification model, we formulate a mechanism with the valuable property of transferability of defense, which allows the underlying classification model to be deployed to new and unknown models without retraining (Gupta et al 2023). The assumption is that a single shared model can be more robust than the individual ones, while also reducing overhead when deploying it into new models.…”

Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data

“…Systematic Probes for Tables. Tables have been utilized previously used to create probes for table grounding (Gupta et al, 2022b) or recasting non-NLI datasets (e.g. question-answering) to NLI (Jena et al, 2022).…”