INFOTABS: Inference on Tables as Semi-structured Data

Abstract

In this paper, we observe that semi-structured tabulated text is ubiquitous; understanding them requires not only comprehending the meaning of text fragments, but also implicit relationships between them. We argue that such data can prove as a testing ground for understanding how we reason about information. To study this, we introduce a new dataset called INFOTABS, comprising of human-written textual hypotheses based on premises that are tables extracted from Wikipedia info-boxes. Our analysis shows that the semi-structured, multi-domain and heterogeneous nature of the premises admits complex, multi-faceted reasoning. Experiments reveal that, while human annotators agree on the relationships between a table-hypothesis pair, several standard modeling strategies are unsuccessful at the task, suggesting that reasoning about tables can pose a difficult modeling challenge.

Incorporating External Knowledge to Enhance Tabular Reasoning

“…Datasets Our experiments uses InfoTabS, a tabular inference dataset from Gupta et al (2020). The dataset is heterogeneous in the types of tables and keys, and relies on background knowledge and common sense.…”

“…We represent the premise table by converting each table row into a sentence, and then appending them into a paragraph, i.e. the Para representation of Gupta et al (2020).…”

“…All improvements are statistically significant with p < 0.05, except α 1 for BPR representation w.r.t to Para (Original). Here the Human and Para results are taken from Gupta et al (2020).…”

“…Better Paragraph Representation (BPR): One way to represent the premise table is to use a universal template to convert each row of the table into sentence which serves as input to a BERT-style model. Gupta et al (2020) suggest that in a table titled t, a row with key k and value v should be converted to a sentence using the template: "The k of t are v." Despite the advantage of simplicity, the approach produces ungrammatical sentences. In our example, the template converts the Founded row to the sentence "The Founded of New York Stock Exchange are May 17, 1792; 226 years ago.…”

“…In this paper, we study the harder problem of reasoning about tabular premises, as instantiated in datasets such as TabFact and InfoTabS (Gupta et al, 2020). This problem is similar to standard NLI, but the premises are Wikipedia tables rather than sentences.…”