Table Search Using a Deep Contextualized Language Model

Abstract

Pretrained contextualized language models such as BERT have achieved impressive results on various natural language processing benchmarks. Benefiting from multiple pretraining tasks and large scale training corpora, pretrained models can capture complex syntactic word relations. In this paper, we use the deep contextualized language model BERT for the task of ad hoc table retrieval. We investigate how to encode table content considering the table structure and input length limit of BERT. We also propose an approach that incorporates features from prior literature on table retrieval and jointly trains them with BERT. In experiments on public datasets, we show that our best approach can outperform the previous state-of-the-art method and BERT baselines with a large margin under different evaluation metrics. CCS CONCEPTS• Information systems → Content analysis and feature selection; Retrieval models and ranking; • Computing methodologies → Search methodologies.

Retrieving Complex Tables with Multi-Granular Graph Representation Learning

“…We analyzed the cell adjacency of these tables and discovered that 1,886 of them came with nested layout structures involving merged cells. Following previous works [15,66], we run 5-fold cross-validation on this dataset. Baselines.…”

“…Sun et al [53] proposed query-specific attention mechanisms to aggregate table cell embeddings, which provided a flexible way to induce the relevance between a query and different parts of a table with softmax classifiers. Through this direction, Chen et al [15] designed an embedding-based feature selection technique to select most relevant content from cells, rows and columns of each table, where BERT [17] was used to encode the concatenated text sequence of selected table content. Chen et al [15] also observed that combining neural network methods and feature-based methods achieved further improvements.…”

“…Sun et al [53] utilized attention mechanism to select cell embeddings over each row and each column. The approaches in prior studies have offered decent performance in intrinsic evaluation settings [15,53,66]. However, they are still quite distant from a generalizable solution to the NLTR problem.…”

Neural ranking models for document retrieval

“…Lin et al (2020) focused mainly on pretrained Transformers (Vaswani et al, 2017) for text ranking, where they showed that a BERT-based multi-stage ranking model is a potential choice for a tradeoff between effectiveness and efficiency of a neural ranking model. Xu et al (2020b) reviewed deep learning models for matching in document retrieval and recommendation systems. The authors grouped the neural ranking models into two categories which are representation learning and matching function learning.…”

“…The authors grouped the neural ranking models into two categories which are representation learning and matching function learning. Compared to the survey of Xu et al (2020b), in addition to grouping neural ranking models into five categories based on the neural components and design (Sect. 6), we summarize multiple models based on nine features that are frequently presented in the neural ranking models (Sect.…”