Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training

Shi, Peng; Ng, Patrick; Wang, Zhiguo; Zhu, Henghui; Li, Alexander Hanbo; Wang, Jun; Santos, Cícero Nogueira dos; Xiang, Bing

doi:10.48550/arxiv.2012.10309

Cited by 8 publications

(10 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on the results, we find that using the generated question as text input is a better choice than these two proposals, thus we did not use them in our main experiments. (Herzig et al 2020;Eisenschlos, Krichene, and Müller 2020;Shi et al 2020;Deng et al 2020;Yin et al 2020;Yu et al 2020;Iida et al 2021;Liu et al 2021). Large scale crawled tables are used for pretraining to enhance the table representation ability of language models.…”

Section: Resultsmentioning

confidence: 99%

“…However, this causes a mismatch between the intermediate pre-training and downstream tasks where questions are one essential component of the tasks. More recently, Shi et al (2020) confirmed that the surrounding text is far from optimal because those texts are dissimilar to the natural language questions in terms of text length, composition and content. The surrounding text of the tables can be quite noisy and may be irrelevant to the tables.…”

Section: Introductionmentioning

confidence: 96%

“…The surrounding text of the tables can be quite noisy and may be irrelevant to the tables. In this work, following Shi et al (2020) and Eisenschlos, Krichene, and Müller (2020), we leverage both sequence-to-sequence generation model and synchronous context-free grammar to generate the questionanswer pairs for intermediate pre-training.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Generation-Focused Table-Based Intermediate Pre-training for Free-Form Question Answering

Shi

Ng²,

Feng³

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

Question answering over semi-structured tables has attracted significant attention in the NLP community. However, most of the existing work focus on questions that can be answered with short-form answer, i.e. the answer is often a table cell or aggregation of multiple cells. This can mismatch with the intents of users who want to ask more complex questions that require free-form answers such as explanations. To bridge the gap, most recently, pre-trained sequence-to-sequence language models such as T5 are used for generating free-form answers based on the question and table inputs. However, these pre-trained language models have weaker encoding abilities over table cells and schema. To mitigate this issue, in this work, we present an intermediate pre-training framework, Generation-focused Table-based Intermediate Pre-training (GENTAP), that jointly learns representations of natural language questions and tables. GENTAP learns to generate via two training objectives to enhance the question understanding and table representation abilities for complex questions. Based on experimental results, models that leverage GENTAP framework outperform the existing baselines on FETAQA benchmark. The pre-trained models are not only useful for free-form question answering, but also for few-shot data-to-text generation task, thus showing good transfer ability by obtaining new state-of-the-art results.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 96%

See 1 more Smart Citation

Generation-Focused Table-Based Intermediate Pre-training for Free-Form Question Answering

Shi

Ng²,

Feng³

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…Methods like TAPAS [38] and TaPEx [69] [94] encodes tables with a tabular graph transformer, contexts with a BERT-based model [30], and queries with a FastText method [57], and calculates the relevance score of the query-table and query-context matching by multi-layer perceptron (MLP). [29], GraPPa [105], GAP [81] promote semantic parsing through pre-training on synthetic or human-labeled table-text data.…”

Section: Table Factmentioning

confidence: 99%

Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks

Dong¹,

Cheng²,

He³

et al. 2022

Preprint

View full text Add to dashboard Cite

Since a vast number of tables can be easily collected from web pages, spreadsheets, PDFs, and various other document types, a flurry of table pre-training frameworks have been proposed following the success of text and images, and they have achieved new state-of-thearts on various tasks such as table question answering, table type recognition, column relation classification, table search, formula prediction, etc. To fully use the supervision signals in unlabeled tables, a variety of pre-training objectives have been designed and evaluated, for example, denoising cell values, predicting numerical relationships, and implicitly executing SQLs. And to best leverage the characteristics of (semi-)structured tables, various tabular language models, particularly with specially-designed attention mechanisms, have been explored. Since tables usually appear and interact with free-form text, table pre-training usually takes the form of table-text joint pre-training, which attracts significant research interests from multiple domains. This survey aims to provide a comprehensive review of different model designs, pre-training objectives, and downstream tasks for table pre-training, and we further share our thoughts and vision on existing challenges and future opportunities.

show abstract

“…The current state-of-the-art on SPIDER [32] dataset are PICARD [24] and SADGA [6]. SADGA is built on pretrained GAP [25] model which is in turn a modification of RAT-SQL [28] framework. While PICARD [24] is a text-to-SQL semantic parser built upon pre-trained encoder-decoder models.…”

Section: Related Workmentioning

confidence: 99%

AskYourDB: An end-to-end system for querying and visualizing relational databases using natural language

Joseph¹,

Raj²,

Yadav³

et al. 2022

Preprint

View full text Add to dashboard Cite

Querying databases for the right information is a time consuming and error-prone task and often requires experienced professionals for the job. Furthermore, the user needs to have some prior knowledge about the database. There have been various efforts to develop an intelligence which can help business users to query databases directly. However, there has been some successes, but very little in terms of testing and deploying those for real world users. In this paper, we propose a semantic parsing approach to address the challenge of converting complex natural language into SQL and institute a product out of it. For this purpose, we modified state-of-the-art models, by various pre and post processing steps which make the significant part when a model is deployed in production. To make the product serviceable to businesses we added an automatic visualization framework over the queried results.

show abstract

Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training

Cited by 8 publications

References 33 publications

Generation-Focused Table-Based Intermediate Pre-training for Free-Form Question Answering

Generation-Focused Table-Based Intermediate Pre-training for Free-Form Question Answering

Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks

AskYourDB: An end-to-end system for querying and visualizing relational databases using natural language

Contact Info

Product

Resources

About