Constrained Language Models Yield Few-Shot Semantic Parsers

Abstract

We explore the use of large pretrained language models as few-shot semantic parsers. The goal in semantic parsing is to generate a structured meaning representation given a natural language input. However, language models are trained to generate natural language.To bridge the gap, we use language models to paraphrase inputs into a controlled sublanguage resembling English that can be automatically mapped to a target meaning representation. Our results demonstrate that with only a small amount of data and very little code to convert into English-like representations, our blueprint for rapidly bootstrapping semantic parsers leads to surprisingly effective performance on multiple community tasks, greatly exceeding baseline methods also trained on the same limited data.

Learning To Retrieve Prompts for In-Context Learning

“…Such approaches include prompt engineering through the use of manual patterns (Petroni et al, 2019;Schick and Schütze, 2021), and also methods for extracting either hard (Shin et al, 2020;Haviv et al, 2021) or soft (Li and Liang, 2021;Zhong et al, 2021;Qin and Eisner, 2021) prompts automatically. Shin et al (2021) used GPT-3 to select training examples for in-context learning. However, their focus was not on training a prompt retriever, but instead on representing logical forms with a pseudo-language, and applying constraints that are based on the formal language at decoding time to improve generation.…”

“…Conversely, our approach takes advantage of the generative LM itself and is thus more general. Shin et al (2021) used GPT-3 to select examples for the prompt in the context of few-shot semantic parsing. However, rather than training a retriever, they randomly sample a large set of question-program pairs from the training set, and choose those that are similar to the target instance question according to GPT-3.…”

On The Ingredients of an Effective Zero-shot Semantic Parser

“…Still, as discussed in §3.1.1, we remark that for users proficient in the grammar formalism, curating a handful of idiomatic production rules is still more efficient than labeling parallel samples to exhaustively cover compositional logical patterns and diverse language style, and the size of annotated samples required could be orders-of-magnitude larger compared to the size of the grammar. Meanwhile, the process of creating production rules could potentially be simplified by allowing users to define them using natural language instead of λ-calculus logical rules, similar in the spirit of the studies on naturalizing programs using canonical language (Wang et al, 2017;Shin et al, 2021;Herzig et al, 2021).…”

“…Like our model, such methods use lexicons to capture alignments between NL phrases and logical predicates (Goldwasser et al, 2011), while our method does not require real utterances. Finally, methods based on OVERNIGHT (Wang et al, 2015) synthesize parallel corpora from SCFGs (Cheng et al, 2019;Xu et al, 2020a) or neural sequence models (Guo et al, 2018), and attempt to bridge the gaps between canonical and real utterances via paraphrase detection and generation (Su and Yan, 2017;Shin et al, 2021), or representation learning (Marzoev et al, 2020).…”

DeepA2: A Modular Framework for Deep Argument Analysis with Pretrained Neural Text2Text Language Models

“…Neural semantic parsing uses sequence models to formalize natural language sentences (Kamath and Das 2019). Shin et al (2021) show that PTLMs are zero-shot parsers, and that intermediate steps which rephrase and streamline the original input before parsing it to a formal language improve accuracy.…”