Importance of Synthesizing High-quality Data for Text-to-SQL Parsing

Selective Demonstrations for Cross-domain Text-to-SQL

“…Previous Work We compare ODIS with state-ofthe-art approaches that either require finetuning on the Spider training set or utilize in-context learning. For the finetuning-based methods, we select SmBoP which utilizes RoBERTa-large as the pretrained model, as well as T5+Picard (Scholak et al, 2021), ShiP+Picard (Zhao et al, 2022), and RESDSQL (Li et al, 2023a), which employ T5-3B (Raffel et al, 2020) as the pretrained model.…”

“…end if 14: end for Synthetic data generation To generate synthetic data, we follow previous work to first sample synthetic SQL queries {y i } and then translate SQL queries into natural language questions {x i } (Zhong et al, 2020b;Zhao et al, 2022). We use SHiP (Zhao et al, 2022) to sample synthetic SQL queries, which extract templates from out-of-domain databases and sample columns and values from the test database to fill the templates. After obtaining synthetic SQL queries, we use the Codex to generate corresponding synthetic NLQs, in the same procedure as in our analysis of in-domain SQL distribution.…”

“…Previous Work with Finetuning SmBoP RB 78.0 T5+Picard (Scholak et al, 2021) T5-3B 79.1 ShiP+Picard (Zhao et al, 2022) T5-3B 81.4 RESDSQL+NatSQL (Li et al, 2023a) T5-3B 84.1…”

“…This suggests that employing a better initial model may lead to further enhancements in the performance of the ODIS framework. Impact of synthetic data To assess the impact of synthetic data quality in ODIS framework, we compare two SQL synthesizing methods SHiP (Zhao et al, 2022) and GAZP (Zhong et al, 2020b). Both methods extract the templates of SQL queries from the Spider training set and fill the template with the database schema and content with a new database, however, SHiP leverages a schema-weighted sampling to make the synthetic SQL queries more realistic by controlling the relationship of sampled columns.…”

“…Data synthesis for Text-to-SQL Due to the high cost of acquiring annotated text-to-SQL data, previous research has explored the use of synthetic data as an alternative approach to enhance the performance of cross-domain text-to-SQL models. These works primarily focused on data augmentation, such as generating additional examples for the training databases (Guo et al, 2018;Yu et al, 2021;Liu et al, 2021b;Zhao et al, 2022). Zhong et al (2020b) first synthesized data for test databases and adapted pretrained text-to-SQL models on the synthetic data associated with the test database before inference.…”