DSGPT: Domain-Specific Generative Pre-Training of Transformers for Text Generation in E-commerce Title and Review Summarization

Abstract

We propose a novel domain-specific generative pre-training (DS-GPT) method for text generation and apply it to the product title and review summarization problems on E-commerce mobile display. First, we adopt a decoder-only transformer architecture, which fits well for fine-tuning tasks by combining input and output all together. Second, we demonstrate utilizing only small amount of pretraining data in related domains is powerful. Pre-training a language model from a general corpus such as Wikipedia or the Common Crawl requires tremendous time and resource commitment, and can be wasteful if the downstream tasks are limited in variety. Our DSGPT is pre-trained on a limited dataset, the Chinese short text summarization dataset (LCSTS). Third, our model does not require product-related human-labeled data. For title summarization task, the state of art explicitly uses additional background knowledge in training and predicting stages. In contrast, our model implicitly captures this knowledge and achieves significant improvement over other methods, after fine-tuning on the public Taobao.com dataset. For review summarization task, we utilize JD.com in-house dataset, and observe similar improvement over standard machine translation methods which lack the flexibility of fine-tuning. Our proposed work can be simply extended to other domains for a wide range of text generation tasks.

Pretrained Language Models for Text Generation: A Survey

“…Goodwin et al [58] study how to generate summaries conditioned on different topics or questions. DSGPT [218] proposes to pretrain in e-commerce scenarios and explore the product title and review summarization. Furthermore, PASS [137] aggregates different reviews of one product into a short summary.…”

Automatic product copywriting for e‐commerce

“…(2020), or on domain‐specific corpus Zhang et al. (2021); Zou et al. (2020) with well‐defined pretraining tasks.…”

“…Recently, the pretraining plus fine-tuning paradigm has gained traction and been widely applied in real-world applications. Under this paradigm, models are first pretrained on large-scale corpus Devlin et al (2019a); Brown et al (2020), or on domain-specific corpus Zhang et al (2021); Zou et al (2020) with well-defined pretraining tasks. The resulting pretrained models are then fine-tuned to adapt to different downstream tasks.…”

Automatic Product Copywriting for E-commerce

“…Recently, the pre-training plus fine-tuning paradigm has gained traction and been widely applied in real-world applications. Under this paradigm, models are first pre-trained on large-scale corpus (Devlin et al 2019a;Brown et al 2020), or on domain-specific corpus (Zhang et al 2021) with welldefined pre-training tasks. The resulting pre-trained models are then fine-tuned to adapt to different downstream tasks.…”