Småprat: DialoGPT for Natural Language Generation of Swedish Dialogue by Transfer Learning

Abstract

Building open-domain conversational systems (or chatbots) that produce convincing responses is a recognized challenge. Recent state-of-the-art (SoTA) transformer-based models for the generation of natural language dialogue have demonstrated impressive performance in simulating human-like, single-turn conversations in English. This work investigates, by an empirical study, the potential for transfer learning of such models to Swedish language. DialoGPT, an English language pre-trained model, is adapted by training on three different Swedish language conversational datasets obtained from publicly available sources. Perplexity score (an automated intrinsic language model metric) and surveys by human evaluation were used to assess the performances of the fine-tuned models, with results that indicate that the capacity for transfer learning can be exploited with considerable success. Human evaluators asked to score the simulated dialogue judged over 57% of the chatbot responses to be human-like for the model trained on the largest (Swedish) dataset. We provide the demos and model checkpoints of our English and Swedish chatbots on the HuggingFace platform for public use.

State-of-the-art in Open-domain Conversational AI: A Survey

“…A turn (or utterance) in a conversation is each single contribution from a speaker (Schegloff, 1968;Jurafsky and Martin, 2020). The data may be from written conversations, such as the MultiWOZ (Eric et al, 2020), transcripts of human-human spoken conversations, such as the Gothenburg Dialogue Corpus (GDC) (Allwood et al, 2003), crowdsourced conversations, such as the EmpatheticDialogues (Rashkin et al, 2019), and social media conversations like Familjeliv 1 or Reddit 2 (Adewumi et al, 2022c;Adewumi et al, 2022a). As already acknowledged that the amount of data needed for training deep ML models is usually large, they are normally first pretrained on large, unstructured text or conversations before being finetuned on specific conversational data.…”

State-of-the-Art in Open-Domain Conversational AI: A Survey

“…A turn (or utterance) in a conversation is each single contribution from a speaker [2,9]. The data may be from written conversations, such as the MultiWOZ [10], transcripts of human-human spoken conversations, such as the Gothenburg Dialogue Corpus (GDC) [11], crowdsourced conversations, such as the EmpatheticDialogues [12], and social media conversations such as Familjeliv (familjeliv.se) or Reddit (reddit.com) [13,14]. As already acknowledged that the amount of data needed for training deep ML models is usually large, they are normally first pretrained on large, unstructured text or conversations before being fine-tuned on specific conversational data.…”

State-of-the-art in Open-domain Conversational AI: A Survey

“…The rest of the paper is organized as follows. The Background Section (2) presents brief details about some topics in conversational AI; the Benefits of Conversational AI Section (3) highlights some of the benefits that motivate research in conversational AI; the Methods Section (4) describes the details of the approach for the two investigations carried out in this survey; two Results of the Survey Sections (5 & 6) then follow with details of the outcome of the methods; thereafter, the Existing Challenges Section (7) shares the prevailing challenges to obtaining "human" performance; Open-domain Conversational AI for Low-resource Languages Section (8) discusses this critical challenge and some of the attempts at solving it; the Related Work Section (9) highlights previous related reviews‚ the Conclusion Section (11) summarizes the study after the limitations are given in the Limitation Section.…”

“…A turn (or utterance) in a conversation is each single contribution from a speaker [2,7]. The data may be from written conversations, such as the MultiWOZ [8], transcripts of human-human spoken conversations, such as the Gothenburg Dialogue Corpus (GDC) [9], crowdsourced conversations, such as the EmpatheticDialogues [10], and social media conversations like Familjeliv 1 or Reddit 2 [11,12]. As already acknowledged that the amount of data needed for training deep ML models is usually large, they are normally first pretrained on large, unstructured text or conversations before being finetuned on specific conversational data.…”