Low-Resource Adaptation of Open-Domain Generative Chatbots

Abstract

Recent work building open-domain chatbots has demonstrated that increasing model size improves performance (Adiwardana et al., 2020;Roller et al., 2020). On the other hand, latency and connectivity considerations dictate the move of digital assistants on the device (Verge, 2021). Giving a digital assistant like Siri, Alexa, or Google Assistant the ability to discuss just about anything leads to the need for reducing the chatbot model size such that it fits on the user's device. We demonstrate that low parameter models can simultaneously retain their general knowledge conversational abilities while improving in a specific domain. Additionally, we propose a generic framework that accounts for variety in question types, tracks reference throughout multiturn conversations, and removes inconsistent and potentially toxic responses. Our framework seamlessly transitions between chatting and performing transactional tasks, which will ultimately make interactions with digital assistants more human-like. We evaluate our framework on 1 internal and 4 public benchmark datasets using both automatic (Perplexity) and human (SSA -Sensibleness and Specificity Average) evaluation metrics and establish comparable performance while reducing model parameters by 90%.

ChatClimate: Grounding conversational AI in climate science

“…This approach has been shown to improve the accuracy and efficiency of LLMs in a range of domains, including healthcare and finance 59 . Domain-specific chatbots use a similar concept, where an agent has access to a carefully curated in-house database (long-term memory) to answer domain-specific questions 60 . These chatbots can provide customized responses based on the available information in their database, allowing for more accurate and relevant answers to user queries.…”