Alexa Prize — State of the Art in Conversational AI

Abstract

To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5 million dollar competition that challenges university teams to build conversational agents, or "socialbots", that can converse coherently and engagingly with humans on popular topics for 20 minutes. The Alexa Prize offers the academic community a unique opportunity to perform research at scale with real conversational data obtained by interacting with millions of Alexa users, along with user-provided ratings and feedback, over several months. This enables teams to effectively iterate, improve and evaluate their socialbots throughout the competition. Sixteen teams were selected for the inaugural competition last year. To build their socialbots, the students combined state-of-the-art techniques with their own novel strategies in the areas of Natural Language Understanding and Conversational AI. This article reports on the research conducted over the 2017-2018 year. While the 20 minute grand challenge was not achieved in the first year, the competition produced several conversational agents that advanced the state of the art, are interesting for everyday users to interact with, and help form a baseline for the second year of the competition.

Predictive Engagement: An Efficient Metric for Automatic Evaluation of Open-Domain Dialogue Systems

“…Engagement estimation has been addressed in many spoken dialogue systems based on a listener's multimodal behavior or acoustic features of conversations (Yu, Aoki, and Woodruff 2004;Inoue et al 2018). Heuristic measurements of engagement scores have been proposed by many researchers, but have their own shortcomings Khatri et al 2018;Ghandeharioun et al 2019). In the Alexa prize competition, the engagement score of dialogue systems is calculated based on the number of turns and the total duration of conversation (Venkatesh et al 2018;Khatri et al 2018).…”

“…Heuristic measurements of engagement scores have been proposed by many researchers, but have their own shortcomings Khatri et al 2018;Ghandeharioun et al 2019). In the Alexa prize competition, the engagement score of dialogue systems is calculated based on the number of turns and the total duration of conversation (Venkatesh et al 2018;Khatri et al 2018). This approach suffers from the weakness that it may classify a long conversation as engaging whereas two interlocutors were simply having difficulty understanding each other.…”

Mind the Gap: Discrepancy Between Customer Expectation and Perception on Commercial Chatbots Usage

“…When a user sends a message with "hello", natural language processing helps chatbots to understand that this is a standard greeting and leverage its artificial intelligence abilities to come up with a suitable response. It is likely that chatbots will likely respond with a greeting in this case (Khatri et al, 2018).…”

What’s in a name and does CUI matter?

“…Firstly, I think we must be cautious in avoiding the previous mistakes of the AI community for we risk another AI winter [6] beset with over-promises and under-deliverance. The 'space race' of building ever-more realistic voices and devices capable of conversing as much as possible [9], risks sidelining efforts to understand the needs and benefits of CUIs-we must not eschew functional improvements in favour of glitz. The promises of a fully-conversant AI around the corner are only palatable for so long and investors will tire, and academics will move on (as they have done before!…”