Personalizing Natural Language Understanding using Multi-armed Bandits and Implicit Feedback

Abstract

Natural Language Understanding (NLU) models on voice-controlled speakers face several challenges. In particular, music streaming services have large catalogs, often containing millions of songs, artists, and albums and several thousands of custom playlists and stations. In many cases there is ambiguity and little structural difference between carrier phrases and entity names. In this work, we describe how we leveraged multi-armed bandits in combination with implicit customer feedback to improve accuracy and personalization of responses to voice request in the music domain. Our models are tested in a large-scale industrial system containing several other components. In particular, we focused on using this technology to correct errors made by upstream NLU models and personalize responses based on customer preferences and music provider functionality. The models resulted in significant improvement of playback rate for Amazon Music and are deployed in systems serving several countries and languages. We further used the implicit feedback of the customers to generate weakly labeled training data for the NLU models. This improved the experience for customers using other music providers on all Alexa devices. CCS CONCEPTS • Computing methodologies → Online learning settings; Learning from implicit feedback; • Information systems → Personalization.

Applying Multi-Armed Bandit algorithms for music recommendations at Spotify

“…The first is the Explore-Then-Commit Algorithm. The core idea of ETC algorithm is to explore by playing each arm a fixed number of times and then exploit by committing to the arm that appeared the best during exploration, which is the same idea as AB testing [7]. During the exploration phase, the learner chooses each arm in a round-robin fashion until all 𝑘 arms are selected 𝑚 times each.…”

Improving Large-Scale Conversational Assistants using Model Interpretation based Training Sample Selection

“…Falke et al (2020) leverage user paraphrasing behavior in dialog systems to automatically collect annotations for long-tail utterances. Moerchen et al (2020) present an approach where implicit negative feedback from the user is used to train a re-ranker that is then applied to pick correct annotations for under-performing utterances.…”