Slice-Aware Neural Ranking

Penha, Gustavo; Hauff, Claudia

doi:10.18653/v1/2020.scai-1.1

“…SBL exhibits better performance than a mixture of experts (Jacobs et al, 1991) and multi-task learning (Caruana, 1997), with reduced run-time cost and parameters (Chen et al, 2019). Recently Gustavo et al (Penha & Hauff, 2020) have employed the concept of SBL to understand failures of ranking models and identify difficult instances in order to improve ranking performance. Our work applies the idea to improve skill routing performance on low traffic but critical intents in conversational AI systems.…”

Section: Slice-based Learningmentioning

confidence: 99%

Handling Long-Tail Queries with Slice-Aware Conversational Systems

Wang,

Kim,

Park

et al. 2021

Preprint

View full text Add to dashboard Cite

We have been witnessing the usefulness of conversational AI systems such as Siri and Alexa, directly impacting our daily lives. These systems normally rely on machine learning models evolving over time to provide quality user experience. However, the development and improvement of the models are challenging because they need to support both high (head) and low (tail) usage scenarios, requiring fine-grained modeling strategies for specific data subsets or slices. In this paper, we explore the recent concept of slice-based learning (SBL) (Chen et al., 2019) to improve our baseline conversational skill routing system on the tail yet critical query traffic. We first define a set of labeling functions to generate weak supervision data for the tail intents. We then extend the baseline model towards a slice-aware architecture, which monitors and improves the model performance on the selected tail intents. Applied to de-identified live traffic from a commercial conversational AI system, our experiments show that the slice-aware model is beneficial in improving model performance for the tail intents while maintaining the overall performance.

show abstract

“…SBL has been recently used in many applications. Penha et al (Penha and Hauff, 2020) proposed to adapt SBL to improve ranking performance and capture the failures of the ranker model. Wang et al (Wang et al, 2021) recently implemented SBL in a commercial conversational AI system in order to handle the long-tail problem of imbalanced distribution in customer queries and further improved the performance of the conversational skill routing components (Li et al, 2021;Kim et al, 2018b,a).…”

Section: Related Workmentioning

confidence: 99%

Learning Slice-Aware Representations with Mixture of Attentions

Wang

¹

,

Lee

²

,

Park

³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

0

View full text Add to dashboard Cite

Real-world machine learning systems are achieving remarkable performance in terms of coarse-grained metrics like overall accuracy and F-1 score. However, model improvement and development often require fine-grained modeling on individual data subsets or slices, for instance, the data slices where the models have unsatisfactory results. In practice, it gives tangible values for developing such models that can pay extra attention to critical or interested slices while retaining the original overall performance. This work extends the recent slice-based learning (SBL) (Chen et al., 2019) with a mixture of attentions (MoA) to learn slice-aware dual attentive representations. We empirically show that the MoA approach outperforms the baseline method as well as the original SBL approach on monitored slices with two natural language understanding (NLU) tasks.

show abstract

“…SBL has been recently used in many applications. Penha et al (Penha and Hauff, 2020) proposed to adapt SBL to improve ranking performance and capture the failures of the ranker model. Wang et al (Wang et al, 2021) recently implemented SBL in a commercial conversational AI system in order to handle the long-tail problem of imbalanced distribution in customer queries and further improved the performance of the conversational skill routing components (Li et al, 2021;Kim et al, 2018b,a).…”

Section: Related Workmentioning

confidence: 99%

Learning Slice-Aware Representations with Mixture of Attentions

Wang

¹

,

Lee

²

,

Park

³

et al. 2021

Preprint

2

0

View full text Add to dashboard Cite

Real-world machine learning systems are achieving remarkable performance in terms of coarse-grained metrics like overall accuracy and F-1 score. However, model improvement and development often require fine-grained modeling on individual data subsets or slices, for instance, the data slices where the models have unsatisfactory results. In practice, it gives tangible values for developing such models that can pay extra attention to critical or interested slices while retaining the original overall performance. This work extends the recent slice-based learning (SBL) (Chen et al., 2019) with a mixture of attentions (MoA) to learn slice-aware dual attentive representations. We empirically show that the MoA approach outperforms the baseline method as well as the original SBL approach on monitored slices with two natural language understanding (NLU) tasks.

show abstract

Slice-Aware Neural Ranking

Cited by 3 publications

References 28 publications

Handling Long-Tail Queries with Slice-Aware Conversational Systems

Handling Long-Tail Queries with Slice-Aware Conversational Systems

Learning Slice-Aware Representations with Mixture of Attentions

Learning Slice-Aware Representations with Mixture of Attentions

Contact Info

Product

Resources

About