Prior Knowledge Driven Label Embedding for Slot Filling in Natural Language Understanding

Abstract

Traditional slot filling in natural language understanding (NLU) predicts a one-hot vector for each word. This form of label representation lacks semantic correlation modelling, which leads to severe data sparsity problem, especially when adapting an NLU model to a new domain. To address this issue, a novel label embedding based slot filling framework is proposed in this paper. Here, distributed label embedding is constructed for each slot using prior knowledge. Three encoding methods are investigated to incorporate different kinds of prior knowledge about slots: atomic concepts, slot descriptions, and slot exemplars. The proposed label embeddings tend to share text patterns and reuses data with different slot labels. This makes it useful for adaptive NLU with limited data. Also, since label embedding is independent of NLU model, it is compatible with almost all deep learning based slot filling models. The proposed approaches are evaluated on three datasets. Experiments on single domain and domain adaptation tasks show that label embedding achieves significant performance improvement over traditional one-hot label representation as well as advanced zero-shot approaches.

A survey of joint intent detection and slot-filling models in natural language understanding

“…[ Zhu et al 2020] noticed that the data sparsity problem can also occur with labels because labels are usually encoded using one-hot vectors, so they also proposed a label embedding which is constructed using prior knowledge including atomic concepts, slot descriptions, and slot exemplars. An atomic concept assumes that each slot can be represented as a set of atoms.…”

What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data

“…[259] utilizes attention mechanism to build a graph generation model based on visual features and knowledge graphs, [191], etc. Zhu et al [280] and provide relation propagation features for FSL. [224] regresses the visual features under the knowledge graph representation before performing ZSL classification.…”

“…In these studies, some utilize an ontology directly as the model output [117], and some use the similarity [31,85] or matching degree [129] between the standard feature and the knowledge feature as the learning target. [280] leverages structured knowledge to customize embedding slot labels for natural language understanding (NLU) tasks. [228] takes word embeddings and knowledge graph as inputs and outputs a visual classifier for each category.…”

“…II. Model Training Many LSL methods use auxiliary knowledge systems to provide help creating teh model [11,19,102,113,129,151,205,224,228,259,280]. For structured data modeling, this idea is more generally embodied as "using knowledge to learn knowledge [40,100,153,183,275]. "…”

Slot Transferability for Cross-domain Slot Filling

“…Traditional supervised learning methods have made great achievements with a large amount of labeled data Mesnil et al, 2015;Hakkani-Tür et al, 2016;Kurata et al, 2016;Liu and Lane, 2016;Goo et al, 2018;E et al, 2019). However, there is little or even no labeled data for a new task, the cross-domain slot filling task which uses labeled data in source tasks to training model for target task is gaining increasing attention (Yazdani and Henderson, 2015;Bapna et al, 2017;Zhu and Yu, 2018;Lee and Jha, 2019;Shah et al, 2019;Liu et al, 2020;Zhu et al, 2020). There are mainly two streams of methods in previous work.…”