Automated Social Text Annotation With Joint Multilabel Attention Networks

Abstract

Automated social text annotation is the task of suggesting a set of tags for shared documents on social media platforms. The automated annotation process can reduce users' cognitive overhead in tagging and improve tag management for better search, browsing, and recommendation of documents. It can be formulated as a multilabel classification problem. We propose a novel deep learning-based method for this problem and design an attention-based neural network with semantic-based regularization, which can mimic users' reading and annotation behavior to formulate better document representation, leveraging the semantic relations among labels. The network separately models the title and the content of each document and injects an explicit, title-guided attention mechanism into each sentence. To exploit the correlation among labels, we propose two semantic-based loss regularizers, i.e., similarity and subsumption, which enforce the output of the network to conform to label semantics. The model with the semantic-based loss regularizers is referred to as the joint multilabel attention network (JMAN). We conducted a comprehensive evaluation study and compared JMAN to the state-of-the-art baseline models, using four large, real-world social media data sets. In terms of F 1 , JMAN significantly outperformed bidirectional gated recurrent unit (Bi-GRU) relatively by around 12.8%-78.6% and the hierarchical attention network (HAN) by around 3.9%-23.8%. The JMAN model demonstrates advantages in convergence and training speed. Further improvement of performance was observed against latent Dirichlet allocation (LDA) and support vector machine (SVM). When applying the semantic-based loss regularizers, the performance of HAN and Bi-GRU in terms of F 1 was also boosted. It is also found that dynamic update of the label semantic matrices (JMAN d) has

Explainable Automated Coding of Clinical Notes using Hierarchical Label-wise Attention Networks and Label Embedding Initialisation

“…Deep learning has become the main approach for multi-label document classification [11,15] in recent years. The advantage of multi-label deep learning models lies in their straightforward problem formulation and strong approximation power on large datasets, resulting in better performance over traditional machine learning approaches, as compared in [16,15,6]. For automated coding, some of the notable neural network models adapted for multi-label classification are variations of CNNs [6,7] and RNNs [1] with attention mechanisms.…”

“…This inspires to jointly learn to represent the important words and sentences while classifying a document in HAN [10], thus also enables to explain the inner working of deep learning models. HAN was adapted to a multi-label classification setting to classify socially shared texts in [15] and for automated medical coding [1]. Founded on the studies above, our approach provides a richer label-wise attention mechanism at both the word and the sentence level for automated medical coding.…”

“…The approach showed performance improvement, however, it is not computationally efficient to assign each neuron in the final hidden layer to represent one of the massive (even the pairwise) patterns of label relations. An alternative method is through regularisation in [15] to enforce the output layer of the neural network to satisfy constraints on label relations. This requires to further tune the hyper-parameters of the regularisers so that a relatively marginal improvement (0.5-1.5% example-based F 1 on scientific paper abstracts and questions in social Q&A platforms) could be achieved.…”

“…The attention scores are based on an alignment (or a similarity computation) of each hidden representation in a sequence to a context vector. The context vector is usually shared for all labels as in [10,15], whereas in medical coding, it is essential to interpret the amount of attention paid regarding a specific medical code to the clinical note.…”

JLAN: medical code prediction via joint learning attention networks and denoising mechanism

“…Deep learning technology has shown substantial advantages over traditional machine learning methods and has been widely used for code allocation [ 13 ]. Most researchers model this task as a multi-label text classification problem based on EHR's free text.…”