Hierarchical Transformers for Long Document Classification

Abstract

BERT, which stands for Bidirectional Encoder Representations from Transformers, is a recently introduced language representation model based upon the transfer learning paradigm. We extend its fine-tuning procedure to address one of its major limitations -applicability to inputs longer than a few hundred words, such as transcripts of human call conversations. Our method is conceptually simple. We segment the input into smaller chunks and feed each of them into the base model. Then, we propagate each output through a single recurrent layer, or another transformer, followed by a softmax activation. We obtain the final classification decision after the last segment has been consumed. We show that both BERT extensions are quick to fine-tune and converge after as little as 1 epoch of training on a small, domain-specific data set. We successfully apply them in three different tasks involving customer call satisfaction prediction and topic classification, and obtain a significant improvement over the baseline models in two of them.

Coupled Hierarchical Transformer for Stance-Aware Rumor Verification in Social Media Conversations

“…Our work extends the latter line of work by proposing a hierarchical Transformer based on the recent pre-trained BERT for this task. Moreover, we notice that our BERT-based hierarchical Transformer is similar to the model proposed in (Pappagari et al, 2019), but we want to point out that our model design in the input and output layers is specific to stance classification, which is different from their work. Rumor Verification: Due to the negative impact of various rumors spreading on social media, rumor verification has attracted increasing attention in recent years.…”

Zero-Shot Learning for Cross-Lingual News Sentiment Classification

“…This novel text representation technique showed improvement on five different datasets. Pappagari et al [33] proposed a modification to the BERT model for long document classification in a monolingual setting. They utilized a segmentation approach to divide the input text sequences into several subsequences.…”

“…We extracted the [CLS] vector representations from the last layer and combined them into a final document representation. This approach is inspired by the work of Pappagari et al [33]. The main difference of our study is in the way the subsequence representations are merged into a document representation.…”

“…The advantage of the first two mapping operations is that, in comparison to the methods proposed in Pappagari et al [33], they are more computationally efficient as we need to perform simple vector norm and averaging calculations to produce the final document representations. The third mapping operations uses a convolutional layer to map the different subsequences into one document representation.…”

Relative and Incomplete Time Expression Anchoring for Clinical Text

“…The neural network reads the text linearly while using an external memory to store relations, and can then use the global context to classify them. Other alternatives could be to still leverage the power of pre-trained transformer models, by using solutions to pass entire, long texts to the model instead (Pappagari et al, 2019). A deep learning approach is potentially not the most appropriate to represent complex temporal relations, for example Li et al (2020) recently reported good results using an ontology.…”