Examining Temporality in Document Classification

Abstract

Many corpora span broad periods of time. Language processing models trained during one time period may not work well in future time periods, and the best model may depend on specific times of year (e.g., people might describe hotels differently in reviews during the winter versus the summer). This study investigates how document classifiers trained on documents from certain time intervals perform on documents from other time intervals, considering both seasonal intervals (intervals that repeat across years, e.g., winter) and non-seasonal intervals (e.g., specific years). We show experimentally that classification performance varies over time, and that performance can be improved by using a standard domain adaptation approach to adjust for changes in time.

Neural Temporality Adaptation for Document Classification: Diachronic Word Embeddings and Domain Adaptation Models

“…We retrieved available data sources from previous publications (Zhang et al, 2014;He and McAuley, 2016;Huang and Paul, 2018). Specifically, we use four different sources in in English-Amazon (music reviews), Yelp (restaurant and hotel reviews), Twitter, and economic newspaper articles ( Figure Eight Inc., 2015)-and one source in Chinese, Dianping (Meituan-Dianping, 2019).…”

“…Following Huang and Paul (2018), we group the corpora into several bins of temporal intervals; specifically, non-repeating time intervals spanning one or more years (Table 1). We encode each temporal domain into the discrete time labels, 1, 2, ...T .…”

“…Document classification models often use feature representations that are derived from words. Therefore, variations in word usage across time will change the distribution of features over time, which can impact the stability of document classifiers (Huang and Paul, 2018). Our goal in this section is to test whether there are temporal variations in our datasets, how strong the effects are,…”

“…In this work, we propose an alternative approach to encoding time into word embeddings. The idea is inspired by the "easy" domain adaptation method (Daume III, 2007), which was shown to be successful at modeling different temporal do-mains (Huang and Paul, 2018), and can be implemented by simply modifying the input data without modifying the training process. In our approach, words in the training data are concatenated with the name of the time interval, and embeddings are trained using a sub-word sharing framework (Bojanowski et al, 2016).…”

Adaptive Ensembling: Unsupervised Domain Adaptation for Political Document Analysis

“…Consistency regularization methods (e.g., self-ensembling) outperform adversarial methods on visual semi-supervised and domain adaptation tasks (Athiwaratkun et al, 2019), but have rarely been applied to textual data (Ko et al, 2019). Finally, Huang and Paul (2018) establish the feasibility of using domain adaptation to label documents from discrete time periods. Our work departs from previous work by proposing an adaptive, time-aware approach to consistency regularization provisioned with causal convolutional networks.…”