Domain Divergences: A Survey and Empirical Analysis

Abstract

Domain divergence plays a significant role in estimating the performance of a model in new domains. While there is a significant literature on divergence measures, researchers find it hard to choose an appropriate divergence for a given NLP application. We address this shortcoming by both surveying the literature and through an empirical study. We develop a taxonomy of divergence measures consisting of three classes -Information-theoretic, Geometric, and Higher-order measures and identify the relationships between them. Further, to understand the common use-cases of these measures, we recognise three novel applications -1) Data Selection, 2) Learning Representation, and 3) Decisions in the Wild -and use it to organise our literature. From this, we identify that Information-theoretic measures are prevalent for 1) and 3), and Higher-order measures are more common for 2). To further help researchers choose appropriate measures to predict drop in performance -an important aspect of Decisions in the Wild, we perform correlation analysis spanning 130 domain adaptation scenarios, 3 varied NLP tasks and 12 divergence measures identified from our survey. To calculate these divergences, we consider the current contextual word representations (CWR) and contrast with the older distributed representations. We find that traditional measures over word distributions still serve as strong baselines, while higher-order measures with CWR are effective.

CDMPP: A Device-Model Agnostic Framework for Latency Prediction of Tensor Programs

“…Based on this distribution-difference-based bound, we can minimize the following objective in our model finetuning [36], to project representations of samples from different domains close together in the representation space: regularization for minimizing representation difference (5) One practical divergence measure for Δ(.) is the Central Moment Discrepancy [67], which is theoretically grounded, efficient to implement and compute, and has shown superior empirical success in learning domain invariant representations [44,74].…”

Experimental Standards for Deep Learning in Natural Language Processing Research

“…Even in lower-stake scenarios a model trained on only a subset of the global data distribution can have inconsistent behaviour when applied to a different target data distribution (D'Amour et al, 2020;Koh et al, 2020). For instance, language or even domain differences can have a noticeable impact on model performance (White & Cotterell, 2021;Ramesh Kashyap et al, 2021). Increased data diversity can improve the generalization ability of models to new domains, e.g., more languages in NLP (Benjamin, 2018).…”

Divergence-Based Domain Transferability for Zero-Shot Classification

“…The most optimal transferability metric would be resource-efficient, such that it is capable of accurately predicting the final performance of the model whilst minimising the amount of processing required to compute it. To this end, several works (Van Asch and Daelemans, 2010;Ruder and Plank, 2017;Ramesh Kashyap et al, 2021) have focused on estimating transferability prior to finetuning, using statistical measures of divergence between the underlying feature spaces of model pairs. Domain divergence measures are used to produce a notion of distance between pairs of domains by comparing their representations and have seen significant usage in works which investigate the correlation between their estimations and performance change (Van Asch and Daelemans, 2010;Ramesh Kashyap et al, 2021).…”