ClusterDataSplit: Exploring Challenging Clustering-Based Data Splits for Model Performance Evaluation

Abstract

This paper adds to the ongoing discussion in the natural language processing community on how to choose a good development set. Motivated by the real-life necessity of applying machine learning models to different data distributions, we propose a clustering-based data splitting algorithm. It creates development (or test) sets which are lexically different from the training data while ensuring similar label distributions. Hence, we are able to create challenging cross-validation evaluation setups while abstracting away from performance differences resulting from label distribution shifts between training and test data. In addition, we present a Python-based tool for analyzing and visualizing data split characteristics and model performance. We illustrate the workings and results of our approach using a sentiment analysis and a patent classification task.

Boosting Transformers for Job Expression Extraction and Classification in a Low-Resource Setting

“…Further, there is a line of work now questioning traditional train-dev splits [6] as well as random splits [16]. More challenging datasplits can be created by clustering the documents based on their similarity, where each split encodes unique information to a certain degree [18]. We use this method to train ensembles of models on these splits in a cross-validation format, such that each model has observed slightly different training instances.…”

“…Then, we train the model using only the train-fraction of all the data and use the held-out validation data to determine the best model, which is then used to annotate the test data. As an alternative to random splits, we follow [18] and create strategic datasplits by clustering the documents according to their similarity. This creates more challenging splits, as more distant documents are left out for validation.…”

“…Our solution for these tasks is a neural sequence tagger based on multilingual transformer models. In particular, we experiment with continuing the masked language modeling pretraining of the multilingual XLM-R model [3] on Spanish texts, transferring trained models between the two tasks [13] and using strategic datasplits [18].…”