Coping with highly imbalanced datasets: A case study with definition extraction in a multilingual setting

Abstract

This paper addresses the task of automatic extraction of definitions by thoroughly exploring an approach that solely relies on machine learning techniques, and by focusing on the issue of the imbalance of relevant datasets. We obtained a breakthrough in terms of the automatic extraction of definitions, by extensively and systematically experimenting with different sampling techniques and their combination, as well as a range of different types of classifiers. Performance consistently scored in the range of 0.95-0.99 of area under the receiver operating characteristics, with a notorious improvement between 17 and 22 percentage points regarding the baseline of 0.73-0.77, for datasets with different rates of imbalance. Thus, the present paper also represents a contribution to the seminal work in natural language processing that points toward the importance of exploring the research path of applying sampling techniques to mitigate the bias induced by highly imbalanced datasets, and thus greatly improving the performance of a large range of tools that rely on them.

Syntactically Aware Neural Architectures for Definition Extraction

“…In the early days of DE, rulebased approaches leveraged linguistic cues observed in definitional data (Rebeyrolle and Tanguy, 2000;Klavans and Muresan, 2001;Malaisé et al, 2004;Saggion and Gaizauskas, 2004;Storrer and Wellinghoff, 2006). However, in order to deal with problems like language dependence and domain specificity, machine learning was incorporated in more recent contributions (Del Gaudio et al, 2013), which focused on encoding informative lexico-syntactic patterns in feature vectors (Cui et al, 2005;Fahmi and Bouma, 2006;Westerhout and Monachesi, 2007;Borg et al, 2009), both in supervised and semi-supervised settings (Reiplinger et al, 2012;.…”

Improving Voting Feature Intervals for Spatial Prediction of Landslides

“…is method has been used successfully in various medical, computer, and natural sciences studies [49,50]. e primary purpose of this approach is to deal with very imbalanced datasets [51].…”

Multilingual SMS-based author profiling: Data and methods

“…Therefore, accuracy is not considered as a good measure if data is highly unsymmetrical or imbalanced. As our original dataset is imbalanced for gender class, so considering only the accuracy is not a good measure to evaluate the performance of author profiling techniques (Del Gaudio, Batista and Branco 2014). Therefore, F 1 measure is also considered to evaluate the performance of author profiling methods because it provides the unbiased reflections for skewed data or an uneven class distribution.…”