Cost-Sensitive BERT for Generalisable Sentence Classification on Imbalanced Data

Madabushi, Harish Tayyar; Kochkina, Elena; Castelle, Michael

doi:10.18653/v1/d19-5018

Cited by 67 publications

(52 citation statements)

References 21 publications

(16 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Hence, improved results may be obtained experimentally by using a hyper-parameter search ( 34 ). Furthermore, in addition to cost-sensitive learning, a variety of methods, such as oversampling and data augmentation, are available to address imbalance problems ( 23 , 35 ). Different imbalance strategies may lead to diverse conclusions.…”

Section: Discussionmentioning

confidence: 99%

Deep Learning-Based Natural Language Processing for Screening Psychiatric Patients

Dai

Lee

et al. 2021

Front. Psychiatry

View full text Add to dashboard Cite

The introduction of pre-trained language models in natural language processing (NLP) based on deep learning and the availability of electronic health records (EHRs) presents a great opportunity to transfer the “knowledge” learned from data in the general domain to enable the analysis of unstructured textual data in clinical domains. This study explored the feasibility of applying NLP to a small EHR dataset to investigate the power of transfer learning to facilitate the process of patient screening in psychiatry. A total of 500 patients were randomly selected from a medical center database. Three annotators with clinical experience reviewed the notes to make diagnoses for major/minor depression, bipolar disorder, schizophrenia, and dementia to form a small and highly imbalanced corpus. Several state-of-the-art NLP methods based on deep learning along with pre-trained models based on shallow or deep transfer learning were adapted to develop models to classify the aforementioned diseases. We hypothesized that the models that rely on transferred knowledge would be expected to outperform the models learned from scratch. The experimental results demonstrated that the models with the pre-trained techniques outperformed the models without transferred knowledge by micro-avg. and macro-avg. F-scores of 0.11 and 0.28, respectively. Our results also suggested that the use of the feature dependency strategy to build multi-labeling models instead of problem transformation is superior considering its higher performance and simplicity in the training process.

show abstract

Section: Discussionmentioning

confidence: 99%

Deep Learning-Based Natural Language Processing for Screening Psychiatric Patients

Dai

Lee

et al. 2021

Front. Psychiatry

View full text Add to dashboard Cite

show abstract

“…S [11] proposes a hybrid imbalanced data learning framework (HIDLF) to deal with the imbalance of views in the movie review dataset, and then classifies the movie reviews by the proposed HIDLT-SVM algorithm. Harish [12] proposed the the BERT model to deal with the problem of data imbalance in text classification. Li [13] proposes a solution to the imbalanced text problem in a multi-classification task.…”

Section: Imbalanced Textmentioning

confidence: 99%

Research on Automatic Labelling of Imbalanced Texts of Customer Complaints Based on Text Enhancement and Layer-By-Layer Semantic Matching

TANG

MOU

LIU

et al. 2021

Preprint

View full text Add to dashboard Cite

[Purpose/meaning] Due to its potential impact on business efficiency, automated customer complaint labeling and classification are of great importance for management decisions and business-level applications. The majority of the current research on automated labeling uses large and well-balanced datasets. However, customer complaints' labels are hierarchical in structure, with many labels at the lowest hierarchy level. Relying on lower-level labels leads to small and imbalanced samples, thus rendering the current automatic labeling practices not applicable to customer complaints. [Methodology/process] This article proposes an automatic labeling model incorporating the BERT and Word2Vec methods. The model is validated on electric utility customer complaints data. Within the model, the BERT method serves to obtain shallow-level text tags. Further, text enhancement is used to mitigate the problem of uneven samples that emerges when the number of labels is large. Finally, the Word2Vec model is utilized for the deep-level text analysis. [Findings/conclusions] The experiments demonstrate the proposed model's efficiency in automating customer complaint labeling. Consequently, the proposed model supports the enterprises in improving their service quality while simultaneously reducing labor costs.

show abstract

“…Previous work with WLS data used oversampling of the minority class to address this imbalance, which was effective with some but not all models (Noorian et al, 2017). As recent work with BERT suggests cost-sensitive learning is an effective alternative to address class imbalance (Madabushi et al, 2019), we evaluate the utility of this method also. Cost-sensitive learning involves adjusting the loss function of a model such that changes in performance on one class are weighted more heavily.…”

Section: Introductionmentioning

confidence: 99%

Crossing the “Cookie Theft” Corpus Chasm: Applying What BERT Learns From Outside Data to the ADReSS Challenge Dementia Detection Task

Guo

Roan

et al. 2021

Front. Comput. Sci.

View full text Add to dashboard Cite

Large amounts of labeled data are a prerequisite to training accurate and reliable machine learning models. However, in the medical domain in particular, this is also a stumbling block as accurately labeled data are hard to obtain. DementiaBank, a publicly available corpus of spontaneous speech samples from a picture description task widely used to study Alzheimer's disease (AD) patients' language characteristics and for training classification models to distinguish patients with AD from healthy controls, is relatively small—a limitation that is further exacerbated when restricting to the balanced subset used in the Alzheimer's Dementia Recognition through Spontaneous Speech (ADReSS) challenge. We build on previous work showing that the performance of traditional machine learning models on DementiaBank can be improved by the addition of normative data from other sources, evaluating the utility of such extrinsic data to further improve the performance of state-of-the-art deep learning based methods on the ADReSS challenge dementia detection task. To this end, we developed a new corpus of professionally transcribed recordings from the Wisconsin Longitudinal Study (WLS), resulting in 1366 additional Cookie Theft Task transcripts, increasing the available training data by an order of magnitude. Using these data in conjunction with DementiaBank is challenging because the WLS metadata corresponding to these transcripts do not contain dementia diagnoses. However, cognitive status of WLS participants can be inferred from results of several cognitive tests including semantic verbal fluency available in WLS data. In this work, we evaluate the utility of using the WLS ‘controls’ (participants without indications of abnormal cognitive status), and these data in conjunction with inferred ‘cases’ (participants with such indications) for training deep learning models to discriminate between language produced by patients with dementia and healthy controls. We find that incorporating WLS data during training a BERT model on ADReSS data improves its performance on the ADReSS dementia detection task, supporting the hypothesis that incorporating WLS data adds value in this context. We also demonstrate that weighted cost functions and additional prediction targets may be effective ways to address issues arising from class imbalance and confounding effects due to data provenance.

show abstract

Cost-Sensitive BERT for Generalisable Sentence Classification on Imbalanced Data

Cited by 67 publications

References 21 publications

Deep Learning-Based Natural Language Processing for Screening Psychiatric Patients

Deep Learning-Based Natural Language Processing for Screening Psychiatric Patients

Research on Automatic Labelling of Imbalanced Texts of Customer Complaints Based on Text Enhancement and Layer-By-Layer Semantic Matching

Crossing the “Cookie Theft” Corpus Chasm: Applying What BERT Learns From Outside Data to the ADReSS Challenge Dementia Detection Task

Contact Info

Product

Resources

About