Early Stopping Based on Unlabeled Samples in Text Classification

Choi, Hongseok; Choi, Dongha; Lee, Hyunju

doi:10.18653/v1/2022.acl-long.52

Cited by 5 publications

(3 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The number of training iterations was 100, and the model performance was checked using the validation set. We used the early-stopping ( 39 ) method during the training. If the effect has not improved for 10 consecutive rounds, then training is terminated.…”

Section: Resultsmentioning

confidence: 99%

Automating venous thromboembolism risk assessment: a dual-branch deep learning method using electronic medical records

Yang

Zhang

2023

Front. Med.

View full text Add to dashboard Cite

BackgroundVenous thromboembolism (VTE) is a prevalent cardiovascular disease. Although risk assessment and preventive measures are effective, manual assessment is inefficient and covers a small population in clinical practice. Hence, it is necessary to explore intelligent methods for VTE risk assessment.MethodsThe Padua scale has been widely used in VTE risk assessment, and we divided its assessment into disease category judgment and comprehensive clinical information judgment according to the characteristics of the Padua scale. We proposed a dual-branch deep learning (DB-DL) assessment method. First, in the disease category branch, we propose a deep learning-based Padua disease classification model (PDCM) for determining patients' Padua disease categories by considering patients' diagnosis, symptoms, and symptom weights. In the branch of comprehensive clinical information, we use the Chinese lexical analysis (LAC) word separation technique, combined with professional corpus and rules, to extract and judge the comprehensive clinical factors in the electronic medical record (EMR).ResultsWe validated the accuracy of the method with the Padua assessment results of 7,690 Chinese clinical EMRs. First, our proposed method allows for a fully automated assessment, and the average time to assess one patient is only 0.37 s. Compared to the gold standard, our method has an Area Under Curve (AUC) value of 0.883, a specificity value of 0.957, and a sensitivity value of 0.816 for assessing the Padua risk patient class.ConclusionOur DB-DL assessment method automates VTE risk assessment, thereby addressing the challenges of time-consuming evaluation and limited population coverage. Thus, this method is highly clinically valuable.

show abstract

Section: Resultsmentioning

confidence: 99%

Automating venous thromboembolism risk assessment: a dual-branch deep learning method using electronic medical records

Yang

Zhang

2023

Front. Med.

View full text Add to dashboard Cite

show abstract

“…This advantage is useful in low-resource settings. Choi et al [41] proposed an early stopping method based on unlabeled samples (BUS-stop), which performed well, particularly in low and imbalanced data settings. Garg et al [42] also leveraged unlabeled data for early stopping but focused on providing a theoretical perspective on generalization.…”

Section: Related Workmentioning

confidence: 99%

“…The confidences are sorted in the order of size before calculating the similarity. The detailed process is the same as the conf-sim method in BUS-stop [41], except that the confidences are based only on training data.…”

Section: B Non-validation Early Stopping Criteriamentioning

confidence: 99%

Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters

Choi

Lee

2023

IEEE Access

Self Cite

View full text Add to dashboard Cite

To improve deep-learning performance in low-resource settings, many researchers have redesigned model architectures or applied additional data (e.g., external resources, unlabeled samples). However, there have been relatively few discussions on how to make good use of small amounts of labeled samples, although it is potentially beneficial and should be done before applying additional data or redesigning models. In this study, we assume a low-resource setting in which only a few labeled samples (i.e., 30-100 per class) are available, and we discuss how to exploit them without additional data or model redesigns. We explore possible approaches in the following three aspects: training validation splitting, early stopping, and weight initialization. Extensive experiments are conducted on six public sentence classification datasets. Performance on various evaluation metrics (e.g., accuracy, loss, and calibration error) significantly varied depending on the approaches that were combined in the three aspects. Based on the results, we propose an integrated method, which is to initialize the model with a weight averaging method and use a nonvalidation stop method to train all samples. This simple integrated method consistently outperforms the competitive methods; e.g., the average accuracy of six datasets of this method was 1.8% higher than those of conventional validation-based methods. In addition, the integrated method further improves the performance when adapted to several state-of-the-art models that use additional data or redesign the network architecture (e.g., self-training and enhanced structural models). Our results highlight the importance of the training strategy and suggest that the integrated method can be the first step in the low-resource setting. This study provides empirical knowledge that will be helpful when dealing with low-resource data in future efforts. Our code is publicly available at https://github.com/DMCB-GIST/exploit_all_samples.

show abstract