Impact of Batch Size on Stopping Active Learning for Text Classification

Beatty, Garrett; Kochis, Ethan; Bloodgood, Michael

doi:10.1109/icsc.2018.00059

Cited by 13 publications

(8 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ong, Smola, and Willamson 2003) have identified that the machine learning performance is sensitive to the kernel width selection in similarity measurement. There has also been research (Beatty, Kochis, and Bloodgood 2018) studying the behavior of batch size in query selection for active learning. Therefore, given a limited number of labeled samples, it is critical to automatically determine a good set of hyperparameters for active learning in order to efficiently maximize the classification performance.…”

Section: Introductionmentioning

confidence: 99%

AutoDAL: Distributed Active Learning with Automatic Hyperparameter Selection

Chen¹,

Wujek²

2020

AAAI

View full text Add to dashboard Cite

Automated machine learning (AutoML) strives to establish an appropriate machine learning model for any dataset automatically with minimal human intervention. Although extensive research has been conducted on AutoML, most of it has focused on supervised learning. Research of automated semi-supervised learning and active learning algorithms is still limited. Implementation becomes more challenging when the algorithm is designed for a distributed computing environment. With this as motivation, we propose a novel automated learning system for distributed active learning (AutoDAL) to address these challenges. First, automated graph-based semi-supervised learning is conducted by aggregating the proposed cost functions from different compute nodes in a distributed manner. Subsequently, automated active learning is addressed by jointly optimizing hyperparameters in both the classification and query selection stages leveraging the graph loss minimization and entropy regularization. Moreover, we propose an efficient distributed active learning algorithm which is scalable for big data by first partitioning the unlabeled data and replicating the labeled data to different worker nodes in the classification stage, and then aggregating the data in the controller in the query selection stage. The proposed AutoDAL algorithm is applied to multiple benchmark datasets and a real-world electrocardiogram (ECG) dataset for classification. We demonstrate that the proposed AutoDAL algorithm is capable of achieving significantly better performance compared to several state-of-the-art AutoML approaches and active learning algorithms.

show abstract

Section: Introductionmentioning

confidence: 99%

AutoDAL: Distributed Active Learning with Automatic Hyperparameter Selection

Chen¹,

Wujek²

2020

AAAI

View full text Add to dashboard Cite

show abstract

“…Following previous work, we set w to three [13], [18]. As [33] advised, if a relatively large batch size is used, a smaller value for w should be used in order to mitigate the degradation in stopping method performance caused when using larger batch sizes. Table I shows the performance of unlabeled data stopping methods.…”

Section: Stopping Methods Parametersmentioning

confidence: 99%

The Use of Unlabeled Data Versus Labeled Data for Stopping Active Learning for Text Classification

Beatty

Kochis

Bloodgood

2019

2019 IEEE 13th International Conference on Semantic Computing (ICSC)

Self Cite

View full text Add to dashboard Cite

Annotation of training data is the major bottleneck in the creation of text classification systems. Active learning is a commonly used technique to reduce the amount of training data one needs to label. A crucial aspect of active learning is determining when to stop labeling data. Three potential sources for informing when to stop active learning are an additional labeled set of data, an unlabeled set of data, and the training data that is labeled during the process of active learning. To date, no one has compared and contrasted the advantages and disadvantages of stopping methods based on these three information sources. We find that stopping methods that use unlabeled data are more effective than methods that use labeled data.

show abstract

“…Again, we see there is not much difference in performance of our forecasting system when we vary batch percent. It is possible that with much larger batch percents we would see a change in forecasting performance, but using larger batch percents is known to have various negative effects on active learning [38], [39], so we did not investigate the impact with larger batch percents that are less likely to be used in practice.…”

Section: A Impact Of Batch Percentmentioning

confidence: 99%

Early Forecasting of Text Classification Accuracy and F-Measure with Active Learning

Orth

Bloodgood

2020

2020 IEEE 14th International Conference on Semantic Computing (ICSC)

Self Cite

View full text Add to dashboard Cite

When creating text classification systems, one of the major bottlenecks is the annotation of training data. Active learning has been proposed to address this bottleneck using stopping methods to minimize the cost of data annotation. An important capability for improving the utility of stopping methods is to effectively forecast the performance of the text classification models. Forecasting can be done through the use of logarithmic models regressed on some portion of the data as learning is progressing. A critical unexplored question is what portion of the data is needed for accurate forecasting. There is a tension, where it is desirable to use less data so that the forecast can be made earlier, which is more useful, versus it being desirable to use more data, so that the forecast can be more accurate. We find that when using active learning it is even more important to generate forecasts earlier so as to make them more useful and not waste annotation effort. We investigate the difference in forecasting difficulty when using accuracy and F-measure as the text classification system performance metrics and we find that F-measure is more difficult to forecast. We conduct experiments on seven text classification datasets in different semantic domains with different characteristics and with three different base machine learning algorithms. We find that forecasting is easiest for decision tree learning, moderate for Support Vector Machines, and most difficult for neural networks.

show abstract

Impact of Batch Size on Stopping Active Learning for Text Classification

Cited by 13 publications

References 8 publications

AutoDAL: Distributed Active Learning with Automatic Hyperparameter Selection

AutoDAL: Distributed Active Learning with Automatic Hyperparameter Selection

The Use of Unlabeled Data Versus Labeled Data for Stopping Active Learning for Text Classification

Early Forecasting of Text Classification Accuracy and F-Measure with Active Learning

Contact Info

Product

Resources

About