Active Learning Approaches for Labeling Text: Review and Assessment of the Performance of Active Learning Approaches

Miller, Blake; Linder, Fridolin; Mebane, Walter R.

doi:10.1017/pan.2020.4

Cited by 49 publications

(42 citation statements)

References 31 publications

(2 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Instead of having to go though millions and millions of tweets, this could thus serve as first rough attempt at separating tweets into hateful and non-hateful tweets. This could, e.g., be combined with the active-labeling approach of Miller, Linder, and Mebane (2020).…”

Section: Conclusion and Discussionmentioning

confidence: 99%

“…Hiring student assistants or engaging other researchers in hand-labeling text is extremely time consuming, in terms of designing the coding scheme, potentially redoing the coding scheme and, of course, the actual task of hand-labeling. Even in the case of so-called "active learning" (Miller, Linder, and Mebane 2020), designing a coding scheme and actively labeling a significant amount of text is still necessary. These costs, furthermore, have conceptual consequences as they limit the complexity of the constructs that are most often assessed with supervised approaches.…”

Section: Supervised Text Classificationmentioning

confidence: 99%

See 1 more Smart Citation

Super-unsupervised text classification for labeling online political hate

Rasmussen¹,

Bor²,

Osmundsen³

et al. 2021

Preprint

View full text Add to dashboard Cite

We live in a world of text but the sheer magnitude of social media data coupled with a need to measure complex psychological constructs have made this important source of data difficult to use for many social scientists. Either researchers engage in costly hand-coding of thousands of texts using supervised techniques or in unsupervised techniques where the measurement of predefined constructs are difficult. We propose a novel approach which we call super-unsupervised learning using the psychologically complex construct online political hate. This approach draws on the best features from both supervised and unsupervised learning techniques: Measurements of complex psychological constructs without a single labelled data source. We first outline the approach and then provide tests of (i) face validity, (ii) convergent and discriminant validity, (iii) criterion validity, (iv) external validity and (v) ecological validity.

show abstract

Section: Conclusion and Discussionmentioning

confidence: 99%

Section: Supervised Text Classificationmentioning

confidence: 99%

Super-unsupervised text classification for labeling online political hate

Rasmussen¹,

Bor²,

Osmundsen³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The key insight to AFSM is leveraging one of the foremost principles of human-computer interaction: have computers do what they do well, and let humans do what they do well Lazar, Feng, and Hochheiser 2017;Norman and Draper 1986). Drawing inspiration from the literatures on adaptive machine learning and text-as-data (Enamorado 2018;Miller, Linder, and Mebane 2019), we find that although computers can quickly identify large sets of possible matches, only humans can quickly identify whether a proposed match is correct (Mozer et al 2018).…”

Section: An Adaptive Algorithmmentioning

confidence: 99%

Adaptive Fuzzy String Matching: How to Merge Datasets with Only One (Messy) Identifying Field

Kaufman¹,

Klevs

2021

Polit. Anal.

View full text Add to dashboard Cite

A single dataset is rarely sufficient to address a question of substantive interest. Instead, most applied data analysis combines data from multiple sources. Very rarely do two datasets contain the same identifiers with which to merge datasets; fields like name, address, and phone number may be entered incorrectly, missing, or in dissimilar formats. Combining multiple datasets absent a unique identifier that unambiguously connects entries is called the record linkage problem. While recent work has made great progress in the case where there are many possible fields on which to match, the much more uncertain case of only one identifying field remains unsolved: this fuzzy string matching problem, both its own problem and a component of standard record linkage problems, is our focus. We design and validate an algorithmic solution called Adaptive Fuzzy String Matching rooted in adaptive learning, and show that our tool identifies more matches, with higher precision, than existing solutions. Finally, we illustrate its validity and practical value through applications to matching organizations, places, and individuals.

show abstract

“…In addition, there is another method called active learning, which selects the instances to be labeled from all unlabeled data to build the most accurate model. The active learning method consists of an iterative process of two operations: query and oracle (Aggarwal et al, 2014;Miller, 2020). In a query operation, it must decide which data instance should be queried for its label.…”

Section: Literature Reviewmentioning

confidence: 99%

“…As a result, our method can build the best training set with a cost limitation. On the contrary, the goal of traditional active learning is usually to build effective classifiers using the fewest number of instances (Miller, 2020).…”

Section: Literature Reviewmentioning

confidence: 99%

Building a training dataset for classification under a cost limitation

Chen

Cheng

Zhang

2021

View full text Add to dashboard Cite

Purpose A necessary preprocessing of document classification is to label some documents so that a classifier can be built based on which the remaining documents can be classified. Because each document differs in length and complexity, the cost of labeling each document is different. The purpose of this paper is to consider how to select a subset of documents for labeling with a limited budget so that the total cost of the spending does not exceed the budget limit, while at the same time building a classifier with the best classification results. Design/methodology/approach In this paper, a framework is proposed to select the instances for labeling that integrate two clustering algorithms and two centroid selection methods. From the selected and labeled instances, five different classifiers were constructed with good classification accuracy to prove the superiority of the selected instances. Findings Experimental results show that this method can establish a training data set containing the most suitable data under the premise of considering the cost constraints. The data set considers both “data representativeness” and “data selection cost,” so that the training data labeled by experts can effectively establish a classifier with high accuracy. Originality/value No previous research has considered how to establish a training set with a cost limit when each document has a distinct labeling cost. This paper is the first attempt to resolve this issue.

show abstract

Active Learning Approaches for Labeling Text: Review and Assessment of the Performance of Active Learning Approaches

Cited by 49 publications

References 31 publications

Super-unsupervised text classification for labeling online political hate

Super-unsupervised text classification for labeling online political hate

Adaptive Fuzzy String Matching: How to Merge Datasets with Only One (Messy) Identifying Field

Building a training dataset for classification under a cost limitation

Contact Info

Product

Resources

About