Learning from Rules Generalizing Labeled Exemplars

Awasthi, Abhijeet; Ghosh, Sabyasachi; Goyal, Rasna; Sarawagi, Sunita

doi:10.48550/arxiv.2004.06025

Cited by 6 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is worth noting that ALL was developed for binary classification tasks and uses weak signals that do not abstain, hence we only run ALL on datasets that satisfy these requirements. We compare to additional baselines used by Awasthi et al (2020) in our second set of experiments.…”

Section: Compared Methodsmentioning

confidence: 99%

“…The methods we compare against are from Awasthi et al (2020). We provide additional details in the appendix explaining each baseline.…”

Section: Baselinesmentioning

confidence: 99%

“…We used the same datasets and weak signals as Awasthi et al (2020). Additional details are provided in the appendix.…”

Section: Datasets and Weak Supervisionmentioning

confidence: 99%

“…However, on the multiclass datasets, we calculate π on validation data. Awasthi et al (2020) used the validation data to tune hyperparameters for their method and the methods they compare against. Table 5 lists the evaluation of DCWS along with the baseline metrics computed by (Awasthi et al, 2020) (Table 2).…”

Section: Datasets and Weak Supervisionmentioning

confidence: 99%

“…Table 5. Comparison of DCWS with the baselines from (Awasthi et al, 2020) on five different datasets. We report standard deviation of DCWS after over trials.…”

Section: Representation Using Cluster Labels As Features Is Anothermentioning

confidence: 99%

See 4 more Smart Citations

Data Consistency for Weakly Supervised Learning

Arachie¹,

Huang²

2022

Preprint

View full text Add to dashboard Cite

In many applications, training machine learning models involves using large amounts of humanannotated data. Obtaining precise labels for the data is expensive. Instead, training with weak supervision provides a low-cost alternative. We propose a novel weak supervision algorithm that processes noisy labels, i.e., weak signals, while also considering features of the training data to produce accurate labels for training. Our method searches over classifiers of the data representation to find plausible labelings. We call this paradigm data consistent weak supervision. A key facet of our framework is that we are able to estimate labels for data examples low or no coverage from the weak supervision. In addition, we make no assumptions about the joint distribution of the weak signals and true labels of the data. Instead, we use weak signals and the data features to solve a constrained optimization that enforces data consistency among the labels we generate. Empirical evaluation of our method on different datasets shows that it significantly outperforms state-of-the-art weak supervision methods on both text and image classification tasks.

show abstract

Section: Compared Methodsmentioning

confidence: 99%

“…The methods we compare against are from Awasthi et al (2020). We provide additional details in the appendix explaining each baseline.…”

Section: Baselinesmentioning

confidence: 99%

“…We used the same datasets and weak signals as Awasthi et al (2020). Additional details are provided in the appendix.…”

Section: Datasets and Weak Supervisionmentioning

confidence: 99%

Section: Datasets and Weak Supervisionmentioning

confidence: 99%

“…Table 5. Comparison of DCWS with the baselines from (Awasthi et al, 2020) on five different datasets. We report standard deviation of DCWS after over trials.…”

Section: Representation Using Cluster Labels As Features Is Anothermentioning

confidence: 99%

See 3 more Smart Citations

Data Consistency for Weakly Supervised Learning

Arachie¹,

Huang²

2022

Preprint

View full text Add to dashboard Cite

show abstract

CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

Li,

Shi,

Ziems

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Annotated data plays a critical role in Natural Language Processing (NLP) in training models and evaluating their performance. Given recent developments in Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot capability on many text-annotation tasks, comparable with or even exceeding human annotators. Such LLMs can serve as alternatives for manual annotation, due to lower costs and higher scalability. However, limited work has leveraged LLMs as complementary annotators, nor explored how annotation work is best allocated among humans and LLMs to achieve both quality and cost objectives. We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Under this framework, we utilize uncertainty to estimate LLMs' annotation capability. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline. For code implementation, see https: //github.com/SALT-NLP/CoAnnotating.

show abstract

Label Augmentation with Reinforced Labeling for Weak Supervision

Solmaz¹,

Cirillo²,

Maresca³

et al. 2022

Preprint

View full text Add to dashboard Cite

Weak supervision (WS) is an alternative to the traditional supervised learning to address the need for ground truth. Data programming is a practical WS approach that allows programmatic labeling data samples using labeling functions (LFs) instead of hand-labeling each data point. However, the existing approach fails to fully exploit the domain knowledge encoded into LFs, especially when the LFs' coverage is low. This is due to the common data programming pipeline that neglects to utilize data features during the generative process. This paper proposes a new approach called reinforced labeling (RL). Given an unlabeled dataset and a set of LFs, RL augments the LFs' outputs to cases not covered by LFs based on similarities among samples. Thus, RL can lead to higher labeling coverage for training an end classifier. The experiments on several domains (classification of YouTube comments, wine quality, and weather prediction) result in considerable gains. The new approach produces significant performance improvement, leading up to +21 points in accuracy and +61 points in F1 scores compared to the state-ofthe-art data programming approach.

show abstract

Learning from Rules Generalizing Labeled Exemplars

Cited by 6 publications

References 0 publications

Data Consistency for Weakly Supervised Learning

Data Consistency for Weakly Supervised Learning

CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

Label Augmentation with Reinforced Labeling for Weak Supervision

Contact Info

Product

Resources

About