Quantifying the Contextualization of Word Representations with Semantic Class Probing

Zhao, Muyu; Dufter, Philipp; Yaghoobzadeh, Yadollah; Schütze, Hinrich

doi:10.18653/v1/2020.findings-emnlp.109

Cited by 11 publications

(7 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, there are studies that investigate the LMs' knowledge about selectional preferences of verbs and semantic types. Their findings suggest that contextual language models encode information about the selectional preferences of verbs (Metheniti et al, 2020;Li et al, 2021;Pedinotti et al, 2021) and the semantic type of the nouns in general (Zhao et al, 2020). Similar to our study, Zhao et al (2020) and trained classifiers using the extracted representations of the BERT model for their tasks and these classifiers achieved high accuracy scores.…”

Section: Selectional Preference and Semantic Type Knowledge Of Lmssupporting

confidence: 78%

Identifying Semantic Argument Types in Predication and Copredication Contexts: A Zero-Shot Cross-Lingual Approach

Ekin Yavas,

Kallmeyer,

Osswald

et al. 2023

Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Proce

View full text Add to dashboard Cite

Identifying semantic argument types in predication contexts is not a straightforward task for several reasons, such as inherent polysemy, coercion, and copredication phenomena. In this paper, we train monolingual and multilingual classifiers with a zero-shot cross-lingual approach to identify semantic argument types in predications using pre-trained language models as feature extractors. We train classifiers for different semantic argument types and for both verbal and adjectival predications. Furthermore, we propose a method to detect copredication using these classifiers through identifying the argument semantic type targeted in different predications over the same noun in a sentence. We evaluate the performance of the method on copredication test data with Food•Event nouns for 5 languages.

show abstract

Section: Selectional Preference and Semantic Type Knowledge Of Lmssupporting

confidence: 78%

Identifying Semantic Argument Types in Predication and Copredication Contexts: A Zero-Shot Cross-Lingual Approach

Ekin Yavas,

Kallmeyer,

Osswald

et al. 2023

Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Proce

View full text Add to dashboard Cite

show abstract

“…Few-shot learners in NLP. Significant progress has been made in developing (Devlin et al, 2019;Peters et al, 2018;Brown et al, 2020), understanding (Liu et al, 2019;Tenney et al, 2019;Belinkov and Glass, 2019;Hewitt and Liang, 2019;Hewitt and Manning, 2019;Zhao et al, 2020a;Rogers et al, 2020), and utilizing (Houlsby et al, 2019;Zhao et al, 2020b;Brown et al, 2020;Li and Liang, 2021;Schick and Schütze, 2021a;Lester et al, 2021;Mi et al, 2021a) PLMs. Brown et al (2020), Schütze (2021a), andLiu et al (2021b) show that PLMs can serve as data-efficient few-shot learners, through priming or prompting (Liu et al, 2021a).…”

Section: Related Workmentioning

confidence: 99%

LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Framework

Zhao¹,

Mi²,

Wang³

et al. 2022

Findings of the Association for Computational Linguistics: NAACL 2022

Self Cite

View full text Add to dashboard Cite

Vast efforts have been devoted to creating highperformance few-shot learners, i.e., large-scale pretrained language models (PLMs) that perform well with little downstream task training data. Training PLMs has incurred significant cost, but utilizing the few-shot learners is still challenging due to their enormous size. This work focuses on a crucial question: How to make effective use of these few-shot learners? We propose LMTurk, a novel approach that treats few-shot learners as crowdsourcing workers. The rationale is that crowdsourcing workers are in fact few-shot learners: They are shown a few illustrative examples to learn about a task and then start annotating. LMTurk employs few-shot learners built upon PLMs as workers. We show that the resulting annotations can be utilized to train models that solve the task well and are small enough to be deployable in practical scenarios. Active learning is integrated into LMTurk to reduce the amount of queries made to PLMs, minimizing the computational cost of running PLM inference passes. Altogether, LMTurk is an important step towards making effective use of current PLMs. 1

show abstract

“…Chronis and Erk (2020) studied the similarity and relatedness of contextual representations in the embedding spaces of BERT, while Brunner et al (2019) studied how identifiable the intermediate representations of BERT are with respect to the input. Zhao et al (2020) quantified the contextual knowledge of BERT and Zhao et al (2021) analyzed the embedding spaces of BERT in order to quantify the non-linearity of its layers.…”

Section: Related Workmentioning

confidence: 99%

On Isotropy Calibration of Transformer Models

Ding¹,

Martinkus²,

Pascual³

et al. 2022

Proceedings of the Third Workshop on Insights From Negative Results in NLP

View full text Add to dashboard Cite

Different studies of the embedding space of transformer models suggest that the distribution of contextual representations is highly anisotropic -the embeddings are distributed in a narrow cone. Meanwhile, static word representations (e.g., Word2Vec or GloVe) have been shown to benefit from isotropic spaces. Therefore, previous work has developed methods to calibrate the embedding space of transformers in order to ensure isotropy. However, a recent study (Cai et al., 2021) shows that the embedding space of transformers is locally isotropic, which suggests that these models are already capable of exploiting the expressive capacity of their embedding space. In this work, we conduct an empirical evaluation of state-of-the-art methods for isotropy calibration on transformers and find that they do not provide consistent improvements across models and tasks. These results support the thesis that, given the local isotropy, transformers do not benefit from additional isotropy calibration.

show abstract

Quantifying the Contextualization of Word Representations with Semantic Class Probing

Cited by 11 publications

References 42 publications

Identifying Semantic Argument Types in Predication and Copredication Contexts: A Zero-Shot Cross-Lingual Approach

Identifying Semantic Argument Types in Predication and Copredication Contexts: A Zero-Shot Cross-Lingual Approach

LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Framework

On Isotropy Calibration of Transformer Models

Contact Info

Product

Resources

About