Evaluation of sentence embeddings in downstream and linguistic probing tasks

Perone, Christian S.; Silveira, Roberto Leal; Paula, Thomas S.

doi:10.48550/arxiv.1806.06259

Cited by 12 publications

(13 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To obtain sentence embeddings from individual words, we perform a weighted average of the word embeddings and use the TF-IDF scores of individual words as weight factors. It is a simple yet effective method to obtain sentence embedding for downstream tasks, as noted by previous work [28], [29]. This is shown in detail as Equation 2, where w j is the vector encoding the GloVe embedding of word x j :…”

Section: B Api Representation Learning and Matchingmentioning

confidence: 99%

SOAR: A Synthesis Approach for Data Science API Refactoring

Ramos

Yang

et al. 2021

Preprint

View full text Add to dashboard Cite

With the growth of the open-source data science community, both the number of data science libraries and the number of versions for the same library are increasing rapidly. To match the evolving APIs from those libraries, open-source organizations often have to exert manual effort to refactor the APIs used in the code base. Moreover, due to the abundance of similar open-source libraries, data scientists working on a certain application may have an abundance of libraries to choose, maintain and migrate between. The manual refactoring between APIs is a tedious and error-prone task. Although recent research efforts were made on performing automatic API refactoring between different languages, previous work relies on statistical learning with collected pairwise training data for the API matching and migration. Using large statistical data for refactoring is not ideal because such training data will not be available for a new library or a new version of the same library. We introduce Synthesis for Open-Source API Refactoring (SOAR), a novel technique that requires no training data to achieve API migration and refactoring. SOAR relies only on the documentation that is readily available at the release of the library to learn API representations and mapping between libraries. Using program synthesis, SOAR automatically computes the correct configuration of arguments to the APIs and any glue code required to invoke those APIs. SOAR also uses the interpreter's error messages when running refactored code to generate logical constraints that can be used to prune the search space. Our empirical evaluation shows that SOAR can successfully refactor 80% of our benchmarks corresponding to deep learning models with up to 44 layers with an average run time of 97.23 seconds, and 90% of the data wrangling benchmarks with an average run time of 17.31 seconds.

show abstract

Section: B Api Representation Learning and Matchingmentioning

confidence: 99%

SOAR: A Synthesis Approach for Data Science API Refactoring

Ramos

Yang

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The standard validation is employed. Being Different from the work in [61] that uses logistic regression for the WC task in the category of surface information, we use the same MLP model to provide simple yet fair comparison.…”

Section: Probing Tasksmentioning

confidence: 99%

SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models

Wang

Kuo

2020

Preprint

View full text Add to dashboard Cite

Sentence embedding is an important research topic in natural language processing (NLP) since it can transfer knowledge to downstream tasks. Meanwhile, a contextualized word representation, called BERT, achieves the state-of-the-art performance in quite a few NLP tasks. Yet, it is an open problem to generate a high quality sentence representation from BERTbased word models. It was shown in previous study that different layers of BERT capture different linguistic properties. This allows us to fusion information across layers to find better sentence representation. In this work, we study the layer-wise pattern of the word representation of deep contextualized models. Then, we propose a new sentence embedding method by dissecting BERT-based word models through geometric analysis of the space spanned by the word representation. It is called the SBERT-WK method. No further training is required in SBERT-WK. We evaluate SBERT-WK on semantic textual similarity and downstream supervised tasks. Furthermore, ten sentencelevel probing tasks are presented for detailed linguistic analysis. Experiments show that SBERT-WK achieves the state-of-the-art performance. Our codes are publicly available at the Github 1 .

show abstract

“…One of the major contributions is called word embedding. There are various types of word embedding in the literature that is well covered by Perone et al (2018).…”

Section: Related Workmentioning

confidence: 99%

A Multi-Resolution Word Embedding for Document Retrieval from Large Unstructured Knowledge Bases

Cakaloglu,

2019

Preprint

View full text Add to dashboard Cite

Deep language models learning a hierarchical representation proved to be a powerful tool for natural language processing, text mining and information retrieval. However, representations that perform well for retrieval must capture semantic meaning at different levels of abstraction or context-scopes. In this paper, we propose a new method to generate multi-resolution word embeddings that represent documents at multiple resolutions in terms of context-scopes. In order to investigate its performance,we use the Stanford Question Answering Dataset (SQuAD) and the Question Answering by Search And Reading (QUASAR) in an open-domain questionanswering setting, where the first task is to find documents useful for answering a given question. To this end, we first compare the quality of various text-embedding methods for retrieval performance and give an extensive empirical comparison with the performance of various non-augmented base embeddings with and without multi-resolution representation. We argue that multi-resolution word embeddings are consistently superior to the original counterparts and deep residual neural models specifically trained for retrieval purposes can yield further significant gains when they are used for augmenting those embeddings.

show abstract

Evaluation of sentence embeddings in downstream and linguistic probing tasks

Cited by 12 publications

References 22 publications

SOAR: A Synthesis Approach for Data Science API Refactoring

SOAR: A Synthesis Approach for Data Science API Refactoring

SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models

A Multi-Resolution Word Embedding for Document Retrieval from Large Unstructured Knowledge Bases

Contact Info

Product

Resources

About