A Systematic Comparison of English Noun Compound Representations

Abstract

Building meaningful representations of noun compounds is not trivial since many of them scarcely appear in the corpus. To that end, composition functions approximate the distributional representation of a noun compound by combining its constituent distributional vectors. In the more general case, phrase embeddings have been trained by minimizing the distance between the vectors representing paraphrases. We compare various types of noun compound representations, including distributional, compositional, and paraphrasebased representations, through a series of tasks and analyses, and with an extensive number of underlying word embeddings. We find that indeed, in most cases, composition functions produce higher quality representations than distributional ones, and they improve with computational power. No single function performs best in all scenarios, suggesting that a joint training objective may produce improved representations.

An Unsupervised Method for Learning Representations of Multi-word Expressions for Semantic Classification

“…Much like distributional similarity approaches for learning word representations (Mikolov et al, 2013a;Bojanowski et al, 2017;Pennington et al, 2014, inter alia), a semantic representation of MWEs can be trained using a distributional approach that treats MWEs as single tokens (Mikolov et al, 2013b). However, this approach cannot handle out of vocabulary (OOV) MWEs, and it is likely to suffer from sparsity (Shwartz, 2019), particularly as the MWEs grow in length.…”

“…Our proposed approach does not suffer from this observer effect, as we learn our compositional function indirectly (through Skip-Gram), without relying on a distributionally learned embedding for MWEs for training. Shwartz (2019) also avoid this reliance on the gold embedding of the multi-word, learning the function indirectly. The compositional function is used to encode the multiword and its paraphrase and is then trained to maximize the cosine similarity between the encodings.…”

“…The paraphrases are generated either using backtranslation (Wieting et al, 2017;Wieting and Gimpel, 2018), or by treating frequent joint co-occurrences as paraphrases. However, this approach is outperformed by much cheaper unsupervised approaches, such as the average of the constituents' emebddings (Shwartz, 2019). Furthermore, the backtranslation approach depends on an external system, adding complexity to the model and restricting the languages and domains of application.…”

“…Multi-word expressions (MWEs) are fundamental to language and, as such, having a robust semantic representation for MWEs is important for any natural language processing task that involves text understanding such as information extraction, or question answering (e.g., da Silva and Souza, 2012;Thurmair, 2018;Subramanian et al, 2018). While MWEs have received attention in recent years, leading to considerable progress in learning MWE representations (Mitchell and Lapata, 2010;Butnariu et al, 2010;Tratz, 2011;Hendrickx et al, 2013;Dima, 2016;Shwartz and Dagan, 2018;Shwartz, 2019), we argue that the proposed methods have limitations. First, some methods require concatenating words in specified MWEs, and treating the resulting MWE phrases as atomic units.…”

“…Following Shwartz (2019), for our experiments we used FastText embeddings (Bojanowski et al, 2017), which, overall, performed the best on the Tratz dataset (Tratz, 2011). The embeddings were trained on the English Wikipedia dump from January 2018 3 to facilitate comparison between this work and that of Shwartz (2019).…”