Distributional vectors encode referential attributes

Abstract

Distributional methods have proven to excel at capturing fuzzy, graded aspects of meaning (Italy is more similar to Spain than to Germany). In contrast, it is difficult to extract the values of more specific attributes of word referents from distributional representations, attributes of the kind typically found in structured knowledge bases (Italy has 60 million inhabitants). In this paper, we pursue the hypothesis that distributional vectors also implicitly encode referential attributes. We show that a standard supervised regression model is in fact sufficient to retrieve such attributes to a reasonable degree of accuracy: When evaluated on the prediction of both categorical and numeric attributes of countries and cities, the model consistently reduces baseline error by 30%, and is not far from the upper bound. Further analysis suggests that our model is able to "objectify" distributional representations for entities, anchoring them more firmly in the external world in measurable ways.

Designing and Interpreting Probes with Control Tasks

“…Early work in probing, (also known as diagnostic classification ,) extracted properties like parts-of-speech, gender, tense, and number from distributional word vector spaces like word2vec and GloVe (Mikolov et al, 2013;Pennington et al, 2014) using linear classifiers (Köhn, Part-of- 2015; Gupta et al, 2015). Soon after, the investigation of intermediate layers of deep models using linear probes was introduced independently by Ettinger et al (2016) and Shi et al (2016) in NLP and Alain and Bengio (2016) in computer vision.…”

What do you know about an alligator when you know the company it keeps?

“…There is some empirical evidence that distributional data can be used for inferring properties in Johns & Jones 2012, Fȃgȃrȃşan, Vecchi & Clark 2015, Gupta et al 2015, and Herbelot & Vecchi 2015. They test whether distributional vectors can be used to predict a word's properties (where, as above, I use the term "properties of a word" to mean properties that apply to all entities in the word's extension).…”

“…Finally, there is a need for more research on distributional models for property inference, to develop efficient models beyond the initial approaches proposed by Johns & Jones (2012), Fȃgȃrȃşan, Vecchi & Clark (2015), Herbelot & Vecchi (2015) and Gupta et al (2015) and to see what kinds of properties can be reliably learned and whether verb properties can be learned as well as noun properties.…”

Building a shared world: mapping distributional to model-theoretic semantic spaces

“…But it implements the mapping as a systematic linear transformation. Our approach is similar to Gupta et al (2015), who predict numerical attributes for unseen concepts (countries and cities) from distributional vectors, getting comparably accurate estimates for features such as the GDP or CO 2 emissions of a country. We complement such research by providing a more formal interpretation of the mapping between language and world knowledge.…”