An Exploration of Semantic Features in an Unsupervised Thematic Fit Evaluation Framework

Abstract

Thematic fit is the extent to which an entity fits a thematic role in the semantic frame of an event, e.g., how well humans would rate "knife" as an instrument of an event of cutting. We explore the use of the SENNA semantic role-labeller in defining a distributional space in order to build an unsupervised model of event-entity thematic fit judgements. We test a number of ways of extracting features from SENNA-labelled versions of the ukWaC and BNC corpora and identify tradeoffs. Some of our Distributional Memory models outperform an existing syntaxbased model (TypeDM) that uses hand-crafted rules for role inference on a previously tested data set. We combine the results of a selected SENNA-based model with TypeDM's results and find that there is some amount of complementarity in what a syntactic and a semantic model will cover. In the process, we create a broad-coverage semantically-labelled corpus.

A Structured Distributional Model of Sentence Meaning and Processing

“…However, it should also be noticed that there is a striking difference between the two subsets of DTFit: while on patients the advantage of the gek models on both the baselines is clear, on locations the results are almost indistinguishable from those of the smoothed additive baseline, which simply adds the nearest neighbours to the vectors of the words in the sentence. This complies with previous studies on thematic fit modeling with dependency-based distributional models (Sayeed et al 2015. Because of the ambiguous nature of the prepositions used to identify potential locations, the role vectors used by SDM can be very noisy.…”

“…However, the differences are clearly minimal, suggesting that the structured knowledge encoded in the C-Phrase embeddings is not a plus for the thematic fit task. Concerning this point, it must be mentioned that most of the current models for thematic fit estimation rely on vectors relying either on syntactic information (Baroni and Lenci 2010, Greenberg et al 2015 or semantic roles (Sayeed et al 2015, Tilk et al 2016). On the other hand, our results comply with studies like Lapesa and Evert (2017), who reported comparable performance for bag-of-words and dependency-based models on several semantic modeling tasks, thus questioning whether the injection of linguistic structure in the word vectors is actually worth its processing cost.…”

Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge

“…In its classical form, the thematic fit estimation task consists in comparing a candidate argument or filler (e.g., wine) with the typical fillers of a given verb role (e.g., agent, patient, etc. ), either in the form of exemplars previously attested in a corpus (Erk, 2007;Vandekerckhove et al, 2009;Erk et al, 2010) or in the form of a vector-based prototype (Baroni and Lenci, 2010;Sayeed and Demberg, 2014;Sayeed et al, 2015;Greenberg et al, 2015a,b;Santus et al, 2017;. Additionally, recent studies explored the use of masked language modeling with BERT for scoring the candidate arguments (Metheniti et al, 2020).…”

Measuring Thematic Fit with Distributional Feature Overlap

“…Starting from the work of Erk et al (2010), several distributional semantic methods have been proposed to compute the extent to which nouns fulfill the requirements of verb-specific thematic roles, and their performances have been evaluated against human-generated judgments (Baroni and Lenci, 2010;Lenci, 2011;Sayeed and Demberg, 2014;Sayeed et al, 2015Greenberg et al, 2015a,b). Most research on thematic fit estimation has focused on count-based vector representations (as distinguished from prediction-based vectors).…”

“…The thematic fit models proposed by Sayeed and Demberg (2014) and Sayeed et al (2015) are similar to Baroni and Lenci's, but their DSMs were built by using the roles assigned by the SENNA semantic role labeler (Collobert et al, 2011) to define the feature space. These authors argued that the prototype-based method with dependencies works well when applied to the agent and to the patient role (which are almost always syntactically realized as subjects and objects), but that it might be problematic to apply it to different roles, such as instruments and locations, as the construction of the prototype would have to rely on prepositional complements as typical fillers, and the meaning of prepositions can be ambiguous.…”