A Semantic and Syntactic Similarity Measure for Political Tweets

Abstract

Measurement of the semantic and syntactic similarity of human utterances is essential in allowing machines to understand dialogue with users. However, human language is complex, and the semantic meaning of an utterance is usually dependent upon the context at a given time and learnt experience of the meaning of the words that are used. This is particularly challenging when automatically understanding the meaning of social media, such as tweets, which can contain non-standard language. Short Text Semantic Similarity measures can be adapted to measure the degree of similarity of a pair of tweets. This work presents a new Semantic and Syntactic Similarity Measure (TSSSM) for political tweets. The approach uses word embeddings to determine semantic similarity and extracts syntactic features to overcome the limitations of current measures which may miss identical sequences of words. A large dataset of tweets focusing on the political domain were collected, pre-processed and used to train the word embedding model, with various experiments performed to determine the optimal model and parameters. A selection of tweet pairs were evaluated by humans for semantic equivalence and correlated against the measure. The new measure can be used in a variety of applications, including for identifying and analyzing political narratives. Experiments on three diverse human-labelled test datasets demonstrate that the measure outperforms an existing measure, performs well on tweets from the political domain and may also generalize outside the political domain.

FastKASSIM: A Fast Tree Kernel-Based Syntactic Similarity Metric

“…This ranges from traditional methods such as TFIDF or Jaccard similarity to modern approaches including BERT embeddings (Devlin et al, 2019;Reimers and Gurevych, 2019;Zhang et al, 2019b) and AMR kernels (Opitz et al, 2021). Some approaches use syntactic features specific to certain domains such as Twitter (Alnajran, 2019;Little et al, 2020) or web documents (Broder et al, 1997;Pereira and Ziviani, 2003). Other metrics include "syntactic elements" which take on various forms of parts-of-speech aggregation (Alnajran, 2019;Pakray et al, 2011).…”

“…An important advantage of CASSIM is that it is generalizable to any corpus; it does not represent syntax using platform-specific features like Alnajran (2019); Little et al (2020). However, the cost of exhaustively using a metric such as Edit Distance is rather penalizing, as its implementations range in asymptotic time complexity from Θ(mn) (Wagner and Fischer, 1974) nen, 1985), where m and n are the string lengths, and s is the maximal Edit Distance.…”

FastKASSIM: A Fast Tree Kernel-Based Syntactic Similarity Metric

“…Alnajran (2019) proposed TREA-SURE, which primarily captured semantic similarity using word embeddings, but also considered syntax in the form of coarse metrics such as counting parts-of-speech and features specific to the social media platform Twitter. Similarly, Little et al (2020) also computed a metric for semantic and syntactic similarity specific to Twitter. They took a weighted average between BERT embeddings and their "syntactic element," which consists of the length of the longest common sequences of partsof-speech, and a feature vector counting the number of hashtags, mentions, and pronouns.…”

“…One of the most imporant advantages of CAS-SIM is that it is generalizable to any corpus, as it doesn't require platform-specific features like Alnajran (2019); Little et al (2020). However, because the metric is exhaustive, the cost of an algorithm such as Edit Distance becomes rather penalizing -the common Edit Distance algorithms range in time complexity from Θ(mn) (Wagner and Fischer, 1974) to O(s×min(m, n)) (Ukkonen, 1985), where m and n are the string lengths, and s is the maximal Edit Distance.…”

Fuzzy Influence in Fuzzy Semantic Similarity Measures

“…Similarity measures combine semantic and syntactic features of natural language to determine a similarity measure of two short texts. Short texts are typically 25 words or less in length [1] and include structured (sentences) and unstructured (tweets) [2,3,4,5]. Substantial research has been undertaken in the field of traditional semantic similarity [6], with methods typically grouped into corpus-based [7], string-based [8], knowledge-based [9], and hybrid [1].…”