Representation learning of writing style

Abstract

In this paper, we introduce a new method of representation learning that aims to embed documents in a stylometric space. Previous studies in the field of authorship analysis focused on feature engineering techniques in order to represent document styles and to enhance model performance in specific tasks. Instead, we directly embed documents in a stylometric space by relying on a reference set of authors and the intra-author consistency property which is one of two components in our definition of writing style. The main intuition of this paper is that we can define a general stylometric space from a set of reference authors such that, in this space, the coordinates of different documents will be close when the documents are by the same author, and spread away when they are by different authors, even for documents by authors who are not in the set of reference authors. The method we propose allows for the clustering of documents based on stylistic clues reflecting the authorship of documents. For the empirical validation of the method, we train a deep neural network model to predict authors of a large reference dataset consisting of news and blog articles. Albeit the learning process is supervised, it does not require a dedicated labeling of the data but it relies only on the metadata of the articles which are available in huge amounts. We evaluate the model on multiple datasets, on both the authorship clustering and the authorship attribution tasks.

Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework

“…Style embedding yields promising results. The method "deepstyle" (Hay et al, 2020) performs well across STEL components (0.66). It performs the worst on the simple/complex dimension (0.55).…”

“…Other Methods. We also experiment with the "deepstyle" model (Hay et al, 2020) by taking the cosine similarity between the style vector representations. Additionally, we consider the following sentence features: NLTK POS Tags (Bird et al, 2009) and share of cased characters (e.g., Sari et al (2018)) via the cosine similarity between the frequency vectors and 1 -the difference between the proportion of cased characters respectively.…”

“…Especially in authorship attribution, recurring style features include character n-grams, punctuation, average word length or function word frequency (Neal et al, 2017;Grieve, 2007;Stamatatos, 2009). Other recurring methods for style measurement include LIWC (Pennebaker et al, 2015;El Baff et al, 2020), and learned vector representations of words and sentences (Akama et al, 2018;Ding et al, 2019;Hay et al, 2020). suggests that style variations are already represented in commonly used neural embeddings.…”

“…There are several general evaluation benchmarks for different linguistic phenomena (e.g., Wang et al (2018Wang et al ( , 2019)) but less emphasis has been put on linguistic style. Nevertheless, natural language processing literature shows a variety of approaches for the evaluation of style measuring methods: They have been tested on whether they group texts by the same authors together (Hay et al, 2020;Bevendorff et al, 2020), whether they can correctly classify the style for ground truth datasets Kang and Hovy, 2021) and whether 'similar style words' are similarly represented (Akama et al, 2018). However, these evaluation approaches are (i) often application-specific, (ii) rarely used to compare different style methods, (iii) usually do not control for content and (iv) often do not test for fine-grained style differences.…”

“…To be evaluated on STEL, style measuring methods need not be able to classify styles directly. Instead, any method that can calculate the style similarity between two sentences can be evaluated: (1) Style (measuring) methods that calculate similarity values directly (e.g., edit distance or cross-encoders Reimers and Gurevych (2019)) and (2) vector representations of a sentence's style (e.g., Hay et al (2020); Ding et al (2019)) by using a distance or similarity measure between them (e.g., cosine similarity). This similarity-based setup also simplifies task extension (c.f.…”