Open set evaluation of web genre identification

Pritsos, Dimitrios A.; Stamatatos, Efstathios

doi:10.1007/s10579-018-9418-y

Cited by 12 publications

(20 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additional high scores were achieved with fourgrams as non-binary features and bag-of-words. Similarly, Pritsos and Stamatatos (2018) recently achieved an F1-score of 79% based on word trigrams and two of the same corpora that Sharoff et al (2010) used. For other, earlier WGI studies, see Stamatatos et al (2000) based on common words, Kanaris and Stamatatos (2007) based on character n-grams, and Santini (2007) based on structural web page information.…”

Section: Detecting Online Registersmentioning

confidence: 89%

“…2.2). Similarly, Pritsos and Stamatatos (2018) reported that their bestperforming feature sets varied by the used corpora, which suggests that their models may have learnt patterns associated with topics rather than registers.…”

Section: Detecting Online Registersmentioning

confidence: 95%

“…In the analysis, we used text classification and linear SVMs (Boser 1992;Vapnik 1998). SVMs have been successfully applied in recent years with state-of-the-art performance in a number of different NLP tasks, including WGI (Petrenz and Webber 2011;Asheghi et al 2014;Pritsos and Stamatatos 2018; also Rodrigues and Couto 2017). Furthermore, linear SVMs have the advantage that they can be used to estimate the importance of a feature set or a feature to the model performance (see Guyon et al 2002;Guyon and Elisseeff 2003).…”

Section: Methodsmentioning

confidence: 99%

“…However, models trained primarily on grammatical features do not achieve very high accuracies (for results on stability, see Petrenz and Webber 2011; on the use of grammatical features in register identification, see . The best-performing models vary across studies, but typically, they are based on lexical information, such as character fourgrams (Sharoff et al 2010) and word trigrams (Pritsos and Stamatatos 2018). However, as shown by Sharoff et al (2010) and Petrenz and Webber (2011), these feature sets tend to learn topical aspects associated with the documents rather than generalizable register characteristics.…”

Section: Introductionmentioning

confidence: 99%

“…To investigate the kinds of features that provide the highest and most stable performance, we use a linear support vector machine (SVM) in the study. In addition to having achieved high performance in register identification tasks (see Sharoff et al 2010;Pritsos and Stamatatos 2018) and in document classification in general (see Joachims 1998), an important property of a linear SVM is that it allows us to estimate the contribution of a given feature or feature set to the model (see Guyon et al, 2002;Guyon and Elisseeff 2003). This is essential in our study, as the objective is not only to automatically identify the registers but also to examine the stability and the role of different linguistic features in the model.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Exploring the role of lexis and grammar for the stable identification of register in an unrestricted corpus of web documents

Laippala

Egbert

Biber

et al. 2021

Lang Resources & Evaluation

View full text Add to dashboard Cite

The Internet offers great possibilities for many scientific disciplines that utilize text data. However, the potential of online data can be limited by the lack of information on the genre or register of the documents, as register—whether a text is, e.g., a news article or a recipe—is arguably the most important predictor of linguistic variation (see Biber in Corpus Linguist Linguist Theory 8:9–37, 2012). Despite having received significant attention in recent years, the modeling of online registers has faced a number of challenges, and previous studies have presented contradictory results. In particular, these have concerned (1) the extent to which registers can be automatically identified in a large, unrestricted corpus of web documents and (2) the stability of the models, specifically the kinds of linguistic features that achieve the best performance while reflecting the registers instead of corpus idiosyncrasies. Furthermore, although the linguistic properties of registers vary importantly in a number of ways that may affect their modeling, this variation is often bypassed. In this article, we tackle these issues. We model online registers in the largest available corpus of online registers, the Corpus of Online Registers of English (CORE). Additionally, we evaluate the stability of the models towards corpus idiosyncrasies, analyze the role of different linguistic features in them, and examine how individual registers differ in these two aspects. We show that (1) competitive classification performance on a large-scale, unrestricted corpus can be achieved through a combination of lexico-grammatical features, (2) the inclusion of grammatical information improves the stability of the model, whereas many of the previously best-performing feature sets are less stable, and that (3) registers can be placed in a continuum based on the discriminative importance of lexis and grammar. These register-specific characteristics can explain the variation observed in previous studies concerning the automatic identification of online registers and the importance of different linguistic features for them. Thus, our results offer explanations for the jungle-likeness of online data and provide essential information on online registers for all studies using online data.

show abstract

Section: Detecting Online Registersmentioning

confidence: 89%

Section: Detecting Online Registersmentioning

confidence: 95%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Exploring the role of lexis and grammar for the stable identification of register in an unrestricted corpus of web documents

Laippala

Egbert

Biber

et al. 2021

Lang Resources & Evaluation

View full text Add to dashboard Cite

show abstract

Open-Set Web Genre Identification Using Distributional Features and Nearest Neighbors Distance Ratio

Pritsos

Rocha

Stamatatos

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Register identification from the unrestricted open Web using the Corpus of Online Registers of English

Laippala

Rönnqvist

Oinonen

et al. 2022

Lang Resources & Evaluation

View full text Add to dashboard Cite

This article examines the automatic identification of Web registers, that is, text varieties such as news articles and reviews. Most studies have focused on corpora restricted to include only preselected classes with well-defined characteristics. These corpora feature only a subset of documents found on the unrestricted open Web, for which register identification has been particularly difficult because the range of linguistic variation on the Web is known to be substantial. As part of this study, we present the first open release of the Corpus of Online Registers of English (CORE), which is drawn from the unrestricted open Web and, currently, is the largest collection of manually annotated Web registers. Furthermore, we demonstrate that the CORE registers can be automatically identified with competitive results, with the best performance being an F1-score of 68% with the deep learning model BERT. The best performance was achieved using two modeling strategies. The first one involved modeling the registers using propagated register labels, that is, repeating the main register label along with its corresponding subregister label in a multilabel model. In the second one, we explored how the length of the document affects model performance, discovering that the beginning provided superior classification accuracy. Overall, the current study presents a systematic approach for the automatic identification of a large number of Web registers from the unrestricted Web, hence providing new pathways for future studies.

show abstract

Open set evaluation of web genre identification

Cited by 12 publications

References 16 publications

Exploring the role of lexis and grammar for the stable identification of register in an unrestricted corpus of web documents

Exploring the role of lexis and grammar for the stable identification of register in an unrestricted corpus of web documents

Open-Set Web Genre Identification Using Distributional Features and Nearest Neighbors Distance Ratio

Register identification from the unrestricted open Web using the Corpus of Online Registers of English

Contact Info

Product

Resources

About