Overview of CLEF 2019 Lab ProtestNews: Extracting Protests from News in a Cross-Context Setting

Hürriyetoğlu, Ali; Yörük, Erdem; Yüret, Deniz; Yoltar, Çağrı; Gürel, Burak; Duruşan, Fırat; Mutlu, Osman; Akdemir, Arda

doi:10.1007/978-3-030-28577-7_32

Cited by 23 publications

(24 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The event category incidence ratios are imbalanced both in individual countries and across countries. This difference is one potential reason for the performance gap of the automated tools for event information collection in crosscontext settings [17]. There are 409 participant and 161 organizer annotations, mostly first and last names, that were not semantically labeled, because they do not carry any semantic information per our annotation schema.…”

Section: Cross-context News Corpus For Protest Event-related Knowledge Base Constructionmentioning

confidence: 99%

“…Additional results that were yielded using some parts of this corpus can be found in various publications. The results obtained in a shared task for the cross-context document and sentence classification and token extraction were reported in the overview paper of the ProtestNews Lab [17], which was held in the scope of Conference and Labs of the Evaluation Forum (CLEF 2019). Participants of this shared task reported comparable results to the performance reported in this paper.…”

Section: Cross-context News Corpus For Protest Event-related Knowledge Base Constructionmentioning

confidence: 99%

“…The corpus has enabled the development of a pipeline of ML models that extract a knowledge base of protest events from archives of news articles. Moreover, some parts of the GSC have enabled shared tasks for cross-context document and sentence classification as well as token extraction [16,17] and event sentence coreference identification [18].…”

mentioning

confidence: 99%

See 2 more Smart Citations

Cross-Context News Corpus for Protest Event-Related Knowledge Base Construction

et al. 2021

Self Cite

View full text Add to dashboard Cite

We describe a gold standard corpus of protest events that comprise various local and international English language sources from various countries. The corpus contains document-, sentence-, and token-level annotations. This corpus facilitates creating machine learning models that automatically classify news articles and extract protest event-related information, constructing knowledge bases that enable comparative social and political science studies. For each news source, the annotation starts with random samples of news articles and continues with samples drawn using active learning. Each batch of samples is annotated by two social and political scientists, adjudicated by an annotation supervisor, and improved by identifying annotation errors semi-automatically. We found that the corpus possesses the variety and quality that are necessary to develop and benchmark text classification and event extraction systems in a cross-context setting, contributing to the generalizability and robustness of automated text processing systems. This corpus and the reported results will establish a common foundation in automated protest event collection studies, which is currently lacking in the literature.

show abstract

Section: Cross-context News Corpus For Protest Event-related Knowledge Base Constructionmentioning

confidence: 99%

Section: Cross-context News Corpus For Protest Event-related Knowledge Base Constructionmentioning

confidence: 99%

See 1 more Smart Citation

Cross-Context News Corpus for Protest Event-Related Knowledge Base Construction

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…Before preparing the data we had to consider the data shared in previous editions of the shared task, copyright issues and possible inference between data of separate subtasks. Some portion of the data in English was shared with academic community in previous shared tasks (Hürriyetoglu et al, 2019b(Hürriyetoglu et al, , 2020 and publications as sample data .…”

Section: Data Preparationmentioning

confidence: 99%

Multilingual Protest News Detection - Shared Task 1, CASE 2021

Hürriyetoğlu¹,

Mutlu²,

Yörük³

et al. 2021

Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-Political Events From Text (CAS

Self Cite

View full text Add to dashboard Cite

Benchmarking state-of-the-art text classification and information extraction systems in multilingual, cross-lingual, few-shot, and zeroshot settings for socio-political event information collection is achieved in the scope of the shared task Socio-political and Crisis Events Detection at the workshop CASE @ ACL-IJCNLP 2021. Socio-political event data is utilized for national and international policyand decision-making. Therefore, the reliability and validity of such datasets are of utmost importance. We split the shared task into three parts to address the three aspects of data collection (Task 1), fine-grained semantic classification (Task 2), and evaluation (Task 3). Task 1, which is the focus of this report, is on multilingual protest news detection and comprises four subtasks that are document classification (subtask 1), sentence classification (subtask 2), event sentence coreference identification (subtask 3), and event extraction (subtask 4). All subtasks have English, Portuguese, and Spanish for both training and evaluation data. Data in Hindi language is available only for the evaluation of subtask 1. The majority of the submissions, which are 238 in total, are created using multi-and cross-lingual approaches. Best scores are between 77.27 and 84.55 F1-macro for subtask 1, between 85.32 and 88.61 F1macro for subtask 2, between 84.23 and 93.03 CoNLL 2012 average score for subtask 3, and between 66.20 and 78.11 F1-macro for subtask 4 in all evaluation settings. The performance of the best system for subtask 4 is above 66.20 F1 for all available languages. Although there is still a significant room for improvement in cross-lingual and zero-shot settings, the best submissions for each evaluation scenario yield remarkable results. Monolingual models outperformed the multilingual models in a few evaluation scenarios, in which there is relatively much training data.

show abstract

“…constructed a corpus of protest events comprising various language sources from various countries. Several systems were submitted to the CLEF ProtestNews Track that consisted of three shared tasks, primarily aimed at identifying and extracting event information spanning to multiple countries (Hürriyetoglu et al, 2019b.…”

Section: Related Workmentioning

confidence: 99%

IIITT at CASE 2021 Task 1: Leveraging Pretrained Language Models for Multilingual Protest Detection

Kalyan¹,

Reddy²,

Hande³

et al. 2021

Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-Political Events From Text (CAS

View full text Add to dashboard Cite

In a world abounding in constant protests resulting from events like a global pandemic, climate change, religious or political conflicts, there has always been a need to detect events/protests before getting amplified by news media or social media. This paper demonstrates our work on the sentence classification subtask of multilingual protest detection in CASE@ACL-IJCNLP 2021. We approached this task by employing various multilingual pre-trained transformer models to classify if any sentence contains information about an event that has transpired or not. Furthermore, we performed soft voting over the models, achieving the best results among the models, accomplishing a macro F1-Score of 0.8291, 0.7578, and 0.7951 in English, Spanish, and Portuguese, respectively. The source codes for our systems are published 1 .

show abstract

Overview of CLEF 2019 Lab ProtestNews: Extracting Protests from News in a Cross-Context Setting

Cited by 23 publications

References 5 publications

Cross-Context News Corpus for Protest Event-Related Knowledge Base Construction

Cross-Context News Corpus for Protest Event-Related Knowledge Base Construction

Multilingual Protest News Detection - Shared Task 1, CASE 2021

IIITT at CASE 2021 Task 1: Leveraging Pretrained Language Models for Multilingual Protest Detection

Contact Info

Product

Resources

About