Inter-Annotator Agreement in Sentiment Analysis: Machine Learning Perspective

Bobicev, Victoria; Sokolova, Marina

doi:10.26615/978-954-452-049-6_015

Cited by 54 publications

(29 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Each document was labeled manually and independently by three Italian native speakers, who are researchers in the e-health domain, with the agreement among the annotators calculated by majority. The global agreement for the entire annotation procedure was measured using the Observed Agreement index [71] which provides a good approximation in multi-annotator contexts, also offering robustness against imperfect (textual) data [72]. In addition to the Observed Agreement index, in order to take into account the level of Inter Annotator Agreement (IAA) in terms of excess over the agreement obtained by chance, the Krippendorff coefficient α [73] was also calculated.…”

Section: B: Annotation Proceduresmentioning

confidence: 99%

A Novel COVID-19 Data Set and an Effective Deep Learning Approach for the De-Identification of Italian Medical Records

et al. 2021

View full text Add to dashboard Cite

In the last years, the need to de-identify privacy-sensitive information within Electronic Health Records (EHRs) has become increasingly felt and extremely relevant to encourage the sharing and publication of their content in accordance with the restrictions imposed by both national and supranational privacy authorities. In the field of Natural Language Processing (NLP), several deep learning techniques for Named Entity Recognition (NER) have been applied to face this issue, significantly improving the effectiveness in identifying sensitive information in EHRs written in English. However, the lack of data sets in other languages has strongly limited their applicability and performance evaluation. To this aim, a new de-identification data set in Italian has been developed in this work, starting from the 115 COVID-19 EHRs provided by the Italian Society of Radiology (SIRM): 65 were used for training and development, the remaining 50 were used for testing. The data set was labelled following the guidelines of the i2b2 2014 de-identification track. As additional contribution, combined with the best performing Bi-LSTM + CRF sequence labeling architecture, a stacked word representation form, not yet experimented for the Italian clinical de-identification scenario, has been tested, based both on a contextualized linguistic model to manage word polysemy and its morpho-syntactic variations and on sub-word embeddings to better capture latent syntactic and semantic similarities. Finally, other cutting-edge approaches were compared with the proposed model, which achieved the best performance highlighting the goodness of the promoted approach. INDEX TERMS clinical de-identification, contextualized embedding, deep learning, Italian language, named entity recognition

show abstract

Section: B: Annotation Proceduresmentioning

confidence: 99%

A Novel COVID-19 Data Set and an Effective Deep Learning Approach for the De-Identification of Italian Medical Records

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The Cohen kappa coefficient [ 36 ] and percent of agreement [ 37 ] were calculated to measure the interobserver agreement between the pediatric radiologists in all investigated ROIs. Statistical analyses were performed using SPSS Statistics (version 24; IBM Corp).…”

Section: Methodsmentioning

confidence: 99%

Chronological Age Assessment in Young Individuals Using Bone Age Assessment Staging and Nonradiological Aspects: Machine Learning Multifactorial Approach

Dallora¹,

Kvist²,

Berglund³

et al. 2020

JMIR Med Inform

View full text Add to dashboard Cite

Background Bone age assessment (BAA) is used in numerous pediatric clinical settings as well as in legal settings when entities need an estimate of chronological age (CA) when valid documents are lacking. The latter case presents itself as critical as the law is harsher for adults and granted rights along with imputability changes drastically if the individual is a minor. Traditional BAA methods have drawbacks such as exposure of minors to radiation, they do not consider factors that might affect the bone age, and they mostly focus on a single region. Given the critical scenarios in which BAA can affect the lives of young individuals, it is important to focus on the drawbacks of the traditional methods and investigate the potential of estimating CA through BAA. Objective This study aims to investigate CA estimation through BAA in young individuals aged 14-21 years with machine learning methods, addressing the drawbacks of research using magnetic resonance imaging (MRI), assessment of multiple regions of interest, and other factors that may affect the bone age. Methods MRI examinations of the radius, distal tibia, proximal tibia, distal femur, and calcaneus were performed on 465 men and 473 women (aged 14-21 years). Measures of weight and height were taken from the subjects, and a questionnaire was given for additional information (self-assessed Tanner Scale, physical activity level, parents' origin, and type of residence during upbringing). Two pediatric radiologists independently assessed the MRI images to evaluate their stage of bone development (blinded to age, gender, and each other). All the gathered information was used in training machine learning models for CA estimation and minor versus adult classification (threshold of 18 years). Different machine learning methods were investigated. Results The minor versus adult classification produced accuracies of 0.90 and 0.84 for male and female subjects, respectively, with high recalls for the classification of minors. The CA estimation for the 8 age groups (aged 14-21 years) achieved mean absolute errors of 0.95 years and 1.24 years for male and female subjects, respectively. However, for the latter, a lower error occurred only for the ages of 14 and 15 years. Conclusions This study investigates CA estimation through BAA using machine learning methods in 2 ways: minor versus adult classification and CA estimation in 8 age groups (aged 14-21 years), while addressing the drawbacks in the research on BAA. The first achieved good results; however, for the second case, the BAA was not precise enough for the classification.

show abstract

“…The tweets analyzed were related to emotional responses of individuals to urban green space. Bobicev and Sokolova [Bobicev and Sokolova 2017] used a limited set of three annotators to analyze texts extracted from an online health forum. Each text can be labeled with one or more sentiment labels: gratitude, encouragement, confusion and facts (this last one indicating neutral content).…”

Section: Literature Reviewmentioning

confidence: 99%

“…Measuring divergence among annotators -Inter-rater Agreement: We adopted the Krippendorff's alpha (α) [Krippendorff 2011] to measure the general agreement level among the independent annotators for each one of the manual labeling tasks, namely: SA classification, OS classification and CA classification. The Krippendorff's alpha (α) agreement coefficient looks at the overall distribution of annotations/labels, not considering which annotators produced these annotations [Bobicev and Sokolova 2017]. Differently from other metrics such as Cohen's Kappa [Cohen 1960] (which computes agreement level between a pair of annotators) and Fleiss' Kappa [Fleiss et al 1981] (which is a generalization of Cohen Kappa and allows more than two annotators), the metric Krippendorff's alpha α can be applied to evaluate labeling agreement among multiple annotators even when there are missing values.…”

Section: Divergence Analysismentioning

confidence: 99%

Measuring the Degree of Divergence when Labeling Tweets in the Electoral Scenario

Santos¹,

Bernardini²,

Paes³

2021

Anais Do X Brazilian Workshop on Social Network Analysis and Mining (BraSNAM 2021)

View full text Add to dashboard Cite

Analyzing electoral trends in political scenarios using social media with data mining techniques has become popular in recent years. A problem in this field is to reliably annotate data during the short period of electoral campaigns. In this paper, we present a methodology to measure labeling divergence and an exploratory analysis of data related to the 2018 Brazilian Presidential Elections. As a result, we point out some of the main characteristics that lead to a high level of divergence during the annotation process in this domain. Our analysis shows a high degree of divergence mainly in regard to sentiment labels. Also, a significant difference was identified between labels obtained by manual annotation and labels obtained using an automatic annotation approach.

show abstract

Inter-Annotator Agreement in Sentiment Analysis: Machine Learning Perspective

Cited by 54 publications

References 10 publications

A Novel COVID-19 Data Set and an Effective Deep Learning Approach for the De-Identification of Italian Medical Records

A Novel COVID-19 Data Set and an Effective Deep Learning Approach for the De-Identification of Italian Medical Records

Chronological Age Assessment in Young Individuals Using Bone Age Assessment Staging and Nonradiological Aspects: Machine Learning Multifactorial Approach

Measuring the Degree of Divergence when Labeling Tweets in the Electoral Scenario

Contact Info

Product

Resources

About