On the creation of a clinical gold standard corpus in Spanish: Mining adverse drug reactions

Oronoz, Maite; Gojenola, Koldo; Pérez, Alicia; Ilarraza, Arantza Díaz de; Casillas, Arantza

doi:10.1016/j.jbi.2015.06.016

Cited by 70 publications

(61 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, Albright et al (2013) achieved an F1 measure of 0.697 in exact match, but of 0.750 in partial match. Overall, our Ogren et al (2008) for English (from 75.7 to 81.4% in entity annotation, exact match) and Oronoz et al (2015) for Spanish (from 88.63 to 90.53% in term annotation). In a POS annotation task of clinical texts, Savkov et al (2016) also obtained similar results (0.76% of F-measure).…”

Section: Inter-annotator Agreement (Iaa)mentioning

confidence: 63%

“…Notable research initiatives, in collaboration with health institutions, have annotated clinical texts: the Mayo Clinic corpus (Ogren et al 2008), the Clinical E-Science Framework (CLEF) (Roberts et al 2009), the THYME (Temporal Histories of Your Medical Events) project (Styler et al 2014), 7 the SHARP Template Annotations (Savova et al 2012), the MiPACQ (Multi-source Integrated Platform for Answering Clinical Questions) (Albright et al 2013), the IxA-Med-GS (Oronoz et al 2015) or the Harvey corpus (Savkov et al 2016). Research challenges have also fuelled the annotation of resources or enrichment of available texts.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)

Campillos

Deléger

Grouin

et al. 2017

Lang Resources & Evaluation

View full text Add to dashboard Cite

Quality annotated resources are essential for Natural Language Processing. The objective of this work is to present a corpus of clinical narratives in French annotated for linguistic, semantic and structural information, aimed at clinical information extraction. Six annotators contributed to the corpus annotation, using a comprehensive annotation scheme covering 21 entities, 11 attributes and 37 relations. All annotators trained on a small, common portion of the corpus before proceeding independently. An automatic tool was used to produce entity and attribute pre-annotations. About a tenth of the corpus was doubly annotated and annotation differences were resolved in consensus meetings. To ensure annotation consistency throughout the corpus, we devised harmonization tools to automatically identify annotation differences to be addressed to improve the overall corpus quality. The annotation project spanned over 24 months and resulted in a corpus comprising 500 documents (148,476 tokens) annotated with 44,740 entities and 26,478 relations. The average inter-annotator agreement is 0.793 F-measure for entities and 0.789 for relations. The performance of the pre-annotation tool for entities reached 0.814 F-measure when sufficient training data was available. The performance of our entity pre-annotation tool shows the value of the corpus to build and evaluate information extraction methods. In addition, we introduced harmonization methods that further improved the quality of annotations in the corpus.Electronic supplementary material The online version of this article

show abstract

Section: Inter-annotator Agreement (Iaa)mentioning

confidence: 63%

Section: Introductionmentioning

confidence: 99%

A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)

Campillos

Deléger

Grouin

et al. 2017

Lang Resources & Evaluation

View full text Add to dashboard Cite

show abstract

“…for English (Uzuner et al, 2011;Pradhan et al, 2013Pradhan et al, , 2014, for Swedish (Skeppstedt et al, 2014), for French (Névéol et al, 2015), for Polish (Mykowiecka et al, 2009) and for German (Roller et al, 2016). Oronoz et al (2015) …”

Section: Previous Workmentioning

confidence: 99%

Annotation of Entities and Relations in Spanish Radiology Reports

Cotik

Filippo²,

Roller

et al. 2017

RANLP 2017 - Recent Advances in Natural Language Processing Meet Deep Learning

View full text Add to dashboard Cite

Radiology reports express the results of a radiology study and contain information about anatomical entities, findings, measures and impressions of the medical doctor. The use of information extraction techniques can help physicians to access this information in order to understand data and to infer further knowledge.Supervised machine learning methods are very popular to address information extraction, but are usually domain and language dependent. To train new classification models, annotated data is required. Moreover, annotated data is also required as an evaluation resource of information extraction algorithms. However, one major drawback of processing clinical data is the low availability of annotated datasets. For this reason we performed a manual annotation of radiology reports written in Spanish. This paper presents the corpus, the annotation schema, the annotation guidelines and further insight of the data.

show abstract

“…In Spanish clinical domain, (Oronoz et al, 2015) reported that 27.58% of diseases presented in a corpus of electronic health records are negated. Although this percentage is not directly comparable with the percentage of negated sentences shown in Table 2, the numbers seem similar.…”

Section: Negation In Spanish Clinical Textsmentioning

confidence: 99%

“…Although the authors mention that 500 medical texts have been manually annotated, they do not provide more information about this goldstandard. Oronoz et al (2015), annotated negated adverse drug reactions, but they did not annotate the words expressing negation. The negation was annotated as a modifier of the disorder or drug.…”

Section: Introductionmentioning

confidence: 99%

Proceedings of the Workshop Computational Semantics Beyond Events and Roles

2017

View full text Add to dashboard Cite

Interpersonal violence (IPV) is a prominent sociological problem that affects people of all demographic backgrounds. By analyzing how readers interpret, perceive, and react to experiences narrated in social media posts, we explore an understudied source for discourse about abuse. We asked readers to annotate Reddit posts about relationships with vs. without IPV for stakeholder roles and emotion, while measuring their galvanic skin response (GSR), pulse, and facial expression. We map annotations to coreference resolution output to obtain a labeled coreference chain for stakeholders in texts, and apply automated semantic role labeling for analyzing IPV discourse. Findings provide insights into how readers process roles and emotion in narratives. For example, abusers tend to be linked with violent actions and certain affect states. We train classifiers to predict stakeholder categories of coreference chains. We also find that subjects' GSR noticeably changed for IPV texts, suggesting that co-collected measurement-based data about annotators can be used to support text annotation. IntroductionMore than one in three women and one in four men in the United States have experienced rape, physical violence, and/or stalking by an intimate partner (Black et al., 2011). One in nine girls and one in 53 boys under the age of eighteen are sexually abused by an adult (Finkelhor et al., 2014). Additionally, approximately one in ten elders in the USA have faced intimidation, isolation, neglect, and threats of violence. 1 Such interpersonal violence (IPV) 2 can lead to injury, depression, post-traumatic stress disorder, substance abuse, sexually transmitted diseases, as well as hospitalization, disability, or death (Black et al., 2011). Most of the science on IPV is based on survey and interview data. However, the nature of IPV relationships can make people feel uncomfortable or unsafe when participating in such studies, leading to inaccurate results. Also, surveys can be costly and time-consuming to carry out (Schrading et al., 2015b).Social media is an understudied source of IPV data. Over 79% of adults that frequent the internet 1 https://www.ncoa.org/public-policy-action/elderjustice/elder-abuse-facts/ 2 For the purposes of this study, we use the WHO definition of IPV, while we recognize that the acronym commonly refers to "Intimate Partner Violence", a subset of this phenomenon. 1 utilize social media (Greenwood et al., 2016). Online, individuals can anonymously share their experiences without fear of embarrassment or repercussions. Such narratives can also provide more details than surveys, and may lead to a deeper understanding of IPV. Nonetheless, it is extremely difficult to establish reference annotations useful for predictive modeling for discourse topics as emotionally charged as IPV.We meet these challenges with a combination of annotator labeling, analyzing annotations, applying semantic processing techniques (coreference resolution, semantic role labeling, sentiment analysis), developing classifiers, and ...

show abstract

On the creation of a clinical gold standard corpus in Spanish: Mining adverse drug reactions

Cited by 70 publications

References 33 publications

A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)

A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI annOtated Text corpus (MERLOT)

Annotation of Entities and Relations in Spanish Radiology Reports

Proceedings of the Workshop Computational Semantics Beyond Events and Roles

Contact Info

Product

Resources

About