Overview of the ShARe/CLEF eHealth Evaluation Lab 2014

Abstract

Abstract. Discharge summaries and other free-text reports in healthcare transfer information between working shifts and geographic locations. Patients are likely to have difficulties in understanding their content, because of their medical jargon, non-standard abbreviations, and ward-specific idioms. This paper reports on an evaluation lab with an aim to support the continuum of care by developing methods and resources that make clinical reports in English easier to understand for patients, and which helps them in finding information related to their condition. This ShARe/CLEFeHealth2013 lab offered student mentoring and shared tasks: identification and normalisation of disorders (1a and 1b) and normalisation of abbreviations and acronyms (2) Overview of the ShARe/CLEF eHealth Evaluation Lab 2013 213 reports with respect to terminology standards in healthcare as well as information retrieval (3) to address questions patients may have when reading clinical reports. The focus on patients' information needs as opposed to the specialised information needs of physicians and other healthcare workers was the main feature of the lab distinguishing it from previous shared tasks. De-identified clinical reports for the three tasks were from US intensive care and originated from the MIMIC II database. Other text documents for Task 3 were from the Internet and originated from the Khresmoi project. Task 1 annotations originated from the ShARe annotations. For Tasks 2 and 3, new annotations, queries, and relevance assessments were created. 64, 56, and 55 people registered their interest in Tasks 1, 2, and 3, respectively. 34 unique teams (3 members per team on average) participated with 22, 17, 5, and 9 teams in Tasks 1a, 1b, 2 and 3, respectively. The teams were from Australia, China, France, India, Ireland, Republic of Korea, Spain, UK, and USA. Some teams developed and used additional annotations, but this strategy contributed to the system performance only in Task 2. The best systems had the F1 score of 0.75 in Task 1a; Accuracies of 0.59 and 0.72 in Tasks 1b and 2; and Precision at 10 of 0.52 in Task 3. The results demonstrate the substantial community interest and capabilities of these systems in making clinical reports easier to understand for patients. The organisers have made data and tools available for future research and development.

Scholarly Influence of the Conference and Labs of the Evaluation Forum eHealth Initiative: Review and Bibliometric Study of the 2012 to 2017 Outcomes (Preprint)

“…CLEF eHealth particularly welcomed and attracted multi-disciplinary teams to collaborate and bridge the academy, government, and industrial researchers, scientists, lecturers, and graduate students with engineers, practitioners, and policy makers [31][32][33][34][35]. For example, in the 33 WNs and 1 CP from the CLEF eHealth 2013 evaluation lab, 162 authors from 10 countries highlighted some leading organizations in health information management, extraction, and retrieval, including,…”

“…The topic of patient-friendly multilingual communication formed the focus of the annual CLEF eHealth evaluation labs in 2013-2017 [31][32][33][34][35], generated the total scholarly influence of 962,559 citations for the 184 CLEF eHealth papers, and reached authors from 33 countries across the world (Table 1, Figure 3) [8]. This influence was computed by multiplying the number of co-authors in the 184 papers (i.e., 741) by the number of citations (i.e., 1,299) these papers had received on 26 October 2017.…”

On the Effects of Low-Quality Training Data on Information Extraction from Clinical Reports

“…In recent years, research on the analysis of clinical texts has been further boosted by the existence of "shared tasks" on this topic, such as the seminal i2b2 series ("Informatics for Integrating Biology and the Bedside" - [Sun et al 2013;Uzuner et al 2012;Uzuner et al 2011]), the 2013 [Suominen et al 2013] and 2014 [Kelly et al 2014] editions of ShARe/CLEF eHealth, and the Semeval-2014 Task 7 "Analysis of Clinical Text" [Pradhan et al 2014]. In these shared tasks the goal is to competitively evaluate information extraction tools that recognise mentions of various concepts of interest (e.g., mentions of diseases and disorders) as appearing in discharge summaries, and in electrocardiogram reports, echocardiograph reports, and radiology reports.…”

“…In the last five years there has been a flurry of work (see e.g., [Kelly et al 2014;Pradhan et al 2014;Sun et al 2013;Suominen et al 2013;Uzuner et al 2012;Uzuner et al 2011]) on information extraction from clinical documents, i.e., on algorithms capable of extracting, from the informal and unstructured texts that are generated during everyday clinical practice (e.g., admission reports, radiological reports, discharge summaries, clinical notes), mentions of concepts relevant to such practice. Most of this literature is about methods based on supervised learning, i.e., methods for training an information extraction system from manually annotated examples.…”

Một Hướng Tiếp Cận Lai Ghép Phân Lớp Mối Quan Hệ Giữa Bệnh Và Thời Gian Viết Tài Liệu Lâm Sàng

“…Các giá trị phân lớp thời gian thể hiện mối quan hệ về mặt thời gian giữa bệnh/rối loạn và thời gian tài liệu lâm sàng được tạo ra [4]. Các giá trị phân lớp thời gian gồm có 5 giá trị: BEFORE, OVERLAP, BEFORE_OVERLAP, AFTER và UNKNOWN (giá trị mặc định).…”