CLEF 2017 eHealth Evaluation Lab Overview

Goeuriot, Lorraine; Kelly, Liadh; Suominen, Hanna; Névéol, Aurélie; Robert, Aude; Kanoulas, Evangelos; Spijker, René; Palotti, João; Zuccon, Guido

doi:10.1007/978-3-319-65813-1_26

Cited by 40 publications

(37 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Evaluation methods for the CLEF eHealth tasks in 2013-2017 (Table 2) were described in detail in [31]- [35], respectively, and are summarised below. The reader is referred to these papers for methodological references.…”

Section: Evaluation Methods From 2013 To 2017mentioning

confidence: 99%

“…CLEF eHealth particularly welcomed and attracted multi-disciplinary teams to collaborate and bridge the academy, government, and industrial researchers, scientists, lecturers, and graduate students with engineers, practitioners, and policy makers [31][32][33][34][35]. For example, in the 33 WNs and 1 CP from the CLEF eHealth 2013 evaluation lab, 162 authors from 10 countries highlighted some leading organizations in health information management, extraction, and retrieval, including,…”

Section: Citation Analysis From 2012 To 2017mentioning

confidence: 99%

“…The topic of patient-friendly multilingual communication formed the focus of the annual CLEF eHealth evaluation labs in 2013-2017 [31][32][33][34][35], generated the total scholarly influence of 962,559 citations for the 184 CLEF eHealth papers, and reached authors from 33 countries across the world (Table 1, Figure 3) [8]. This influence was computed by multiplying the number of co-authors in the 184 papers (i.e., 741) by the number of citations (i.e., 1,299) these papers had received on 26 October 2017.…”

Section: Citation Analysis From 2012 To 2017mentioning

confidence: 99%

“…In 2017, the following three tasks were organised [35] to continue the 2016 Tasks 2 and 3 and introduce a new pilot task: The 2017 Task 1 explored the problem of multilingual text processing by considering this year the extraction of causes of death from both FR and EN death reports to ease clinicians understanding of these reports. The 2017 Task 3 developed medical web search techniques to address the challenge posed by patients (or their next-of-kin) in locating relevant and reliable medical content on the web.…”

Section: Citation Analysis From 2012 To 2017mentioning

confidence: 99%

“…With an aim to lower the entry barrier and encourage novelty and creativity in problem solutions, CLEF eHealth began providing participants with software and code for method evaluation, record text annotation, and document relevance assessment in 2013 and extended this to also release processing code in 2016 [31]- [35]. The reader is referred to these papers [31], [32], [33], [34], and [35], and references therein, for further information about the software and code releases in 2013, 2014, 2015, 2016, and 2017, respectively.…”

Section: Software Releases and Submission From 2013 To 2017mentioning

confidence: 99%

See 4 more Smart Citations

Scholarly Influence of the Conference and Labs of the Evaluation Forum eHealth Initiative: Review and Bibliometric Study of the 2012 to 2017 Outcomes (Preprint)

Suominen¹,

Kelly²,

Goeuriot³

2018

Preprint

Self Cite

View full text Add to dashboard Cite

Background: The eHealth initiative of the Conference and Labs of the Evaluation Forum (CLEF) has aimed since 2012 to gather researchers working on health text analytics and to provide them with annual workshop, shared development challenges/tasks, benchmark datasets, and software for processing and evaluation. The overall purpose of this initiative is to ease and support patients, their next-ofkin, clinical staff, health scientists, and healthcare policy makers in accessing, understanding, using, and authoring health information in a multilingual setting. Objective: This original research paper reports on the outcomes of the first six installations of CLEF eHealth from 2012 to 2017. The focus is on measuring and analysing the scholarly influence by reviewing CLEF eHealth papers, together with relevant citation metrics. Methods: A review and bibliometric study of the CLEF eHealth proceedings, working notes, and author-declared paper extensions was conducted. Citation data for these publications were collected from Google Scholar. Citation content analysis was used for the publications and their citations. Results: The large number of registrations, submissions, and citations demonstrate the substantial community interest in the tasks and their resources. In total, 718 teams have registered their interest in the tasks, leading to 130 teams submitting to the 15 tasks. 184 papers using CLEF eHealth data generated 1,299 citations, yielding a total scholarly citation influence of almost 963,000 citations for the 741 co-authors and included authors from 33 countries across the world. The tasks' evaluation outcomes contribute to the knowledge of the difficulty of the research challenges the tasks address and the applicability of particular methods in solving these challenges, with typically statistically significant improvements in processing quality. Conclusions: These outcomes encourage continuing to develop these technologies to address patient needs. Consequently, data and tools have been opened for future research and development and the CLEF eHealth initiative continues to run new challenges. Keywords: Evaluation Studies as Topic; Health Records; Information Extraction; Information Storage and Retrieval; Information Visualization; Patient Education as Topic; Speech Recognition; Systematic Reviews; Test-set Generation; Text Classification IntroductionHealth information refers to all health-related content in all data formats, document types, information systems, publication media, and languages from all organisations, states, and countries. The privacy-sensitive, official part of health information consists of data recorded in healthcare services when describing a given patient's health or healthcare (Figure 1). The accessibility of this data is defined as limited (i.e., private or confidential information) and it is recorded either on paper in health records or electronically in Electronic Health (eHealth) records. Some common synonyms or related terms include eHealth charts, data, documents, information, ...

show abstract

Section: Evaluation Methods From 2013 To 2017mentioning

confidence: 99%

Section: Citation Analysis From 2012 To 2017mentioning

confidence: 99%

Section: Citation Analysis From 2012 To 2017mentioning

confidence: 99%

Section: Citation Analysis From 2012 To 2017mentioning

confidence: 99%

Section: Software Releases and Submission From 2013 To 2017mentioning

confidence: 99%

See 3 more Smart Citations

Scholarly Influence of the Conference and Labs of the Evaluation Forum eHealth Initiative: Review and Bibliometric Study of the 2012 to 2017 Outcomes (Preprint)

Suominen¹,

Kelly²,

Goeuriot³

2018

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

CLEF eHealth 2019 Evaluation Lab

Kelly

Goeuriot

Suominen

et al. 2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Since 2012 CLEF eHealth has focused on evaluation resource building efforts around the easing and support of patients, their next-ofkins, clinical staff, and health scientists in understanding, accessing, and authoring eHealth information in a multilingual setting. This year's lab offers three tasks: Task 1 on multilingual information extraction; Task 2 on technology assisted reviews in empirical medicine; and Task 3 on consumer health search in mono-and multilingual settings. Herein, we describe the CLEF eHealth evaluation series to-date and then present the 2019 tasks, evaluation methodology, and resources.

show abstract