Overview of the CLEF eHealth Evaluation Lab 2015

Abstract

Abstract. This paper reports on the 3rd CLEFeHealth evaluation lab, which continues our evaluation resource building activities for the medical domain. In this edition of the lab, we focus on easing patients and nurses in authoring, understanding, and accessing eHealth information. The 2015 CLEFeHealth evaluation lab was structured into two tasks, focusing on evaluating methods for information extraction (IE) and information retrieval (IR). The IE task introduced two new challenges. Task 1a focused on clinical speech recognition of nursing handover notes; Task 1b focused on clinical named entity recognition in languages other than English, specifically French. Task 2 focused on the retrieval of health information to answer queries issued by general consumers seeking information to understand their health symptoms or conditions. The number of teams registering their interest was 47 in Tasks 1 (2 teams in Task 1a and 7 teams in Task 1b) and 53 in Task 2 (12 teams) for a total of 20 unique teams. The best system recognized 4, 984 out of 6, 818 test words correctly and generated 2, 626 incorrect words (i.e., 38.5% error) in Task 1a; had the F-measure of 0.756 for plain entity recognition, 0.711 for normalized entity recognition, and 0.872 for entity normalization in Task 1b; and resulted in P@10 of 0.5394 and nDCG@10 of 0.5086 in Task 2. These results demonstrate the substantial community interest and capabilities of these systems in addressing challenges faced by patients and nurses. As in previous years, the organizers have made data and tools available for future research and development.

An analysis of evaluation campaigns in ad-hoc medical information retrieval: CLEF eHealth 2013 and 2014

“…Each year CLEF eHealth offers IR, information extraction (IE) and information management tasks to volunteer task participants which aim to evaluate systems that support laypeople in searching for and understanding health information [15,27,28,57]. In 2013 and 2014 the tasks were built around …”

Scholarly Influence of the Conference and Labs of the Evaluation Forum eHealth Initiative: Review and Bibliometric Study of the 2012 to 2017 Outcomes (Preprint)

“…CLEF eHealth particularly welcomed and attracted multi-disciplinary teams to collaborate and bridge the academy, government, and industrial researchers, scientists, lecturers, and graduate students with engineers, practitioners, and policy makers [31][32][33][34][35]. For example, in the 33 WNs and 1 CP from the CLEF eHealth 2013 evaluation lab, 162 authors from 10 countries highlighted some leading organizations in health information management, extraction, and retrieval, including,…”

“…The topic of patient-friendly multilingual communication formed the focus of the annual CLEF eHealth evaluation labs in 2013-2017 [31][32][33][34][35], generated the total scholarly influence of 962,559 citations for the 184 CLEF eHealth papers, and reached authors from 33 countries across the world (Table 1, Figure 3) [8]. This influence was computed by multiplying the number of co-authors in the 184 papers (i.e., 741) by the number of citations (i.e., 1,299) these papers had received on 26 October 2017.…”

“…In 47 out of the 184 papers (25.5%), this co-authoring collaboration was international and sometimes even across continents (i.e., Africa -Europe, Asia -Australia, Asia -Europe, AsiaNorth America, Australia -Europe, Australia -Europe -North America, and Europe -South America). In 2015 [33] and 2016 [34] CLEF eHealth expanded its scope from EN to multilingual text processing, medical web search, and speech-to-text conversion to ease both patients (and their next-of-kin) and clinicians' understanding of various types of medical content. Again, three tasks per year were organised.…”

“…The reader is referred to these papers [31], [32], [33], [34], and [35], and references therein, for further information about the software and code releases in 2013, 2014, 2015, 2016, and 2017, respectively.…”