Empirical justification of the gain and discount function for nDCG

Kanoulas, Evangelos; Aslam, Javed A.

doi:10.1145/1645953.1646032

Cited by 45 publications

(45 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The performance of the candidate symptom list ranking is evaluated using nDCG (normalized discounted cumulative gain) as formalized in Eq. (4) 30 , where the unique candidate symptoms are weighted by their partial overlap likelihood with the gold-standard symptoms (e.g., if “depression” overlaps a symptom annotation 34 of 46 times in the corpus, its “gold” score is 0.739). nDCG is a widely employed evaluation criterion in Information Retrieval (IR) 30 .…”

Section: Methodsmentioning

confidence: 99%

Psychiatric symptom recognition without labeled data using distributional representations of phrases and on-line knowledge

Zhang

et al. 2017

Journal of Biomedical Informatics

View full text Add to dashboard Cite

Objective Mental health is becoming an increasingly important topic in healthcare. Psychiatric symptoms, which consist of subjective descriptions of the patient’s experience, as well as the nature and severity of mental disorders, are critical to support the phenotypic classification for personalized prevention, diagnosis, and intervention of mental disorders. However, few automated approaches have been proposed to extract psychiatric symptoms from clinical text, mainly due to (a) the lack of annotated corpora, which are time-consuming and costly to build, and (b) the inherent linguistic difficulties that symptoms present as they are not well-defined clinical concepts like diseases. The goal of this study is to investigate techniques for recognizing psychiatric symptoms in clinical text without labeled data. Instead, external knowledge in the form of publicly available “seed” lists of symptoms is leveraged using unsupervised distributional representations. Materials and Methods First, psychiatric symptoms are collected from three online repositories of healthcare knowledge for consumers—MedlinePlus, Mayo Clinic, and the American Psychiatric Association—for use as seed terms. Candidate symptoms in psychiatric notes are automatically extracted using phrasal syntax patterns. In particular, the 2016 CEGS N-GRID challenge data serves as the psychiatric note corpus. Second, three corpora—psychiatric notes, psychiatric forum data, and MIMIC II—are adopted to generate distributional representations with . Finally, semantic similarity between the distributional representations of the seed symptoms and candidate symptoms is calculated to assess the relevance of a phrase. Experiments were performed on a set of psychiatric notes from the CEGS N-GRID 2016 Challenge. Results & Conclusion Our method demonstrates good performance at extracting symptoms from an unseen corpus, including symptoms with no word overlap with the provided seed terms. Semantic similarity based on the distributional representation outperformed baseline methods. Our experiment yielded two interesting results. First, distributional representations built from social media data outperformed those built from clinical data. And second, the distributional representation model built from sentences resulted in better representations of phrases than the model built from phrase alone.

show abstract

Section: Methodsmentioning

confidence: 99%

Psychiatric symptom recognition without labeled data using distributional representations of phrases and on-line knowledge

Zhang

et al. 2017

Journal of Biomedical Informatics

View full text Add to dashboard Cite

show abstract

“…NDCG, which is also adopted in TREC and INEX, has seen increasing use in evaluations based on non-binary scales of relevance. Despite being a well-established and reliable evaluation measure, using NDCG requires deciding on the values of the gain vector and the discount function as well as producing an ideal ranked list of the results (Voorhees, 2001;Zhou et al, 2012;Kanoulas and Aslam, 2009). These details are unclear for both initiatives 25 ; our best guess for the discount function is that both used the default one: log of the document's rank (see Section 3.5.2).…”

Section: Methodsmentioning

confidence: 99%

An overview of semantic search evaluation initiatives

Elbedweihy

Wrigley

Clough

et al. 2015

Journal of Web Semantics

View full text Add to dashboard Cite

“…A popular gain function is the exponential (2 i k − 1), which promotes the highest relevance levels, and a typical discount function is the the logarithmic discount 1/ log 2 (k + 1) or the zipfian discount 1/k. Further information can be found in [14]. These functions are wellchosen heuristics, and we propose to use the relevance weight P…”

Section: Graded Relevance Metrics and The Udmmentioning

confidence: 99%

Exploiting user disagreement for web search evaluation

Demeester

Aly

Hiemstra

et al. 2014

Proceedings of the 7th ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

To express a more nuanced notion of relevance as compared to binary judgments, graded relevance levels can be used for the evaluation of search results. Especially in Web search, users strongly prefer top results over less relevant results, and yet they often disagree on which are the top results for a given information need. Whereas previous works have generally considered disagreement as a negative effect, this paper proposes a method to exploit this user disagreement by integrating it into the evaluation procedure.First, we present experiments that investigate the user disagreement. We argue that, with a high disagreement, lower relevance levels might need to be promoted more than in the case where there is global consensus on the top results. This is formalized by introducing the User Disagreement Model, resulting in a weighting of the relevance levels with a probabilistic interpretation. A validity analysis is given, and we explain how to integrate the model with well-established evaluation metrics. Finally, we discuss a specific application of the model, in the estimation of suitable weights for the combined relevance of Web search snippets and pages.

show abstract

Empirical justification of the gain and discount function for nDCG

Cited by 45 publications

References 13 publications

Psychiatric symptom recognition without labeled data using distributional representations of phrases and on-line knowledge

Psychiatric symptom recognition without labeled data using distributional representations of phrases and on-line knowledge

An overview of semantic search evaluation initiatives

Exploiting user disagreement for web search evaluation

Contact Info

Product

Resources

About