A Test Collection for Research on Depression and Language Use

Losada, David E.; Crestani, Fábio

doi:10.1007/978-3-319-44564-9_3

Cited by 146 publications

(170 citation statements)

References 10 publications

Supporting

Mentioning

133

Contrasting

Unclassified

Order By: Relevance

“…Furthermore, different document representations and predictive models should be tested for CPI and DMC. Finally, we should test this framework in a more recent and competitive early classification corpus like the one presented by Losada and Crestiani [5] and also on other data sets where ETC approaches can be critical like the detection of sexual predators in chats or detection of suicidal discourse.…”

Section: Discussionmentioning

confidence: 99%

“…Recently, in [5] the CPI and DMC aspects are both addressed by learning the CPI component and using a simple heuristic rule for DMC that consists in classifying a text as positive when exceeding a specific confidence threshold in the prediction of the classifier. The problem with that DMC approach is that is very dependent on the problem and put all the burden of selecting the appropriate thresholds on the ETC system's implementer.…”

Section: Related Workmentioning

confidence: 99%

“…Among these is the evaluation of the model, since there is no measure to evaluate the temporary performance of it in a multi-class context. There exists, nonetheless, an evaluation metric for binary early classification [5] that considers the accuracy of the prediction and the delay taken by the system to make the decision. Here the delay is measured by counting the number of terms seen before giving the answer.…”

Section: Evaluation Metricmentioning

confidence: 99%

“…To date, only a few papers have approached this kind of scenarios [2,3,5]. Despite its low popularity, this topic has a major potential in practical applications.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Learning When to Classify for Early Text Classification

Loyola

Errecalde

Gómez

2018

Communications in Computer and Information Science

View full text Add to dashboard Cite

Abstract. The problem of classification in supervised learning is a widely studied one. Nonetheless, there are scenarios that received little attention despite its applicability. One of such scenarios is early text classification, where one needs to know the category of a document as soon as possible. The importance of this variant of the classification problem is evident in tasks like sexual predator detection, where one wants to identify an offender as early as possible. This paper presents a framework for early text classification which highlights the two main pieces involved in this problem: classification with partial information and deciding the moment of classification. In this context, a novel approach that learns the second component (when classify) and an adaptation of a temporal measurement for multi-class problems are introduced. Results with a classical text classification corpus in comparison against a model that reads the entire documents confirm the feasibility of our approach.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Evaluation Metricmentioning

confidence: 99%

“…To date, only a few papers have approached this kind of scenarios [2,3,5]. Despite its low popularity, this topic has a major potential in practical applications.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Learning When to Classify for Early Text Classification

Loyola

Errecalde

Gómez

2018

Communications in Computer and Information Science

View full text Add to dashboard Cite

show abstract

“…Shared test collections are pervasive in well-known evaluation campaigns, such as TREC (Voorhees & Harman, 2005), or NTCIR (Kando, Sakai, & Sanderson, 2016). Furthermore, research teams sometimes need to build their own testbeds, for instance, to evaluate retrieval algorithms in specific domains (Balog & Neumayer, 2013;Losada & Crestani, 2016). However, creating an IR test collection is expensive and time-consuming.…”

Section: Introductionmentioning

confidence: 99%

When to stop making relevance judgments? A study of stopping methods for building information retrieval test collections

Losada

Parapar

Barreiro

2018

Asso for Info Science & Tech

Self Cite

View full text Add to dashboard Cite

In information retrieval evaluation, pooling is a well‐known technique to extract a sample of documents to be assessed for relevance. Given the pooled documents, a number of studies have proposed different prioritization methods to adjudicate documents for judgment. These methods follow different strategies to reduce the assessment effort. However, there is no clear guidance on how many relevance judgments are required for creating a reliable test collection. In this article we investigate and further develop methods to determine when to stop making relevance judgments. We propose a highly diversified set of stopping methods and provide a comprehensive analysis of the usefulness of the resulting test collections. Some of the stopping methods introduced here combine innovative estimates of recall with time series models used in Financial Trading. Experimental results on several representative collections show that some stopping methods can reduce up to 95% of the assessment effort and still produce a robust test collection. We demonstrate that the reduced set of judgments can be reliably employed to compare search systems using disparate effectiveness metrics such as Average Precision, NDCG, P@100, and Rank Biased Precision. With all these measures, the correlations found between full pool rankings and reduced pool rankings is very high.

show abstract

References

2019

Automatic Detection of Irony

View full text Add to dashboard Cite

A Test Collection for Research on Depression and Language Use

Cited by 146 publications

References 10 publications

Learning When to Classify for Early Text Classification

Learning When to Classify for Early Text Classification

When to stop making relevance judgments? A study of stopping methods for building information retrieval test collections

References

Contact Info

Product

Resources

About