Evaluating and enhancing cross-domain rank predictability of textual entailment datasets

Lee, Cheng-Wei; Lin, Chuan-Jie; Shima, Hideki; Hsu, Wen-Lian

doi:10.1109/iri.2012.6302990

Cited by 2 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the previous RITE task, RITE4QA was measured by accuracy. Lee et al [12] compared the ranking of systems working on RITE4QA dataset to the other datasets and found that, with careful control, adding a certain portion of "artificial pairs" might increase the cross-domain rank predictability of a dataset. We did the same observation this time.…”

Section: Rank Correlation Regarding Artificial Pairsmentioning

confidence: 99%

See 1 more Smart Citation

Rank correlation analysis of NTCIR-10 RITE-2 Chinese datasets and evaluation metrics

Lin

Lee

Shih

et al. 2013

2013 IEEE 14th International Conference on Information Reuse &Amp; Integration (IRI)

Self Cite

View full text Add to dashboard Cite

Textual Entailment (TE) is the task of recognizing entailment, paraphrase, and contradiction relations between a given text pair. The goal of textual entailment research is to develop a core inference component that can be applied to various domains such as QA.We observed several rank correlations on the data and system results in the NTCIR-10 RITE-2 task, trying to find out correlations between datasets and evaluation metrics. We also constructed RITE4QA datasets in the RITE-2 task under the scenario of QA in order to see the applicability of RITE systems in QA.We find that datasets created from different sources and different ways can hardly predict each other. However, the system ranking on the dataset consisting of expert-made artificial pairs has moderate correlation with the ranking on QA metrics. Both RITE metrics and QA metrics are stable in terms of their own subtasks.

show abstract

Section: Rank Correlation Regarding Artificial Pairsmentioning

confidence: 99%

“…Lee et al [12] proposed using "artificial pairs" to estimate RITE system rankings. These "artificial pairs" are variations of some existing pairs created manually by human.…”

Section: Introductionmentioning

confidence: 99%