Overview of Touché 2023: Argument and Causal Retrieval

Bondarenko, Alexander; Fröbe, Maik; Kiesel, Johannes; Schlatt, Ferdinand; Barriere, Valentin; Ravenet, Brian; Hemamou, Léo; Luck, Simon; Reimer, Jan Heinrich; Stein, Benno; Potthast, Martin; Hagen, Matthias

doi:10.1007/978-3-031-42448-9_31

Cited by 4 publications

(2 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The documents were pooled using the top-5 pooling strategy from 12 ranked results submitted by participants. While the argument retrieval track in Touché has more recent versions [7,8,10], in our work, we use the Touché 2020 dataset, due to its availability in the BEIR benchmark [61]. 2 Retrieval Models.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Systematic Evaluation of Neural Retrieval Models on the Touché 2020 Argument Retrieval Subset of BEIR

Thakur,

Bonifacio,

Fröbe

et al. 2024

Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

The zero-shot effectiveness of neural retrieval models is often evaluated on the BEIR benchmark-a combination of different IR evaluation datasets. Interestingly, previous studies found that particularly on the BEIR subset Touché 2020, an argument retrieval task, neural retrieval models are considerably less effective than BM25. Still, so far, no further investigation has been conducted on what makes argument retrieval so "special". To more deeply analyze the respective potential limits of neural retrieval models, we run a reproducibility study on the Touché 2020 data. In our study, we focus on two experiments: (i) a black-box evaluation (i.e., no model retraining), incorporating a theoretical exploration using retrieval axioms, and (ii) a data denoising evaluation involving post-hoc relevance judgments. Our black-box evaluation reveals an inherent bias of neural models towards retrieving short passages from the Touché 2020 data, and we also find that quite a few of the neural models' results are unjudged in the Touché 2020 data. As many of the short Touché passages are not argumentative and thus non-relevant per se, and as the missing judgments complicate fair comparison, we denoise the Touché 2020 data by excluding very short passages (less than 20 words) and by augmenting the unjudged data with post-hoc judgments following the Touché guidelines. On the denoised data, the effectiveness of the neural models improves by up to 0.52 in nDCG@10, but BM25 is still more effective. Our code and the augmented Touché 2020 dataset are available at https://github.com/castorini/touche-error-analysis. CCS CONCEPTS• Information systems → Retrieval models and ranking; Evaluation of retrieval results.

show abstract

Section: Methodsmentioning

confidence: 99%

“…6 Finally, for CITADEL+ (with distillation and hard negative mining), we use the original dpr-scale repository for reproduction. 7 Apart from DRAGON+, in our work, we successfully reproduce the nDCG@10 on Touché 2020. 8…”

Section: Methodsmentioning

confidence: 99%