Towards Enriching Responses with Crowd-sourced Knowledge for Task-oriented Dialogue

He, Yingxu; Liao, Lizi; Zhang, Zheng; Chua, Tat-Seng

doi:10.1145/3475959.3485392

Cited by 4 publications

(3 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The obvious dimensions are for example the quality of recommendations using accuracy measures, like Recall, Hit Rate, Precision, etc, [7,18,36]. Similarly, linguistic aspects such as fluency, diversity, etc., as a proxy to an overall conversational quality of a system were evaluated using offline metrics like Perplexity, N-Gram, BLEU scores, see e.g., [12,19,2,7].…”

Section: Crs Evaluationmentioning

confidence: 99%

“…Common quality aspects, e.g., to evaluate the meaningfulness or consistency of system-responses are investigated in recent work on CRS, see, e.g., , [11,24,33]. Therefore, current works on CRS usually rely on a combined approach in their evaluations using both offline metrics and case studies with humans, see e.g., [7,11,12].…”

Section: Crs Evaluationmentioning

confidence: 99%

“…The elicited preferences and domain knowledge are then leveraged by the recommender algorithm in order to compute and provide items of interest to the users. Moreover, domain specific concepts and entities also contribute in producing meaningful and coherent responses, especially in knowledge-aware CRS, see also [9,10,11,12].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

INSPIRED2: An Improved Dataset for Sociable Conversational Recommendation

Manzoor¹,

Jannach²

2022

Preprint

View full text Add to dashboard Cite

Conversational recommender systems (CRS) that interact with users in natural language utilize recommendation dialogs collected with the help of paired humans, where one plays the role of a seeker and the other as a recommender. These recommendation dialogs include items and entities to disclose seekers' preferences in natural language. However, in order to precisely model the seekers' preferences and respond consistently, mainly CRS rely on explicitly annotated items and entities that appear in the dialog, and usually leverage the domain knowledge. In this work, we investigate INSPIRED, a dataset consisting of recommendation dialogs for the sociable conversational recommendation, where items and entities were explicitly annotated using automatic keyword or pattern matching techniques. To this end, we found a large number of cases where items and entities were either wrongly annotated or missing annotations at all. The question however remains to what extent automatic techniques for annotations are effective. Moreover, it is unclear what is the relative impact of poor and improved annotations on the overall effectiveness of a CRS in terms of the consistency and quality of responses. In this regard, first, we manually fixed the annotations and removed the noise in the INSPIRED dataset. Second, we evaluate the performance of several benchmark CRS using both versions of the dataset. Our analyses suggest that with the improved version of the dataset, i.e., INSPIRED2, various benchmark CRS outperformed and that dialogs are rich in knowledge concepts compared to when the original version is used. We release our improved dataset (INSPIRED2) publicly at https://github.com/ahtsham58/INSPIRED2.

show abstract