Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models

Abstract

The recent success of large language models (LLMs) has shown great potential to develop more powerful conversational recommender systems (CRSs), which rely on natural language conversations to satisfy user needs. In this paper, we embark on an investigation into the utilization of ChatGPT for CRSs, revealing the inadequacy of the existing evaluation protocol. It might overemphasize the matching with ground-truth items annotated by humans while neglecting the interactive nature of CRSs.To overcome the limitation, we further propose an interactive Evaluation approach based on LLMs, named iEvaLM, which harnesses LLM-based user simulators. Our evaluation approach can simulate various system-user interaction scenarios. Through the experiments on two public CRS datasets, we demonstrate notable improvements compared to the prevailing evaluation protocol. Furthermore, we emphasize the evaluation of explainability, and Chat-GPT showcases persuasive explanation generation for its recommendations. Our study contributes to a deeper comprehension of the untapped potential of LLMs for CRSs and provides a more flexible and realistic evaluation approach for future research about LLMbased CRSs. The code is available at https: //github.com/RUCAIBox/iEvaLM-CRS.

Artificial Intelligence in the 21st Century

“…However, the study also found that GPT-4 is less proficient in tasks that require complex reasoning or specific domain knowledge, highlighting the limitations of these models [24]. Recent research has addressed various limitations of large language models, including the hand-crafting of task-specific demonstrations [25], the evaluation of code synthesis [26], the cost barrier associated with large models [27], the evaluation protocol for conversational recommendation systems [28], and the context window restriction for off-the-shelf LLMs [29].…”