Extrinsic Summarization Evaluation: A Decision Audit Task

Abstract

In this work we describe a large-scale extrinsic evaluation of automatic speech summarization technologies for meeting speech. The particular task is a decision audit, wherein a user must satisfy a complex information need, navigating several meetings in order to gain an understanding of how and why a given decision was made. We compare the usefulness of extractive and abstractive technologies in satisfying this information need, and assess the impact of automatic speech recognition (ASR) errors on user performance. We employ several evaluation methods for participant performance, including post-questionnaire data, human subjective and objective judgments, and a detailed analysis of participant browsing behavior. We find that while ASR errors affect user satisfaction on an information retrieval task, users can adapt their browsing behavior to complete the task satisfactorily. Results also indicate that users consider extractive summaries to be intuitive and useful tools for browsing multi-modal meeting data. We discuss areas in which automatic summarization techniques can be improved in comparison with gold-standard meeting abstracts.

Abstractive Summarization of Voice Communications

“…ROUGE). We agree with [30] that the evaluation of abstractive summaries must be extrinsic, i.e. based on human judgement or indirectly observed as its helpfulness in performing a given task.…”

The CALO Meeting Assistant System

“…Hence, another trend is to use the sentences selected in the summaries as starting point for browsing the meetings. This helps users recontextualize the information and improve their ability to locate information as shown by [71]. To this end, in [69], we proposed a user interface for improving the capture of a user's information need by presenting automatically extracted keyphrases that can be refined and used to generate summaries for meeting browsing.…”

Automatic analysis of multiparty meetings

“…However, ROUGE does not correlate well with human judgements for meeting summarisation (Liu and Liu 2008). In order to better evaluate our meeting summarisation systems, we carried out a subjective evaluation based on a decision audit task (Murray et al 2009). Users were asked to find the factors that led to a particular decision that resulted from a sequence of four meetings in the AMI corpus.…”