A meeting browser evaluation test

Recognition and understanding of meetings the AMI and AMIDA projects

“…In such cases, extrinsic evaluation approaches may be preferred, in which a task is evaluated in the context of a larger scenario, such as a meeting browser. In AMI we have developed a framework for extrinsic evaluation of browser components, that we call the Browser Evaluation Test (BET) [36]. The BET provides a framework for the comparison of arbitrary meeting browser setups, where setups differ in terms of which content extraction or abstraction components are employed.…”

Automatic Summarization

“…This browser is intended as one part of a large system that would allow users to explore a corpus of meetings, for example to search for the most relevant pieces across meetings to allow the user to answer a specific question. More information on AMI/AMIDA and the JFerret meeting browser can be found in [70,119,292,293].…”

Towards an Objective Test for Meeting Browsers: The BET4TQB Pilot Experiment

“…The BET is an extensive framework containing guidelines and tools that allow evaluators to construct a browser-independent evaluation task, and then to test the performances of a given browser on that task [1]. Each evaluation task is meeting-specific and consists of a set of observations of interest determined by a pool of observers who have watched closely the meeting recording, and have noted the most salient facts and events that occurred in the meeting (observers are not meeting participants).…”

“…From the very first BET experiments [1], two important parameters characterized the subjects' performance. The first one is precision (or accuracy), i.e.…”

“…At this stage, some of the observations can be rejected for a number of reasons, which are carefully explicited to ensure that they are browser-neutral, and do not select observations that are better suited to a particular kind of browsing technique. These reasons are: (1) statements that are true at one moment but false at another moment of the meeting; (2) statements that are considered incomprehensible to native English speakers because of serious grammatical or typographical errors, or unclear formulation; (3) statements that are too easily guessable; (4) true and false statements that aren't parallel enough, or are not mutually exclusive; (5) statements based on "censored" material, i.e. on segments which participants had asked to be left out of the recording.…”