INTRODUCTION The MUC-4 evaluation metrics measure the performance of the message understanding systems. This paper describes the scoring algorithms used to arrive at the metrics as well as the improvements that were made to th e MUC-3 methods. MUC-4 evaluation metrics were stricter than those used in MUC-3. Given the differences in scoring between MUC-3 and MUC-4, the MUC-4 systems' scores represent a larger improvement over MUC-3 performance than the numbers themselves suggest. The major improvements in the scoring of MUC-4 were the automation of the scoring of set fill slots, partia l automation of the scoring of string fill slots, content-based mapping enforced across the board, the focus on the AL L TEMPLATES score as opposed to the MATCHED/MISSING score in MUC-3, the exclusion of the template id scores from the score tallies, and the addition of the object level scores, string fills only scores, text filtering scores , and F-measures. These improvements and their effects on the scores are discussed in detail in this paper. SCORE REPORT The MUC-4 Scoring System produces score reports in various formats. These reports show the scores for the templates and messages in the test set. Varying amounts of detail can be reported. The scores that are of the most interest are those that appear in the comprehensive summary report. Figure 1 shows a sample summary score report. The rows and columns of this report are explained below.

No abstract

No abstract

INTRODUCTIO N Purpos eThe MUC-3 evaluation metrics are measures of performance for the MUC-3 template fill task . Obtaining summary measures of performance necessitates the los s of information about many details of performance . The utility of summary measure s for comparison of performance over time and across systems should outweigh thi s loss of detail . The template fill task is complex because of the varying nature of th e fills for each slot and the interdependencies of the slots . The evaluation metrics used in MUC-3 were adapted from traditional measures in information retrieval and signa l procesing and were still evolving to fit the more complex data extraction task of MUC -3 when the evaluation was performed . The scoring of the template fill task and th e calculation of the metrics used in MUC-3 will be described here . This description i s meant to assist in the analysis of the MUC-3 results and in the further evolution of the evaluation metrics . Metric sThe measures of performance chosen for use in MUC-3 were recall, precision , fallout, and overgeneration .Recall, precision, and fallout were adapted based o n their use in information retrieval . Overgeneration was developed as a measure fo r MUC-3 . Recall is a measure of the completeness of the template fill . Precision is a measure of the accuracy of the fill . Fallout is a measure of the false alarm rate fo r the slots which can be filled from finite sets of slot fillers .Overgeneration is a measure of spurious generation . These measures will be described in greater detai l below . SCORE REPORTA semi-automated scoring system was developed for MUC-3 . The scorin g system displayed the answer key templates, the response templates, and the message s using a flexibly customized emacs interface . During scoring, the user was asked to enter the score for displayed mismatches between the key and the respons e templates . Fills could generally be scored as matches, partial matches, or mismatches . Depending on the type of slot fill, the scoring system may or may not have allowe d full credit to be given .The interactive scoring was carried out following welldefined scoring guidelines .Depending on the scoring guidelines, full, partial, or n o credit may have been allowed for each mismatch .After the interactive scoring wa s complete, the scoring system produced an official score report containing template by template score reports and a summary score report for the official record .A sample summary score report produced for human comparison against the ke y appears in Figure 1 . The following sections discuss the contents of the score report .

No abstract