Abstract. RAVE (Real-time Answer Validation Engine) is a logic-based answer validator/selector designed for real-time question answering. Instead of proving a hypothesis for each answer, RAVE uses logic only for checking if a considered passage supports a correct answer at all. In this way parsing of the answers is avoided, yielding low validation/selection times. Machine learning is used for assigning local validation scores based on logical and shallow features. The subsequent aggregation of these local scores strives to be robust to duplicated information in the support passages. To achieve this, the effect of aggregation is controlled by the lexical diversity of the support passages for a given answer.
Description of the Validation TaskThe Answer Validation Exercise (AVE) [1] introduces a test set of validation items i ∈ I comprising the question q i , answer candidate a i and supporting snippet s i . Let Q = {q i : i ∈ I} be the set of all questions and I q = {i ∈ I : q i = q} the set of validation items for a question q ∈ Q. The validator must assign a validation decision v i ∈ {REJECTED, SELECTED, VALIDATED} and confidence score c i ∈ [0, 1] to each i ∈ I. At most one answer per question can be selected. Answers can only be validated if an answer was selected as best answer.
The RAVE ValidatorThe input to the validator comprises a question together with answer candidates for the question and the supporting text snippets, as represented by I q . Let A q = {a i : q i = q} denote the set of answer candidates for q in the test set. The AVE 2008 test set is redundancy-free, i.e. for each a ∈ A q , there is only one item i ∈ I q supporting a. As the basis for aggregation, the IRSAW system [2] was used to actively search for additional supporting snippets for each of the answer candidates. Answers were clustered into groups of minor variants with the same 'answer key' κ(a i ) by applying a simplification function κ (which drops accents etc.) For example, κ(in the year 2001) = 2001. The result is a set of auxiliary items i ∈ I q with q i = q and κ(a i ) = κ(a j ) for some original answer a j ∈ A q , and supporting snippet s i for a i found by IRSAW. The original and auxiliary validation items are joined into the enhanced validation pool I * q = I q ∪ I q .