Semantic coherence scoring using an ontology

Abstract

In this paper we present ONTOSCORE, a system for scoring sets of concepts on the basis of an ontology. We apply our system to the task of scoring alternative speech recognition hypotheses (SRH) in terms of their semantic coherence. We conducted an annotation experiment and showed that human annotators can reliably differentiate between semantically coherent and incoherent speech recognition hypotheses. An evaluation of our system against the annotated data shows that, it successfully classifies 73.2% in a German corpus of 2.284 SRHs as either coherent or incoherent (given a baseline of 54.55%).

Towards a method for evaluating naturalness in conversational dialog systems

“…Although at the present time spoken interaction may not be as efficient at accomplishing tasks as written interaction, Le Bigot et al [15] suggest that such interaction promotes collaboration rather than placing emphasis on the task and its performance without regards to the dialogue quality. It can be inferred from their findings that this may be a result of both the lower informational density of speech and the elimination of essential terms for grounding 'shared knowledge' that occurs in human-computer speech interactions.…”

“…Gurevych demonstrates, through an evaluation of the semantic coherence of ontology based speech recognition systems in [14], that a gap exists in recognition that is effectively and semantically coherent, partially due to the arbitrary nature of human speech and understanding of context [15].…”

“…An obvious impediment to obtaining contextually relevant data arises from imperfect transcripts from speech recognition. Semantic checks on the retrieved audio will hinder the system's interpretation of facts and its ability to validate context as seen in [14] and [15]. A simple experiment on this would show the input of an errorneously transcribed script into a chatbot and the effect on the bot's coherence.…”

“…Such a principle falls within the metadata frameworks advocated by [18]. As can be seen from the results of Gurevych [15], the 'gold standard' for this would be the consensus of human judges with the system's interpretation of the domain.…”

A Graph-Theoretic Framework for Semantic Distance

“…We hypothesize that, because ontological relations play an integral role in our semantic distance measure, the measure is less effective when the semantic profile for a text (the set of corresponding concepts) lacks semantic coherence. Other work has explored ways to measure the semantic coherence of a set of concepts in terms of their connectedness within an ontology (Gurevych et al 2003). Because a semantic profile in our work includes both ontological (relational) and distributional (frequency) knowledge, we require a measure of semantic coherence that takes both into account.…”