Using long runs as predictors of semantic coherence in a partial document retrieval system

Abstract

We propose a method for dealing with semantic complexities occurring in information retrieval systems on the basis of linguistic observations. Our method follows from an analysis indicating that long runs of content words appear in a stopped document cluster, and our observation that these long runs predominately originate from the prepositional phrase and subject complement positions and as such, may be useful predictors of semantic coherence. From this linguistic basis, we test three statistical hypotheses over a small collection of documents from different genre. By coordinating thesaurus semantic categories (SEMCATs) of the long run words to the semantic categories of paragraphs, we conclude that for paragraphs containing both long runs and short runs, the SEMCAT weight of long runs of content words is a strong predictor of the semantic coherence of the paragraph.

Entropy and Graph Based Modelling of Document Coherence using Discourse Entities

“…There is extensive work on local coherence that uses different approaches, including bag of words methods at sentence level [22], sequences of content words (of length ≥ 3) at paragraph level [53], local lexical cohesion information [2], local syntactic cues [17], and combining local lexical and syntactic features, e.g., term co-occurrence [38,55]. Overall, various aspects of CT have long been used to model local coherence [37,51], including the well-known entity approaches that rank the repetition and syntactic realisation of entities in adjacent sentences [3,17].…”