Computing Discriminating and Generic Words

Abstract

Abstract. We study the following three problems of computing generic or discriminating words for a given collection of documents. Given a pattern P and a threshold d, we want to report (i) all longest extensions of P which occur in at least d documents, (ii) all shortest extensions of P which occur in less than d documents, and (iii) all shortest extensions of P which occur only in d selected documents. For these problems, we propose efficient algorithms based on suffix trees and using advanced data structure techniques. For problem (i), we propose an optimal solution with constant running time per output word.

Succinct indexes for reporting discriminating and generic words

“…In [8] the authors showed how the problem of identifying minimal discriminating words can be reduced to orthogonal segment intersection problem. In this article, we rely on this key insight for both problems under consideration and use the result summarized in the lemma below for segment intersection.…”

“…In this section, we first review a linear space index, which is based on the ideas from the previous results [8]. Later we show how to employ sampling techniques to achieve a space efficient solution.…”

“…strings which have P as a prefix. These problems were introduced by Kucherov et al [8], and they proposed indexes of size O (n log n) bits. The query processing time is optimal O (p + output) for reporting all maximal generic words, and is near-optimal O (p + log log n + output) for reporting all minimal discriminating words, where output is the size of the output.…”

“…For finding maximal generic words corresponding to the marked nodes, we store data structure similar to [8] described above only for the marked nodes, which takes space linear to the number of marked nodes. 2.…”

“…Discriminating and generic words also find applications in text mining and automated text classification. For further details, we refer to [8].…”