The bag-of-repeats representation of documents

Abstract

n-gram representations of documents may improve over a simple bag-of-word representation by relaxing the independence assumption of word and introducing context. However, this comes at a cost of adding features which are nondescriptive, and increasing the dimension of the vector space model exponentially.We present new representations that avoid both pitfalls. They are based on sound theoretical notions of stringology, and can be computed in optimal asymptotic time with algorithms using data structures from the suffix family. While maximal repeats have been used in the past for similar tasks, we show how another equivalence class of repeats -largestmaximal repeats -obtain similar or better results, with only a fraction of the features. This class acts as a minimal generative basis of all repeated substrings. We also report their use for topic modeling, showing easier to interpret models.

Disease Inference and Community-based Health Services

“…As a repercussion, we separate inquiries of this kind that require illness derivation from different sorts. It merits underscoring that large-scale data frequently prompts blast of highlight space in the lights of ngram representation [5] [6] particularly for the group created conflicting data. To keep away from this issue, we use the medicinal phrasings to speak to our information.…”