Poisson mixtures

Abstract

Shannon (1948) showed that a wide range of practical problems can be reduced to the problem of estimating probability distributions of words and ngrams in text. It has become standard practice in text compression, speech recognition, information retrieval and many other applications of Shannon's theory to introduce a “bag-of-words” assumption. But obviously, word rates vary from genre to genre, author to author, topic to topic, document to document, section to section, and paragraph to paragraph. The proposed Poisson mixture captures much of this heterogeneous structure by allowing the Poisson parameter θ to vary over documents subject to a density function φ. φ is intended to capture dependencies on hidden variables such genre, author, topic, etc. (The Negative Binomial is a well-known special case where φ is a Г distribution.) Poisson mixtures fit the data better than standard Poissons, producing more accurate estimates of the variance over documents (σ2), entropy (H), inverse document frequency (IDF), and adaptation (Pr(x ≥ 2/x ≥ 1)).

A Logical Inference Approach to Query Expansion with Social Tags

“…4 is one suggestion for estimating tag salience, which we evaluate experimentally in Section 3. Further alternative estimations are possible, for instance by relatively straight-forward extensions to IDF, such as RIDF [2], or by more elaborate approximations of tag topicality, such as Zhou et al's approach [6] that uses Bayesian Inference.…”

Language Technology for eLearning

“…A statistics which is frequently used to model the distribution of words in texts is Poisson or, alternatively, a mixture of Poissons (cf. [2]). While the distribution of function words is close to the expected distribution under these models, good keyword candidates deviate significantly from it.…”

Improving the Search for Learning Objects with Keywords and Ontologies

“…Documents which are assigned keywords of a wide variety might also be searched by such a wide variety of search terms. 4 Ontological support of searching Current eLearning systems oer only full-text or keyword-based search facilities. We will outline in this section the steps we took in order to implement an ontology search facility and also describe planned extensions for crosslingual search.…”