Negative Selection of Written Language Using Character Multiset Statistics

Abstract

We study the combination of symbol frequence analysis and negative selection for anomaly detection of discrete sequences where conventional negative selection algorithms are not practical due to data sparsity. Theoretical analysis on ergodic Markov chains is used to outline the properties of the presented anomaly detection algorithm and to predict the probability of successful detection. Simulations are used to evaluate the detection sensitivity and the resolution of the analysis on both generated artificial data and real-world language data including the English Wikipedia. Simulation results on large reference corpora are used to study the effects of the assumptions made in the theoretical model in comparison to real-world data.

The People’s Encyclopedia Under the Gaze of the Sages: A Systematic Review of Scholarly Research on Wikipedia

“…Their article reported the results of experiments done by different participants. Pöllä and Honkela (2010) applied English Wikipedia in their study on "the combination of symbol frequency analysis and negative selection algorithm for anomaly detection of discrete sequences" (2010, p.1256), concluding that "the baseline result of the Wikipedia edit detection experiment is promising" (2010, p.1265).…”

Excavating the mother lode of human-generated text: A systematic review of research that uses the wikipedia corpus

“…To overcome the limitations of negative selection-based anomaly detection techniques in sparse data cases, Pöllä and Honkela [104] proposed a combination of symbol frequency analysis and negative selection. Wikipedia was employed as a real-world data to evaluate 8 the sensitivity of the proposed anomaly detection algorithm.…”