Revisiting the effect of history on learning performance: the problem of the demanding lord

Abstract

Abstract. In a variety of settings ranging from recommendation systems to information filtering, approaches which take into account feedback have been introduced to improve services and user experience. However, as also indicated in the machine learning literature, there exist several settings where the requirements and target concept of a learning system changes over time, which consists a case of "concept drift". In several systems a sliding window over the training instances has been used to follow drifting concepts. However, no general analytic study has been performed on the relation between the size of the sliding window and the average performance of a learning system, since previous works have focused on instantaneous performance and specific underlying learners and data characteristics.This work proposes an analytic model that describes the effect of memory window size on the prediction performance of a learning system that is based on iterative feedback. The analysis considers target concepts changing over time, either periodically or randomly, using a formulation termed "the problem of the demanding lord". Using a signal-to-noise approach to sketch learning ability of underlying machine learning algorithms, we estimate the average performance of a learning system regardless of its underlying algorithm, and, as a corollary, propose a stepping stone towards finding the memory window that maximizes the average performance for a given drift setting and a specific modeling system. We experimentally support the proposed methodology with very promising results on three synthetic and four real datasets, using a variety of learning algorithms including Support Vector Machines, Naive Bayes, Nearest Neighbor and Decision Trees on classification and regression tasks. The results validate the analysis and indicate very good estimation performance in different settings.

Review of the Classification of Massive Chinese Texts Based on Spark

“…It has many advantages in solving small sample, nonlinear and high-dimensional pattern recognition problems, and has been used in pattern recognition, regression estimation, probability density function estimation and so on. In the field of text classification, support vector machine classifier has better classification performance and generalization ability, and is widely used in classification field [16][17][18][19][20][21] . The Liblinear classifier based on the theoretical design of quadratic soft-interval support vector machine designed by Lin Zhiren et.…”

dbTrento

“…Building on this work, we subsequently proposed an analytic model that describes the effect of the memory window size on the average prediction performance of a learning system, regardless of its underlying algorithm [22,23]. We have additionally identified simple criteria, some of which are tied to specific data characteristics, that can be used by our framework in order to compare the behavior of learning algorithms in the presence of varying levels of noise [34].…”