Learning Entropy: Multiscale Measure for Incremental Learning

Abstract

First, this paper recalls a recently introduced method of adaptive monitoring of dynamical systems and presents the most recent extension with a multiscale-enhanced approach. Then, it is shown that this concept of real-time data monitoring establishes a novel non-Shannon and non-probabilistic concept of novelty quantification, i.e., Entropy of Learning, or in short the Learning Entropy. This novel cognitive measure can be used for evaluation of each newly measured sample of data, or even of whole intervals. The Learning Entropy is quantified in respect to the inconsistency of data to the temporary governing law of system behavior that is incrementally learned by adaptive models such as linear or polynomial adaptive filters or neural networks. The paper presents this novel concept on the example of gradient descent learning technique with normalized learning rate.

Another Adaptive Approach to Novelty Detection in Time Series

“…This is justified by the notion that the prediction error of the model carries the information about the size of inaccuracy, while the adaptive weight increments carry information about how much the model tried to adapt on new data. Thus, even when the model error may be high, this may not necessarily be related with the adaptive weight increments (for more on this principle, please refer to the works of [20]- [22]). In case of a high error value, caused by common phenomena (what is out of the models capability to learn), novelty detection doesn't mark those particular samples, because the model has already recognized its inability to learn, which also further justifies the functionality of this method for novelty detection.…”

“…the Adaptation Plot [20] that has been recently enhanced with multi-scale approach [21]. A most recent method is the Learning Entropy, i.e., a multi-scale approach to evaluation of unusual behaviour of adaptive parameters of a learning model is introduced in [22]. This paper however introduces another, different approach to novelty detection, that is neither based on statistical approaches, nor is it based on evaluation of error residual.…”

Novelty detection based on learning entropy

“…The first theoretical formula for estimating LE was proposed as a multifractal (multiscale) measure for the slopes of power‐law quantification in a log‐log plot . Later on, cumulative sums were proposed for practical approximations of LE, ie, where is an indicator function, and is the mean absolute value of the recent history of the i th weight adaptation increment (for details, see Section 3) and α is the multiscale sensitivity parameter α ∈ { α 1 , α 2 ,…, α n }, α 1 < α 2 < ⋯ < α n .…”

“…The first theoretical formula for estimating LE was proposed as a multifractal (multiscale) measure for the slopes of power-law quantification in a log-log plot. 13 Later on, cumulative sums were proposed for practical approximations of LE, ie,…”

“…This paper introduces a bridge between Learning Entropy (LE), which was introduced a few years ago as a novel concept of a “nonprobabilistic information measure” for the learning process of incremental learning systems and a classical multivariate SPC framework for sequential online change point detection. Using multivariate SPC methods, we are able to quantify and detect “unusual learning effort”—hence the change of underlying process behavior—during the process monitoring without any consideration for the underlying process.…”