On the learning mechanism of adaptive filters

Abstract

Abstract-This paper highlights, both analytically and by simulations, some interesting phenomena regarding the behavior of ensemble-average learning curves of adaptive filters that may have gone unnoticed. Among other results, the paper shows that even ensemble-average learning curves of single-tap LMS filters actually exhibit two distinct rates of convergence: one for the initial time instants and another, faster one, for later time instants. In addition, such curves tend to converge faster than predicted by mean-square theory and can converge even when a mean-square stability analysis predicts divergence. These effects tend to be magnified by increasing the step size. Two of the conclusions that follow from this work are 1) mean-square stability alone may not be the most appropriate performance measure, especially for larger step sizes. A combination of mean-square stability and almost sure (a.s.) stability seems to be more appropriate. 2) Care is needed while interpreting ensemble-average curves for larger step sizes. The curves can lead to erroneous conclusions unless a large number of experiments are averaged (at times of the order of tens of thousands or higher).Index Terms-Adaptive filter, almost-sure convergence, Chebyshev's inequality, law of large numbers, learning curve, mean square convergence, rate of convergence.

Energy Conservation and the Learning Ability of LMS Adaptive Filters

“…Assume further that the entries {u(n)} are iid and uniform in the interval [−0.5, 0.5] so that . It is seen that in both cases, the averaged curves converge, as opposed to the theoretical curve, which is divergent for this value of the step-size (see [1]). Observe in addition that the larger the value of L, the longer the averaged curve stays closer to the theoretical curve before ultimately converging away from it.…”

“…However, these distinctions disappear for infinitesimally small step-sizes, which explains why the phenomena described before can pass unnoticed at this level of adaptation. This is a consequence of the fact that under some reasonable assumptions about the probability density function of the random variable {u}, it holds that (see, e.g., [1])…”

“…A second result pertains to the learning abilities of adaptive filters, especially for larger step-sizes, where several interesting phenomena arise that seem to indicate that the learning behavior of adaptive filters is more sophisticated than was originally thought. More details can be found in [1], such as extensions of the arguments to the vector case.…”

“…In this way, one can conclude that even under these controlled conditions, the differences still exist. Actually, the differences occur even for situations where the independence assumptions are not satisfied (see [1]). …”

“…For this reason, the discussion focuses in some detail on a special case that helps illustrate and explain the desired phenomena in their simplest forms. Readers interested in more advanced cases, and in additional details, are referred to the article [1] and the textbook [20].…”