Analysis of error-gradient adaptive linear estimators for a class of stationary dependent processes

On the learning mechanism of adaptive filters

“…We will refer to the resulting rate of convergence of , viz., , as the MS rate of convergence. For ease of comparison with a later condition [see (12)], we will rewrite the requirement in the equivalent form (in terms of the natural logarithm) (8) Observe further that the logarithm of the MS rate of convergence is equal to [which is a result that we will also invoke later-in the discussion following (12)]. We summarize the above conclusions in the following statement.…”

“…Now, since does not depend on the time , the first exponential in (11) dominates the second when is large, which implies that the upper bound tends to zero if, and only if, . We thus conclude that a typical curve converges to zero a.s. (or with probability one) if, and only if, the step size is such that (12) This leads to a different condition on than the one derived for mean-square stability in (8). In addition, note that for large , when is already close to or inside , the rate of convergence of a typical curve is dictated primarily by the term .…”

“…iii) The noise sequence is independent of the input sequence. Despite the fact that these assumptions are seldom satisfied in practice, it is known that the learning curves that are obtained using the independence theory are good approximations for the true learning curves when the step size is vanishingly small (see, e.g., [7], [8], and [15]). However, what about the case of larger step sizes?…”

“…We will normally refer to this plot also as "the learning curve," or as the MSD learning curve, if we need to distinguish between the plots for the MSE and for the MSD. The MSD EALC is For vanishingly small step sizes, it is known [7], [8], [15] that an average of a few tens of experiments is enough to obtain experimental learning curves that are good approximations for the actual learning curve (we will also give an analytical justification for this fact later in Theorem 2). It is thus common in the literature to use the average of a few independent repeated experiments to predict or confirm theoretical results by means of simulations (a few relatively recent examples include [3]and [25], which use 10-20 independent experiments, and [24], [27], and [28], which use 100 independent experiments).…”

On the Learning Behavior of Adaptive Networks—Part II: Performance Analysis

“…Condition (4) means that the matrices {A 0 , A 1 , A 2 } are left-stochastic (i.e., the entries on each of their columns add up to one). We also explained in Part I [2] that different choices for A 1 , A 0 and A 2 correspond to different distributed strategies, such as the such as the traditional consensus [7], [8], [11]- [14], [27] and diffusion (ATC and CTA) [17]- [22], [25], [26] algorithms -see Table I.…”