The stochastic approximation method for the estimation of a multivariate probability density

Abstract

We apply the stochastic approximation method to construct a large class of recursive kernel estimators of a probability density, including the one introduced by Hall and Patil [1994. On the efficiency of on-line density estimators. IEEE Trans. Inform. Theory 40, 1504-1512]. We study the properties of these estimators and compare them with Rosenblatt's nonrecursive estimator. It turns out that, for pointwise estimation, it is preferable to use the nonrecursive Rosenblatt's kernel estimator rather than any recursive estimator. A contrario, for estimation by confidence intervals, it is better to use a recursive estimator rather than Rosenblatt's estimator.

Plug‐in bandwidth selector for recursive kernel regression estimators defined by stochastic approximation method

“…Let us first underline that it follows from that the stepsize, which minimize the variance of m n is ( γ n ) = ([1 − a ] n −1 ); using this stepsize, the variance of m n is equal to Now, using the special stepsize ( γ n ) = ( n −1 ) ( MOKKADEM et al , ; SLAOUI ), the variance of m n is equal to Let us recall that under the Assumptions ( A 1), ( A 2)(ii), and ( A 3), we have which shows that the recursive estimator gives smaller variance than the non‐recursive estimator . Similar results was given by Mokkadem et al () and Slaoui () in the framework of the densi...…”

“…Assumption (A2) on the stepsize and the bandwidth was used in the recursive framework for the estimation of the density function (Mokkadem et al 2009a;Slaoui, 2013Slaoui, , 2014a and for the estimation of the distribution function (Slaoui (2014b)).…”

“…Now, using the special stepsize ( n ) = (n −1 ) (Mokkadem et al, 2009a;Slaoui 2013), the variance of m n is equal to…”

“…Following Robbins–Monro's procedure, this algorithm is defined by setting , and, for all n ≥1, where W n ( x ) is an ‘observation’ of the function h at the point m n − 1 ( x ), and the stepsize ( γ n ) is a sequence of positive real numbers that goes to zero. To define W n ( x ), we follow the approach of RéVéSZ (), Tsybakov () and of MOKKADEM et al () and introduces a kernel K (that is, a function satisfying ), a bandwidth ( h n ) (that is, a sequence of positive real numbers that goes to zero), and sets . Then, the estimator m n to recursively estimate the regression function m at the point x can be written as This recursive property is particularly useful in large sample size because m n can be easily updated with each additional observation.…”

“…Following Robbins-Monro's procedure, this algorithm is defined by setting m 0 (x) ∈ R, and, for all n ≥ 1, m n (x) = m n−1 (x) + n W n , where W n (x) is an 'observation' of the function h at the point m n−1 (x), and the stepsize ( n ) is a sequence of positive real numbers that goes to zero. To define W n (x), we follow the approach of Révész (1973Révész ( , 1977, Tsybakov (1990) and of Mokkadem et al (2009aMokkadem et al ( , 2009b and introduces a kernel K (that is, a function satisfying ∫ R K(x) dx = 1), a bandwidth (h n ) (that is, a sequence of positive real numbers that goes to zero), and sets W n (x) = h −1 n Y n K ( h −1 n (x − X n ) ) − m n−1 (x). Then, the estimator m n to recursively estimate the regression function m at the point x can be written as…”