Universal linear least squares prediction: upper and lower bounds

Singer, Andrew C.; Kozat, Süleyman S.; Feder, Meir

doi:10.1109/tit.2002.800489

Cited by 38 publications

(42 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…in the upper bound cannot be improved [52]. It is also shown in [52] that one can reach the optimal upper bound (with exact scaling terms) by using a slightly modified version of (2.1)…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Boosted adaptive filters

Kari

Mirza

Khan

et al. 2018

Digital Signal Processing

View full text Add to dashboard Cite

We investigate boosted online regression and propose a novel family of regression algorithms with strong theoretical bounds. In addition, we implement several variants of the proposed generic algorithm. We specifically provide theoretical bounds for the performance of our proposed algorithms that hold in a strong mathematical sense. We achieve guaranteed performance improvement over the conventional online regression methods without any statistical assumptions on the desired data or feature vectors. We demonstrate an intrinsic relationship, in terms of boosting, between the adaptive mixture-of-experts and data reuse algorithms. Furthermore, we introduce a boosting algorithm based on random updates that is significantly faster than the conventional boosting methods and other variants of our proposed algorithms while achieving an enhanced performance gain. Hence, the random updates method is specifically applicable to the fast and high dimensional streaming data. Specifically, we investigate Recursive Least Squares (RLS)-based and Least Mean Squares (LMS)-based linear regression algorithms in a mixture-of-experts setting, and provide several variants of these well known adaptation methods. Moreover, we extend the proposed algorithms to other filters. Specifically, we investigate the effect of the proposed algorithms on piecewise linear filters. Furthermore, we provide theoretical bounds for the computational complexity of our proposed algorithms. We demonstrate substantial performance gains in terms of mean square error over the constituent filters through an extensive set of benchmark real data sets and simulated examples.

show abstract

“…in the upper bound cannot be improved [52]. It is also shown in [52] that one can reach the optimal upper bound (with exact scaling terms) by using a slightly modified version of (2.1)…”

Section: Related Workmentioning

confidence: 99%

“…Note that the extension (2.3) of (2.1) is a forward algorithm (Section 5 of [53]) and one can show that, in the scalar case, the predictions of (2.3) are always bounded (which is not the case for (2.1)) [52].…”

Section: Related Workmentioning

confidence: 99%

Boosted adaptive filters

Kari

Mirza

Khan

et al. 2018

Digital Signal Processing

View full text Add to dashboard Cite

show abstract

“…As an example, in [2] we investigated linear regression of real-valued data under the square error loss. We presented a regression algorithm whose accumulated error is asymptotically as small as the best fixed linear regressor for that sequence, taken from the class of all linear regressors of a given order.…”

Section: Introductionmentioning

confidence: 99%

“…Unlike [2], [4], here we try to exploit the time varying nature of the best choice of algorithm for any given realization, since the choice of best algorithm from a class of static algorithms can change over time. Nevertheless, instead of trying to find the best partition (possible best switching points) or best number of transitions, our objective is simply to achieve the performance of the best partition directly.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we show that the performance weighting approach used in [1], [5] can be generalized to yield algorithms that can efficiently compete with the best partition over all partitions for more general loss functions. We demonstrate that the universal probability assignment introduced in [1], [5] can be effectively merged into the prediction framework by using the methodology introduced in [2]. Although we investigate the continuous class of linear regressors or that of a finite number of adaptive filters as our competition class and use the square error loss, the methodology introduced in this paper can be extended to arbitrary competition classes, such as that of certain nonlinear regressors considered in [6], [7] or to more general loss functions as in [8].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Universal Switching Linear Least Squares Prediction

Kozat

Singer

2008

IEEE Trans. Signal Process.

Self Cite

View full text Add to dashboard Cite

Abstract-We consider sequential regression of individual sequences under the square error loss. Using a competitive algorithm framework, we construct a sequential algorithm that can achieve the performance of the best piecewise (in time) linear regression algorithm tuned to the underlying individual sequence. The sequential algorithm we construct does not need the data length, number of piecewise linear regions, or the locations of the transition times, however, it can asymptotically achieve the performance of the best piecewise (in time) linear regressor that can choose number of segments, duration of these segments and best regressor in each segment, based on observation of the whole sequence in advance. We use a transition diagram similar to that of [Willems '96] to effectively combine an exponential number of competing algorithms, with a complexity that is only linear in the data length. We demonstrate that the regret of this approach is at most O(4 ln(n)) per transition for not knowing the best transition times and at most O(ln(n)) for estimating the best linear regressor in each segment, where n is the total length of the observation process. Lower bounds for any sequential algorithm demonstrate a form of minmax optimality in certain settings. We then extend these results to include a finite collection of competing algorithms within each time segment, rather than linear regressors. I. INTRODUCTIONA common approach to applications in adaptive signal processing is to take the viewpoint of turning the problem at hand, such as equalization, prediction or some other sequential decision problem, and turn it into an associated parametric modeling or estimation problem. By forcing the problem into this form, we then have to live with the associated performance of the resulting parameter estimation problem, which in general is worse, often significantly worse, than that which could have been obtained by directly addressing the problem at hand. Moreover, if the assumptions in the model do not match reality, then the performance of the algorithm tuned to the assumed statistical model may deteriorate considerably.In this paper, we approach the problem of prediction from a competitive algorithm point of view. By defining a competitive framework, we try to achieve the performance of the best algorithm from a large class of candidate algorithms, rather than attempting to fit a given model to the data at hand. The performance measure of interest is then defined with respect to the best from this class, instead of the usual parametric modeling error between the output of the modeling algorithm and the desired signal directly. We will show that by not forcing the algorithms to make hard decisions about a set of parameters at each step, but rather permitting a competition among many candidate models, we can obtain algorithms that

show abstract

Parametric Estimation

Kozat

Singer

2014

Academic Press Library in Signal Processing

View full text Add to dashboard Cite

Universal linear least squares prediction: upper and lower bounds

Cited by 38 publications

References 13 publications

Boosted adaptive filters

Boosted adaptive filters

Universal Switching Linear Least Squares Prediction

Parametric Estimation

Contact Info

Product

Resources

About