Universal Switching Linear Least Squares Prediction

Abstract

Abstract-We consider sequential regression of individual sequences under the square error loss. Using a competitive algorithm framework, we construct a sequential algorithm that can achieve the performance of the best piecewise (in time) linear regression algorithm tuned to the underlying individual sequence. The sequential algorithm we construct does not need the data length, number of piecewise linear regions, or the locations of the transition times, however, it can asymptotically achieve the performance of the best piecewise (in time) linear regressor that can choose number of segments, duration of these segments and best regressor in each segment, based on observation of the whole sequence in advance. We use a transition diagram similar to that of [Willems '96] to effectively combine an exponential number of competing algorithms, with a complexity that is only linear in the data length. We demonstrate that the regret of this approach is at most O(4 ln(n)) per transition for not knowing the best transition times and at most O(ln(n)) for estimating the best linear regressor in each segment, where n is the total length of the observation process. Lower bounds for any sequential algorithm demonstrate a form of minmax optimality in certain settings. We then extend these results to include a finite collection of competing algorithms within each time segment, rather than linear regressors. I. INTRODUCTIONA common approach to applications in adaptive signal processing is to take the viewpoint of turning the problem at hand, such as equalization, prediction or some other sequential decision problem, and turn it into an associated parametric modeling or estimation problem. By forcing the problem into this form, we then have to live with the associated performance of the resulting parameter estimation problem, which in general is worse, often significantly worse, than that which could have been obtained by directly addressing the problem at hand. Moreover, if the assumptions in the model do not match reality, then the performance of the algorithm tuned to the assumed statistical model may deteriorate considerably.In this paper, we approach the problem of prediction from a competitive algorithm point of view. By defining a competitive framework, we try to achieve the performance of the best algorithm from a large class of candidate algorithms, rather than attempting to fit a given model to the data at hand. The performance measure of interest is then defined with respect to the best from this class, instead of the usual parametric modeling error between the output of the modeling algorithm and the desired signal directly. We will show that by not forcing the algorithms to make hard decisions about a set of parameters at each step, but rather permitting a competition among many candidate models, we can obtain algorithms that

Linear MMSE-Optimal Turbo Equalization Using Context Trees

“…Hence, as a corollary to the theorem, taking the expectation of both sides of (14) with respect to any distribution on yields the following: Corollary: (15) Equation (14) is true for all , and given for any , , , i.e., (16) since (14) is true for the minimizing and equalizer vectors. Taking the expectation of both sides of (16) and minimizing with respect to and , , yields the corollary.…”

“…Context trees and context tree weighting are extensively used in data compression [15], coding and data prediction [16]- [18]. In the context of source coding and universal probability assignment, the context tree weighting method is mainly used to calculate a weighted mixture of probabilities generated by the piecewise Markov models represented on the tree [15].…”

“…In the context of source coding and universal probability assignment, the context tree weighting method is mainly used to calculate a weighted mixture of probabilities generated by the piecewise Markov models represented on the tree [15]. In nonlinear prediction, context trees are used to represent piecewise linear models by partitioning the space of past regressors [16], [18], specifically for labeling the past observations based on a certain criteria. Note that although we use the notion of context trees for nonlinear modeling as in [15], [17]- [20], our results and usage of context trees differ from [15], [17]- [19] in a number of important ways.…”

Boosted adaptive filters

“…For presentation purposes, we assume that d t ∈ [−1, 1], however, our derivations hold for any bounded but arbitrary desired data sequences. In our framework, we do not use any statistical assumptions on the input feature vectors or on the desired data such that our results are guaranteed to hold in an individual sequence manner [48].…”

Learning-based approximation of interconnect delay and slew in signoff timing tools

“…We derive wire delay and slew models by using least-squares regression (LSQR) [18] to fit values from the ST. We generate training samples from benchmark netlists that have no slack violations and contain heterogeneous mixes of cell sizes and types. From the ST, we obtain delay and slew at every pin in the netlist, and fit our models to the data from ST. We use 50% of these data points for training, derive models using LSQR, test the models on all data points, and compute the estimation errors.…”