Finite Sample Deviation and Variance Bounds for First Order Autoregressive Processes

González, Rodrigo A.; Rojas, Cristian R.

doi:10.1109/icassp40776.2020.9053095

Cited by 4 publications

(4 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, some assume that the process is auto-regressive (e.g. Lai and Wei (1983); Goldenshluger and Zeevi (2001); González and Rojas (2020)) or ergodic (e.g. Yu (1994); Duchi et al (2012)).…”

Section: Introductionmentioning

confidence: 99%

Learning from many trajectories

Tu¹,

Frostig²,

Soltanolkotabi³

2022

Preprint

View full text Add to dashboard Cite

We initiate a study of supervised learning from many independent sequences ("trajectories") of non-independent covariates, reflecting tasks in sequence modeling, control, and reinforcement learning. Conceptually, our multi-trajectory setup sits between two traditional settings in statistical learning theory: learning from independent examples and learning from a single auto-correlated sequence. Our conditions for efficient learning generalize the former settingtrajectories must be non-degenerate in ways that extend standard requirements for independent examples. They do not require that trajectories be ergodic, long, nor strictly stable.For linear least-squares regression, given n-dimensional examples produced by m trajectories, each of length T , we observe a notable change in statistical efficiency as the number of trajectories increases from a few (namely m n) to many (namely m n). Specifically, we establish that the worst-case error rate this problem is Θ(n/mT ) whenever m n. Meanwhile, when m n, we establish a (sharp) lower bound of Ω(n 2 /m 2 T ) on the worst-case error rate, realized by a simple, marginally unstable linear dynamical system. A key upshot is that, in domains where trajectories regularly reset, the error rate eventually behaves as if all of the examples were independent altogether, drawn from their marginals. As a corollary of our analysis, we also improve guarantees for the linear system identification problem.

show abstract

Section: Introductionmentioning

confidence: 99%

Learning from many trajectories

Tu¹,

Frostig²,

Soltanolkotabi³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In this case, the modified algorithm can estimate unknown parameters and the unknown noise simultaneously, with less computational cost and better accuracy. Non-asymptotic deviations bounds for least-squared estimation in Gaussian AR processes have been recently studied (see [68]). The study relies on martingale concentration inequalities and tail bound for χ 2 -distributed variables; in the end, they provided a concentration bound for the sample covariance matrix of the process output.…”

Section: Robust Version Of Maximum Likelihood Methods With Whitening:...mentioning

confidence: 99%

Robust Parametric Identification for ARMAX Models with Non-Gaussian and Coloured Noise: A Survey

Escobar

Poznyak

2022

Mathematics

View full text Add to dashboard Cite

In this paper the Cramer-Rao information bound for ARMAX (Auto-Regression-Moving-Average-Models-with-Exogenuos-inputs) under non-Gaussian noise is derived. It is shown that the direct application of the Least Squares Method (LSM) leads to incorrect (shifted) parameter estimates. This inconsistency can be corrected by the implementation of the parallel usage of the MLMW (Maximum Likelihood Method with Whitening) procedure, applied to all measurable variables of the model, and a nonlinear residual transformation using the information on the distribution density of a non-Gaussian noise, participating in Moving Average structure. The design of the corresponding parameter-estimator, realizing the suggested MLMW-procedure is discussed in details. It is shown that this method is asymptotically optimal, that is, reaches this information bound. If the noise distribution belongs to some given class, then the Huber approach (min-max version of MLM) may be effectively applied. A numerical example illustrates the suggested approach.

show abstract

“…We mention two recent works related to this special case. Finite sample analysis of OLS estimator for AR models with known order appeared in [32], where the parameter error in terms of ℓ ∞norm is analyzed as opposed to the ℓ 2 -norm error in our case. Non-asymptotic results for using lasso estimator to learn AR models appeared in [33], which requires mixing time and a slightly different excitation condition, which can be less interpretable for control audience than system-related properties in our work.…”

Section: A Special Casesmentioning

confidence: 99%

Sample Complexity Analysis and Self-regularization in Identification of Over-parameterized ARX Models

Liu

Weitze

et al. 2022

2022 IEEE 61st Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

AutoRegressive eXogenous (ARX) models form one of the most important model classes in control theory, econometrics, and statistics, but they are yet to be understood in terms of their finite sample identification analysis. The technical challenges come from the strong statistical dependency not only between data samples at different time instances but also between elements within each individual sample. In this work, for ARX models with potentially unknown orders, we study how ordinary least squares (OLS) estimator performs in terms of identifying model parameters from data collected from either a single length-T trajectory or N i.i.d. trajectories. Our main results show that as long as the orders of the model are chosen optimistically, i.e., we are learning an over-parameterized model compared to the ground truth ARX, the OLS will converge with the optimal rate O(1/ √ T ) (or O(1/ √ N )) to the true (low-order) ARX parameters. This occurs without the aid of any regularization, thus is referred to as self-regularization. Our results imply that the oracle knowledge of the true orders and usage of regularizers are not necessary in learning ARX models -over-parameterization is all you need.

show abstract

Finite Sample Deviation and Variance Bounds for First Order Autoregressive Processes

Cited by 4 publications

References 30 publications

Learning from many trajectories

Learning from many trajectories

Robust Parametric Identification for ARMAX Models with Non-Gaussian and Coloured Noise: A Survey

Sample Complexity Analysis and Self-regularization in Identification of Over-parameterized ARX Models

Contact Info

Product

Resources

About