Sparsest Piecewise-Linear Regression of One-Dimensional Data

Abstract

We study the problem of interpolating one-dimensional data with total variation regularization on the second derivative, which is known to promote piecewise-linear solutions with few knots. In a first scenario, we consider the problem of exact interpolation. We thoroughly describe the form of the solutions of the underlying constrained optimization problem, including the sparsest piecewise-linear solutions, i.e., with the minimum number of knots. Next, we relax the exact interpolation requirement, and consider a penalized optimization problem with a strictly convex data-fidelity cost function. We show that the underlying penalized problem can be reformulated as a constrained problem, and thus that all our previous results still apply. We propose a simple and fast two-step algorithm to reach a sparsest solution of this constrained problem. Our theoretical and algorithmic results have implications in the field of machine learning, more precisely for the study of popular ReLU neural networks. Indeed, it is well known that such networks produce an input-output relation that is a continuous piecewise-linear function, as does our interpolation algorithm.

Nonsmooth convex optimization to estimate the Covid-19 reproduction number space-time evolution with robustness against low quality data

“…Such a property ensures that, besides an estimate of R at day t, one also gets an estimation of a local trend, assessing whether the pandemic is growing or decreasing. Following [8,12], to favor piecewise linear temporal evolution of the estimates, we penalize the 1 -norm of the multivariate time-domain Laplacian D 2 : R D×T → R D×(T −2) of the estimates of R:…”

“…To study the theoretical properties of Problem (12), and to derive a minimization algorithm, it is useful to recast Eq. ( 12) into a generic formulation:…”

“…The 1 -norm and the indicator functions in the objective function make the overall functional non-smooth, preventing from the use of standard gradient descent. Minimizing (12) thus requires more advanced tools, handling nondifferentiable functions, such as proximal algorithms.…”

TV-based Reconstruction of Periodic Functions

“…Splines are piecewisesmooth functions with finite rates of innovation, whose smoothness can be adapted by adequately choosing the differential operator [1,38]. Since then, algorithmic schemes have been developed [39][40][41][42], and important extensions have been proposed, generalizing the framework to other Banach spaces such as measure spaces over spherical domains [43,44], hybrid spaces [45], multivariate settings [46], or more general abstract settings [47][48][49]. Applications include geophysical and astronomical data reconstruction [43,44], neural networks [50,51], and image analysis [52].…”

A Fast and Scalable Polyatomic Frank-Wolfe Algorithm for the LASSO

“…In compressed sensing setups [12], the number of measurements L is typically much smaller than the dimension N of the problem. The solution set to the LASSO problem (1) can be shown to be non-empty, and the closed convex hull of (at most) L-sparse extreme points; see for example [16,Theorem 6] or [17,Theorem 6.8] as well as [18]- [25] for generalizations. If the design matrix coefficients are furthermore drawn according to a continuous probability distribution, then the LASSO solution is unique with probability one, and guaranteed to be at most L-sparse [11], [26].…”