A Comprehensive Approach to Universal Piecewise Nonlinear Regression Based on Trees

Abstract

Abstract-In this paper, we investigate adaptive nonlinear regression and introduce tree based piecewise linear regression algorithms that are highly efficient and provide significantly improved performance with guaranteed upper bounds in an individual sequence manner. We use a tree notion in order to partition the space of regressors in a nested structure. The introduced algorithms adapt not only their regression functions but also the complete tree structure while achieving the performance of the "best" linear mixture of a doubly exponential number of partitions, with a computational complexity only polynomial in the number of nodes of the tree. While constructing these algorithms, we also avoid using any artificial "weighting" of models (with highly data dependent parameters) and, instead, directly minimize the final regression error, which is the ultimate performance goal. The introduced methods are generic such that they can readily incorporate different tree construction methods such as random trees in their framework and can use different regressor or partitioning functions as demonstrated in the paper.Index Terms-Nonlinear regression, nonlinear adaptive filtering, binary tree, universal, adaptive.

Boosted adaptive filters

“…However, one can use a method similar to that in [36] to make the partitioning adaptive. As an example, suppose that each constituent filter is defined on a 2-region partition, as shown in Fig.…”

“…These filters mitigate the overfitting, stability and convergence issues tied to nonlinear models [38,39,40], while effectively improving the modeling power relative to linear filters [36]. Nevertheless, in order to justify the boosting effect of our algorithm, we use linear base learners with exactly the same parameters and demonstrate that even in this case we can get performance improvement by our algorithm since any gain obtained in this way reflects the sole effect of the boosting mechanism.…”

“…Note that although linear filters have a low complexity, piecewise linear filters deliver a significantly superior performance in real life applications [36,37], with a comparable complexity. These filters mitigate the overfitting, stability and convergence issues tied to nonlinear models [38,39,40], while effectively improving the modeling power relative to linear filters [36].…”

A tree-based solution to nonlinear regression problem

“…Tammlanan bu birle~tirme ile sundugumuz algoritma veri tizerinde hi<;bir istatistiki varsaYllua dayanmadan asimptotik olarak en iyi temel slmflandlflcmm performansma ula~maktadlr. Sunulan algoritmanm hesaplama karma~lkhgl bUtiln veriler i<;in verinin boyutuna ve kullamlan hiyerar~ik modelin derinligine baglt olarak dogrusal bir ~ekilde degi~mektedir ve biz hiyerar~ik yapldaki dogrusal modellerin birle~imi i<;in slmrh saYlda bir birle~im kullandlglllllz i<;in algoritma iyi bir genelleme yapabilir ve veri Uzerine a~1fl uyum saglamaz (ya da slmrlt bir a~1fl uyum yapar) [3], [4].…”

Highly efficient nonlinear regression for big data with lexicographical splitting

“…However, in real-world applications, there is no prior information about the optimal partition of the regressor space. Using tree-based regression algorithms [2,7] is a common approach in such scenarios. These algorithms form a class of models over a set of partitions of the regressor space and adaptively combine these models, such that the combination asymptotically achieves the performance of the best model in the class.…”