Multi-Output Tree Chaining: An Interpretative Modelling and Lightweight Multi-Target Approach

Abstract

Multi-target regression (MTR) regards predictive problems with multiple numerical targets. To solve this, machine learning techniques can model solutions treating each target as a separated problem based only on the input features. Nonetheless, modelling inter-target correlation can improve predictive performance. When performing MTR tasks using the statistical dependencies of targets, several approaches put aside the evaluation of each pair-wise correlation between those targets, which may differ for each problem. Besides that, one of the main drawbacks of the current leading MTR method is its high memory cost. In this paper, we propose a novel MTR method called Multi-output Tree Chaining (MOTC) to overcome the mentioned disadvantages. Our method provides an interpretative internal tree-based structure which represents the relationships between targets denominated Chaining Trees (CT). Different from the current techniques, we compute the outputs dependencies, one-by-one, based on the Random Forest importance metric. Furthermore, we proposed a memory friendly approach which reduces the number of required regression models when compared to a leading method, reducing computational cost. We compared the proposed algorithm against three MTR methods (Single-target-ST; Multi-Target Regressor Stacking-MTRS; and Ensemble of Regressor Chains-ERC) on 18 benchmark datasets with two base regression algorithms (Random Forest and Support Vector Regression). The obtained results show that our method is superior to the ST approach regarding predictive performance, whereas, having no significant difference from ERC and MTRS. Moreover, the interpretative tree-based structures built by MOTC pose as great insight on the relationships among targets. Lastly, the proposed solution used significantly less memory than ERC being very similar in predictive performance.

Stock Portfolio Prediction by Multi-Target Decision Support

“…In the case of MTAS, multiple techniques can be used, which increases its costs when compared with SST. ERC and MOTC do not differ in their complexity mag-12 nitude order, but as empirically demonstrated by MOTC's authors [Mastelini et al 2018], their method expends less regressors than ERC, being consequently, lighter than the latter. DRS was the most costly evaluated method, since it uses a deep learning strategy, as previously mentioned.…”

“…The Multi-output Tree Chaining (MOTC) [Mastelini et al 2018] method was proposed as an extension to the ERC approach, which, instead of randomly defining target orders (as in ERC), employs heuristics (a correlation measurement metric and a statistical bound to rank target relevance) to build a tree structure to represent inter-target dependencies. These trees are called Chaining Trees (CT) and they also represent the strategy in which the regressors are going to be created: the process start from the leaves up to the root; the inner nodes employ the predictions of their descendent's regressors as additional features.…”

Online Multi-target regression trees with stacked leaf models

“…Local algorithms combine traditional STR solutions and often manipulate or modify the input space to insert inter-target dependency information within the modelling process. Thus, local algorithms use multiple ST regressors to solve an MTR task, often more than one regressor for each target variable (Spyromitros-Xioufis et al, 2016;Mastelini et al, 2017;Santana et al, 2018;Mastelini et al, 2018). As a result, they have a higher cost than global algorithms.…”

“…MTR have been widely used in batch learning applications (Borchani et al, 2015;Spyromitros-Xioufis et al, 2016;Mastelini et al, 2017;Melki et al, 2017;Mastelini et al, 2018;Santana et al, 2018), since they are related to several real-life problems, such as prediction of river flow properties, online sales, airline ticket prices and poultry meat properties (Spyromitros-Xioufis et al, 2016;Santana et al, 2018).…”

Improved prediction of soil properties with Multi-target Stacked Generalisation on EDXRF spectra

“…This idea is similar to the Bayesian network inference in design and was designed initially for classification problems (Read et al, 2009). Different strategies were proposed in the recent MTR literature (Spyromitros-Xioufis et al, 2016;Mastelini et al, 2018a). The Ensemble of Regressor Chains (ERC) constructs multiple randomly ordered target chains (Spyromitros-Xioufis et al, 2016).…”