CARTopt: a random search method for nonsmooth unconstrained optimization

Robertson, B. L.; Price, C. J.; Reale, Marco

doi:10.1007/s10589-013-9560-9

Cited by 16 publications

(17 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We extend the oblique DT method used in the CARTopt optimisation algorithm of [17] in a number of ways to de- First, we explain the basic concept of our algorithm for a two class classification problem. The algorithm easily generalises to the multi-class problem.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

HHCART: An oblique decision tree

Wickramarachchi

Robertson

Reale

et al. 2016

Computational Statistics & Data Analysis

Self Cite

View full text Add to dashboard Cite

Decision trees are a popular technique in statistical data classification. They recursively partition the feature space into disjoint sub-regions until each sub-region becomes homogeneous with respect to a particular class. The basic Classification and Regression Tree (CART) algorithm partitions the feature space using axis parallel splits. When the true decision boundaries are not aligned with the feature axes, this approach can produce a complicated boundary structure. Oblique decision trees use oblique decision boundaries to potentially simplify the boundary structure. The major limitation of this approach is that the tree induction algorithm is computationally expensive. In this article we present a new decision tree algorithm, called HHCART. The method utilizes a series of Householder matrices to reflect the training data at each node during the tree construction. Each reflection is based on the directions of the eigenvectors from each classes' covariance matrix. Considering axis parallel splits in the reflected training data provides an efficient way of finding oblique splits in the unreflected training data. Experimental results show that the accuracy and size of the HHCART trees are comparable with some benchmark methods in the literature. The appealing feature of HHCART is that it can handle both qualitative and quantitative features in the same oblique split.

show abstract

Section: Methodsmentioning

confidence: 99%

“…Axis parallel splits can then be searched in the reflected feature space to find the best split. This split will be oblique in the original feature space [17].…”

Section: Methodsmentioning

confidence: 99%

HHCART: An oblique decision tree

Wickramarachchi

Robertson

Reale

et al. 2016

Computational Statistics & Data Analysis

Self Cite

View full text Add to dashboard Cite

show abstract

“…Consider splitting a node t containing p quantitative features and C classes. Each reflected feature space at node t is defined using a Householder matrix (Robertson, Price & Reale ):

H = I - 2 \frac{false(bold-italice - d_{ik} false) {(e - {bold-italicd}_{italicik})}^{⊤}}{false‖ bold-italice - {bold-italicd}_{italicik} {false‖}^{2}},

where e is the first column of the p ‐dimensional identity matrix I and d ik is the i th unit scaled eigenvector of the estimated covariance matrix of class k examples at node t . Each feature vector x (column vector) at node t is reflected using H x and the best axis parallel split in the reflected training examples is found.…”

Section: Related Decision Treesmentioning

confidence: 99%

“…Each feature vector x (column vector) at node t is reflected using H x and the best axis parallel split in the reflected training examples is found. Reflecting the feature vectors in this way makes d ik parallel to e and provides a simple and effective way to find oblique splits (Robertson, Price & Reale ; Wickramarachchi et al . ).…”

Section: Related Decision Treesmentioning

confidence: 99%

A reflected feature space for CART

Wickramarachchi¹,

Robertson

Reale

et al. 2019

Aus NZ J of Statistics

Self Cite

View full text Add to dashboard Cite

We present an algorithm for learning oblique decision trees, called HHCART(G). Our decision tree combines learning concepts from two classification trees, HHCART and Geometric Decision Tree (GDT). HHCART(G) is a simplified HHCART algorithm that uses linear structure in the training examples, captured by a modified GDT angle bisector, to define splitting directions. At each node, we reflect the training examples with respect to the modified angle bisector to align this linear structure with the coordinate axes. Searching axis parallel splits in this reflected feature space provides an efficient and effective way of finding oblique splits in the original feature space. Our method is much simpler than HHCART because it only considers one reflected feature space for node splitting. HHCART considers multiple reflected feature spaces for node splitting making it more computationally intensive to build. Experimental results show that HHCART(G) is an effective classifier, producing compact trees with similar or better results than several other decision trees, including GDT and HHCART trees.

show abstract

“…To potentially simplify the CART partition, Robertson et al (2013a) reflect the training data using a Householder matrix…”

Section: Introductionmentioning

confidence: 99%

Stochastic global optimization using random forests

Bl¹,

Price²,

Reale³

2017

Syme, G., Hatton MacDonald, D., Fulton, B. And Piantadosi, J. (Eds) MODSIM2017, 22nd International Congress on Modelling and Si

View full text Add to dashboard Cite

Global optimization problems occur in many fields i ncluding m athematics, s tatistics, computer science, engineering, and economics. The purpose of a global optimization algorithm is to efficiently find an objective function's global minimum. In this article we consider bound constrained global optimization, where the search is performed in a box, denoted Ω. The global optimization problem is deceptively simple and it is usually difficult to find the global mi nimum. One of the difficulties is that there is often no way to verify that a local minimum is indeed the global minimum. If the objective function is convex, the local minimum is also the global minimum. However, many optimization problems are not convex. Of particular interest in this article are objective functions that lack any special properties such as continuity, smoothness, or a Lipschitz constant. The CARTopt algorithm is a stochastic algorithm for bound constrained global optimization. This algorithm alternates between partition and sampling phases. At each iteration, points sampled from Ω are classified low or high based on their objective function values. These classified p oints d efine tr aining da ta th at is us ed to partition Ω in boxes using classification and regression trees (CART). The objective function is then evaluated at a number of points drawn from the boxes classified low and some are drawn from Ω i tself. Drawing points from the low boxes focuses the search in regions where the objective function is known to be low. Sampling Ω reduces the risk of missing the global minimum and is necessary to establish convergence. The new points are then added to the existing training data and the method repeats. In this article we provide an alternative partitioning strategy for the CARTopt algorithm. Rather than using a CART partition at each iteration, we propose using a random forest partition. Pure CART partitions are known to over-fit t raining d ata a nd h ave l ow b ias a nd h igh v ariance. A r andom f orest p artition h as l ess variance than a CART partition, and hence provides a more stable approximation to where the objective function is expected to be low. We also provide a method for sampling low regions defined by random forest partitions. A preliminary simulation study showed that using random forests in the CARTopt algorithm was an effective strategy for solving a variety of global optimization test problems. The authors are currently refining the method and extending the set of test problems.

show abstract

CARTopt: a random search method for nonsmooth unconstrained optimization

Cited by 16 publications

References 18 publications

HHCART: An oblique decision tree

HHCART: An oblique decision tree

A reflected feature space for CART

Stochastic global optimization using random forests

Contact Info

Product

Resources

About