How effective is Cauchy-EDA in high dimensions?

Abstract

We consider the problem of high dimensional blackbox optimisation via Estimation of Distribution Algorithms (EDA) and the use of heavy-tailed search distributions in this setting. Some authors have suggested that employing a heavy tailed search distribution, such as a Cauchy, may make EDA better explore a high dimensional search space. However, other authors have found Cauchy search distributions are less effective than Gaussian search distributions in high dimensional problems. In this paper, we set out to resolve this controversy. To achieve this we run extensive experiments on a battery of high-dimensional test functions, and develop some theory which shows that small search steps are always more likely to move the search distribution towards the global optimum than large ones and, in particular, large search steps in high-dimensional spaces nearly always do badly in this respect. We hypothesise that, since exploration by large steps is mostly counterproductive in high dimensions, and since the fraction of good directions decays exponentially fast with increasing dimension, instead one should focus mainly on finding the right direction in which to move the search distribution. We propose a minor change to standard Gaussian EDA which implicitly achieves this aim, and our experiments on a sequence of test functions confirm the good performance of our new approach.

A Review of Population-Based Metaheuristics for Large-Scale Black-Box Global Optimization—Part I

“…They reported that although a multivariate Cauchy performs better than a univariate Cauchy, they both perform worse than a multivariate Gaussian distribution on large-scale problems. In another subsequent study, they provided a theoretical explanation for the poor performance of Cauchy distribution on large-scale problems, which was shown to be consistent with empirical results [96]. The study showed that unlike Gaussian norms, Cauchy norms lack a good concentration property causing a disproportionate number of very large steps, which results in an inefficient search strategy as the dimensions increase.…”

“…Among these, the Cauchy distribution has been used more widely with EDAs. Although the literature is clear on the efficacy of Cauchy sampling on low-dimensional problem [94,95], there is controversy in its utility on high-dimensional problems [96]. Hansen et al [97] reported that Cauchy's long jumps are almost invariably ineffective, while other studies found it beneficial [73,74].…”

Large-Scale Estimation of Distribution Algorithms with Adaptive Heavy Tailed Random Projection Ensembles

“…The increased exploration abilities of heavy tailed distributions are well known in evolutionary search, starting from pioneering work by Yao et al [19] in the univariate setting, and more recent work in the multivariate EDA framework -see e.g. [20][21][22] and references therein. However, in the regime of large search spaces of dimensionality well beyond 100 variables, the use of such heavy tailed distributions in EDA is not straightforward -as demonstrated in [22], a heavy tailed search distribution becomes increasingly counter-productive as it loses sight of the direction of the search.…”

“…[20][21][22] and references therein. However, in the regime of large search spaces of dimensionality well beyond 100 variables, the use of such heavy tailed distributions in EDA is not straightforward -as demonstrated in [22], a heavy tailed search distribution becomes increasingly counter-productive as it loses sight of the direction of the search. Here we avoid these problems as we employ heavy tailed distributions in a combination of RPs rather than directly in the role of a search distribution.…”

Large scale continuous EDA using mutual information

“…For the experiments in this section, we set m c = 100 and θ = 0.3 for WI, because they are recommended in [10]. For SM, c = min{D/5, M/15} is used, for reasons expained in [16]. Some experimental analysis on the influence of θ will be given in Section V. All the experiments in this section were ran 25 times, and the reported results are the average values ± standard deviations of the best fitness value found in these 25 independent repetitions.…”