Non-sparse Multiple Kernel Learning for Fisher Discriminant Analysis

Abstract

Sparsity-inducing multiple kernel Fisher discriminant analysis (MK-FDA) has been studied in the literature. Building on recent advances in non-sparse multiple kernel learning (MKL), we propose a non-sparse version of MK-FDA, which imposes a general ℓ p norm regularisation on the kernel weights. We formulate the associated optimisation problem as a semi-infinite program (SIP), and adapt an iterative wrapper algorithm to solve it. We then discuss, in light of latest advances in MKL optimisation techniques, several reformulations and optimisation strategies that can potentially lead to significant improvements in the efficiency and scalability of MK-FDA. We carry out extensive experiments on six datasets from various application areas, and compare closely the performance of ℓ p MK-FDA, fixed norm MK-FDA, and several variants of SVM-based MKL (MK-SVM). Our results demonstrate that ℓ p MK-FDA improves upon sparse MK-FDA in many practical situations. The results also show that on image categorisation problems, ℓ p MK-FDA tends to outperform its SVM counterpart. Finally, we also discuss the connection between (MK-)FDA and (MK-)SVM, under the unified framework of regularised kernel machines.

&#x2113;<inf>p</inf> norm multiple kernel Fisher discriminant analysis for object and image categorisation

“…However, similar to 1 MK-SVM, it suffers from the "over-selectiveness" problem. This is overcome by its 2 counterpart in [31], but the formulation in [31] is for binary problems only. It is thus the goal of this paper to extend existing MK-FDA methods to a general p regularisation for both binary and multiclass problems.…”

“…In parallel to MK-SVMs, another line of research focuses on multiple kernel learning for Fisher discriminant analysis [13,33,31]. In MK-FDA, the FDA type of class separation criterion, i.e., the ratio of the projected between and within class scatters, is considered instead of the margin criterion in SVM.…”

“…In this paper, we extend [13,33,31] to a general p norm regularisation. This is achieved by approximating the p norm constraint in the optimisation problem using Taylor expansion.…”

“…1 regularisation tends to produce sparse selection results, which may lead to a loss of information. [31] recently proposed to replace the 1 regularisation with an 2 version. Experiments in [31] suggest that the regularisation norm can have a significant impact on the classifier performance, and one should choose 1 or 2 regularisation based on the intrinsic sparsity of the given set of base kernels.…”

“…Experiments in [31] suggest that the regularisation norm can have a significant impact on the classifier performance, and one should choose 1 or 2 regularisation based on the intrinsic sparsity of the given set of base kernels. The 2 MK-FDA formulation in [31] is only for binary problems.…”