Transfer Extreme Learning Machine with Output Weight Alignment

Abstract

Extreme Learning Machine (ELM) as a fast and efficient neural network model in pattern recognition and machine learning will decline when the labeled training sample is insufficient. Transfer learning helps the target task to learn a reliable model by using plentiful labeled samples from the different but relevant domain. In this paper, we propose a supervised Extreme Learning Machine with knowledge transferability, called Transfer Extreme Learning Machine with Output Weight Alignment (TELM-OWA). Firstly, it reduces the distribution difference between domains by aligning the output weight matrix of the ELM trained by the labeled samples from the source and target domains. Secondly, the approximation between the interdomain ELM output weight matrix is added to the objective function to further realize the cross-domain transfer of knowledge. Thirdly, we consider the objective function as the least square problem and transform it into a standard ELM model to be efficiently solved. Finally, the effectiveness of the proposed algorithm is verified by classification experiments on 16 sets of image datasets and 6 sets of text datasets, and the result demonstrates the competitive performance of our method with respect to other ELM models and transfer learning approach.

TSTELM: Two-Stage Transfer Extreme Learning Machine for Unsupervised Domain Adaptation

“…In order to further improve cross-domain knowledge transferring, similar to [ 38 ], we introduce a transformation matrix M to align the output weights of ELM between the source domain and the target domain. The function is established as follows: where ‖•‖ F 2 is Frobenius norm.…”

“…It can be seen: Compared with the classical ELM, TSTELM reduces the distribution difference between domains and transfers knowledge across domains via adopting MMD, output weight alignment, parameter approximation, and ∑ c =0 C ‖ H T ( c ) β T − H S _ av ( c ) β 1 ‖ 2 . Though TELM-OWA proposed by Zang et al [ 38 ] also applies output weight alignment and parameter approximation for domain adaptation, it is a supervised domain adaptation algorithm requiring few target labeled samples unlike TSTELM. In addition, TSTELM replaces ‖ H T β T − Y T ‖ 2 with ∑ c =0 C ‖ H T ( c ) β T − H S _ av ( c ) β 1 ‖ 2 , which is different from TELM-OWA.…”

“…Though TELM-OWA proposed by Zang et al [ 38 ] also applies output weight alignment and parameter approximation for domain adaptation, it is a supervised domain adaptation algorithm requiring few target labeled samples unlike TSTELM. In addition, TSTELM replaces ‖ H T β T − Y T ‖ 2 with ∑ c =0 C ‖ H T ( c ) β T − H S _ av ( c ) β 1 ‖ 2 , which is different from TELM-OWA.…”

“…Adaptive ELM (AELM) [ 37 ] was proposed by introducing the manifold regularization term into ELM for image classification. Zang et al [ 38 ] proposed a supervised extreme learning machine called transfer extreme learning machine with output weight alignment (TELM-OWA), which aligned the output weight matrix of the ELM between domains and added the approximation between the inter-domain ELM parameters for knowledge transferring. However, these approaches are developed to solve semi-supervised domain adaptation problems because they require few labeled samples from the target domains.…”

“…In this article, inspired by pioneering works [ 38 , 42 ], we propose a novel method denoted as two-stage transfer extreme learning machine (TSTELM), in which there are two stages of domain adaptation: statistical matching and subspace alignment. At the statistical matching stage, we first learn a domain adaptation ELM classifier via utilizing the MMD to simultaneously minimize the marginal and conditional distribution between domains.…”