Optimizing Kernel Machines Using Deep Learning

Abstract

Building highly nonlinear and nonparametric models is central to several state-of-the-art machine learning systems. Kernel methods form an important class of techniques that induce a reproducing kernel Hilbert space (RKHS) for inferring non-linear models through the construction of similarity functions from data. These methods are particularly preferred in cases where the training data sizes are limited and when prior knowledge of the data similarities is available. Despite their usefulness, they are limited by the computational complexity and their inability to support end-to-end learning with a task-specific objective. On the other hand, deep neural networks have become the de facto solution for end-to-end inference in several learning paradigms. In this paper, we explore the idea of using deep architectures to perform kernel machine optimization, for both computational efficiency and end-to-end inferencing. To this end, we develop the deep kernel machine optimization framework, that creates an ensemble of dense embeddings using Nyström kernel approximations and utilizes deep learning to generate task-specific representations through the fusion of the embeddings. Intuitively, the filters of the network are trained to fuse information from an ensemble of linear subspaces in the RKHS. Furthermore, we introduce the kernel dropout regularization to enable improved training convergence. Finally, we extend this framework to the multiple kernel case, by coupling a global fusion layer with pretrained deep kernel machines for each of the constituent kernels. Using case studies with limited training data, and lack of explicit feature sources, we demonstrate the effectiveness of our framework over conventional model inferencing techniques.

A Deep Learning Approach to Recover High-g Shock Signals From the Faulty Accelerometer

“…The inspiration of adopting deep learning to correct faulty shock signals comes from the aphorism of "diligence redeems stupidity" that humans or even biological entities can improve their skills through continuous training and effort. Moreover, recent research in applying deep learning to process time-series signals [15], [24] also proves the concept in related domains, and paves a new avenue towards the goal of this paper. Similarly, the DNN is trained with the collected shock signal datasets.…”

“…Over the past decade, deep learning has achieved great success in a variety of fields. Examples include computer vision [15], fault diagnosis in mechanics [16], aerospace engineering [17], etc. However, few works have been done to introduce deep learning to the field of calibrating industrial high-g accelerometer.…”

Key issues of support vector machines and future prospects

“…尽管文献 [31] 为学习最优核矩阵提供了良好的开端, 但实践中, 如 何选择或构造出合适的核矩阵基底并求得最优核等问题尚未解决. 目前, 利用深度学习和启发式学 习 [38,39] 等方式的研究为最优核学习提供了一个新的方向. 此外, 核方法在其他机器学习问题中也大有可为.…”

Triplet Network with Attention for Speaker Diarization

“…Despite its effectiveness, it is important to note that the feature extraction process is disentangled from the metric learning network and hence cannot support endto-end inferencing. However, recent success of such end-to-end learning systems in computer vision applications [20,21,22] motivates the design of a deep metric learning architecture that works directly on the temporal sequences. Long Short-Term Memory (LSTM) based recurrent networks have become the de facto solution to sequence modeling tasks including acoustic modeling [23], speech recognition [24] and Natural Language Processing (NLP) [25].…”