Multimodal Task-Driven Dictionary Learning for Image Classification

Abstract

Dictionary learning algorithms have been successfully used for both reconstructive and discriminative tasks, where an input signal is represented with a sparse linear combination of dictionary atoms. While these methods are mostly developed for single-modality scenarios, recent studies have demonstrated the advantages of feature-level fusion based on the joint sparse representation of the multimodal inputs. In this paper, we propose a multimodal task-driven dictionary learning algorithm under the joint sparsity constraint (prior) to enforce collaborations among multiple homogeneous/heterogeneous sources of information. In this task-driven formulation, the multimodal dictionaries are learned simultaneously with their corresponding classifiers. The resulting multimodal dictionaries can generate discriminative latent features (sparse codes) from the data that are optimized for a given task such as binary or multiclass classification. Moreover, we present an extension of the proposed formulation using a mixed joint and independent sparsity prior, which facilitates more flexible fusion of the modalities at feature level. The efficacy of the proposed algorithms for multimodal classification is illustrated on four different applications--multimodal face recognition, multi-view face recognition, multi-view action recognition, and multimodal biometric recognition. It is also shown that, compared with the counterpart reconstructive-based dictionary learning algorithms, the task-driven formulations are more computationally efficient in the sense that they can be equipped with more compact dictionaries and still achieve superior performance.

Cost-Sensitive Feature Selection by Optimizing F-Measures

“…Gain [18] are among the most representative algorithms. (2) Wrapper methods, which use the prediction method as a black box to score the feature subsets, such as correlation-based feature selection (CFS) [19] and support vector machine recursive feature elimination (SVM-RFE) [20]. (3) Embedded methods, which directly incorporate the feature selection procedure into the model training process, such as regularized regressionbased feature selection methods [21], [22].…”

Recent progress on tactile object recognition

“…Such coding strategies has already been successfully used in face recognition and identity validation. 61 Very recently, Liu et al 62 attempted to develop joint sparse coding method for visual-tactile fusion and solved the intrinsic weakly pairing problem by relaxing the joint coding requirements. In the left panel of Figure 10, we list some representative experimental objects.…”

Joint Facial Action Unit Detection and Feature Fusion: A Multi-Conditional Learning Approach

“…Another possible direction toward feature fusion is to exploit the notion of joint sparse representations or learn multi-modal dictionaries based on joint sparsity constraints. Based on these techniques, the authors in [31], [32] managed to fuse the information from various biometrics in order to perform more accurate face verification. A similar approach is to perform joint dimensionality reduction and project the multiple features on a common subspace.…”