A Multi-modal Sparse Coding Classifier Using Dictionaries with Different Number of Atoms

Abstract

Most of classification methods including the ones based on sparse representation (SRC), look at every training sample and its extracted modalities as a single point in a high dimensional space and a collection of these points build the training space used to train the classifier. In a multimodality classification problem, there might be lots of redundancies associated with different modalities of the training data which degrade the performance of the classifier. This paper considers the problem of multi-modality classification in a sparse representation framework which separately looks at each modality space and builds abstract and in the same time, precise models with different number of representatives for each individual modality. An optimization is also introduced to solve the classification problem using these models. In comparison to SRC which directly uses one modality from the training samples, the proposed method utilizes multiple abstract modalities to form efficient and comprehensive representation of the data in order to increase both the accuracy and efficiency of the classification process. Experimental results on face and digit recognition applications show that the proposed method has higher recognition rate compared to single-modality as well as multi-modality methods based on SRC.

Cross-modal subspace learning for fine-grained sketch-based image retrieval

“…In particular, many researchers use deep Convolutional Neural Network (CNN) to conduct cross-modal matching [84,85,86] or SBIR which is essentially to learn some feature subspaces to match multi-modal data. Moreover, the convolutional sparse coding technology can also learn subspace satisfying certain qualities [87,88,89], which illustrates the convolutional idea and subspace learning can be reasonably combined. Therefore, it is natural to also utilize cross-modal subspace learning concepts to improve CNN for SBIR, and potentially incorporating saliency information [90,91] to improve partlevel examination in the same network.…”

Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)

“…Originated to explaining neuronal activation of human visual system Olshausen and Field [1997], sparse coding model has been widely used in machine learning applications to model different uni-modal data, such as image, text and audio Mairal et al [2010], Yang et al [2009], Yogatama et al [2015, Arora et al [2018], Whitaker andAnderson [2016], Grosse et al [2012]. Also, there is a line of research to develop sparse representations for multiple modalities simultaneously Yuan et al [2012], Shafiee et al [2015], Gwon et al [2016].…”