MIML-FCN+: Multi-Instance Multi-Label Learning via Fully Convolutional Networks with Privileged Information

Abstract

Multi-instance multi-label (MIML) learning has many interesting applications in computer visions, including multi-object recognition and automatic image tagging. In these applications, additional information such as bounding-boxes, image captions and descriptions is often available during training phrase, which is referred as privileged information (PI). However, as existing works on learning using PI only consider instance-level PI (privileged instances), they fail to make use of bag-level PI (privileged bags) available in MIML learning. Therefore, in this paper, we propose a two-stream fully convolutional network, named MIML-FCN+, unified by a novel PI loss to solve the problem of MIML learning with privileged bags. Compared to the previous works on PI, the proposed MIML-FCN+ utilizes the readily available privileged bags, instead of hard-to-obtain privileged instances, making the system more general and practical in real world applications. As the proposed PI loss is convex and SGDcompatible and the framework itself is a fully convolutional network, MIML-FCN+ can be easily integrated with stateof-the-art deep learning networks. Moreover, the flexibility of convolutional layers allows us to exploit structured correlations among instances to facilitate more effective training and testing. Experimental results on three benchmark datasets demonstrate the effectiveness of the proposed MIML-FCN+, outperforming state-of-the-art methods in the application of multi-object recognition.

Deep Learning Under Privileged Information Using Heteroscedastic Dropout

“…Information Dropout [1]: a regularization method that utilizes injection of multiplicative noise in the activations of a deep neural network (but as a function of the input x, not x ). MIML-FCN [44]: a CNN-based LUPI framework designed for multi-instance problems. Our problem is not multiple instance; however, we still make a comparison for the sake of completeness.…”

Discriminative Multi-View Privileged Information Learning for Image Re-Ranking

“…In the computer vision community, PI, which is interpreted as the auxiliary information about the training data, can be used for learning better recognition systems. Recently, extensive efforts are devoted to exploring PI cues for enhancing the model training in a variety of vision tasks [22,8,13,14,9,10,11,6,5,4].…”

Weakly Supervised Person Re-Identification

“…During the past few years, many related algorithms have been investigated and developed for MIML problems [3,2,16]. The MIML formulation has also been applied in many practical vision domains, such as image annotation [36,25] and classification tasks [37,6,38,41].…”