Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles

Abstract

Abstract. In this paper we study the problem of image representation learning without human annotation. By following the principles of selfsupervision, we build a convolutional neural network (CNN) that can be trained to solve Jigsaw puzzles as a pretext task, which requires no manual labeling, and then later repurposed to solve object classification and detection. To maintain the compatibility across tasks we introduce the context-free network (CFN), a siamese-ennead CNN. The CFN takes image tiles as input and explicitly limits the receptive field (or context) of its early processing units to one tile at a time. We show that the CFN includes fewer parameters than AlexNet while preserving the same semantic learning capabilities. By training the CFN to solve Jigsaw puzzles, we learn both a feature mapping of object parts as well as their correct spatial arrangement. Our experimental evaluations show that the learned features capture semantically relevant content. Our proposed method for learning visual representations outperforms state of the art methods in several transfer learning benchmarks.

DeepPermNet: Visual Permutation Learning

“…Interestingly, when finer grid cells are used (e.g., 4 × 4), we do not observe any improvement in recognition performance. Moreover, the first layer filters learned by our method and the ones learned by Noroozi and Favaro [32] seems somewhat similar (see Figure 5) despite the different number of permutations learned. This fact suggests that 3 × 3 grid partition and a well chosen subset of permutations are enough to learn filters which produce state-of-theart results for self-supervised representation learning.…”

“…This fact suggests that 3 × 3 grid partition and a well chosen subset of permutations are enough to learn filters which produce state-of-theart results for self-supervised representation learning. However, DeepPermNet is a generic method than the method proposed by Noroozi and Favaro [32], and our method can be used to solve many different computer vision tasks as shown in our experiments.…”

“…A recent work closely related to ours is Noroozi and Favaro [32] that also proposes to train CNNs for solving image-based jigsaw puzzles. However, different from us, they train a CNN to predict only a tiny subset of possible permutations generated from an image shuffling grid of size 3 × 3 (specifically, they use only 100 permutations from 362k possible permutations).…”

“…Following the literature on self-supervised pre-training [2] 52.9 41.8 -Doersch et al [10] 55.3 46.6 -Wang and Gupta [45] 58.4 44.0 -Pathak et al [34] 56.5 44.5 29.7 Donahue et al [11] 58.9 45.7 34.9 Zhang et al [48] 65.6 47.9 35.6 Noroozi and Favaro [32] 68 [10,11,34,32], we test our models on the commonly used self-supervised benchmarks on the PASCAL Visual Object Challenge and compare against supervised and selfsupervised procedures for pre-training. We transfer our learned weights to initialize from Conv1 to Conv5 layers of AlexNet [24], Fast-RCNN [17] and Fully Convolutional Network [27] models and fine-tune them for object classification, object detection, and object segmentation tasks respectively, using their default training parameters.…”

“…Instead it uses the spatial structure as a supervisory signal. Such a learning task is popularly known as self-supervised learning [10,16,30,32], and is very useful to learn rich features, especially in the context of training deep learning models, which often require large amounts of human annotated datasets.…”