A Random CNN Sees Objects: One Inductive Bias of CNN and Its Applications

Abstract

This paper starts by revealing a surprising finding: without any learning, a randomly initialized CNN can localize objects surprisingly well. That is, a CNN has an inductive bias to naturally focus on objects, named as Tobias ("The object is at sight") in this paper. This empirical inductive bias is further analyzed and successfully applied to self-supervised learning (SSL). A CNN is encouraged to learn representations that focus on the foreground object, by transforming every image into various versions with different backgrounds, where the foreground and background separation is guided by Tobias. Experimental results show that the proposed Tobias significantly improves downstream tasks, especially for object detection. This paper also shows that Tobias has consistent improvements on training sets of different sizes, and is more resilient to changes in image augmentations.

DisWOT: Student Architecture Search for Distillation WithOut Training

“…To clarify this, we carefully analyze the disparities in vanilla and distillation performance for each model: (1) for overall search space, vanilla accuracy only preserves 85% correlations with actual distillation performance. (2) for a particular instance, as shown in Figure 1 (Right), ResNet20 with 3 res-blocks in each stage (i.e., ResNet [3,3,3]) has more parameters and better standalone performance but is weaker than ResNet [7,1,3] in the distillation process. Considering that ResNet [7,1,3] has more layers than ResNet20, we seek to understand the above phenomenon regarding the vanilla-distillation accuracy gap from the perspective of semantic matching [42].…”

“…(2) for a particular instance, as shown in Figure 1 (Right), ResNet20 with 3 res-blocks in each stage (i.e., ResNet [3,3,3]) has more parameters and better standalone performance but is weaker than ResNet [7,1,3] in the distillation process. Considering that ResNet [7,1,3] has more layers than ResNet20, we seek to understand the above phenomenon regarding the vanilla-distillation accuracy gap from the perspective of semantic matching [42]. ResNet [7,1,3] enjoys a larger effective receptive field and more excellent matched knowledge with teacher, resulting in significant distillation gains.…”

“…Considering that ResNet [7,1,3] has more layers than ResNet20, we seek to understand the above phenomenon regarding the vanilla-distillation accuracy gap from the perspective of semantic matching [42]. ResNet [7,1,3] enjoys a larger effective receptive field and more excellent matched knowledge with teacher, resulting in significant distillation gains. Encouraged by this understanding, we strive to design a new zero-proxy regarding the semantic matching of teacher-student.…”

“…To address this issue, Knowledge Distillation (KD) has been proposed as a means of transferring knowledge from a high-capacity teacher model to a low-capacity target student model, providing a more optimal accuracy-efficiency trade-off during runtime [5,8,84]. The [7,1,3] and ResNet [3,3,3] on search space S0. original KD method [26] utilizes the logit outputs of the teacher network as the source of knowledge.…”

Multiple Thinking Achieving Meta-Ability Decoupling for Object Navigation

“…Numerous techniques have been developed to advance this field by incorporating different inductive biases (Figure 1 (a)) due to the task's complexity. However, regrettably, the object navigation field does not form a unified inductive bias paradigm similar to the CV (Cao & Wu, 2022;d'Ascoli et al, 2021) or NLP (Levine et al, 2022;Kharitonov & Chaabouni, 2021) fields. Inspired by the flaw, through the induction and sublimation of the current mainstream methods, we propose a meta-ability decoupling (MAD) paradigm, hoping to unify and connect various object navigation methods.…”