Tree-like Decision Distillation

Song, Jie; Zhang, Haofei; Wang, Xinchao; Xue, Mengqi; Chen, Ying; Li, Sun; Tao, Dacheng; Song, Mingli

doi:10.1109/cvpr46437.2021.01328

Cited by 19 publications

(10 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Over the last several years, most works devote themselves to the exploration of "what to distill", i.e., forms of knowledge for distillation. Representative forms include soft targets [1], features [6], [12], attention [7], factors [9], activation boundary [10], sample relationship [3], [13], [14] and so on. By imitating the teacher to behave in a similar way, the student model achieves comparable performance to the teacher model even with much fewer parameters.…”

Section: A Knowledge Distillationmentioning

confidence: 99%

Spot-Adaptive Knowledge Distillation

Song

Chen

et al. 2022

IEEE Trans. on Image Process.

Self Cite

View full text Add to dashboard Cite

Knowledge distillation (KD) has become a well established paradigm for compressing deep neural networks. The typical way of conducting knowledge distillation is to train the student network under the supervision of the teacher network to harness the knowledge at one or multiple spots (i.e., layers) in the teacher network. The distillation spots, once specified, will not change for all the training samples, throughout the whole distillation process. In this work, we argue that distillation spots should be adaptive to training samples and distillation epochs. We thus propose a new distillation strategy, termed spotadaptive KD (SAKD), to adaptively determine the distillation spots in the teacher network per sample, at every training iteration during the whole distillation period. As SAKD actually focuses on "where to distill" instead of "what to distill" that is widely investigated by most existing works, it can be seamlessly integrated into existing distillation methods to further improve their performance. Extensive experiments with 10 state-of-theart distillers are conducted to demonstrate the effectiveness of SAKD for improving their distillation performance, under both homogeneous and heterogeneous distillation settings. Code is available at https://github.com/zju-vipa/spot-adaptive-pytorch.

show abstract

Section: A Knowledge Distillationmentioning

confidence: 99%

Spot-Adaptive Knowledge Distillation

Song

Chen

et al. 2022

IEEE Trans. on Image Process.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Our goal is to teach a lightweight RL-CC tree-based policy that imitates a NN-based policy over a representative distribution of inputs. Previous work has shown that model distillation with trees can work well on various tasks [6,12,27,31,43]. The biggest challenge in distilling the policy is the LSTM layer, which specializes in incorporating past information.…”

Section: Model Distillation With Boosting Treesmentioning

confidence: 99%

Implementing Reinforcement Learning Datacenter Congestion Control in NVIDIA NICs

Fuhrer¹,

Yuval²,

Tessler³

et al. 2022

Preprint

View full text Add to dashboard Cite

Cloud datacenters are exponentially growing both in numbers and size. This increase results in a network activity surge that warrants better congestion avoidance. The resulting challenge is two-fold: (i) designing algorithms that can be custom-tuned to the complex traffic patterns of a given datacenter; but, at the same time (ii) run on low-level hardware with the required low latency of effective Congestion Control (CC). In this work, we present a Reinforcement Learning (RL) based CC solution that learns from certain traffic scenarios and successfully generalizes to others. We then distill the RL neural network policy into binary decision trees to achieve the desired 𝜇sec decision latency required for realtime inference with RDMA. We deploy the distilled policy on NVIDIA NICs in a real network and demonstrate state-of-theart performance, balancing all tested metrics simultaneously: bandwidth, latency, fairness, and packet drops.

show abstract

“…They point out that the soft targets which reveal how teachers tend to generalize, regularize the student. To sufficiently distill knowledge, FitNets [3] additionally use intermediate features (also known as hints) of the teacher and the following works [4]- [6] extract deeper level of information from the intermediate layers of teachers to supervise the student in different aspects.…”

Section: B Model Reusingmentioning

confidence: 99%

“…Following this teacher-student paradigm, FitNets [3] utilize intermediate features, also known as hints, to sufficiently supervise hidden layers of the student. Besides, a number of methods [4]- [6] have managed to extract deeper level of information from the intermediate layers and demonstrated promising results. Except for image classification, researchers are exploring KD towards many other tasks, such as semantic segmentation [7], [8]; object detection [9]- [12]; natural language processing [13]- [15]; and reinforcement learning [16], [17].…”

Section: Introductionmentioning

confidence: 99%

Knowledge Amalgamation for Object Detection with Transformers

Zhang¹,

Mao²,

Xue³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Knowledge amalgamation (KA) is a novel deep model reusing task aiming to transfer knowledge from several well-trained teachers to a multi-talented and compact student. Currently, most of these approaches are tailored for convolutional neural networks (CNNs). However, there is a tendency that transformers, with a completely different architecture, are starting to challenge the domination of CNNs in many computer vision tasks. Nevertheless, directly applying the previous KA methods to transformers leads to severe performance degradation. In this work, we explore a more effective KA scheme for transformer-based object detection models. Specifically, considering the architecture characteristics of transformers, we propose to dissolve the KA into two aspects: sequence-level amalgamation (SA) and task-level amalgamation (TA). In particular, a hint is generated within the sequence-level amalgamation by concatenating teacher sequences instead of redundantly aggregating them to a fixed-size one as previous KA works. Besides, the student learns heterogeneous detection tasks through soft targets with efficiency in the tasklevel amalgamation. Extensive experiments on PASCAL VOC and COCO have unfolded that the sequence-level amalgamation significantly boosts the performance of students, while the previous methods impair the students. Moreover, the transformerbased students excel in learning amalgamated knowledge, as they have mastered heterogeneous detection tasks rapidly and achieved superior or at least comparable performance to those of the teachers in their specializations.

show abstract

Tree-like Decision Distillation

Cited by 19 publications

References 20 publications

Spot-Adaptive Knowledge Distillation

Spot-Adaptive Knowledge Distillation

Implementing Reinforcement Learning Datacenter Congestion Control in NVIDIA NICs

Knowledge Amalgamation for Object Detection with Transformers

Contact Info

Product

Resources

About