Open-source deep-learning frameworks are prevalent in designing, training, and deploying deep neural networks (DNNs) on general-purpose computing devices, such as CPU, GPU, and DSP. However, for custom-designed reconfigurable hardware accelerators, there is no existing universal framework, capable of optimizing DNN deployment configuration and guiding the hardware design with specific accuracy and efficiency requirements. In the paper, we proposed a crossplatform framework, which can convert deep-learning models from popular open-source frameworks to intermediate representation and optimize weight/activation dynamic ranges and quantization strategy to achieve better efficiency and accuracy based on a baseline reference design of hardware accelerator. With a few unlabeled data, the proposed framework can analyze the statistical inference information, compare different bit-width impacts, and optimize network structure. We further illustrate the detailed experiment results using the framework, showing mAP and top-1 accuracy loss is less than 1.5% and 1.2% with 12-bit and 8-bit activation-constrained quantization schemes respectively for object detection and image classification.
Video question answering (QA) is a cross-modal task that requires understanding the video content to answer questions. Current techniques address this challenge by employing stacked modules, such as attention mechanisms and graph convolutional networks. These methods reason about the semantics of video features and their interaction with text-based questions, yielding excellent results. However, these approaches often learn and fuse features representing different aspects of the video separately, neglecting the intra-interaction and overlooking the latent complex correlations between the extracted features. Additionally, the stacking of modules introduces a large number of parameters, making model training more challenging. To address these issues, we propose a novel multimodal knowledge distillation method that leverages the strengths of knowledge distillation for model compression and feature enhancement. Specifically, the fused features in the larger teacher model are distilled into knowledge, which guides the learning of appearance and motion features in the smaller student model. By incorporating cross-modal information in the early stages, the appearance and motion features can discover their related and complementary potential relationships, thus improving the overall model performance. Despite its simplicity, our extensive experiments on the widely used video QA datasets, MSVD-QA and MSRVTT-QA, demonstrate clear performance improvements over prior methods. These results validate the effectiveness of the proposed knowledge distillation approach.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.