Block Walsh–Hadamard Transform-based Binary Layers in Deep Neural Networks

Abstract

Convolution has been the core operation of modern deep neural networks. It is well-known that convolutions can be implemented in the Fourier Transform domain. In this paper, we propose to use binary block Walsh-Hadamard transform (WHT) instead of the Fourier transform. We use WHT-based binary layers to replace some of the regular convolution layers in deep neural networks. We utilize both one-dimensional (1-D) and two-dimensional (2-D) binary WHTs in this paper. In both 1-D and 2-D layers, we compute the binary WHT of the input feature map and denoise the WHT domain coefficients using a nonlinearity which is obtained by combining soft-thresholding with the tanh function. After denoising, we compute the inverse WHT. We use 1D-WHT to replace the 1 × 1 convolutional layers, and 2D-WHT layers can replace the 3 × 3 convolution layers and Squeeze-and-Excite layers. 2D-WHT layers with trainable weights can be also inserted before the Global Average Pooling (GAP) layers to assist the dense layers. In this way, we can reduce the number of trainable parameters significantly with a slight decrease in trainable parameters. In this paper, we implement the WHT layers into MobileNet-V2, MobileNet-V3-Large, and ResNet to reduce the number of parameters significantly with negligible accuracy loss. Moreover, according to our speed test, the 2D-FWHT layer runs about 24 times as fast as the regular 3 × 3 convolution with 19.51% less RAM usage in an NVIDIA Jetson Nano experiment.

ADC/DAC-Free Analog Acceleration of Deep Neural Networks With Frequency Transformation

“…WHT presents a computational challenge when the dimension of the input vector is not a power of two. A technique called blockwise WHTs (BWHTs) was introduced to address this issue in [28]. The BWHT approach divides the transform matrix into multiple blocks, each sized to an integer power of two.…”

“…On the other hand, the projection operation employs a 1-D-BWHT layer to reduce the dimensional to make the network computationally efficient while retaining essential features. In Pan et al's study [28], these transformations maintained a matching accuracy under frequency transforms while achieving significant compression than standard implementation on benchmark datasets such as CIFAR-10, CIFAR-100, and ImageNet. The number of parameters in the BWHT layer is thus proportional to the thresholding parameter T , which is significantly smaller than the number of parameters in a 1 × 1 convolution layer.…”

“…Moreover, WHT, as considered in this example, is well-suited for low-power and computationally efficient processing, as the transformation matrices only consist of binary values. More detailed characterization of WHT-based frequency transformation was presented in [28] and [29]. Motivated by these findings, this work primarily focuses on WHT-based model compression.…”