Enabling Highly Efficient Capsule Networks Processing Through A PIM-Based Architecture Design

Abstract

In recent years, the CNNs have achieved great successes in the image processing tasks, e.g., image recognition and object detection. Unfortunately, traditional CNN's classification is found to be easily misled by increasingly complex image features due to the usage of pooling operations, hence unable to preserve accurate position and pose information of the objects. To address this challenge, a novel neural network structure called Capsule Network has been proposed, which introduces equivariance through capsules to significantly enhance the learning ability for image segmentation and object detection. Due to its requirement of performing a high volume of matrix operations, CapsNets have been generally accelerated on modern GPU platforms that provide highly optimized software library for common deep learning tasks. However, based on our performance characterization on modern GPUs, CapsNets exhibit low efficiency due to the special program and execution features of their routing procedure, including massive unshareable intermediate variables and intensive synchronizations, which are very difficult to optimize at software level. To address these challenges, we propose a hybrid computing architecture design named PIM-CapsNet. It preserves GPU's on-chip computing capability for accelerating CNN types of layers in CapsNet, while pipelining with an off-chip in-memory acceleration solution that effectively tackles routing procedure's inefficiency by leveraging the processing-in-memory capability of today's 3D stacked memory. Using routing procedure's inherent parallellization feature, our design enables hierarchical improvements on CapsNet inference efficiency through minimizing data movement and maximizing parallel processing in memory. Evaluation results demonstrate that our proposed design can achieve substantial improvement on both performance and energy savings for CapsNet inference, with almost zero accuracy loss. The results also suggest good performance scalability in optimizing the routing procedure with increasing network size.

A Study on the Performance of Unconstrained Very Low Resolution Face Recognition: Analyzing Current Trends and New Research Directions

“…Efficient CapsNet research is not present in the state of the art and can benefit from adding robustness and more accurate performance for face recognition in embedded systems. Early research on efficient Capsule Networks [123] has suggested that the major bottleneck is the dynamic routing procedure (which determines which capsule is a vector going to connect). This procedure is very intensive on memory calls.…”

Shifting Capsule Networks from the Cloud to the Deep Edge

“…Attending to the specifications of these devices we estimate that, at maximum power, these MCUs consume less than 1% of the energy consumed by NVIDIA GTX 980 Ti also at maximum power. To get a better insight into the energy 6 Related Work Zhang et al [2020] proposed a hybrid computing architecture to accelerate the routing procedure of CapsNets during inference in GPU platforms. The authors claim that even in modern GPUs, provided with highly optimized software libraries for DL tasks, CapsNets exhibit low efficiency due to their routing procedure.…”

“…In contrast to CNNs, which are invariant to translation, combining these features makes Capsnets equivariant, understanding proportion and pose changing [El Alaoui-Elfels, Omaima and Gadi, Taoufiq, 2021]. As a consequence, CapsNets have shown promising results to solve real-life problems and outperform CNNs in astronomy [Katebi et al, 2019], autonomous cars [Dinesh Kumar, 2018, Pari S. et al, 2019, machine translation [Wang et al, 2018], healthcare [Toraman et al, 2020], and so on [Kwabena Patrick et al, 2019, Zhang et al, 2020.…”

“…Despite being promoted as the next ground-breaking algorithm in DL [Kwabena Patrick et al, 2019], CapsNets in their original form are computationally intensive in terms of memory requirements (bandwidth and size) and energy consumption [Marchisio et al, 2020a,b, 2021, Zhang et al, 2020, Park et al, 2020, Kakillioglu et al, 2020. For the same dataset, the original CapsNet has a ratio of multiply-accumulate (MAC) operations per memory near 100x higher than AlexNet [Marchisio et al, 2020a], which is also a heavy CNN.…”

ClickTrain

“…The ever-increasing scale and complexity of the networks with large-scale training datasets such as ImageNet-2012 [25] are bringing more and more challenges to the cost of DNN training, which requires large amounts of computations and resources such as memory, storage, and I/O [41,58,[70][71][72]. Previous works have also tried some optimization methods to try to overcome these challenges [9,10,17,26,51,59].…”