TinyADC: Peripheral Circuit-aware Weight Pruning Framework for Mixed-signal DNN Accelerators

Yuan, Geng; Behnam, Payman; Cai, Yuxuan; Shafiee, Ali; Fu, Jingyan; Liao, Zhiheng; Li, Zhengang; Ma, Xiaolong; Deng, Jieren; Wang, Jinhui; Bojnordi, Mahdi Nazm; Wang, Yanzhi; Ding, Caiwen

doi:10.23919/date51398.2021.9474235

Cited by 20 publications

(5 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, the limited computing resources and capabilities of these devices render it difficult to meet realtime requirements while accomplishing complex tasks. While neural network compression techniques-e.g., pruning [59,60] and quantization [61,62]-together with operating system level optimizations [63,64] are the potential solutions for cost-effective continual learning when using edge devices [65], it is important to note that the research to date has mainly focused on computer vision tasks.…”

Section: Energy Efficiency and Computation Capabilitymentioning

confidence: 99%

Continual Learning with Deep Neural Networks in Physiological Signal Data: A Survey

Li,

Yuan

2024

Healthcare

Self Cite

View full text Add to dashboard Cite

Deep-learning algorithms hold promise in processing physiological signal data, including electrocardiograms (ECGs) and electroencephalograms (EEGs). However, healthcare often requires long-term monitoring, posing a challenge to traditional deep-learning models. These models are generally trained once and then deployed, which limits their ability to adapt to the dynamic and evolving nature of healthcare scenarios. Continual learning—known for its adaptive learning capabilities over time—offers a promising solution to these challenges. However, there remains an absence of consolidated literature, which reviews the techniques, applications, and challenges of continual learning specific to physiological signal analysis, as well as its future directions. Bridging this gap, our review seeks to provide an overview of the prevailing techniques and their implications for smart healthcare. We delineate the evolution from traditional approaches to the paradigms of continual learning. We aim to offer insights into the challenges faced and outline potential paths forward. Our discussion emphasizes the need for benchmarks, adaptability, computational efficiency, and user-centric design in the development of future healthcare systems.

show abstract

Section: Energy Efficiency and Computation Capabilitymentioning

confidence: 99%

Continual Learning with Deep Neural Networks in Physiological Signal Data: A Survey

Li,

Yuan

2024

Healthcare

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, it induces remarkable hardware overhead as all mechanisms such as row indexing, routing controls, and word-line controls are hardware-managed. TinyADC [40] proposes a pruning solution that fixes the number of non-zero weights in each column of the ReRAM crossbar while their positions can vary. This helps to decrease the accumulated value and required ADC resolution.…”

Section: A Reram Crossbar and Reram-based Dnn Accelerationmentioning

confidence: 99%

“…However, the ADC does not scale as fast as the CMOS technology does [18,42]. The recent study [40,42] reports the ADC/DAC blocks may become the major contributor to the total chip area and power. To save power and area, many designs share an ADC with many crossbar columns (e.g., 128 columns in ISAAC).…”

Section: B Challengesmentioning

confidence: 99%

FORMS: Fine-grained Polarized ReRAM-based In-situ Computation for Mixed-signal DNN Accelerator

Yuan

Behnam

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recent work demonstrated the promise of using resistive random access memory (ReRAM) as an emerging technology to perform inherently parallel analog domain in-situ matrix-vector multiplication-the intensive and key computation in deep neural networks (DNNs). One key problem is the weights that are signed values. However, in a ReRAM crossbar, weights are stored as conductance of the crossbar cells, and the in-situ computation assumes all cells on each crossbar column are of the same sign. The current architectures either use two ReRAM crossbars for positive and negative weights (PRIME), or add an offset to weights so that all values become positive (ISAAC). Neither solution is ideal: they either double the cost of crossbars, or incur extra offset circuity. To better address this problem, we propose FORMS, a fine-grained ReRAM-based DNN accelerator with algorithm/hardware co-design. Instead of trying to represent the positive/negative weights, our key design principle is to enforce exactly what is assumed in the in-situ computationensuring that all weights in the same column of a crossbar have the same sign. It naturally avoids the cost of an additional crossbar. Such polarized weights can be nicely generated using alternating direction method of multipliers (ADMM) regularized optimization during the DNN training, which can exactly enforce certain patterns in DNN weights. To achieve high accuracy, we divide the crossbar into logical sub-arrays and only enforce this property within the fine-grained sub-array columns. Crucially, the small sub-arrays provides a unique opportunity for input zeroskipping, which can significantly avoid unnecessary computations and reduce computation time. At the same time, it also makes the hardware much easier to implement and is less susceptible to nonidealities and noise than coarse-grained architectures. Putting all together, with the same optimized DNN models, FORMS achieves 1.50× and 1.93× throughput improvement in terms of GOP s s×mm 2 and GOP s W compared to ISAAC, and 1.12× ∼ 2.4× speed up in terms of frame per second over optimized ISAAC with almost the same power/area cost. Interestingly, FORMS optimization framework can even speed up the original ISAAC from 10.7× up to 377.9×, reflecting the importance of software/hardware co-design optimizations.

show abstract

“…Deep Neural Networks have played a significant role in advancing many applications (Yuan et al, 2021;Ding et al, 2017). The field of Natural Language Processing (NLP) leverages Recurrent Neural Networks (RNNs) and Transformers to achieve outstanding performance on many tasks.…”

Section: Introductionmentioning

confidence: 99%

A Secure and Efficient Federated Learning Framework for NLP

Wang¹,

Deng²,

Meng³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

Self Cite

View full text Add to dashboard Cite

In this work, we consider the problem of designing secure and efficient federated learning (FL) frameworks. Existing solutions either involve a trusted aggregator or require heavyweight cryptographic primitives, which degrades performance significantly. Moreover, many existing secure FL designs work only under the restrictive assumption that none of the clients can be dropped out from the training protocol. To tackle these problems, we propose SEFL, a secure and efficient FL framework that (1) eliminates the need for the trusted entities; (2) achieves similar and even better model accuracy compared with existing FL designs; (3) is resilient to client dropouts. Through extensive experimental studies on natural language processing (NLP) tasks, we demonstrate that the SEFL achieves comparable accuracy compared to existing FL solutions, and the proposed pruning technique can improve runtime performance up to 13.7×.

show abstract

TinyADC: Peripheral Circuit-aware Weight Pruning Framework for Mixed-signal DNN Accelerators

Cited by 20 publications

References 13 publications

Continual Learning with Deep Neural Networks in Physiological Signal Data: A Survey

Continual Learning with Deep Neural Networks in Physiological Signal Data: A Survey

FORMS: Fine-grained Polarized ReRAM-based In-situ Computation for Mixed-signal DNN Accelerator

A Secure and Efficient Federated Learning Framework for NLP

Contact Info

Product

Resources

About