FACT: FFN-Attention Co-optimized Transformer Architecture with Eager Correlation Prediction

Qin, Yubin; Wang, Yang; Deng, Dazheng; Zhao, Zijian; Yang, Xuan; Liu, Leibo; Wei, Shaojun; Hu, Yang; Yin, Shouyi

doi:10.1145/3579371.3589057

Cited by 9 publications

(3 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some work [17,24,32,34,44,45] lay more emphasis on accelerating sparse attention. They design specialized architectures to fully utilize the pre-defined static attention pattern [17,32] or dynamically generated attention pattern [34,37,44,45]. Recently, FACT [37] points out the importance of compressing linear layers with mixed-precision quantization to help reduce latency.…”

Section: Related Workmentioning

confidence: 99%

“…They design specialized architectures to fully utilize the pre-defined static attention pattern [17,32] or dynamically generated attention pattern [34,37,44,45]. Recently, FACT [37] points out the importance of compressing linear layers with mixed-precision quantization to help reduce latency. However, these methods cannot accelerate the decode stage of LLMs since they mainly focus on the prefill stage for discriminative models, like medium-sized BERT [16] model.…”

Section: Related Workmentioning

confidence: 99%

“…SOTA Accelerator Baselines. We also selected three domainspecific accelerators targeting at accelerating attention mechanism: DFX [25], FACT [37], and CTA [44]. It has to be mentioned that DFX is a multi-FPGA acceleration work, and we only evaluate its hardware performance of a single card.…”

Section: Evaluation 61 Evaluation Setupmentioning

confidence: 99%

See 2 more Smart Citations

FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs

Zeng,

Liu,

Dai

et al. 2024

Proceedings of the 2024 ACM/SIGDA International Symposium on Field Programmable Gate Arrays

View full text Add to dashboard Cite

Transformer-based Large Language Models (LLMs) have made a significant impact on various domains. However, LLMs' efficiency suffers from both heavy computation and memory overheads. Compression techniques like sparsification and quantization are commonly used to mitigate the gap between LLM's computation/memory overheads and hardware capacity. However, existing GPU and transformer-based accelerators cannot efficiently process compressed LLMs, due to the following unresolved challenges: low computational efficiency, underutilized memory bandwidth, and large compilation overheads.This paper proposes FlightLLM, enabling efficient LLMs inference with a complete mapping flow on FPGAs. In FlightLLM, we highlight an innovative solution that the computation and memory overhead of LLMs can be solved by utilizing FPGA-specific resources (e.g., DSP48 and heterogeneous memory hierarchy). We propose a configurable sparse DSP chain to support different sparsity patterns with high computation efficiency. Second, we propose an always-on-chip decode scheme to boost memory bandwidth with mixed-precision support. Finally, to make FlightLLM available

show abstract