Cerebras Architecture Deep Dive: First Look Inside the HW/SW Co-Design for Deep Learning : Cerebras Systems

Lie, Sean

doi:10.1109/hcs55958.2022.9895479

Cited by 15 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Clear examples are mapping and resource allocation techniques, whose success greatly depend on aspects such as the algorithm structure, qubit connectivity, gate fidelities, or the or the cost of moving qubits within and across chips. Similarly to recent trends in co-design in deep learning [25], one could conceive techniques where the algorithms, the compiler, and the runtime techniques are co-designed with the interconnect.…”

Section: Discussionmentioning

confidence: 99%

Interconnect Fabrics for Multi-Core Quantum Processors

Escofet,

Rached,

Rodrigo

et al. 2023

Proceedings of the 16th International Workshop on Network on Chip Architectures

View full text Add to dashboard Cite

Quantum computing has revolutionized the field of computer science with its extraordinary ability to handle classically intractable problems. To realize its potential, however, quantum computers need to scale to millions of qubits, a feat that will require addressing fascinating yet extremely challenging interconnection problems. In this paper, we provide a context analysis of the nascent quantum computing field from the perspective of communications, with the aim of encouraging the on-chip networks community to contribute and pave the way for truly scalable quantum computers in the decades to come. CCS CONCEPTS• Hardware → Quantum computation; Interconnect; • Computer systems organization → Multicore architectures; • Networks → Network on chip.

show abstract

Section: Discussionmentioning

confidence: 99%

Interconnect Fabrics for Multi-Core Quantum Processors

Escofet,

Rached,

Rodrigo

et al. 2023

Proceedings of the 16th International Workshop on Network on Chip Architectures

View full text Add to dashboard Cite

show abstract

“…In this section the focus will be on Cerebras Systems Wafer Scale Engine (WSE) [11,12] as it is the most advanced large scale silicon die currently made. This is a system where the processor itself if of the size of a wafer and is primarily made for ML models and computations.…”

Section: Large Scale Silicon Diementioning

confidence: 99%

Performance Scaling of Silicon after Moore’s Law

Venkatachalam

2023

Preprint

View full text Add to dashboard Cite

Semiconductors play a very important role in modern society. The improvement in performance of semiconductors is what has enabled the world to grow at such a fast pace. For decades scaling has been done through Moore’s law i.e performance improvements have been obtained due to transistor scaling where the number of transistors per unit area increase is what has contributed to the increase in performance. However, with transistor scaling reaching its limitations we need to find alternate methods to enhance silicon design to suit the computational need of modern society. To address this problem we will be looking into a few methods that enable performance scaling with a focus on analysing the most popular implementation of each in the industry.

show abstract

“…Companies like Graphcore [50], Cerebras [52], Groq [27], and earlier generations of SambaNova's RDU [63], [64] offer alternate AI accelerators. However, they all lack the three-tier memory system required to execute CoEs efficiently.…”

Section: Related Workmentioning

confidence: 99%

SambaNova SN10 RDU:Accelerating Software 2.0 with Dataflow

Prabhakar

Jairath²

2021

2021 IEEE Hot Chips 33 Symposium (HCS)

View full text Add to dashboard Cite

Monolithic large language models (LLMs) like GPT-4 have paved the way for modern generative AI applications. Training, serving, and maintaining monolithic LLMs at scale, however, remains prohibitively expensive and challenging. The disproportionate increase in computeto-memory ratio of modern AI accelerators have created a memory wall, necessitating new methods to deploy AI. Recent research has shown that a composition of many smaller expert models, each with several orders of magnitude fewer parameters, can match or exceed the capabilities of monolithic LLMs. Composition of Experts (CoE) is a modular approach that lowers the cost and complexity of training and serving. However, this approach presents two key challenges when using conventional hardware: (1) without fused operations, smaller models have lower operational intensity, which makes high utilization more challenging to achieve; and (2) hosting a large number of models can be either prohibitively expensive or slow when dynamically switching between them.In this paper, we describe how combining CoE, streaming dataflow, and a three-tier memory system scales the AI memory wall. We describe Samba-CoE, a CoE system with 150 experts and a trillion total parameters. We deploy Samba-CoE on the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) -a commercial dataflow accelerator architecture that has been codesigned for enterprise inference and training applications. The chip introduces a new three-tier memory system with on-chip distributed SRAM, on-package HBM, and off-package DDR DRAM. A dedicated inter-RDU network enables scaling up and out over multiple sockets. We demonstrate speedups ranging from 2× to 13× on various benchmarks running on eight RDU sockets compared with an unfused baseline. We show that for CoE inference deployments, the 8-socket RDU Node reduces machine footprint by up to 19×, speeds up model switching time by 15× to 31×, and achieves an overall speedup of 3.7× over a DGX H100 and 6.6× over a DGX A100.

show abstract

Cerebras Architecture Deep Dive: First Look Inside the HW/SW Co-Design for Deep Learning : Cerebras Systems

Cited by 15 publications

References 0 publications

Interconnect Fabrics for Multi-Core Quantum Processors

Interconnect Fabrics for Multi-Core Quantum Processors

Performance Scaling of Silicon after Moore’s Law

SambaNova SN10 RDU:Accelerating Software 2.0 with Dataflow

Contact Info

Product

Resources

About