Mobius: Fine Tuning Large-Scale Models on Commodity GPU Servers

Feng, Yangyang; Xie, Min; Tian, Zijie; Wang, Shuo; Lu, Youyou; Shu, Jiwu

doi:10.1145/3575693.3575703

Cited by 8 publications

(1 citation statement)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The method proposed by Wang et al [68] involves decomposing the original communication and computational operations into more fine-grained tasks, thereby achieving an overlap between communication and computation that effectively reduces data communication overhead. Mobius [69] introduces a pipeline strategy for heterogeneous memory, which overlaps communication with computation by prefetching data from the CPU to the GPU memory for the next stage. Additionally, it employs a Cross-mapping strategy to reduce communication contention, further optimizing overall performance.…”

Section: Communication Optimizationmentioning

confidence: 99%

Training and Serving System of Foundation Models: A Comprehensive Survey

Zhou,

Chen,

Hong

et al. 2024

IEEE Open J. Comput. Soc.

View full text Add to dashboard Cite

Foundation models (e.g., ChatGPT, DALL-E, PengCheng Mind, PanGu-Σ) have demonstrated extraordinary performance in key technological areas, such as natural language processing and visual recognition, and have become the mainstream trend of artificial general intelligence. This has led more and more major technology giants to dedicate significant human and financial resources to actively develop their foundation model systems, which drives continuous growth of these models' parameters. As a result, the training and serving of these models have posed significant challenges, including substantial computing power, memory consumption, bandwidth demands, etc. Therefore, employing efficient training and serving strategies becomes particularly crucial. Many researchers have actively explored and proposed effective methods. So, a comprehensive survey of them is essential for system developers and researchers. This paper extensively explores the methods employed in training and serving foundation models from various perspectives. It provides a detailed categorization of these state-of-the-art methods, including finer aspects such as network, computing, and storage. Additionally, the paper summarizes the challenges and presents a perspective on the future development direction of foundation model systems. Through comprehensive discussion and analysis, it hopes to provide a solid theoretical basis and practical guidance for future research and applications, promoting continuous innovation and development in foundation model systems.

show abstract

Section: Communication Optimizationmentioning

confidence: 99%