PanGu-Coder: Program Synthesis with Function-Level Language Modeling

Abstract

We present PANGU-CODER, a pretrained decoder-only language model adopting the PANGU-α architecture for text-to-code generation, i.e. the synthesis of programming language solutions given a natural language problem description. We train PANGU-CODER using a two-stage strategy: the first stage employs Causal Language Modelling (CLM) to pre-train on raw programming language data, while the second stage uses a combination of Causal Language Modelling and Masked Language Modelling (MLM) training objectives that focus on the downstream task of text-to-code generation and train on loosely curated pairs of natural language program definitions and code functions. Finally, we discuss PANGU-CODER-FT, which is fine-tuned on a combination of competitive programming problems and code with continuous integration tests. We evaluate PANGU-CODER with a focus on whether it generates functionally correct programs and demonstrate that it achieves equivalent or better performance than similarly sized models, such as CodeX [16], while attending a smaller context window and training on less data.

From Large Language Models to Large Multimodal Models: A Literature Review

“…It introduces an input mechanism that employs distinct embeddings for distinct domains, which is coupled with a two-level routing design in the Random Routed Experts (RRE) framework. The pretraining corpus of Pangu-Σ, totalling 329 billion tokens, primarily encompasses diverse data formats of bilingual Chines-English, content from [24,47,48] and code from [49,50].…”

“…Chinese companies adopt an ecosystem-diverse and systematic approach, typically rolling out a series of models to create a holistic technological ecosystem. Examples include Baidu's ERNIE series [164][165][166] of large models and its derivative ERNIEBot, Huawei's PanGu series [10,11,49] model, Alibaba's Tongyi series, Tencent's HunYuan series. Additionally, Chinese universities actively participate in large model development and research, partner-ing with tech companies or independently creating multiple large models, with Tsinghua University's CPM [36,37,167] series and GLM [168] being one such instance.…”

Deep reinforcement with spectrum series learning control for a mode-locked fiber laser

“…Recent research has delved into leveraging pretrained large language models (LLMs) from the natural language processing (NLP) field to automate program synthesis tasks, using vast-scale code corpus data mined from open-source repositories. Notably, there are several prominent examples of such pretrained models including the encoder-only CodeBERT (Feng et al, 2020), decoder-only CodeGPT (Lu et al, 2021), Code-Gen (Nijkamp et al, 2022), PaLM-Coder (Chowdhery et al, 2022), PanGu-Coder (Christopoulou et al, 2022), CodeGeex (Zheng et al, 2023), andSantaCoder (Allal et al, 2023), as well as encoder-decoder transformer architectures like PLABRT (Ahmad et al, 2021) and CodeT5 (Wang et al, 2021). These pretrained probabilistic language (PL) models are already capable of generating code that appears visually impressive and well-structured.…”

“…AlphaCode , for instance, aspires to address competitive-level programming challenges, while InCoder (Fried et al, 2022) enables code insertion at arbitrary junctures utilizing bidirectional contexts. Other acclaimed models include CodeT5 (Wang et al, 2021), CodeGen (Nijkamp et al, 2022), PaLM-Coder (Chowdhery et al, 2022), PanGu-Coder (Christopoulou et al, 2022), CodeGeex (Zheng et al, 2023), andSantaCoder (Allal et al, 2023). As the size of these LLMs increases, they demonstrate emergent competencies, including human-like programming prowess and debugging aptitude Saunders et al, 2022).…”

Predicting typeScript type annotations and definitions with machine learning

“…F. Xu et al, 2022]. In particular, large language models (LLMs) have been successful in a variety of code generation tasks [Athiwaratkun et al, 2023;Austin et al, 2021;Cassano, Gouwar, et al, 2023;Christopoulou et al, 2022;Izadi et al, 2022;Nijkamp et al, 2023;F. F. Xu et al, 2022].…”