Toolformer: Language Models Can Teach Themselves to Use Tools

Schick, Timo; Dwivedi-Yu, Jane; Dessì, Roberto; Raileanu, Roberta; Lomelí, María; Zettlemoyer, Luke; Cancedda, Nicola; Scialom, Thomas

doi:10.48550/arxiv.2302.04761

Cited by 56 publications

(70 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Allowing such systems to further exploit execution information could make them more efficient. NPR systems could also be extended in other ways, for example by feeding them data from previous executions, perhaps using mechanisms similar to "scratch pads" [55] or by allowing them to use tools, like the Toolformer [66].…”

Section: Discussionmentioning

confidence: 99%

Can OpenAI's codex fix bugs?

Prenner

Babii

Robbes

2022

Proceedings of the Third International Workshop on Automated Program Repair

View full text Add to dashboard Cite

Recently, we can notice a transition to data-driven techniques in Automated Program Repair (APR), in particular towards deep neural networks. This entails training on hundreds of thousands or even millions of non-executable code fragments. We would like to bring more attention to an aspect of code often neglected in Neural Program Repair (NPR), namely its execution. Code execution has several significant advantages. It allows for test-based evaluation of candidate fixes and can provide valuable information to aid repair. In this work we present a fully executable dataset of 450,000 small buggy/fixed program pairs originally submitted to programming competition websites written in eight different programming languages. Along with the dataset we provide infrastructure to compile, safely execute and test programs as well as fine-grained bug-type labels. To give a point of reference, we provide basic evaluation results for two baselines, one based on a generate-andvalidate approach and one on deep learning. With this dataset we follow several goals: we want to lift Neural Program Repair beyond fully static code representations, foster the use of executionbased features and, by including several different languages, counterbalance the predominance of Java in the current landscape of APR datasets and benchmarks. Keywords automated program repair, data-driven software engineering, fault localizationRecently, more and more APR research builds on deep learning. So much so that this sub-field has been given its own name: Neural Program Repair (NPR). NPR systems are trained on up to millions of buggy/fixed code fragment pairs. So far, there is a strong focus on static code features, in particular textual features [13,47,32], less commonly, tree or graph representations [43,17].Because NPR systems are data hungry and manually collecting and isolating bugs is infeasible on a large scale, copious amounts of bug data are mined from open source code repositories (e.g., GitHub) [71].

show abstract

Section: Discussionmentioning

confidence: 99%

Can OpenAI's codex fix bugs?

Prenner

Babii

Robbes

2022

Proceedings of the Third International Workshop on Automated Program Repair

View full text Add to dashboard Cite

show abstract

“…These advancements enhance the models' ability to discern user intent, rendering them more user-friendly and practical. Moreover, recent studies reveal the potential of LLMs to program and control other digital tools, such as APIs, search engines, and even other generative AI systems (Schick et al, 2023;Mialon et al, 2023;Chase, 2022). This enables seamless integration of individual components for better utility, performance, and generalization.…”

Section: The Advancement Of Large Language Modelsmentioning

confidence: 99%

GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models

Eloundou¹,

Manning²,

Mishkin³

et al. 2023

Preprint

View full text Add to dashboard Cite

We investigate the potential implications of large language models (LLMs), such as Generative Pretrained Transformers (GPTs), on the U.S. labor market, focusing on the increased capabilities arising from LLM-powered software compared to LLMs on their own. Using a new rubric, we assess occupations based on their alignment with LLM capabilities, integrating both human expertise and GPT-4 classifications. Our findings reveal that around 80% of the U.S. workforce could have at least 10% of their work tasks affected by the introduction of LLMs, while approximately 19% of workers may see at least 50% of their tasks impacted. We do not make predictions about the development or adoption timeline of such LLMs. The projected effects span all wage levels, with higher-income jobs potentially facing greater exposure to LLM capabilities and LLM-powered software. Significantly, these impacts are not restricted to industries with higher recent productivity growth. Our analysis suggests that, with access to an LLM, about 15% of all worker tasks in the US could be completed significantly faster at the same level of quality. When incorporating software and tooling built on top of LLMs, this share increases to between 47 and 56% of all tasks. This finding implies that LLM-powered software will have a substantial effect on scaling the economic impacts of the underlying models. We conclude that LLMs such as GPTs exhibit traits of general-purpose technologies, indicating that they could have considerable economic, social, and policy implications.

show abstract

“…An LLM agent employs the capabilities of state-of-the-art language models to perform complex tasks and make informed decisions. These agents can autonomously determine which actions to take, including utilizing various tools and observing their outputs or providing responses to user queries (Schick et al 2023). By leveraging the LLM's vast knowledge and understanding of natural language, agents can efficiently navigate through an array of tools and select the most appropriate one based on the given context.…”

Section: Llm Agentsmentioning

confidence: 99%

“…In the field of climate change, accurate and up-to-date information is of paramount importance to ensure that decision-makers, researchers, and the public can make informed choices and develop effective strategies (Financial Stability Board 2017b). Recent transformer models have started to be employed within the climate change domain, witnessing improved accuracy in typical classification tasks (Kölbel et al 2020;Bingler et al 2022;Callaghan et al 2021;Webersinke et al 2022;Stammbach et al 2022). These models are capable of accounting for the context of words, enabling them to detect complex and implicit topic patterns in addition to many trivial cases.…”

Section: Nlp In Climate Changementioning

confidence: 99%