The great Transformer: Examining the role of large language models in the political economy of AI

Abstract

In recent years, AI research has become more and more computationally demanding. In natural language processing (NLP), this tendency is reflected in the emergence of large language models (LLMs) like GPT-3. These powerful neural network-based models can be used for a range of NLP tasks and their language generation capacities have become so sophisticated that it can be very difficult to distinguish their outputs from human language. LLMs have raised concerns over their demonstrable biases, heavy environmental footprints, and future social ramifications. In December 2020, critical research on LLMs led Google to fire Timnit Gebru, co-lead of the company’s AI Ethics team, which sparked a major public controversy around LLMs and the growing corporate influence over AI research. This article explores the role LLMs play in the political economy of AI as infrastructural components for AI research and development. Retracing the technical developments that have led to the emergence of LLMs, we point out how they are intertwined with the business model of big tech companies and further shift power relations in their favour. This becomes visible through the Transformer, which is the underlying architecture of most LLMs today and started the race for ever bigger models when it was introduced by Google in 2017. Using the example of GPT-3, we shed light on recent corporate efforts to commodify LLMs through paid API access and exclusive licensing, raising questions around monopolization and dependency in a field that is increasingly divided by access to large-scale computing power.

A Roadmap for Big Model

“…[199]. For training GPT-3, the supercomputer developed for OpenAI is a single system with more than 285,000 CPU cores, 10,000 NVIDIA V100 GPUs and 400 gigabits per second of network connectivity for each GPU server [200]. Megatron-Turing Natural Language Generation (MLT-NLG) model is an AI model with a whopping 530 billion parameters [201].…”

The Paradox of Anthroponormative Restriction: Artistic Artificial Intelligence and Literary Writing

“…-Kehlmann's use of CTRL was, one is lead to speculate, in no small part a PR campaign of McCann on behalf of CTRL; given the virtually total insignificance of CTRL compared to other large language models today, the collaboration appears to have been not just an artistic but also a business failure. 4 CTRL has 1.6 billion parameters -or 'neurons' in its neural network -while GPT-3 boasts 175 billion; CTRL was trained on 140 gigabytes of text, GPT-3 on 570 gigabytes; PaLM, introduced by Google in April 2022, has 540 billion parameters and was trained on 780 gigabytes of text. For CTRL, see [8]; for GPT-3, see [9]; for PaLM, see [10].…”

“…In 2021, he turned to an entirely different subject. In the essay 1 A useful introduction to large language models is [4].…”

“…3 Indeed, Kehlmann's collaboration with Bryan McCann, the founder of CTRL, was presented under the heading of 'AI Storytelling', and Kehlmann himself stated that the Even GPT-3, which made a splash in 2020 as the state of the art in text AI, is a hundred times larger and would have produced much better results in terms of coherence; and the most recent models, like WuDao 2.0 or Google's PaLM, are even more comprehensive. 4 More interesting than such technical quibbles is a paradox that is behind Kehlmann's disappointment and that can frequently be discerned in discussions about art-making AI: the more one expects from Artificial Intelligence, the more human it is thought to be, but the less it is appreciated as a phenomenon in its own right; one may call this the paradox of anthroponormative restriction. A truly powerful artistic AI would not extend Kehlmann, but actually replace him -and it would not necessitate new aesthetics, but merely repeat the old ones.…”

ESG investing in the AI era: Features of developed and developing countries

“…While in the past, industrial revolutions improved only production technologies, now-under the conditions of the fourth Industrial Revolution-management technologies are also improved. AI ensures the intellectual support for decision making, in particular investment decisions (Luitse and Denkena, 2021;Som, 2021).…”