The hardware lottery

Hooker, Sara

doi:10.1145/3467017

Cited by 72 publications

(43 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One common method in this space is to operate over a fully connected graph, i.e. setting N u = V. This allows the model to rediscover the edges it needs, and has become quite popular in the context of graph Transformers (Ying et al, 2021;Kreuzer et al, 2021;Mialon et al, 2021), which allows for building GNNs that are able to "win the hardware lottery" (Hooker, 2021). Conveniently, the fully connected view also encompasses spectrally defined graph convolutions, such as the graph Fourier transform (Bruna et al, 2013).…”

Section: Graph Rewiringmentioning

confidence: 99%

Message passing all the way up

Veličković¹

2022

Preprint

View full text Add to dashboard Cite

The message passing framework is the foundation of the immense success enjoyed by graph neural networks (GNNs) in recent years. In spite of its elegance, there exist many problems it provably cannot solve over given input graphs. This has led to a surge of research on going "beyond message passing", building GNNs which do not suffer from those limitations-a term which has become ubiquitous in regular discourse. However, have those methods truly moved beyond message passing? In this position paper, I argue about the dangers of using this termespecially when teaching graph representation learning to newcomers. I show that any function of interest we want to compute over graphs can, in all likelihood, be expressed using pairwise message passing -just over a potentially modified graph, and argue how most practical implementations subtly do this kind of trick anyway. Hoping to initiate a productive discussion, I propose replacing "beyond message passing" with a more tame term, "augmented message passing".

show abstract

Section: Graph Rewiringmentioning

confidence: 99%

Message passing all the way up

Veličković¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Consider for instance how the availability of projects like scikit-learn, TensorFlow, PyTorch, and many others, allowed for a wide ML adoption and quicker improvement of models through more standardized processes: where before implementing a ML model required years of work for highly skilled ML researchers, now the same can be accomplished in few lines of code that most developers would be able to write. In a recent paper Hooker [2020] argues that availability of accelerator hardware determines the success of ML algorithms potentially more than their intrinsic merits. We agree with that assessment, and we add that availability of easy to use software packages tailored to ML algorithms has been at least as important for their success and adoption, if not more important.…”

Section: Software Engineering Meets MLmentioning

confidence: 99%

Declarative Machine Learning Systems

Molino

Ré

2021

Queue

View full text Add to dashboard Cite

The people training and using ML models now are typically experienced developers with years of study working within large organizations, but the next wave of ML systems should allow a substantially larger number of people, potentially without any coding skills, to perform the same tasks. These new ML systems will not require users to fully understand all the details of how models are trained and used for obtaining predictions, but will provide them a more abstract interface that is less demanding and more familiar. Declarative interfaces are well-suited for this goal, by hiding complexity and favoring separation of interest, and ultimately leading to increased productivity.

show abstract

“…One hardware innovation in particular -Graphics Processor Units (GPUs) developed for video-games applicationsbecame an important enabler for deep learning techniques that benefit from parallelisation of tasks. Hooker [2020] argues that the lag between the development of key research ideas about artificial neural networks and their application in deep learning are an example of 'hardware lotteries' in AI research: this refers to the fact that the adoption of an idea depends not only on its merits but also on the availability of suitable complements: hardware and software to implement the idea, and in the case of machine learning methods, large datasets for training.…”

Section: Directionality In Ai Researchmentioning

confidence: 99%

A narrowing of AI research?

Klinger¹,

Mateos-García²,

Stathoulopoulos³

2020

Preprint

View full text Add to dashboard Cite

Artificial Intelligence (AI) is being hailed as the latest example of a General Purpose Technology that could transform productivity and help tackle important societal challenges. This outcome is however not guaranteed: a myopic focus on short-term benefits could lock AI into technologies that turn out to be sub-optimal in the longer-run. For this reason, it may be valuable to preserve diversity in the AI trajectories that are explored until there is more information about their relative merits and dangers. Recent controversies about the dominance of deep learning methods and private labs in AI research suggest that the field may be getting narrower, but the evidence base is lacking. We seek to address this gap with an analysis of the thematic diversity of AI research in arXiv, a widely used pre-prints site. Having identified 110,000 AI papers in this corpus, we use hierarchical topic modelling to estimate the thematic composition of AI research, and this composition to calculate various metrics of research diversity. Our analysis suggests that diversity in AI research has stagnated in recent years, and that AI research involving private sector organisations tends to be less diverse than research in academia. This appears to be driven by a small number of prolific and narrowly-focused technology companies. Diversity in academia is bolstered by smaller institutions and research groups that may have less incentives to 'race' and lower levels of collaboration with the private sector. We also find that private sector AI researchers tend to specialise in data and computationally intensive deep learning methods at the expense of research involving other (symbolic and statistical) AI methods, and of research that considers the societal and ethical implications of AI or applies it in domains like health. Our results suggest that there may be a rationale for policy action to prevent a premature narrowing of AI research that could reduce its societal benefits, but we note the incentive, information and scale hurdles standing in the way of such interventions.

show abstract

The hardware lottery

Abstract: After decades of incentivizing the isolation of hardware, software, and algorithm development, the catalysts for closer collaboration are changing the paradigm.

Cited by 72 publications

References 22 publications

Message passing all the way up

Message passing all the way up

Declarative Machine Learning Systems

A narrowing of AI research?

Contact Info

Product

Resources

About