LightOn Optical Processing Unit: Scaling-up AI and HPC with a Non von Neumann co-processor

Brossollet, Charles; Cappelli, Alessandro; Carron, Igor; Chaintoutis, Charidimos; Chatelain, Amélie; Daudet, Laurent; Gigan, Sylvain; Hesslow, Daniel; Krząkała, Florent; Launay, Julien; Mokaadi, Safa; Moreau, Fabien; Müller, Kilian; Ohana, Ruben; Pariente, G.; Poli, Iacopo; Tommasone, Elena L.

doi:10.48550/arxiv.2107.11814

Cited by 3 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, using a diffuser that generates speckle patterns can improve the scalability and generalization when using convolutional neural networks [17]. Additionally, hybrid data processing architectures have also been developed, such as LightOn's optical processing unit which functions as a fast external preprocessor [18,19]. Motivated by these photonic implementations, we look into the opportunities offered by photonic reservoir computing (RC) as preprocessor for DNNs.…”

Section: Introductionmentioning

confidence: 99%

Using photonic reservoirs as preprocessors for deep neural networks

et al. 2022

View full text Add to dashboard Cite

Artificial neural networks are very time consuming and energy intensive to train, especially when increasing the size of the neural network in an attempt to improve the performance. In this paper, we propose to preprocess the input data of a deep neural network using a reservoir, which has originally been introduced in the framework of reservoir computing. The key idea of this paper is to use such a reservoir to transform the input data into a state in a higher dimensional state-space, which allows the deep neural network to process the data with improved performance. We focus on photonic reservoirs because of their fast computation times and low-energy consumption. Based on numerical simulations of delay-based reservoirs using a semiconductor laser, we show that using such preprocessed data results in an improved performance of deep neural networks. Furthermore, we show that we do not need to carefully fine-tune the parameters of the preprocessing reservoir.

show abstract

Section: Introductionmentioning

confidence: 99%

Using photonic reservoirs as preprocessors for deep neural networks

et al. 2022

View full text Add to dashboard Cite

show abstract

“…In a language model, frozen parameters are one third cheaper than fully trainable ones: assuming the backward pass is normally twice the cost in FLOP of the forward (backpropagation + update) [6], the update step is skipped for frozen parameters. Furthermore, multiplication with random matrices can be offloaded to specialized co-processors [17].…”

mentioning

confidence: 99%

Is the Number of Trainable Parameters All That Actually Matters?

Chatelain¹,

Djeghri²,

Hesslow³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recent work has identified simple empirical scaling laws for language models, linking compute budget, dataset size, model size, and autoregressive modeling loss. The validity of these simple power laws across orders of magnitude in model scale provides compelling evidence that larger models are also more capable models. However, scaling up models under the constraints of hardware and infrastructure is no easy feat, and rapidly becomes a hard and expensive engineering problem. We investigate ways to tentatively cheat scaling laws, and train larger models for cheaper. We emulate an increase in effective parameters, using efficient approximations: either by doping the models with frozen random parameters, or by using fast structured transforms in place of dense linear layers. We find that the scaling relationship between test loss and compute depends only on the actual number of trainable parameters; scaling laws cannot be deceived by spurious parameters. IntroductionPredictably linking model and dataset size with generalization error is a long-standing open question. Discrepancies between classical bias-variance trade-off models and modern practices have been identified [1], with phenomena such as deep double descent [2] providing glimpses into a deeper understanding. However, actionable insights for machine learning practitioners have remained elusive.Recently, simple and general empirical scaling laws for deep learning models have been uncovered. Starting with first relationships for modern convolutional networks [3], guidelines informing optimal architectures-such as EfficientNet [4]-have been derived. Importantly, these laws can extrapolate model performance across scale, motivating the training of increasingly large and capable models [5].We focus on seminal work on large generative language models [6], establishing and using scaling laws to predict the computational requirements for a specific performance level-as pioneered by [7]. This work showed that the performance of an auto-regressive language model follows a remarkably simple relationship, linking model and dataset size, compute budget, and modeling loss across many orders of magnitude in scale. This make it possible to answer questions central to the efficient training of extreme-scale models, such as: (1) which model size achieves the best loss given a fixed compute budget; or (2) how many samples are necessary to train a model of a given size optimally.Unfortunately, these empirical laws also show that training models much larger than current ones comes at a prohibitive cost. State-of-the-art language models such as GPT-3 have required several thousand PF-days to train [8], and model size has plateaued since then [9,10]. Proposed distributed * Work done while at LightOn. Preprint. Under review.

show abstract

Linear optical random projections without holography

et al. 2023

View full text Add to dashboard Cite

We introduce what we believe to be a novel method to perform linear optical random projections without the need for holography. Our method consists of a computationally trivial combination of multiple intensity measurements to mitigate the information loss usually associated with the absolute-square non-linearity imposed by optical intensity measurements. Both experimental and numerical findings demonstrate that the resulting matrix consists of real-valued, independent, and identically distributed (i.i.d.) Gaussian random entries. Our optical setup is simple and robust, as it does not require interference between two beams. We demonstrate the practical applicability of our method by performing dimensionality reduction on high-dimensional data, a common task in randomized numerical linear algebra with relevant applications in machine learning.

show abstract

LightOn Optical Processing Unit: Scaling-up AI and HPC with a Non von Neumann co-processor

Cited by 3 publications

References 0 publications

Using photonic reservoirs as preprocessors for deep neural networks

Using photonic reservoirs as preprocessors for deep neural networks

Is the Number of Trainable Parameters All That Actually Matters?

Linear optical random projections without holography

Contact Info

Product

Resources

About