Differentiable Multiple Shooting Layers

Massaroli, Stefano; Poli, Michael; Sonoda, Shozo; Taji, Suzuki,; Park, Jinkyoo; Yamashita, Atsushi; Asama, Hajime

doi:10.48550/arxiv.2106.03885

Cited by 2 publications

(2 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, Neural Collages can be framed as a compactly-parametrized operator variant of deep equilibrium networks (DEQs) (Bai et al, 2019), with parameters produced by a hypernetwork (Ha et al, 2016). Huang et al (2021) uses implicit models as implicit representations, with computational advantages gained via fixed-point tracking (Massaroli et al, 2021). Further speedups for Neural Collages compressors could similarly be found via tracking during training.…”

Section: Related Work and Discussionmentioning

confidence: 99%

Self-Similarity Priors: Neural Collages as Differentiable Fractal Representations

Poli¹,

Xu²,

Massaroli³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Many patterns in nature exhibit self-similarity: they can be compactly described via self-referential transformations. Said patterns commonly appear in natural and artificial objects, such as molecules, shorelines, galaxies and even images. In this work, we investigate the role of learning in the automated discovery of self-similarity and in its utilization for downstream tasks. To this end, we design a novel class of implicit operators, Neural Collages, which (1) represent data as the parameters of a self-referential, structured transformation, and (2) employ hypernetworks to amortize the cost of finding these parameters to a single forward pass. We investigate how to leverage the representations produced by Neural Collages in various tasks, including data compression and generation. Neural Collage image compressors are orders of magnitude faster than other self-similarity-based algorithms during encoding and offer compression rates competitive with implicit methods. Finally, we showcase applications of Neural Collages for fractal art and as deep generative models.

show abstract

Section: Related Work and Discussionmentioning

confidence: 99%

Self-Similarity Priors: Neural Collages as Differentiable Fractal Representations

Poli¹,

Xu²,

Massaroli³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Neural Operators for PDEs. Deep learning has found application in the domain of differential equations and scientific computing [85], with methods developed for prediction and control problems [56,71], as well as acceleration of numerical schemes [83,51]. Specific to the partial differential equations (PDEs) are approaches designed to learn solution operators [87,29,65], and hybridized solvers [59], evaluated primarily on classical fluid dynamics.…”

Section: A Extended Related Workmentioning

confidence: 99%

Monarch: Expressive Structured Matrices for Efficient and Accurate Training

Dao¹,

Chen²,

Sohoni³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Large neural networks excel in many domains, but they are expensive to train and fine-tune. A popular approach to reduce their compute/memory requirements is to replace dense weight matrices with structured ones (e.g., sparse, low-rank, Fourier transform). These methods have not seen widespread adoption (1) in end-to-end training due to unfavorable efficiency-quality tradeoffs, and (2) in denseto-sparse fine-tuning due to lack of tractable algorithms to approximate a given dense weight matrix. To address these issues, we propose a class of matrices (Monarch) that is hardware-efficient (they are parameterized as products of two block-diagonal matrices for better hardware utilization) and expressive (they can represent many commonly used transforms). Surprisingly, the problem of approximating a dense weight matrix with a Monarch matrix, though nonconvex, has an analytical optimal solution. These properties of Monarch matrices unlock new ways to train and fine-tune sparse and dense models. We empirically validate that Monarch can achieve favorable accuracy-efficiency tradeoffs in several end-to-end sparse training applications: speeding up ViT and GPT-2 training on ImageNet classification and Wikitext-103 language modeling by 2× with comparable model quality, and reducing the error on PDE solving and MRI reconstruction tasks by 40%. In sparse-to-dense training, with a simple technique called "reverse sparsification," Monarch matrices serve as a useful intermediate representation to speed up GPT-2 pretraining on OpenWebText by 2× without quality drop. The same technique brings 23% faster BERT pretraining than even the very optimized implementation from Nvidia that set the MLPerf 1.1 record. In dense-to-sparse fine-tuning, as a proof-of-concept, our Monarch approximation algorithm speeds up BERT fine-tuning on GLUE by 1.7× with comparable accuracy.

show abstract

Differentiable Multiple Shooting Layers

Cited by 2 publications

References 38 publications

Self-Similarity Priors: Neural Collages as Differentiable Fractal Representations

Self-Similarity Priors: Neural Collages as Differentiable Fractal Representations

Monarch: Expressive Structured Matrices for Efficient and Accurate Training

Contact Info

Product

Resources

About