Efficient and Modular Implicit Differentiation

Blondel, Mathieu; Berthet, Quentin; Cuturi, Marco; Frostig, Roy; Hoyer, Stephan; Llinares-López, Felipe; Pedregosa, Fabián; Vert, Jean-Philippe

doi:10.48550/arxiv.2105.15183

Cited by 39 publications

(56 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Implicit differentiation. Implicit differentiation is a technique that allows efficient differentiation of certain types of optimization problems (Krantz & Parks, 2012;Blondel et al, 2021). We replace the normal backward pass of DSPN with implicit differentiation and call our method implicit deep set prediction networks (iDSPN).…”

Section: Implicit Deep Set Prediction Networkmentioning

confidence: 99%

“…For example, when each set element corresponds to a class label, we would like to enforce that every vector is in the probability simplex. As suggested by Blondel et al (2021), we can incorporate such constraints through constrained optimization in implicit differentiation by changing Equation 8 to the following and then following the standard derivation in Appendix D:…”

Section: Implicit Deep Set Prediction Networkmentioning

confidence: 99%

“…Recent works in deep learning have successfully applied implicit differentiation to large-scale language modelling , semantic segmentation (Bai et al, 2020), meta-learning (Rajeswaran et al, 2019), neural architecture search (Zhang et al, 2021b), and more (Agrawal et al, 2019). The implicit function theorem allows for seamless integration of optimization problems into larger end-to-end trainable neural networks (Gould et al, 2016;Amos & Kolter, 2017;Blondel et al, 2021;El Ghaoui et al, 2021). Beyond the sets and multisets considered in this work, implicit differentiation has been applied to other non-Euclidean domains, such as hyperbolic space (Lou et al, 2020) and graphs (Gu et al, 2020).…”

Section: Additional Related Workmentioning

confidence: 99%

“…We start by deriving the Jacobians for z and θ with implicit differentiation and present the approximation for the Hessian afterwards. Blondel et al (2021) describe a straightforward and efficient recipe for applying implicit differentiation to optimization problems. We follow their approach for the standard implicit differentiation derivation and efficiency concerns.…”

Section: Derivation Of Implicit Dspnmentioning

confidence: 99%

See 3 more Smart Citations

Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation

Zhang¹,

Zhang²,

Lacoste-Julien³

et al. 2021

Preprint

View full text Add to dashboard Cite

Most set prediction models in deep learning use set-equivariant operations, but they actually operate on multisets. We show that set-equivariant functions cannot represent certain functions on multisets, so we introduce the more appropriate notion of multiset-equivariance. We identify that the existing Deep Set Prediction Network (DSPN) can be multiset-equivariant without being hindered by set-equivariance and improve it with approximate implicit differentiation, allowing for better optimization while being faster and saving memory. In a range of toy experiments, we show that the perspective of multiset-equivariance is beneficial and that our changes to DSPN achieve better results in most cases. On CLEVR object property prediction, we substantially improve over the state-of-the-art Slot Attention from 8% to 77% in one of the strictest evaluation metrics because of the benefits made possible by implicit differentiation.

show abstract

Section: Implicit Deep Set Prediction Networkmentioning

confidence: 99%

Section: Implicit Deep Set Prediction Networkmentioning

confidence: 99%

Section: Additional Related Workmentioning

confidence: 99%

Section: Derivation Of Implicit Dspnmentioning

confidence: 99%

See 2 more Smart Citations

Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation

Zhang¹,

Zhang²,

Lacoste-Julien³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…First introduced in the field of economic game theory by Stackelberg (1934), this problem has recently received increasing attention in the machine learning community (Domke, 2012;Gould et al, 2016;Liao et al, 2018;Blondel et al, 2021;Liu et al, 2021;Shaban et al, 2019). Indeed, many machine learning applications can be reduced to (1) including hyper-parameter optimization (Feurer and Hutter, 2019), meta-learning (Bertinetto et al, 2018), reinforcement learning (Hong et al, 2020b;Liu et al, 2021) or dictionary learning (Mairal et al, 2011;Lecouat et al, 2020a;b).…”

Section: Introductionmentioning

confidence: 99%

Amortized Implicit Differentiation for Stochastic Bilevel Optimization

Arbel,

Mairal

2021

Preprint

View full text Add to dashboard Cite

We study a class of algorithms for solving bilevel optimization problems in both stochastic and deterministic settings when the inner-level objective is strongly convex. Specifically, we consider algorithms based on inexact implicit differentiation and we exploit a warm-start strategy to amortize the estimation of the exact gradient. We then introduce a unified theoretical framework inspired by the study of singularly perturbed systems (Habets, 1974) to analyze such amortized algorithms. By using this framework, our analysis shows these algorithms to match the computational complexity of oracle methods that have access to an unbiased estimate of the gradient, thus outperforming many existing results for bilevel optimization. We illustrate these findings on synthetic experiments and demonstrate the efficiency of these algorithms on hyper-parameter optimization experiments involving several thousands of variables.

show abstract

Tcf7l2 in hepatocytes regulates de novo lipogenesis in diet-induced non-alcoholic fatty liver disease in mice

Lee

Kim

et al. 2023

Diabetologia

View full text Add to dashboard Cite

Aims/hypothesis Non-alcoholic fatty liver disease (NAFLD) associated with type 2 diabetes may more easily progress towards severe forms of non-alcoholic steatohepatitis (NASH) and cirrhosis. Although the Wnt effector transcription factor 7-like 2 (TCF7L2) is closely associated with type 2 diabetes risk, the role of TCF7L2 in NAFLD development remains unclear. Here, we investigated how changes in TCF7L2 expression in the liver affects hepatic lipid metabolism based on the major risk factors of NAFLD development. Methods Tcf7l2 was selectively ablated in the liver of C57BL/6N mice by inducing the albumin (Alb) promoter to recombine Tcf7l2 alleles floxed at exon 5 (liver-specific Tcf7l2-knockout [KO] mice: Alb-Cre;Tcf7l2f/f). Alb-Cre;Tcf7l2f/f and their wild-type (Tcf7l2f/f) littermates were fed a high-fat diet (HFD) or a high-carbohydrate diet (HCD) for 22 weeks to reproduce NAFLD/NASH. Mice were refed a standard chow diet or an HCD to stimulate de novo lipogenesis (DNL) or fed an HFD to provide exogenous fatty acids. We analysed glucose and insulin sensitivity, metabolic respiration, mRNA expression profiles, hepatic triglyceride (TG), hepatic DNL, selected hepatic metabolites, selected plasma metabolites and liver histology. Results Alb-Cre;Tcf7l2f/f essentially exhibited increased lipogenic genes, but there were no changes in hepatic lipid content in mice fed a normal chow diet. However, following 22 weeks of diet-induced NAFLD/NASH conditions, liver steatosis was exacerbated owing to preferential metabolism of carbohydrate over fat. Indeed, hepatic Tcf7l2 deficiency enhanced liver lipid content in a manner that was dependent on the duration and amount of exposure to carbohydrates, owing to cell-autonomous increases in hepatic DNL. Mechanistically, TCF7L2 regulated the transcriptional activity of Mlxipl (also known as ChREBP) by modulating O-GlcNAcylation and protein content of carbohydrate response element binding protein (ChREBP), and targeted Srebf1 (also called SREBP1) via miRNA (miR)-33-5p in hepatocytes. Eventually, restoring TCF7L2 expression at the physiological level in the liver of Alb-Cre;Tcf7l2f/f mice alleviated liver steatosis without altering body composition under both acute and chronic HCD conditions. Conclusions/interpretation In mice, loss of hepatic Tcf7l2 contributes to liver steatosis by inducing preferential metabolism of carbohydrates via DNL activation. Therefore, TCF7L2 could be a promising regulator of the NAFLD associated with high-carbohydrate diets and diabetes since TCF7L2 deficiency may lead to development of NAFLD by promoting utilisation of excess glucose pools through activating DNL. Data availability RNA-sequencing data have been deposited into the NCBI GEO under the accession number GSE162449 (www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE162449). Graphical abstract

show abstract

Efficient and Modular Implicit Differentiation

Cited by 39 publications

References 30 publications

Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation

Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation

Amortized Implicit Differentiation for Stochastic Bilevel Optimization

Tcf7l2 in hepatocytes regulates de novo lipogenesis in diet-induced non-alcoholic fatty liver disease in mice

Contact Info

Product

Resources

About