Gradient-Based Inference for Networks with Output Constraints

Lee, Jay Yoon; Mehta, Sanket Vaibhav; Wick, Michael; Tristan, Jean-Baptiste; Carbonell, Jaime G.

doi:10.1609/aaai.v33i01.33014147

Cited by 18 publications

(12 citation statements)

References 16 publications

(24 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This work is complementary to the work presented in a companion paper that studied enforcing statistical constraints [36]. While precise constraints have previously been used as regularization in various fields (e.g., natural language processing [37], lake temperature modeling [38,39], general dynamical systems [25], fluid flow simulations [9,11,12,40], and more specifically in turbulent flow simulations and generation [41][42][43]), the effects, performances, and best practices of imposing imprecise constraints in generative models still need further investigations.…”

Section: Scope and Contributions Of Present Workmentioning

confidence: 82%

Enforcing Imprecise Constraints on Generative Adversarial Networks for Emulating Physical Systems

Zeng¹

2021

CiCP

View full text Add to dashboard Cite

Generative adversarial networks (GANs) were initially proposed to generate images by learning from a large number of samples. Recently, GANs have been used to emulate complex physical systems such as turbulent flows. However, a critical question must be answered before GANs can be considered trusted emulators for physical systems: do GANs-generated samples conform to the various physical constraints? These include both deterministic constraints (e.g., conservation laws) and statistical constraints (e.g., energy spectrum of turbulent flows). The latter have been studied in a companion paper (Wu et al., Enforcing statistical constraints in generative adversarial networks for modeling chaotic dynamical systems. Journal of Computational Physics. 406, 109209, 2020). In the present work, we enforce deterministic yet imprecise constraints on GANs by incorporating them into the loss function of the generator. We evaluate the performance of physics-constrained GANs on two representative tasks with geometrical constraints (generating points on circles) and differential constraints (generating divergence-free flow velocity fields), respectively. In both cases, the constrained GANs produced samples that conform to the underlying constraints rather accurately, even though the constraints are only enforced up to a specified interval. More importantly, the imposed constraints significantly accelerate the convergence and improve the robustness in the training, indicating that they serve as a physics-based regularization. These improvements are noteworthy, as the convergence and robustness are two well-known obstacles in the training of GANs.

show abstract

Section: Scope and Contributions Of Present Workmentioning

confidence: 82%

Enforcing Imprecise Constraints on Generative Adversarial Networks for Emulating Physical Systems

Zeng¹

2021

CiCP

View full text Add to dashboard Cite

show abstract

“…The second, more general method to build constraint-aware neural networks is based on the idea of penalizing deviations from the constraint (11) during the training procedure (as mentioned in [27,22]) . To this end, we introduce the constraint loss function…”

Section: Constraint-adapted Loss Methods (Cal)mentioning

confidence: 99%

“…which will be the constraint that a surrogate model for (21) should satisfy. As in (12) the constraint target function Φ cubic : × × , corresponding to (22), is given by…”

Section: Cubic Flux Model: Undercompressive Wavementioning

confidence: 99%

See 1 more Smart Citation

Constraint-aware neural networks for Riemann problems

Magiera

Ray

Hesthaven

et al. 2020

Journal of Computational Physics

View full text Add to dashboard Cite

Neural networks are increasingly used in complex (data-driven) simulations as surrogates or for accelerating the computation of classical surrogates. In many applications physical constraints, such as mass or energy conservation, must be satis ed to obtain reliable results. However, standard machine learning algorithms are generally not tailored to respect such constraints. We propose two di erent strategies to generate constraint-aware neural networks. We test their performance in the context of front-capturing schemes for strongly nonlinear wave motion in compressible uid ow. Precisely, in this context so-called Riemann problems have to be solved as surrogates. Their solution describes the local dynamics of the captured wave front in numerical simulations. Three model problems are considered: a cubic ux model problem, an isothermal two-phase ow model, and the Euler equations. We demonstrate that a decrease in the constraint deviation correlates with low discretization errors for all model problems, in addition to the structural advantage of ful lling the constraint.

show abstract

“…a) Incorporating constraints: Many SP tasks require the structured outputs to satisfy some constraints (e.g., valid trees in parsing). Incorporating these constraints is a major challenge for methods that perform gradient-based inference and learning [Lee et al, 2017]. These constraints can be classified into three main categories: relational, logical, and scientific.…”

Section: Deep Learning ∩ Structured Predictionmentioning

confidence: 99%

Learning and Inference for Structured Prediction: A Unifying Perspective

Deshwal

Doppa

Roth

2019

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

In a structured prediction problem, one needs to learn a predictor that, given a structured input, produces a structured object, such as a sequence, tree, or clustering output. Prototypical structured prediction tasks include part-of-speech tagging (predicting POS tag sequence for an input sentence) and semantic segmentation of images (predicting semantic labels for pixels of an input image). Unlike simple classification problems, here there is a need to assign values to multiple output variables accounting for the dependencies between them. Consequently, the prediction step itself (aka ``inference" or ``decoding") is computationally-expensive, and so is the learning process, that typically requires making predictions as part of it. The key learning and inference challenge is due to the exponential size of the structured output space and depend on its complexity. In this paper, we present a unifying perspective of the different frameworks that address structured prediction problems and compare them in terms of their strengths and weaknesses. We also discuss important research directions including integration of deep learning advances into structured prediction, and learning from weakly supervised signals and active querying to overcome the challenges of building structured predictors from small amount of labeled data.

show abstract

Gradient-Based Inference for Networks with Output Constraints

Cited by 18 publications

References 16 publications

Enforcing Imprecise Constraints on Generative Adversarial Networks for Emulating Physical Systems

Enforcing Imprecise Constraints on Generative Adversarial Networks for Emulating Physical Systems

Constraint-aware neural networks for Riemann problems

Learning and Inference for Structured Prediction: A Unifying Perspective

Contact Info

Product

Resources

About