Training Fully Connected Neural Networks is $\exists\mathbb{R}$-Complete

Bertschinger, Daniel; Hertrich, Christoph; Jungeblut, Paul; Miltzow, Tillmann; Weber, Simon

doi:10.48550/arxiv.2204.01368

Cited by 5 publications

(3 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Important ∃R-completeness results include the realizability of abstract order types [48,59] and geometric linkages [53], as well as the recognition of geometric segment [36,44], unit-disk [34,46], and ray intersection graphs [19]. More results appeared in the graph drawing community [22,23,41,54], regarding the Hausdorff distance [33], regarding polytopes [21,51], the study of Nash-equilibria [6,9,10,25,55], training neural networks [3,8], matrix factorization [20,56,57,58,61], or continuous constraint satisfaction problems [47]. In computational geometry, we would like to mention geometric packing [4], the art gallery problem [2], and covering polygons with convex polygons [1].…”

Section: Background and Related Workmentioning

confidence: 99%

Representing Matroids over the Reals is $\exists \mathbb R$-complete

Kim¹,

Mesmay²,

Miltzow³

2023

Preprint

View full text Add to dashboard Cite

A matroid M is an ordered pair (E, I), where E is a finite set called the ground set and a collection I ⊂ 2 E called the independent sets which satisfy the conditions: (I1) ∅ ∈ I, (I2) I ⊂ I ∈ I implies I ∈ I, and (I3) I1, I2 ∈ I and |I1| < |I2| implies that there is an e ∈ I2 such that I1 ∪ {e} ∈ I. The rank rk(M ) of a matroid M is the maximum size of an independent set. We say that a matroid M = (E, I) is representable over the reals if there is a map ϕ : E → R rk(M ) such that I ∈ I if and only if ϕ(I) forms a linearly independent set.We study the problem of Matroid R-Representability over the reals. Given a matroid M , we ask whether there is a set of points in the Euclidean space representing M . We show that Matroid R-Representability is ∃R-complete, already for matroids of rank 3. The complexity class ∃R can be defined as the family of algorithmic problems that is polynomial-time equivalent to determining if a multivariate polynomial with integer coefficients has a real root.Our methods are similar to previous methods from the literature. Yet, the result itself was never pointed out and there is no proof readily available in the language of computer science.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Representing Matroids over the Reals is $\exists \mathbb R$-complete

Kim¹,

Mesmay²,

Miltzow³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…However, for a full theoretical understanding of this fundamental machine learning model it is necessary to understand what functions can be exactly expressed with different NN architectures. For instance, insights about exact representability have boosted our understanding of the computational complexity of the task to train an NN with respect to both, algorithms [4,36] and hardness results [9,18,20]. It is known that a function can be expressed with a ReLU NN if and only if it is continuous and piecewise linear (CPWL) [4].…”

Section: Introductionmentioning

confidence: 99%

ReLU Neural Networks of Polynomial Size for Exact Maximum Flow Computation

Hertrich

Sering

2023

Integer Programming and Combinatorial Optimization

View full text Add to dashboard Cite

This paper studies the expressive power of artificial neural networks with rectified linear units. In order to study them as a model of real-valued computation, we introduce the concept of Max-Affine Arithmetic Programs and show equivalence between them and neural networks concerning natural complexity measures. We then use this result to show that two fundamental combinatorial optimization problems can be solved with polynomial-size neural networks. First, we show that for any undirected graph with n nodes, there is a neural network (with fixed weights and biases) of size O(n 3 ) that takes the edge weights as input and computes the value of a minimum spanning tree of the graph. Second, we show that for any directed graph with n nodes and m arcs, there is a neural network of size O(m 2 n 2 ) that takes the arc capacities as input and computes a maximum flow. Our results imply that these two problems can be solved with strongly polynomial time algorithms that solely uses affine transformations and maxima computations, but no comparison-based branchings.

show abstract

“…However, the implications on the computational complexity are limited since their result requires the number of hidden neurons to be very large. Bertschinger, Hertrich, Jungeblut, Miltzow, and Weber (2022) show that training 2-layer neural networks is complete for the complexity class ∃R (existential theory of the reals), implying that the problem is presumably not contained in NP. They generalize a previous result by Abrahamsen, Kleist, and Miltzow (2021), who showed the same fact for specifically designed, more complex architectures.…”

Section: Introductionmentioning

confidence: 99%

The Computational Complexity of ReLU Network Training Parameterized by Data Dimensionality

Froese¹,

Hertrich

Niedermeier

2022

jair

Self Cite

View full text Add to dashboard Cite

Understanding the computational complexity of training simple neural networks with rectified linear units (ReLUs) has recently been a subject of intensive research. Closing gaps and complementing results from the literature, we present several results on the parameterized complexity of training two-layer ReLU networks with respect to various loss functions. After a brief discussion of other parameters, we focus on analyzing the influence of the dimension d of the training data on the computational complexity. We provide running time lower bounds in terms of W[1]-hardness for parameter d and prove that known brute-force strategies are essentially optimal (assuming the Exponential Time Hypothesis). In comparison with previous work, our results hold for a broad(er) range of loss functions, including lp-loss for all p ∈ [0, ∞]. In particular, we improve a known polynomial-time algorithm for constant d and convex loss functions to a more general class of loss functions, matching our running time lower bounds also in these cases.

show abstract

Training Fully Connected Neural Networks is $\exists\mathbb{R}$-Complete

Cited by 5 publications

References 48 publications

Representing Matroids over the Reals is $\exists \mathbb R$-complete

Representing Matroids over the Reals is $\exists \mathbb R$-complete

ReLU Neural Networks of Polynomial Size for Exact Maximum Flow Computation

The Computational Complexity of ReLU Network Training Parameterized by Data Dimensionality

Contact Info

Product

Resources

About