Tropical Geometry and Machine Learning

Maragos, Petros; Charisopoulos, Vasileios; Theodosis, Emmanouil

doi:10.1109/jproc.2021.3065238

Cited by 34 publications

(16 citation statements)

References 86 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More speculatively, tropical structures might provide insights about how large language models actually learn semantic information. Recently, it has been discovered [5,27,36,45] that feedforward neural networks with ReLU activation functions compute tropical rational maps, opening a new way to study the mathematical structure underlying them. LLMs use neural architectures to learn probabability distributions on text continuations.…”

Section: Theoremmentioning

confidence: 99%

An Enriched Category Theory of Language: From Syntax to Semantics

2022

View full text Add to dashboard Cite

State of the art language models return a natural language text continuation from any piece of input text. This ability to generate coherent text extensions implies significant sophistication, including a knowledge of grammar and semantics. In this paper, we propose a mathematical framework for passing from probability distributions on extensions of given texts, such as the ones learned by today's large language models, to an enriched category containing semantic information. Roughly speaking, we model probability distributions on texts as a category enriched over the unit interval. Objects of this category are expressions in language, and hom objects are conditional probabilities that one expression is an extension of another. This category is syntactical-it describes what goes with what. Then, via the Yoneda embedding, we pass to the enriched category of unit interval-valued copresheaves on this syntactical category. This category of enriched copresheaves is semantic-it is where we find meaning, logical operations such as entailment, and the building blocks for more elaborate semantic concepts.

show abstract

Section: Theoremmentioning

confidence: 99%

An Enriched Category Theory of Language: From Syntax to Semantics

2022

View full text Add to dashboard Cite

show abstract

“…In Section 4 we present a novel application of tropical concepts to understand neural networks. We refer to Maragos et al [2021] for a recent survey of connections between machine learning and tropical geometry, as well as to the textbooks by Maclagan and Sturmfels [2015] and Joswig [2022] for in-depth introductions to tropical geometry and tropical combinatorics.…”

Section: Related Workmentioning

confidence: 99%

Towards Lower Bounds on the Depth of ReLU Neural Networks

Hertrich¹,

Basu²,

Summa³

et al. 2021

Preprint

View full text Add to dashboard Cite

We contribute to a better understanding of the class of functions that is represented by a neural network with ReLU activations and a given architecture. Using techniques from mixed-integer optimization, polyhedral theory, and tropical geometry, we provide a mathematical counterbalance to the universal approximation theorems which suggest that a single hidden layer is sufficient for learning tasks. In particular, we investigate whether the class of exactly representable functions strictly increases by adding more layers (with no restrictions on size). This problem has potential impact on algorithmic and statistical aspects because of the insight it provides into the class of functions represented by neural hypothesis classes. However, to the best of our knowledge, this question has not been investigated in the neural network literature. We also present upper bounds on the sizes of neural networks required to represent functions in these neural hypothesis classes.

show abstract

“…These include studies of the tropicalization of stochastic processes (Akian et al, 1994) or of Gaussian measures (Tran, 2020), tropical support vector machines (Yoshida et al, 2021), tropical principal component analysis (Yoshida et al, 2019) inspired by phylogenetic studies, quantification of the expressivity of deep neural networks (Zhang et al, 2018;Montúfar et al, 2021) or their approximation (Calafiore et al, 2020) through tropical methods. A survey of some of these approaches can be found in Maragos et al (2021). The proper tropical analogue of RKHSs still remained elusive nonetheless.…”

Section: Introductionmentioning

confidence: 99%

Tropical reproducing kernels and optimization

Aubin-Frankowski¹,

Gaubert²

2022

Preprint

View full text Add to dashboard Cite

Hilbertian kernel methods and their positive semidefinite kernels have known an extensive use in various fields of applied mathematics and machine learning, owing to their several equivalent characterizations. We here unveil an analogy with concepts from tropical geometry, proving that tropical positive semidefinite kernels are also endowed with equivalent viewpoints, stemming from Fenchel-Moreau conjugations. We give a tropical analogue of Aronszajn theorem, showing that these kernels correspond to a feature map, define monotonous operators, and generate max-plus function spaces endowed with a reproducing property. They furthermore include all the Hilbertian kernels classically studied as well as Monge arrays. However, two relevant notions of tropical reproducing kernels must be distinguished, based either on linear or sesquilinear interpretations. The sesquilinear interpretation is the most expressive one, since reproducing spaces then encompass classical max-plus spaces, such as those of (semi)convex functions. In contrast, in the linear interpretation, the reproducing kernels are characterized by a restrictive condition, von Neumann regularity.

show abstract

Tropical Geometry and Machine Learning

Abstract: This article deals with tropical geometry that has recently emerged as a tool in the analysis and extension of several classes of problems in both classical machine learning and deep learning.

Cited by 34 publications

References 86 publications

An Enriched Category Theory of Language: From Syntax to Semantics

An Enriched Category Theory of Language: From Syntax to Semantics

Towards Lower Bounds on the Depth of ReLU Neural Networks

Tropical reproducing kernels and optimization

Contact Info

Product

Resources

About