Measure and Cardinality

Briggs, James M.; Schaffter, Thomas

doi:10.1080/00029890.1979.11994929

Cited by 4 publications

(3 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Consequently, validating Proposition 1 translates to showing that any given w i does not lie in the span of W span −i ; that is, w i and W span −i are linearly independent. First, it is easy to note that P (w 1 = 0) = 1, since the Lebesgue measure of a singleton set is zero [79]; thus, P (w 1 / ∈ W span −i ) = 1. Furthermore, for p ∈ {2, ..., n − 1}, let W −1,p = {w 1 , ..., w p } be the set of first p vectors, excluding vector i, such that W −1,p is linearly independent with a probability of 1.…”

Section: Main Findings Onmentioning

confidence: 99%

Why Is Everyone Training Very Deep Neural Network With Skip Connections?

Oyedotun

Ismaeil

Aouada

2023

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Recent deep neural networks (DNNs) with several layers of feature representations rely on some form of skip connections to simultaneously circumnavigate optimization problems and improve generalization performance. However, the operations of these models are still not clearly understood, especially in comparison to DNNs without skip connections referred to as PlainNets that are absolutely untrainable beyond some depth. As such, the exposition of this paper is the theoretical analysis of the role of skip connections in training very DNNs using concepts from linear algebra and random matrix theory. In comparison with PlainNets, the results of our investigation directly unravel the following (i) why DNNs with skip connections are easier to optimize (ii) why DNNs with skip connections exhibit improved generalization. Our investigation results concretely show that the hidden representations of PlainNets progressively suffer from information loss via singularity problems with depth increase, and thus making their optimization difficult. In contrast, as model depth increases, the hidden representations of DNNs with skip connections circumnavigate singularity problems to retain full information that reflects in improved optimization and generalization. For theoretical analysis, this paper studies in relation to PlainNets two popular skip-connection based DNNs that are residual networks (ResNets) and residual network with aggregated features (ResNeXt).

show abstract

Section: Main Findings Onmentioning

confidence: 99%

Why Is Everyone Training Very Deep Neural Network With Skip Connections?

Oyedotun

Ismaeil

Aouada

2023

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

“…Consequently, proofing Proposition 1 translates to showing that any given w i does not lie in the span of W span −i ; that is, w i and W span −i are linearly independent. First, it is easy to note that P (w 1 = 0) = 1, since the Lebesgue measure of a singleton set is zero [57]; thus, P (w 1 / ∈ W span −i ) = 1. Furthermore, for p ∈ {2, ..., n−1}, let W −1,p = [w 1 , ..., w p ] be the matrix whose columns are the first p vectors, excluding vector w i , such that the columns of W −1,p are linearly independent with a probability of 1.…”

Section: Appendixmentioning

confidence: 99%

Training very deep neural networks: Rethinking the role of skip connections

2021

View full text Add to dashboard Cite

State-of-the-art deep neural networks (DNNs) typically consist of several layers of features representations, and especially rely on skip connections to avoid the difficulty of model optimization. Despite the proliferation of different DNN models that employ various forms of skip connections to achieve remarkable results on benchmarking datasets, a concrete explanation for the successful operation and improved generalization capability of these DNNs is surprisingly still lacking. In this paper, we focus on investigating the role of skip connections for training very deep DNNs. Our exposition directly provides interesting insights and new interpretations to the following important questions (i) why model optimization is easier (ii) why model generalization is better. Theoretical results reveal that skip connections allow DNNs to circumnavigate the singularity of latent representations that translate to optimization and generalization problems, which plague models without skip connections referred to as PlainNets. For substantiating our analysis, our investigation puts into context some of the most successful skip-connection based DNNs, which include residual networks (ResNets) and residual network with aggregated features (ResNeXt) in relation to PlainNets. Experimental evaluations of these models support the theoretical analysis.

show abstract

“…It is still true without the continuum hypothesis, but it is more difficult to prove. See[1].3 In general, still assuming axiom of choice, you cannot get a set with some infinite cardinality κ, by taking finite products of sets with smaller cardinality, or by taking union of µ < κ sets with smaller cardinality.…”

mentioning

confidence: 99%

Cauchy’s Functional Equation

2001

Conjecture and Proof

View full text Add to dashboard Cite

Cauchy's functional equation, f (x + y) = f (x) + f (y), f : R → R looks very simple, and it has a class of simple solutions, f (x) = λx, but there are many other and more interesting solutions. In these notes, I will show you what some of these 'wild' solutions look like, and I will use them to prove that there exist a set A ⊂ R, such that neither A nor R \ A contains a measurable subset with positive measure. Section 1 is about Cauchy's functional equation on the rational numbers, in section 2 I show that there some wild solutions on R, and in section 3 I will show that their graphs are dense in R 2. In section 4 I'll show that these functions are ugly from a measure theoretical point of view, and in section 5, I'll show that some of these functions are wilder than others. E.g., I will prove that there is a solution to Cauchy's functional equation, that intersects any continuous function from R to R.

show abstract

Measure and Cardinality

Cited by 4 publications

References 4 publications

Why Is Everyone Training Very Deep Neural Network With Skip Connections?

Why Is Everyone Training Very Deep Neural Network With Skip Connections?

Training very deep neural networks: Rethinking the role of skip connections

Cauchy’s Functional Equation

Contact Info

Product

Resources

About