Skewed alpha-stable distribution for natural texture modeling and segmentation in contourlet domain

Sarafrazi, Katayoon; Yazdi, Mehran

doi:10.1186/s13640-019-0412-0

Cited by 3 publications

(2 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[Samorodnitsky and Taqqu, 1994, Property 1.2.15]). The family of α-stable distributions include normal, Lévy and Cauchy distributions as special cases, and can be used to model many complex stochastic phenomena [Sarafrazi and Yazdi, 2019, Fiche et al, 2013, Farsad et al, 2015.…”

Section: Proof Of Theoremmentioning

confidence: 99%

Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variance

Wang,

Gürbüzbalaban,

Zhu

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent studies have provided both empirical and theoretical evidence illustrating that heavy tails can emerge in stochastic gradient descent (SGD) in various scenarios. Such heavy tails potentially result in iterates with diverging variance, which hinders the use of conventional convergence analysis techniques that rely on the existence of the second-order moments. In this paper, we provide convergence guarantees for SGD under a state-dependent and heavy-tailed noise with a potentially infinite variance, for a class of strongly convex objectives. In the case where the p-th moment of the noise exists for some p ∈ [1, 2), we first identify a condition on the Hessian, coined 'p-positive (semi-)definiteness', that leads to an interesting interpolation between positive semi-definite matrices (p = 2) and diagonally dominant matrices with nonnegative diagonal entries (p = 1). Under this condition, we then provide a convergence rate for the distance to the global optimum in L p . Furthermore, we provide a generalized central limit theorem, which shows that the properly scaled Polyak-Ruppert averaging converges weakly to a multivariate α-stable random vector. Our results indicate that even under heavy-tailed noise with infinite variance, SGD can converge to the global optimum without necessitating any modification neither to the loss function or to the algorithm itself, as typically required in robust statistics. We demonstrate the implications of our results to applications such as linear regression and generalized linear models subject to heavy-tailed data.

show abstract

Section: Proof Of Theoremmentioning

confidence: 99%

Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variance

Wang,

Gürbüzbalaban,

Zhu

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…By their flexibility, such distributions can model many complex stochastic phenomena for which exact analytic forms are intractable (Sarafrazi and Yazdi, 2019;Fiche et al, 2013).…”

Section: Introductionmentioning

confidence: 99%

Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections

Camuto,

Wang,

Zhu

et al. 2021

Preprint

View full text Add to dashboard Cite

Gaussian noise injections (GNIs) are a family of simple and widely-used regularisation methods for training neural networks, where one injects additive or multiplicative Gaussian noise to the network activations at every iteration of the optimisation algorithm, which is typically chosen as stochastic gradient descent (SGD). In this paper we focus on the so-called 'implicit effect' of GNIs, which is the effect of the injected noise on the dynamics of SGD. We show that this effect induces an asymmetric heavy-tailed noise on SGD gradient updates. In order to model this modified dynamics, we first develop a Langevin-like stochastic differential equation that is driven by a general family of asymmetric heavy-tailed noise. Using this model we then formally prove that GNIs induce an 'implicit bias', which varies depending on the heaviness of the tails and the level of asymmetry. Our empirical results confirm that different types of neural networks trained with GNIs are well-modelled by the proposed dynamics and that the implicit effect of these injections induces a bias that degrades the performance of networks.

show abstract