Local linear convergence of stochastic higher-order methods for convex optimization

Lupu, Daniela; Necoara, Ion

doi:10.23919/ecc54610.2021.9654966

Cited by 2 publications

(9 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…, w r , and use the same decorrelation procedure as in FastICA. For this stochastic method one can derive the following convergence and descent results (see [11], [12]). Theorem 1: Assume that the functions f i have the second derivatives Lipschitz over the unit ball B.…”

Section: B Stochastic Second-order Taylor-based Methodsmentioning

confidence: 99%

“…FastICA is one of the most used ICA algorithms, which is based on full batch Newton type iterations [7]. The second algorithm is a stochastic Newton type method with a proper cubic regularization term that guarantees descent, originally developed in the paper [11].…”

Section: Optimization Algorithms For Icamentioning

confidence: 99%

“…Hence, in the next section we describe a stochastic variant of the cubic regularized Newton method for solving the finite sum problem (4). The algorithm, in full generality, was developed in [11], [12].…”

Section: Increase K Endmentioning

confidence: 99%

“…Various strategies have been recently proposed to scale-up inferential problems from big datasets. Besides parallelized and distributed approaches exploiting hardware architectures, several variants of the stochastic gradient descent method have been designed for accelerating the optimization [3], [11]. We develop a stochastic second-order Taylor-based algorithm adapted to the loss functions used in ICA, inspired from [11] (see also the extended version [12] of this paper).…”

Section: Introductionmentioning

confidence: 99%

“…Besides parallelized and distributed approaches exploiting hardware architectures, several variants of the stochastic gradient descent method have been designed for accelerating the optimization [3], [11]. We develop a stochastic second-order Taylor-based algorithm adapted to the loss functions used in ICA, inspired from [11] (see also the extended version [12] of this paper). In particular, we show that the ICA loss functions have the third derivatives bounded on the unit ball.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Dimensionality reduction of hyperspectral images using an ICA-based stochastic second-order optimization algorithm

Lupu,

Necoara,

Ionescu

et al. 2023

2023 European Control Conference (ECC)

View full text Add to dashboard Cite

Hyperspectral imaging is one of the advanced remote sensing techniques whose goal is to obtain the spectrum for each pixel in the image of a scene, with the purpose of finding objects, identifying materials or detecting processes. However, the high dimensional nature of hyperspectral images makes their analysis complex. Various methods have been developed to reduce the dimension of hyperspectral images. Most commonly used dimension reduction techniques are Principal Component Analysis (PCA) and Independent Component Analysis (ICA). PCA is a method to reduce the dimensionality by removing the correlation among the bands, while ICA finds additively independent components. FastICA is one of the most used ICA algorithms. It is based on maximizing a loss derived from the fourth order statistical moment (kurtosis) or negentropy, which are both non-convex functions. Moreover, FastICA can find irrelevant stationary points (no maxima) and is not scalable as it uses at each iteration the whole set of pixels. In this paper, we present a stochastic second-order Taylor-based algorithm adapted to such ICA non-convex loss functions. Our algorithm guarantees ascent, hence it usually identifies (local) maxima. Moreover, the algorithm since it is stochastic, is scalable. Detailed numerical simulations show the superior performance of our method compared to FastICA.

show abstract

Section: B Stochastic Second-order Taylor-based Methodsmentioning

confidence: 99%

Section: Optimization Algorithms For Icamentioning

confidence: 99%

Section: Increase K Endmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Dimensionality reduction of hyperspectral images using an ICA-based stochastic second-order optimization algorithm

Lupu,

Necoara,

Ionescu

et al. 2023

2023 European Control Conference (ECC)

View full text Add to dashboard Cite

show abstract

Stochastic Higher-Order Independent Component Analysis for Hyperspectral Dimensionality Reduction

Lupu

Necoara

Garrett

et al. 2022

IEEE Trans. Comput. Imaging

View full text Add to dashboard Cite

Hyperspectral imaging is a remote sensing technique that measures the spectrum of each pixel in the image of a scene. It can be used to detect objects or classify materials based on their optical reflectance spectra. Various methods have been developed to reduce the spectral dimension of hyperspectral images in order to facilitate their analysis. Independent Component Analysis (ICA) is a class of algorithms which extract statistically independent features. FastICA, is one of the most used ICA algorithms because it is simple and fast. However, FastICA often finds irrelevant stationary points (e.g., minima instead of maxima) and is not scalable as it uses at each iteration the whole set of pixels. In this paper, we present a new stochastic algorithm, called SHOICA, which smoothly approximates the non-convex loss functions of ICA using higher-order Taylor minorizers. Because SHOICA guarantees ascent of its objective function, it identifies (local) maxima. Moreover, because SHOICA is stochastic, it facilitates minibatching and thus is scalable and appropriate for large datasets. The quality of features extracted, as well as the time and epochs required by both FastICA and SHOICA are compared on dimensionality reduction and classification tasks of real hyperspectral images.

show abstract

Local linear convergence of stochastic higher-order methods for convex optimization

Cited by 2 publications

References 12 publications

Dimensionality reduction of hyperspectral images using an ICA-based stochastic second-order optimization algorithm

Dimensionality reduction of hyperspectral images using an ICA-based stochastic second-order optimization algorithm

Stochastic Higher-Order Independent Component Analysis for Hyperspectral Dimensionality Reduction

Contact Info

Product

Resources

About