Stochastic Bigger Subspace Algorithms for Nonconvex Stochastic Optimization

Abstract

It is well known that the stochastic optimization problem can be regarded as one of the most hard problems since, in most of the cases, the values of f and its gradient are often not easily to be solved, or the F (•, ξ) is normally not given clearly and (or) the distribution function P is equivocal. Then an effective optimization algorithm is successfully designed and used to solve this problem that is an interesting work. This paper designs stochastic bigger subspace algorithms for solving nonconvex stochastic optimization problems. A general framework for such algorithm is presented for convergence analysis, where the socalled the sufficient descent property, the trust region feature, and the global convergence of the stationary points are proved under the suitable conditions. In the worst-case, we will turn out that the complexity is competitive under a given accuracy parameter. We will proved that the SF O-calls complexity of the presented algorithm with diminishing steplength is O( − 1 1−β ) and the SF O-calls complexity of the given algorithm with random constant steplength is O( −2 ) respectively, where β ∈ (0.5, 1) and is accuracy and the needed conditions are weaker than the quasi-Newton methods and the normal conjugate gradient algorithms. The detail algorithm framework with variance reduction is also proposed for experiments and the nonconvex binary classification problem is done to demonstrate the performance of the given algorithm.

Value Gain-based Power Line Subcarrier Aggregation Method under Colored Noise

“…According to the importance of each indicator to the value gain,  ,  ,  , and  are set to 5,  ,  ,  , and  are set to 10. The proposed algorithm is compared with no value gain subcarrier packet aggregation optimization algorithm (NVG-SPA) [11] and the random subcarrier packet aggregation optimization algorithm (R-SPA) [12]. NVG-SPA does not consider the gain value and network topology changes when optimizing subcarrier packet aggregation.…”

Wasserstein‐metric‐based distributionally robust optimization method for unit commitment considering wind turbine uncertainty

“…In the defined model, the following constraints should be satisfied. ) , (10) where P load n,t is the load power. The direct current power flow algorithm, discussed in Reference 35, describes the system's power balance.…”

“…It allows constraints containing uncertain variables of the wind turbine to hold at a certain confidence level during the optimization process, thus transforming uncertain optimization into deterministic optimization with a better economy 9 . However, it is hard to get the complete probability distribution information of uncertain parameters in practice 10 . Reference 11 constructed SO models based on advanced scene selection algorithms.…”

“…9 However, it is hard to get the complete probability distribution information of uncertain parameters in practice. 10 Reference 11 constructed SO models based on advanced scene selection algorithms. Reference 12 proposed SO models based on decomposition techniques.…”

Stochastic three-term conjugate gradient method with variance technique for non-convex learning

“…In order to overcome the shortcomings of the full gradient descent algorithm, many improved methods have appeared [12,13,31,32]. At the same time, in order to increase the convergence rate of SGD algorithms, Le Roux [35] proposed an SGD method with variance technology, and based on this work, more gradient methods with variance technology such as CGVR [8], SCGN [9], SCGA [1], and their common feature is that they all use the stochastic conjugate gradient method. Among them, CGVR and SCGA are both hybrid conjugate gradient methods, and both achieve linear convergence rates under strong convex conditions.…”