Chunk Dynamic Updating for Group Lasso with ODEs

Abstract

Group Lasso is an important sparse regression method in machine learning which encourages selecting key explanatory factors in a grouped manner because of the use of L-2,1 norm. In real-world learning tasks, some chunks of data would be added into or removed from the training set in sequence due to the existence of new or obsolete historical data, which is normally called dynamic or lifelong learning scenario. However, most of existing algorithms of group Lasso are limited to offline updating, and only one is online algorithm which can only handle newly added samples inexactly. Due to the complexity of L-2,1 norm, how to achieve accurate chunk incremental and decremental learning efficiently for group Lasso is still an open question. To address this challenging problem, in this paper, we propose a novel accurate dynamic updating algorithm for group Lasso by utilizing the technique of Ordinary Differential Equations (ODEs), which can incorporate or eliminate a chunk of samples from original training set without retraining the model from scratch. Specifically, we introduce a new formulation to reparameterize the adjustment procedures of chunk incremental and decremental learning simultaneously. Based on the new formulation, we propose a path following algorithm for group Lasso regarding to the adjustment parameter. Importantly, we prove that our path following algorithm can exactly track the piecewise smooth solutions thanks to the technique of ODEs, so that the accurate chunk incremental and decremental learning can be achieved. Extensive experimental results not only confirm the effectiveness of proposed algorithm for the chunk incremental and decremental learning, but also validate its efficiency compared to the existing offline and online algorithms.

When Online Learning Meets ODE: Learning without Forgetting on Variable Feature Space

“…Comparison with Homotopy Method While there are likely to have some mathematical similarities, our DFLS differs from the family of homotopy algorithms (Watson 1986;Nocedal and Wright 1999;Watson 2001). As mentioned previously, the work of (Garrigues and Ghaoui 2008) (for Lasso) and (Hofleitner et al 2014(Hofleitner et al , 2013Li and Gu 2022) (for generalized Lasso) also incorporate instrumental variable(s) in order to realize the online learning, however, the authors achieve this by a series of linear systems, rather than specifying an ODE-based structure as in our approach. More to the point, they are all focused on the sample-wise online learning and limited to mere square loss.…”