“…Stochastic gradient descent (SGD) is a prototypical stochastic optimization algorithm widely used for solving large scale data science problems [1,2,3,4,5,6], not only for its scalability to large datasets, but also due to its surprising capability of identifying parameters of deep neural network models with better generalization behavior than adaptive gradient methods [7,8,9]. The past decade has witnessed growing interests in accelerating this simple yet powerful optimization scheme [10,11,12,13,14,15], as well as better understanding its dynamics, through the lens of either discrete Markov chains [16,17] or continuous stochastic differential equations [18,19,20,21]. This paper introduces new techniques into the theoretical framework of diffusion approximation, which provides weak approximation to SGD algorithms through the solution of a modified stochastic differential equation (SDE).…”