Smart Gradient - An adaptive technique for improving gradient estimation

Abstract

<p style='text-indent:20px;'>Computing the gradient of a function provides fundamental information about its behavior. This information is essential for several applications and algorithms across various fields. One common application that requires gradients are optimization techniques such as stochastic gradient descent, Newton's method and trust region methods. However, these methods usually require a numerical computation of the gradient at every iteration of the method which is prone to numerical errors. We propose a simple limited-memory technique for improving the accuracy of a numerically computed gradient in this gradient-based optimization framework by exploiting (1) a coordinate transformation of the gradient and (2) the history of previously taken descent directions. The method is verified empirically by extensive experimentation on both test functions and on real data applications. The proposed method is implemented in the <inline-formula><tex-math id="M1">\begin{document}$\texttt{R} $\end{document}</tex-math></inline-formula> package <inline-formula><tex-math id="M2">\begin{document}$ \texttt{smartGrad}$\end{document}</tex-math></inline-formula> and in C<inline-formula><tex-math id="M3">\begin{document}$ \texttt{++} $\end{document}</tex-math></inline-formula>.</p>

A new avenue for Bayesian inference with INLA

“…Latent Gaussian models are a broad class of models such as Gaussian processes for time series, spatial and spatio-temporal data or clustered random effects models for data with a grouping structure. The formulation given in (1) can be seen as any model where each one of the f k (.) terms can be written in matrix form as A A A k u u u k .…”

“…that includes the linear predictors as the first n elements of X X X and the m model components as defined in (1). This augmentation results in a singular covariance matrix of X X X since η η η is a deterministic function of the other elements of X X X .…”

“…Based on the work of [1] we use the smart gradient approach to more accurately and efficiently calculate the gradient, and hence find the mode and the Hessian at the mode more accurately.…”

Parallelized integrated nested Laplace approximations for fast Bayesian inference

“…Often the directional derivatives are computed according to this canonical basis, however, noncanonical bases can just as well be used. INLA makes use of knowledge from previous iterations to choose directional derivatives exhibiting more robust numerical properties and hence faster overall convergence, see [16] for details. Independently of the choice of basis, the directional derivatives are computed for each component of θ and each time entail one or two function evaluations of f .…”

Approximate Bayesian Inference for the Interaction Types 1, 2, 3 and 4 with Application in Disease Mapping

“…The estimated gradient ∇π(θ θ θ|y y y) at each iteration for this algorithm and the estimated Hessian ∇ 2 π(θ θ θ|y y y) of π(θ θ θ|y y y) at θ θ θ * are approximated using numerical differentiation methods boosted by the Smart Gradient and Hessian Techniques [40]. We need to correct for the constraints at every configuration of the hyperparameter θ θ θ. INLA computes uncorrected mean x x x * un and uncorrected covariance Σ Σ Σ * un using sparse solvers, then it uses kriging technique to find x x x * and Σ Σ Σ * .…”