On a constraint manifold we give an explicit formula for the Hessian matrix of a cost function that involves the Hessian matrix of a prolonged function and the Hessian matrices of the constraint functions. We give an explicit formula for the case of the orthogonal group O(n) by using only Euclidean coordinates on R n 2 . An optimization problem on SO(3) is completely carried out. Its applications to nonlinear stability problems are also analyzed.MSC: 53Bxx, 58A05, 58E50, 51F25, 34K20.