We study the critical behavior of a random field O(N ) spin model with a second-rank random anisotropy term in spatial dimensions 4 < d < 6, by means of the replica method and the 1/N expansion. We obtain a replica-symmetric solution of the saddle-point equation, and we find the phase transition obeying dimensional reduction. We study the stability of the replica-symmetric saddle point against the fluctuation induced by the second-rank random anisotropy. We show that the eigenvalue of the Hessian at the replica-symmetric saddle point is strictly positive. Therefore, this saddle point is stable and the dimensional reduction holds in the 1/N expansion. To check the consistency with the functional renormalization group method, we obtain all fixed points of the renormalization group in the large N limit and discuss their stability. We find that the analytic fixed point yielding the dimensional reduction is practically singly unstable in a coupling constant space of the given model with large N . Thus, we conclude that the dimensional reduction holds for sufficiently large N .