We propose and analyze a new numerical method for computing the ground state of the modified Gross-Pitaevskii equation for modeling the Bose-Einstein condensate with a higher order interaction by adapting the density function formulation and the accelerated projected gradient method. By reformulating the energy functional E(φ) with φ, the wave function, in terms of the density ρ = |φ| 2 , the original non-convex minimization problem for defining the ground state is then reformulated to a convex minimization problem. In order to overcome the semi-smoothness of the function √ ρ in the kinetic energy part, a regularization is introduced with a small parameter 0 < ε 1. Convergence of the regularization is established when ε → 0. The regularized convex optimization problem is discretized by the second order finite difference method. The convergence rates in terms of the density and energy of the discretization are established. The accelerated projected gradient method is adapted for solving the discretized optimization problem. Numerical results are reported to demonstrate the efficiency and accuracy of the proposed numerical method. Our results show that the proposed method is much more efficient than the existing methods in the literature, especially in the strong interaction regime.