Autonomous celestial navigation has been exploited for orbit determination of deep space exploration. Geometric constraints in celestial measurement are the inherent attributes in actual comprehensive autonomous celestial navigation physical systems; they are usually neglected in autonomous navigation systems which causes a loss of information in measurement. For the purpose of high-precision autonomous celestial navigation, the geometric constraints should be utilized as fully as possible. This paper proposes a geometric coplanar constraint for the mutually dependent celestial measurement (line of sight or vectors), and the geometric coplanar constraint model is established. The sequence quadratic program (SQP) algorithm based on the geometric coplanar constraint is put forward to eliminate the dependence of multiple celestial measurements, and suppress the noises in celestial measurement geometrically. Taking both geometry coplanar constraints of celestial measurement and the nonlinear characteristics of system models into account, cubature Kalman filter with measurement optimization is proposed for decreasing the random noise in measurements geometrically and statistically. Simulations demonstrate that the proposed geometric coplanar constraints-aided autonomous celestial navigation method can effectively eliminate the measurement noise geometrically and statistically, and achieve high-precision performance.