The conventional gravity compensation algorithm requires precise dynamic parameters and a complex matrix transformation operation, which is difficult in applications to real-time control. In this paper, a simple and practical gravity compensation algorithm is proposed based on the space geometry characteristics of a mechanical arm and the principle of torque balance. This algorithm does not require a complex calculation of space coordinate transformation and does not require obtaining all accurate dynamic models and parameters. It only requires estimating the maximum gravity moment of the mechanical arm and simply calculating the trigonometric function. Thus, this algorithm can be extended to a non-parallel shaft mechanical arm, which is suitable for N joints in space. To verify the control effect after gravity compensation, the most easily comprehensible proportional-derivative controller combined with gravity compensation is used to control two-joint and three-joint mechanical arms for simulation. With the gravity compensation and non-compensation of the mechanical arm and with a comparison with other compensation methods, such as the fixed gravity compensation algorithm, the results show that the gravity compensation algorithm can achieve better trajectory tracking control, higher steady-state precision, an effectively reduced work burden of the controller and improved system stability.