The safety of the quadrotor transport system (QTS) is always threatened by the suspension load swing under the fuselage during flight. At the same time, the QTS is affected by uncertainties such as changes in desired position and cable length. Trajectory planning is often neglected during the development of the controller. Moreover, there are some shortcomings in the most control methods such as the maximum speed, acceleration, payload swing, and transportation efficiency of the system that cannot be theoretically guaranteed. Therefore, a nonlinear tracking control method is proposed using the segmented energy, which can make the quadrotor move to the target position quickly and smoothly, meanwhile, suppress the heavy swing of the payload. Furthermore, it is proved that the method proposed in this paper can achieve the asymptotic stability of the system equilibrium point and converge to the expected value by Lyapunov theorem and LaSalle's invariance principle. Simulation results show that the control method achieves good control performance and strong robustness against system parameter uncertainty.