A new adaptive tracking control problem of a nonlinear teleoperation system is considered in this article, which differs from the previous ones in that both kinematics and dynamics of manipulators are uncertain and perturbation of the actuator parameters exists due to overheating of the actuator. To overcome the first which is also the key problem, we propose a new adaptive tracking control framework which additionally contains a neural network mechanism and a robust adaptive mechanism. Based on the framework, an adaptive controller is further established to compensate for the perturbation of the actuator parameters online. By employing the neural networks and adaption method in control design, an extended teleoperation control system is effectively accommodated. With the Lyapunov theory, the tracking error is shown to converge to a residual around zero as the time goes to infinity. The effectiveness of the proposed scheme has been verified in the physical platform.