Fault tolerant control (FTC) is essential for rotorcraft UAV (RUAV). Due to the inherently unstable dynamics, either flight test or real application of a RUAV is in high risk while a minutial failure may lead to the whole system collapse. In this paper, a novel adaptive unscented Kalman filter (AUKF) is proposed for onboard failure coefficient estimation and a new FTC method is designed against the actuator failure of RUAV. In the AUKF, the error between the covariance matrices of innovation and their corresponding estimations/predictions in normal UKF is utilized as a cost function. Based on the MIT rule, an adaptive algorithm is developed to update the covariance of process noise by minimizing the cost function. The updated covariance is then fed back into the normal UKF. Such an adaptive mechanism intends to release the dependence of UKF on a prior knowledge of the noise environment and improve the convergence speed and estimation accuracy of normal UKF. By introducing the actuator health coefficients (AHCs) into the dynamics equation of a RUAV, the proposed AUKF is utilized to online estimate both the flight states and the AHCs. A fault adaptive control is further designed based on the estimated states and AHCs. Simulations are conducted on the dynamics of a model helicopter developed in Shenyang Institute of Automation. The comparisons between the adaptive-UKF-based FAC and the normal-UKF-based one show the effectiveness and improvements of the proposed method.