A robust fault-tolerant motion/force controller is vital for a fully-actuated unmanned aerial vehicle performing contact-based aerial manipulation tasks (e.g., window cleaning, infrastructure inspection and so on) in the presence of lumped disturbances (including external disturbances, model uncertainties, and actuator faults) with unknown boundaries. To address this problem, a disturbance-observer-based adaptive sliding mode impedance controller is proposed in this article. Using the impedance model as a reference model, the sliding mode impedance controller is constructed with a modified adaptive super-twisting gain technique to reduce the chattering phenomenon. To avoid overestimation of the unknown disturbance bounds, a novel adaptive third-order extended state observer is developed to actively estimate the disturbances and compensate for the controller without any priori knowledge of the disturbance bounds. Convergence of the proposed observer and stability of the closed-loop system are analyzed by the Lyapunov method. The effects of parameters in observers and controllers are presented under measurement noise through numerical simulations. In addition, guidelines for parameter selections are also provided. The advantages and effectiveness of the proposed control strategy are also demonstrated by simulating a push-and-slide scenario in the presence of lumped disturbances.