Active multi-focal vision systems are designed to perform visual tasks such as saccades and smooth pursuit movements. Saccades are fast movements to a given position, which is a time-optimal control problem. Smooth pursuit movement is used to follow moving objects and is a tracking control problem. Therefore we use a hybrid approach for the control of our vision system. For saccades we use sliding-mode control with a switching line, designed such that nearly time-optimal control performance is achieved. For smooth pursuit movement we use state-space control to attain adequate tracking performance. To widen the applicability of our multi-focal vision system, it is designed such that cameras are easily exchangeable. Thus system dynamics is subject to change, which implies the need for adaptive control. This paper presents the design of the resulting hybrid adaptive controller together with experimental results.