Salp Swarm Algorithm (SSA) is a new nature-inspired swarm optimization algorithm that mimics the swarming behavior of salps navigating and foraging in the oceans. SSA has been proved to enable to avoid local optima and enhance convergence speed benefiting from the adaptive nonlinear mechanism and salp chains. In this paper, visual tracking is considered to be a process of locating the optimal position through the interaction between leaders and followers in successive images. A novel SSA-based tracking framework is proposed and the analysis and adjustment of parameters are discussed experimentally. Besides, the qualitative analysis and quantitative analysis are performed to demonstrate the tracking effect of our proposed approach by comparing with ten classical tracking algorithms. Extensive comparative experimental results show that our algorithm has good performance in visual tracking, especially for abrupt motion tracking.