Autonomous aerial refueling (AAR) is an essential application of unmanned aerial vehicles for both military and civilian domains. In this paper, a hybrid algorithm of the pigeon-inspired optimization (PIO) and lateral inhibition (LI), called LI-PIO, is proposed for image matching problem of AAR. LI is adopted for image pre-processing to enhance the edges and contrast of images. PIO, inspired from the homing characteristics of pigeons, is a novel bio-inspired swarm intelligence algorithm. To demonstrate the effectiveness and feasibility of our proposed algorithm, we make extensive comparative experiments with particle swarm optimization (PSO), particle swarm optimization based on lateral inhibition (LI-PSO), and PIO. It can be concluded from the experimental results that our proposed LI-PIO has excellent performances for image matching problem of AAR, especially in convergent rate and computation speed.