The infrared imaging system needs non-uniformity correction due to problems such as detector manufacturing process. The traditional correction method based on reference source will interrupt the imaging process. The scene-based correction method uses the algorithm to perform real-time non-uniformity correction update on the image data. The image transmission process will not be interrupted, but the ghost problem is always difficult to be solved. This paper analyzes the causes of ghosting in traditional scene correction algorithms and proposes an adaptive infrared scene correction algorithm based on neighborhood bias update. The algorithm uses the gray value of image pixels in adjacent frames to perform inter-frame motion estimation to avoid the ghosting problem caused by insufficient scene motion in the iterative process of the algorithm. The scene correction algorithm based on neighborhood bias update corrects the non-uniformity of the image. The experimental results show that the algorithm proposed in this paper has good convergence, and can avoid the generation of ghosts and effectively improve the image quality.