Pedestrian detection and tracking based on computer vision has gradually become an international pattern recognition, which is one of the most active research topics in the field of computer vision and artificial intelligence. Using the theoretical results in the field of pattern recognition and computer vision technology, we are committed to detect and track pedestrians from video sequences. In addition to computer vision-based passer-by detection and tracking technology as the key, in the advanced computer vision action and analysis, it has a direct impact on the accuracy and robustness of its understanding. We analyzed various targets, such as subsequent recognition motion and pedestrian motion, and described them as high-level application processing, such as action understanding. In addition, because of the unique texture of human clothes compared with the surrounding natural landscape, they are highly “prominent” from the perspective of human visual system, and they are particularly prominent in the peripheral part of human contact with the background. In this paper, a binary function based on importance is proposed. As the space representation of image itself is not sensitive to noise and local signal, space representation is used. In addition, as an observation model, it can reduce the adverse effects of background noise and local noise on the tracking algorithm. Through the function block tracking, the pedestrian’s body can be tracked in detail. At the same time, the color band learning method is used to update the target template online to deal with the changes of target appearance caused by sunshine, pedestrian posture, and other factors. According to the experimental results, even if the appearance and posture of pedestrians change greatly, it has a stable tracking effect.