Gaze perception includes both recognizing the location where another person gazes at in the surrounding environment and the judgment of whether another person is looking at ourselves or not.Previous studies have shown that gaze perception is influenced by many factors from the perspectives of the gazer and observer. In this paper, we reviewed and discussed these factors. Specifically, from the perspective of the gazer, we discussed the influence of the gazers' facial emotions and attractiveness and the physical characteristics of the eyes and the head on gaze perception. From the perspective of the observer, we discussed the influence of individuals' psychological disorders, current states, and the observer's gender, race and culture differences on gaze perception. Future research could develop innovative experimental designs and tasks to explore more about the role of stimulus type, the clinical application, and cross-cultural differences in the field of gaze perception.