This paper presents an in-depth analysis of tennis match scene classification using an adaptive Gaussian mixture model parameter estimation simulation algorithm. We divided the main components of semantic analysis into type of motion, distance of motion, speed of motion, and landing area of the tennis ball. Firstly, for the problem that both people and tennis balls in the video frames of tennis matches from the surveillance viewpoint are very small, we propose an adaptive Gaussian mixture model parameter estimation algorithm, which has good accuracy and speed on small targets. Secondly, in this paper, we design a sports player tracking algorithm based on role division and continuously lock the target player to be tracked and output the player region. At the same time, based on the displacement information of the key points of the player’s body and the system running time, the distance and speed of the player’s movement are obtained. Then, for the problem that tennis balls are small and difficult to capture in high-speed motion, this paper designs a prior knowledge-based algorithm for predicting tennis ball motion and landing area to derive the landing area of tennis balls. Finally, this paper implements a prototype system for semantic analysis of real-time video of tennis matches and tests and analyzes the performance indexes of the system, and the results show that the system has good performance in real-time, accuracy, and stability.