Applying computer vision and machine learning techniques into sport tests is an effective way to realize “intelligent sports.” Facing practical application, we design a real-time and lightweight deep learning network to realize intelligent pull-ups test in this study. The main contributions are as follows: (1) a new self-produced pull-ups dataset is established under the requirement of including a human body and horizontal bar. In addition, a semiautomatic annotating software is developed to enhance annotation efficiency and increase labeling accuracy. (2) A novel lightweight deep network named PEPoseNet is designed to estimate and analyze a human pose in real time. The backbone of the network is made up of the heatmap network and key point network, which conduct human pose estimation based on the key points extracted from a human body and horizontal bar. The depth-wise separable convolution is adopted to speed up the training and convergence. (3) An evaluation criterion of intelligent pull-ups test is defined based on action quality assessment (AQA). The action quality of five states, i.e., ready or end, hang, pull, achieved, and resume in one pull-ups test cycle is automatically graded using a random forest classifier. A mobile application is developed to realize intelligent pull-ups test in real time. The performance of the proposed model and software is confirmed by verification and ablation experiments. The experimental results demonstrated that the proposed PEPoseNet has competitive performance to the state of the art. Its PCK @ 0.2 and frames per second (FPS) achieved were 83.8 and 30 fps, respectively. The mobile application has promising application prospects in pull-ups test under complex scenarios.