Sidestep cuts between 60° and 180° and one-leg landings have been identified as the main mechanisms of ACL injuries in several sports. This study sought to determine intra- and inter-rater reliability of a qualitative tool to assess high-risk movements in a 90° change of direction when the test is applied in a real framework of sport practice. Female footballers from two teams (n = 38) participated in this study and were asked to perform 90° cutting trials to each side, which were simultaneously filmed from a frontal and a sagittal view. A total of 61 cases were selected for 2D qualitative observational analysis by three raters. Poor reliability was found among each pair of raters as well as moderate reliability when the Cutting Movement Assessment Score (CMAS) was given by the same rater at different moments, but with too high a minimum detectable change. On the other hand, raters presented a significant, as well as moderate-to-good intra-rater reliability for most items of the CMAS tool. There was, however, non-significant reliability between observers in rating most check-points of the tool. For these reasons, more objective guidelines and clearer definitions for each criterion within the CMAS, as well as a longer, standardised training period for novel observers, would be highly recommended to improve the reliability of this tool in an applied context with female footballers.