Higher physical activity (PA) levels will obtain more health-related benefits for children and adolescents with intellectual disabilities (ID). The mastery of fundamental movement skills (FMS) potentially correlates with PA. This study aimed to examine the associations of FMS with moderate-to-vigorous intensity physical activity (MVPA) levels in children and adolescents with moderate to severe ID. Moreover, this research analyzes whether there are gender and age differences in the association between these two variables. A total of 93 children and adolescents with ID, aged 8–17 years (mean age = 13.27; SD = 3.35), were recruited from a special school located in western China. The time spent in MVPA was measured using waist-worn accelerometers. FMS proficiency was assessed using the Test of Gross Motor Development 2 (TGMD-2). Children and adolescents with ID tend to have delayed maturity of FMS patterns (locomotor skills t (92) = −16.91, p < 0.001, d = 2.48; object control skills t (92) = −25.39, p < 0.001, d = 3.72; total FMS t (92) = −21.83, p < 0.001, d = 3.20) and lower proficiency in objective control skills (t (92) = 3.989, p < 0.001, d = 0.29). A significant positive correlation was found between MVPA and FMS, and this association was moderated by gender and age. For boys, object control skills were a significant predictor of MVPA time (B = 0.842, p < 0.01), whereas locomotor skills were a significant predictor of MVPA time (B = 0.472, p < 0.05) for girls. For children with ID, object control skills were a significant predictor of MVPA time (B = 0.736, p < 0.05). Proficiency in FMS has a positive effect on increasing the level of MVPA in children and adolescents with ID. Gender and age factors should be considered when implementing FMS intervention programs.