This review aims to systematically review the effects of physical exercise on the cognitive performance of patients with Alzheimer's disease (AD) and its mechanisms of action. Databases such as Web of Science, PubMed, EMBASE, and the Cochrane Central Register of Controlled Trials were searched until December 2021. A randomized controlled trial (RCT) to assess the effect of an exercise intervention (compared with no exercise) on patients with AD. The measures included cognitive function [Mini-Mental State Examination (MMSE), Alzheimer's Disease assessment scale-cognitive (ADAS-Cog), Montreal cognitive assessment scale (MoCA) and Executive Function (EF)]. The methodological quality of the included literature was assessed using the Physiotherapy Evidence Database (PEDro) scale. Twenty-two studies (n = 1647, mean age: 77.1 ± 6.3 years) were included in the systematic review, sixteen of which were included in the meta-analysis. A systematic review and meta-analysis revealed that physical exercise positively affects cognitive performance in older patients with AD. However, the positive effects of the intervention should be interpreted with caution considering the differences in methodological quality, type, frequency, and duration of exercise in the included studies. Future studies should consider the design rigor and specification of RCT protocols.