Some nursing homes for the elderly provide holistic care that integrates several interventions, including physical exercise. The aim of this systematic review is to summarize the effects of physical exercise or multicomponent exercise programs on the mental health (wellbeing, anxiety and depression) and cognitive functions of older adults with/without dementia who live in a nursing home and do/do not require wheelchair assistance. To this end, PubMed, PsycInfo and Web of Science are using to identify clinical trials and randomized controlled studies conducted during the period January 2011 to December 2021 to examine the progression of research in this field over the past ten years. In total, 2597 articles are identifying and 21 are including in the systematic review. After selecting articles according to the PRISMA standards, the data extraction and methodological quality assessment of the eligible studies are performing individually by two reviewers and then pooled together. The synthesis of the studies shows that physical exercise or multicomponent exercise programs have a beneficial effect on mental health and cognitive functions. However, this effect is more common among older adults without dementia, compared to their counterparts who have dementia or use wheelchairs. The explanatory mechanisms, lack of benefits from physical exercise and the need to standardize methods are discussing in this regard. Finally, future studies must examine the effects of an innovative and adaptive physical activity program on the mental health and cognitive functions of older adults living in nursing homes.