Rural settlement development in mountainous areas is the key to eliminating global hunger and poverty. The spatial reconstruction of rural settlements in mountainous areas can promote rural development in mountainous areas. In this study, the Panxi area—a typical mountainous area in China—was chosen as the study area. The driving forces for the spatial reconstruction of rural settlements in mountainous areas were explored from the perspective of peasant households by combining participatory rural appraisal (PRA) with structural equation modeling (SEM). Results showed that: (1) 62.03% of the 266 peasant households included were willing to have spatial reconstruction, indicating that most peasant households in mountainous areas have a very strong intention towards the spatial reconstruction of rural settlements. (2) Infrastructure, medical conditions, living environment, farming culture, and dietary habits significantly influenced the reconstruction intention of peasant households. In contrast, development opportunities, place attachment, language, and living mode each had a slight influence. (3) Geological disasters were the main driving force for the spatial reconstruction of rural settlements in mountainous areas, whilst the driving force of living cohesion was the smallest. This study provides insights for future planning and construction of rural settlements in the Panxi area and spatial reconstruction practices. It has important practical significance for overcoming poverty and realizing rural revitalization in mountainous areas.