Based on the sample data from 2005 to 2019, this paper calculates the poverty nature of contiguous destitute areas through FGT index and its decomposition and systematically analyzes the impact of economic growth, inequality, and population change on poverty change. From the decomposition results of poverty change, we can see that, first, economic growth, inequality, and population change have different impacts on poverty change in counties and rural areas, and inequality and population mobility have widened the gap between them; second, population factor has always played a key role in the change of poverty, and the deceleration of population growth has a more significant impact on poverty change; third, the impact of the mobility on the poverty change of the counties is different from that of the rural areas. Accordingly, the paper puts forward some countermeasures and suggestions, such as promoting the organic connection between rural revitalization and poverty alleviation, speeding up rural governance, and promoting the process of urbanization.