City shrinkage, as an ongoing worldwide phenomenon, is an issue for urban planning and regional development. City shrinkage is remarkable in Japan, with over 85% of municipalities experiencing population loss from 2005 to 2015. As Japan’s society ages and with its low fertility rate, city shrinkage has had a tremendous negative effect on economic development and urban planning. Understanding the spatial dependence and spatial heterogeneity of city shrinkage and its determinants is essential for ensuring the sustainable development of a city or region. In this study, a semiparametric geographically weighted regression (SGWR) model was adopted to explore the spatiotemporal differences in determinants of city shrinkage. The results reveal that the SGWR model incorporating the global and local variables is more interpretive compared to ordinary least squares and geographically weighted regression models in exploring the correlates of city shrinkage. We found the spatial dependence and heterogeneity of shrinking cities resulted from demographic, economy, and social factors, and revealed low fertility, the ageing population, and enterprise change ratio influenced city shrinkage in different regions at different times in Japan, whereas foreign population ratio, industry structure, and social welfare had global impacts. The findings provide useful information for understanding city shrinkage at global and local scales.