As the COVID-19 pandemic continues, an increasing number of different research studies focusing on various aspects of the pandemic are emerging. Most of the studies focus on the medical aspects of the pandemic, as well as on the impact of COVID-19 on various areas of life; less emphasis is put on analyzing the influence of socio-environmental factors on the spread of the pandemic. In this paper, using the geographically weighted regression method, the extent to which demographic, social, and environmental factors explain the number of cases of SARS-CoV-2 is explored. The research was performed for the case-study area of Poland, considering the administrative division of the country into counties. The results showed that the demographic factors best explained the number of cases of SARS-CoV-2; the social factors explained it to a medium degree; and the environmental factors explained it to the lowest degree. Urban population and the associated higher amount and intensity of human contact are the most influential factors in the development of the COVID-19 pandemic. The analysis of the factors related to the areas burdened by social problems resulting primarily from the economic exclusion revealed that poverty-burdened areas are highly vulnerable to the development of the COVID-19 pandemic. Using maps of the local R2 it was possible to visualize how the relationships between the explanatory variables (for this research—demographic, social, and environmental factors) and the dependent variable (number of cases of SARS-CoV-2) vary across the study area. Through the GWR method, counties were identified as particularly vulnerable to the pandemic because of the problem of economic exclusion. Considering that the COVID-19 pandemic is still ongoing, the results obtained may be useful for local authorities in developing strategies to counter the pandemic.