The accurate distinction between cities and villages plays a great role in the sustainable development of cities. The physical urban area is not artificially delimited, and reflects the objective distribution of the spatial scope of the urban area. Although there are many methods that have studied and explained the physical urban area to a certain extent, the influencing factors of identification threshold are not studied deeply. In this paper, 22 administrative districts in central and western Chongqing Municipality are taken as the research object. Based on vector building data, the urban expansion curve method is used to identify the optimal distance threshold to extract the physical urban area of Chongqing Municipality. And the geographical detector technique is used to detect respectively that how and to what extent the urban spatial structure factors, geographical environment factors and social economic factors affect the optimal distance threshold of 22 administrative districts. The results show that the optimal distance threshold for identifying physical urban area of Chongqing Municipality based on the vector building is 146m, and the physical urban area is 1598.43km2, with the error of 2.1% compared with the built-up area. Taking 22 administrative districts as the individual research objects, it is found that the road network density and building density in urban spatial structure factors, urbanization rate and urban population density in socio-economic factors, and their interaction with regional GDP play a critical role in the optimal distance threshold, and the index value of influence degree ≥ 0.79. Under the influence of different factors, the optimal distance thresholds of 22 administrative districts show adaptive characteristics. Looking forward to the future, quantitative analysis of the relationship between the structural scale, economic indicators, geographical environment and the optimal distance threshold of different levels of cities will help to understand the driving mechanism that affects physical urban area identification more deeply. The adaptive characteristics of the optimal distance threshold proposed in this paper also provide a theoretical basis for further study on the morphological characteristics and distribution laws of multi-scale cities.