The conflict of interests in agricultural land use based on the diversity of needs of private and public interest is the main problem of the current protection of agricultural land in Slovakia. Therefore, the aim of the paper is to identify factors affecting the withdrawal of agricultural land, i.e., conversion of the agricultural land to non-agricultural purposes, and to initiate a professional discussion on the concept of protection and use of the agricultural land in Slovakia. Through panel regression models, the developmental, territorial, and legislative factors affecting land withdrawal for the purpose of housing, industry, transport, mining, and other purposes were analyzed. Research has shown that developmental factors, compared to legislative ones, affect the total volume of agricultural land withdrawn in bigger scope. From the perspective of the conflict of interests between the individuals and state regarding land protection, the private interest prevails over the public one. As a consequence, agricultural land is withdrawn in suburbanized and attractive areas, where the land of the highest quality is mostly located. In accordance with the precautionary principle, the state should adopt a long-term conceptual document defining the areas of agricultural land use taking into account the impact of the developmental factors on the land protection.