Improving agricultural productivity is a priority concern in promoting the sustainable development of agriculture in developing countries. In this study, we first apply stochastic frontier analysis (SFA) to analyze the growth of agricultural total factor productivity (TFP) and its three components (technical change—TC, technical efficiency change—TEC and scale change—SC) in 15 south and southeast Asian countries covering the period 2002 to 2016. Then, the determinants of agricultural TFP growth are identified using dynamic panel data models. The results reveal that the south and southeast Asian countries witnessed an overall decline in agricultural productivity during the sample period, thereby creating concerns over sustaining future agricultural growth. Technical progress was the major source of TFP growth, but its contribution has slowed in recent years. On the other hand, declining scale change and technical efficiency change resulted in the deterioration of productivity over time. Variable levels of productivity performances were observed for individual countries, mainly driven by technological progress. Overall, southeast Asia achieved a more stable and sustained agricultural growth as compared to south Asia. Among the determinants, human capital, level of urbanization, and development flow to agriculture positively influenced agricultural TFP growth, while the level of economic development and agricultural import were negatively associated with TFP growth. Policy recommendations include the suggestions that south and southeast Asian countries should increase investment in human capital, focus on technological innovation and make use of financial assistance and development flow to agriculture to increase and sustain agricultural productivity. In addition, frontier countries of the two regions (e.g., India and Indonesia) should take the lead on regional agricultural development ventures by enhancing cooperation with neighboring countries on technological innovations, and countries facing diseconomies of scale (i.e., Afghanistan and Iran) should consider the rational reallocation of agricultural inputs.