Large changes have taken place in smallholder farming systems in South Asia’s coastal areas in recent decades, particularly related to cropping intensity, input availability, climate risks, and off-farm activities. However, few studies have investigated the extent to which these changes have impacted farm-level crop productivity, which is a key driver of food security and poverty in rainfed, low-input, rice-based systems. The objective of this study was to conduct an integrated assessment of variables related to socioeconomic status, farm characteristics, and crop management practices to understand the major factors influencing crop productivity and identify promising leverage points for sustainable development in coastal Bangladesh. Using a panel survey dataset of 32 variables from 502 farm households located within polder (coastal embankment) and outside polder systems during 2005–2015, we employed statistical factor analysis to characterize five independent latent factors named here as Farming Challenges, Economic Status, Crop Management Practices, Asset Endowment, and Farm Characteristics. The factor Farming Challenges explained the most variation among households (31%), with decreases observed over time, specifically households located outside polders. Individual variables contributing to this factor included perceived cyclone severity, household distance to main roads and input-output markets, cropping intensity, and access to extension services. The most important factors for increasing crop productivity on a household and per unit area basis were Asset Endowment and Crop Management Practices, respectively. The former highlights the need for increasing gross cropped area, which can be achieved through greater cropping intensity, while the latter was associated with increased fertilizer, labor, and pesticide input use. Despite the importance of these factors, household poverty trajectory maps showed that changes in off-farm income had played the strongest role in improving livelihoods in this coastal area. This study can help inform development efforts and policies for boosting farm-level crop productivity, specifically through agricultural intensification (higher cropping intensity combined with appropriate and efficient use of inputs) and expanding opportunities for off-farm income as key pathways to bring smallholder households out of poverty.