Little is known about the mechanisms through which neighborhood-level factors (e.g., social support, economic opportunity) relate to suboptimal availability of healthy foods in low-income urban communities. We engaged a diverse group of chain and local food outlet owners, residents, neighborhood organizations, and city agencies based in Baltimore, MD. Eighteen participants completed a series of exercises based on a set of pre-defined scripts through an interactive, iterative group model building process over a two-day community-based workshop. This process culminated in the development of causal loop diagrams, based on participants’ perspectives, illustrating the dynamic factors in an urban neighborhood food system. Synthesis of diagrams yielded 21 factors and their embedded feedback loops. Crime played a prominent role in several feedback loops within the neighborhood food system: contributing to healthy food being “risky food,” supporting unhealthy food stores, and severing social ties important for learning about healthy food. Findings shed light on a new framework for thinking about barriers related to healthy food access and pointed to potential new avenues for intervention, such as reducing neighborhood crime.