A healthy diet is associated with lower risk of chronic disease. African Americans generally have poor diet quality and experience a higher burden of many chronic diseases. We examined the associations of demographic and psychosocial factors and barriers to diet quality among African American adults. This cross-sectional study included 100 African American adults in a southeastern metropolitan area. Psychosocial factors (social support, self-efficacy), and barriers to healthy eating were assessed with validated measures. Diet quality was assessed using the Healthy Eating Index (HEI-2010). Nested linear regressions were used to examine the association between the variables of interest and HEI scores. Participants reported having social support (M (mean) = 2.0, SD (standard deviation) = 0.6, range 0–3), high levels of self-efficacy (M = 3.1, SD = 0.7, range 1–4), and low barriers (M = 1.4, SD = 0.6, range 0–4) to engage in healthy eating but total mean HEI scores needed improvement (M = 54.8, SD = 10.9, range 27.1–70.0). Participants consumed significantly higher empty calories and lower whole fruits, dairy, and total protein foods than the national average. Barriers to healthy eating (b = −12.13, p = 0.01) and the interaction between age and barriers (b = 0.25, p = 0.02) were most strongly associated with lower HEI scores. Younger African Americans with the highest barriers to healthy eating had the lowest HEI scores. Culturally appropriate interventions targeting empty calories, barriers to healthy eating, and knowledge of the Dietary Guidelines for Americans are needed for African Americans.