Background The goals of glycemic management for patients with diabetes are to prevent or delay complications and optimize quality of life. However, in clinical practice, the recommended glycemic control target is difficult to achieve. Therefore, it is important to identify factors that influence the outcomes of glycemia to improve the quality of diabetic management. The study aimed to evaluate the level and factors associated with glycemic control among type 2 diabetic outpatients at Tikur Anbessa Specialized Hospital, Addis Ababa, Ethiopia. Methods A hospital-based cross-sectional study was conducted among systematically selected 325 patients with type 2 diabetes who attended diabetic clinics at Tikur Anbessa Specialized Hospital. Pretested, structured, and interviewer-administered questionnaires were used to collect sociodemographic and diabetes-related information from March 1 to May 30, 2021. HbA1c was used to assess glycemic control according to the HbA1c target of < 7% (‘good’ control) as recommended by the American Diabetes Association for non-pregnant adults. The HbA1c level in the range of 7–8% was defined as ‘inadequate’ control and ‘poor’ at levels > 8%. Data entry and analysis were performed using SPSS v26. Multivariate logistic regression analysis was used to identify determinants of glycemic control. Results The median level of HbA1c of the participants was 8.4% (IQR 6.8–10.1). And approximately three-quarters (73.8%) of the patients had inadequate and poor glycemic control (HbA1c ≥ 7%). Older age (AOR: 2.46, 95% CI: 1.28–6.01), DM duration of > 10 years (AOR: 3.15, 95% CI: 2.22–6.54), insulin therapy (AOR: 3.07, 95% CI: 2.10–6.12), poor diet compliance (AOR: 1.97, 95% CI: 1.28–3.52) and failure to set goals for glycemic control (AOR: 3.42, 95% CI: 2.17–5.97) were factors associated with inadequate and poor glycemic control. Conclusions The study revealed that a significant number of diabetic patients had inadequate and poor glycemic control levels. And this was associated with older age, longer duration of DM, insulin therapy, poor diet compliance, and failure to set control goals. This requires a focus on the associated factors identified and tailored management mechanisms to maintain good glycemic control.