Atrial Fibrillation (AF) patients could reduce their risk of stroke by using oral antithrombotic therapy. However, many older people with AF experience cognitive impairment and have limited health literacy, which can lead to non-adherence to antithrombotic treatment. This study aimed to investigate the influence of cognitive impairment and health literacy on non-adherence to antithrombotic therapy. The study performed a secondary analysis of baseline data from a cross-sectional survey of AF patients’ self-care behaviors at a tertiary university hospital in 2018. Data were collected from a total of 277 AF patients aged 65 years and older, through self-reported questionnaires administered by face-to-face interviews. Approximately 50.2% of patients were non-adherent to antithrombotic therapy. Multiple logistic regression analysis revealed that cognitive impairment independently increased the risk of non-adherence to antithrombotic therapy (odds ratio = 2.628, 95% confidence interval = 1.424–4.848) after adjustment for confounding factors. However, health literacy was not associated with non-adherence to antithrombotic therapy. Cognitive impairment is a significant risk factor for poor adherence to antithrombotic therapy. Thus, health professionals should periodically assess both cognitive function after AF diagnosis and adherence to medication in older patients. Further studies are needed to identify the factors that affect cognitive decline and non-adherence among AF patients.