Neuropsychological test batteries are designed to assess cognition in detail by measuring cognitive performance in multiple domains. This study examines the factor structure of tests from the ARIC-NCS battery overall and across informative subgroups defined by demographic and vascular risk factors in a population of older adults. We analyzed neuropsychological test scores from 6413 participants in the Atherosclerosis Risk in Communities Neurocognitive Study (ARIC-NCS) examined in 2011-2013. Confirmatory Factor Analysis (CFA) was used to assess the fit of an a priori hypothesized three-domain model, and fit statistics were calculated and compared to one- and two-domain models. Additionally, we tested for stability (invariance) of factor structures among different subgroups defined by diabetes, hypertension, age, sex, race, and education. Mean age of participants was 76 years, 76% were White, and 60% were female. CFA on the a priori hypothesized three-domain structure, including memory, sustained attention and processing speed, and language, fit the data better (CFI=0.973, RMSEA=0.059) than the two-domain (CFI=0.960, RMSEA=0.070) and one-domain (CFI=0.947, RMSEA=0.080) models. BIC value was lowest, and QQ-Plots indicated better fit, for the three-domain model. Additionally, multiple-group CFA supported a common structure across the tested demographic subgroups, and indicated strict invariance by diabetes and hypertension status. In this community-based population of older adults with varying levels of cognitive performance, the a priori hypothesized three-domain structure fit the data well. The identified factors were configurally invariant by age, sex, race, and education, and strictly invariant by diabetes and hypertension status.