Objective: On continuous recognition tasks, changing the context objects are embedded in impairs memory. Older adults are worse on pattern separation tasks requiring identification of similar objects compared to younger adults. However, how contexts impact pattern separation in aging is unclear. The apolipoprotein (APOE) ϵ4 allele may exacerbate possible age-related changes due to early, elevated neuropathology. The goal of this study is to determine how context and APOE status affect pattern separation among younger and older adults. Method: Older and younger ϵ4 carriers and noncarriers were given a continuous object recognition task. Participants indicated if objects on a Repeated White background, Repeated Scene, or a Novel Scene were old, similar, or new. The proportions of correct responses and the types of errors made were calculated. Results: Novel scenes lowered recognition scores compared to all other contexts for everyone. Younger adults outperformed older adults on identifying similar objects. Older adults misidentified similar objects as old more than new, and the repeated scene exacerbated this error. APOE status interacted with scene and age such that in repeated scenes, younger carriers produced less false alarms, and this trend switched for older adults where carriers made more false alarms. Conclusions: Context impacted recognition memory in the same way for both age groups. Older adults underutilized details and over relied on holistic information during pattern separation compared to younger adults. The triple interaction in false alarms may indicate an even greater reliance on holistic information among older adults with increased risk for Alzheimer’s disease.