Cognitive impairment and especially memory disruption is a major complicating feature of the epilepsies. In this review we begin with a focus on the problem of memory impairment in temporal lobe epilepsy. We start with a brief overview of the early development of knowledge regarding the anatomic substrates of memory disorder in temporal lobe epilepsy, followed by discussion of the refinement of that knowledge over time as informed by the outcomes of epilepsy surgery (anterior temporal lobectomy) and the clinical efforts to predict those patients at greatest risk of adverse cognitive outcomes following epilepsy surgery. These efforts also yielded new theoretical insights regarding the function of the human hippocampus and a few examples of these insights are touched on briefly. Finally, the vastly changing view of temporal lobe epilepsy is examined including findings demonstrating that anatomic abnormalities extend far outside the temporal lobe, cognitive impairments extend beyond memory function, with linkage of these distributed cognitive and anatomic abnormalities pointing to a new understanding of the anatomic architecture of cognitive impairment in epilepsy. Challenges remain in understanding the origin of these cognitive and anatomic abnormalities, their progression over time, and most importantly, how to intervene to protect cognitive and brain health in epilepsy.