<b><i>Introduction:</i></b> Brief, Web-based, and self-administered cognitive assessments hold promise for early detection of cognitive decline in individuals at risk for dementia. The current study describes the design, implementation, and convergent validity of a fWeb-based cognitive assessment tool, the Survey for Memory, Attention, and Reaction Time (SMART), for older adults. <b><i>Methods:</i></b> A community-dwelling sample of older adults (<i>n</i> = 69) was included, classified as cognitively intact (<i>n</i> = 44) or diagnosed with mild cognitive impairment (MCI, <i>n</i> = 25). Participants completed the SMART at home using their computer, tablet, or other Internet-connected device. The SMART consists of 4 face-valid cognitive tasks available in the public domain assessing visual memory, attention/processing speed, and executive functioning. Participants also completed a battery of standardized neuropsychological tests, a cognitive screener, and a daily function questionnaire. Primary SMART outcome measures consisted of subtest completion time (CT); secondary meta-metrics included outcomes indirectly assessed or calculated within the SMART (e.g., click count, total CT, time to complete practice items, and time of day the test was completed). <b><i>Results:</i></b> Regarding validity, total SMART CT, which includes time to complete test items, practice items, and directions, had the strongest relationship with global cognition (β = −0.47, <i>p</i> < 0.01). Test item CT was significantly greater for the MCI group (<i>F =</i> 5.20, <i>p</i> = 0.026). Of the SMART tasks, the executive functioning subtests had the strongest relationship with cognitive status as compared to the attention/processing speed and visual memory subtests. The primary outcome measures demonstrated fair to excellent test-retest reliability (intraclass correlation coefficient = 0.50–0.76). <b><i>Conclusions:</i></b> This study provides preliminary evidence for the use of the SMART protocol as a feasible, reliable, and valid assessment method to monitor cognitive performance in cognitively intact and MCI older adults.