Education and related proxies for cognitive reserve (CR) are confounded by associations with environmental factors that correlate with cerebrovascular disease possibly explaining discrepancies between studies examining their relationships to cognitive aging and dementia. In contrast, sex-related memory differences may be a better proxy. Since they arise developmentally, they are less likely to reflect environmental confounds. Women outperform men on verbal and men generally outperform women on visuospatial memory tasks. Furthermore, memory declines during the preclinical stage of AD, when it is clinically indistinguishable from normal aging. To determine whether CR mitigates age-related memory decline, we examined the effects of gender and APOE genotype on longitudinal memory performances. Memory decline was assessed in a cohort of healthy men and women enriched for APOE ɛ4 who completed two verbal [Rey Auditory Verbal Learning Test (AVLT), Buschke Selective Reminding Test (SRT)] and two visuospatial [Rey-Osterrieth Complex Figure Test (CFT), and Benton Visual Retention Test (VRT)] memory tests, as well as in a separate larger and older cohort [National Alzheimer’s Coordinating Center (NACC)] who completed a verbal memory test (Logical Memory). Age-related memory decline was accelerated in APOE ɛ4 carriers on all verbal memory measures (AVLT, p=.03; SRT p<.001; logical memory p<.001) and on the VRT p=.006. Baseline sex associated differences were retained over time, but no sex differences in rate of decline were found for any measure in either cohort. Sex-based memory advantage does not mitigate age-related memory decline in either APOE ɛ4 carriers or non-carriers. (JINS, 2015, 21, 95–104)