Introduction: White matter hyperintensities (WMHs) were commonly seen in brain magnetic resonance imaging (MRI) of the elderly. Many studies found that WMHs were associated with cognitive decline and dementia. However, the association between WMHs in different brain regions and cognitive decline remains debated. Methods: We explored the association of the severity of WMHs and cognitive decline in 115 non-demented elderly (≥50 years old) sampled from the Wuliqiao Community located in urban area of Shanghai. MRI scans were done during 2009–2011 at the beginning of the study. Severity of WMHs in different brain regions was scored by Improved Scheltens Scale and Cholinergic Pathways Hyperintensities Scale (CHIPS). Cognitive function was evaluated by Mini-Mental State Examination (MMSE) every 2 to 4 years during 2009–2018. Results: After adjusting for confounding factors including age, gender, education level, smoking status, alcohol consumption, depression, hypertension, diabetes, hyperlipidemia, brain infarcts, brain atrophy, apoE4 status, and baseline MMSE score, periventricular and subcortical WMH lesions as well as WMHs in cholinergic pathways were significantly associated with annual MMSE decline ( p < 0.05), in which the severity of periventricular WMHs predicted a faster MMSE decline (–0.187 points/year, 95% confidence interval: –0.349, –0.026, p = 0.024). Conclusions: The severity of WMHs at baseline was associated with cognitive decline in the non-demented elderly over time. Interventions on WMH lesions may offer some benefits for cognitive deterioration.