Objectives: The extended face network contains clusters of neurons that perform distinct functions on facial stimuli. Regions in the posterior ventral visual stream appear to perform basic perceptual functions on faces, while more anterior regions, such as the ventral anterior temporal lobe and amygdala, function to link mnemonic and affective information to faces. Anterior and posterior regions are interconnected by a long-range white matter tracts; however, it is not known if variation in connectivity of these pathways explains cognitive performance. Methods: Here, we used diffusion imaging and deterministic tractography in a cohort of 28 neurologically normal adults ages 18–28 to examine microstructural properties of visual fiber pathways and their relationship to certain mnemonic and affective functions involved in face processing. We investigated how inter-individual variability in two tracts, the inferior longitudinal fasciculus (ILF) and the inferior fronto-occipital fasciculus (IFOF), related to performance on tests of facial emotion recognition and face memory. Results: Results revealed that microstructure of both tracts predicted variability in behavioral performance indexed by both tasks, suggesting that the ILF and IFOF play a role in facilitating our ability to discriminate emotional expressions in faces, as well as to remember unique faces. Variation in a control tract, the uncinate fasciculus, did not predict performance on these tasks. Conclusions: These results corroborate and extend the findings of previous neuropsychology studies investigating the effects of damage to the ILF and IFOF, and demonstrate that differences in face processing abilities are related to white matter microstructure, even in healthy individuals. (JINS, 2016, 22, 180–190)