The epsilon 4 (e4) isoform of apolipoprotein E (ApoE) is a known genetic risk factor for suboptimal brain health. Morphometry studies of brains with Alzheimer’s disease have reported significant alterations in temporal lobe brain structure of e4 carriers, yet it remains unclear if the presence of an e4 allele is associated with alterations in the microstructure of white matter fiber bundles in healthy populations. The present study used quantitative tractography based on diffusion tensor imaging (qtDTI) to examine the influence of the e4 allele on temporal lobe fiber bundle lengths (FBLs) in 64 healthy older adults with at least one e4 allele (carriers, N = 23) versus no e4 allele (non-carriers, N = 41). Subtests from the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) were also analyzed to examine memory performance between groups. Analyses revealed shorter FBLs in the left uncinate fasciculus (UF) (p = .038) of e4 carriers compared to non-carriers. By contrast, neither FBLs specific to the temporal lobe nor memory performances differed significantly between groups. Increased age correlated significantly with shorter FBL in the temporal lobe and UF, and with decreased performance on tests of memory. This is the first study to utilize qtDTI to examine relationships between FBL and ApoE genotype. Results suggest that FBL in the UF is influenced by the presence of an ApoE e4 allele (ApoE4) in healthy older adults. Temporal lobe FBLs, however, are more vulnerable to aging than the presence of an e4 allele.