Introduction. Although increased mean diffusivity of the white matter has been repeatedly linked to Alzheimer’s disease pathology, the underlying mechanism is not known. Methods. Here, we used ADNI-3 multishell diffusion magnetic resonance imaging data to separate the diffusion signal of the parenchyma from less hindered fluid pools within the white matter such as perivascular space fluid and fluid-filled cavities. Results. We found that the source of the pathological increase of the mean diffusivity is the increased nonparenchymal fluid, often found in lacunes and perivascular spaces. In this cohort, the cognitive decline was significantly associated with the fluid increase and not with the microstructural changes of the white matter parenchyma itself. The white matter fluid increase was dominantly observed in the sagittal stratum and anterior thalamic radiation. Discussion. These findings are positive steps toward understanding the pathophysiology of white matter alteration and its role in the cognitive decline.