We present and evaluate a bilateral filter for smoothing diffusion MRI fiber orientations with preservation of anatomical boundaries and support for multiple fibers per voxel. Two challenges in the process are the geometric structure of fiber orientations and the combinatorial problem of matching multiple fibers across voxels. To address these issues, we define distances and local estimators of weighted collections of multi-fiber models and show that these provide a basis for an efficient bilateral filtering algorithm for orientation data. We evaluate our approach with experiments testing the effect on tractography-based reconstruction of fiber bundles and response to synthetic noise in computational phantoms and clinical human brain data. We found this to significantly reduce the effects of noise and to avoid artifacts introduced by linear filtering. This approach has potential applications to diffusion MR tractography, brain connectivity mapping, and cardiac modeling.