Olfactory loss and traumatic brain injury (TBI) both lead to anatomical brain alterations in humans. Little research has been done on the structural brain changes for TBI patients with olfactory loss. Using voxel-based morphometry, the gray matter (GM) density was examined for 22 TBI patients with hyposmia, 24 TBI patients with anosmia, and 22 age-matched controls. Olfactory bulb (OB) volumes were measured by manual segmentation of acquired T2-weighted coronal slices using a standardized protocol. Brain lesions in the olfactory-relevant areas also were examined for TBI patients. Results showed that patients with anosmia have more frequent lesions in the OB, orbitofrontal cortex (OFC), and the temporal lobe pole, compared with patients with hyposmia. GM density in the primary olfactory area was decreased in both groups of patients. In addition, compared with controls, patients with anosmia showed GM density reduction in several secondary olfactory eloquent regions, including the gyrus rectus, medial OFC, anterior cingulate cortex, insula, and cerebellum. However, patients with hyposmia showed a lesser degree of GM reduction, compared with healthy controls. Smaller OB volumes were found for patients with olfactory loss, compared with controls. TBI patients with anosmia had the smallest OB volumes, which were caused by the lesions for OB. In addition, post-TBI duration was negatively correlated with GM density in the secondary olfactory areas in patients with hyposmia, but was positively correlated with GM density in the frontal and temporal gyrus in patients with anosmia. The GM density and OB volume reduction among TBI patients with olfactory loss was largely dependent on the location and severity of brain lesions in olfactory-relevant regions. Longer post-TBI duration had an impact on brain GM density changes, which indicate a decreased olfactory function in patients with hyposmia and possible compensatory mechanisms in patients with anosmia.