Interaural time difference (ITD) is a critical cue to sound-source localization. Traditional models assume that sounds leading at one ear, and perceived on that side, are processed in the opposite midbrain. Using functional magnetic resonance imaging we demonstrate that as the ITDs of sounds increase, midbrain activity can switch sides, even though perceived location remains on the same side. The data require a new model for human ITD processing.