Human voice recognition is critical for many aspects of social communication. Recently, a rare disorder, developmental phonagnosia, which describes the inability to recognise a speaker's voice, has been discovered. The underlying neural mechanisms are unknown. Here, we used two functional magnetic resonance imaging experiments to investigate brain function in two behaviourally well characterised phonagnosia cases, both 32 years old: AS has apperceptive and SP associative phonagnosia. We found distinct malfunctioned brain mechanisms in AS and SP matching their behavioural profiles. In apperceptive phonagnosia, right-hemispheric auditory voice-sensitive regions (i.e., Heschl's gyrus, planum temporale, superior temporal gyrus) showed lower responses than in matched controls (nAS=16) for vocal versus non-vocal sounds and for speaker versus speech recognition. In associative phonagnosia, the connectivity between voice-sensitive (i.e. right posterior middle/inferior temporal gyrus) and supramodal (i.e. amygdala) regions was reduced in comparison to matched controls (nSP=16) during speaker versus speech recognition. Additionally, both cases recruited distinct potential compensatory mechanisms. Our results support a central assumption of current two-system models of voice-identity processing: They provide the first evidence that dysfunction of voice-sensitive regions and impaired connectivity between voice-sensitive and supramodal person recognition regions can selectively contribute to deficits in person recognition by voice.