Progressive hearing loss is a common phenomenon in healthy aging and may affect the perception of emotions expressed in speech. Elderly with mild to moderate hearing loss often rate emotional expressions as less emotional and display reduced activity in emotion-sensitive brain areas (e.g., amygdala). However, it is not clear how hearing loss affects cognitive and emotional control mechanisms engaged in multimodal speech processing. In previous work we showed that negative, task-relevant and -irrelevant emotion modulates the two types of control in younger and older adults without hearing loss. To further explore how reduced hearing capacity affects emotional and cognitive control, we tested whether moderate hearing loss (>30 dB) at frequencies relevant for speech impacts cognitive and emotional control. We tested two groups of older adults with hearing loss (HL; N = 21; mean age = 70.5) and without hearing loss (NH; N = 21; mean age = 68.4). In two EEG experiments participants observed multimodal video clips and either categorized pronounced vowels (cognitive conflict) or their emotions (emotional conflict). Importantly, the facial expressions were either matched or mismatched with the corresponding vocalizations. In both conflict tasks, we found that negative stimuli modulated behavioral conflict processing in the NH but not the HL group, while the HL group performed at chance level in the emotional conflict task. Further, we found that the amplitude difference between congruent and incongruent stimuli was larger in negative relative to neutral N100 responses across tasks and groups. Lastly, in the emotional conflict task, neutral stimuli elicited a smaller N200 response than emotional stimuli primarily in the HL group. Consequently, age-related hearing loss not only affects the processing of emotional acoustic cues but also alters the behavioral benefits of emotional stimuli on cognitive and emotional control, despite preserved early neural responses. The resulting difficulties in the multimodal integration of incongruent emotional stimuli may lead to problems in processing complex social information (irony, sarcasm) and impact emotion processing in the limbic network. This could be related to social isolation and depression observed in the elderly with age-related hearing loss.