Humans have the capacity to share others' emotions, be they positive or negative. Elicited by the observed or imagined emotion of another person, an observer develops a similar emotional state herself. This capacity, empathy, is one of the pillars of social understanding and interaction as it creates a representation of another's inner, mental state. Empathy needs to be dissociated from other social emotions and, crucially, also from cognitive mechanisms of understanding others, the ability to take others' perspective. Here, we describe the conceptual distinctions of these constructs and review behavioral and neural evidence that dissociates them. The main focus of the present review lies on the intraindividual changes in empathy and perspective-taking across the lifespan and on interindividual differences on subclinical and clinical levels. The data show that empathy and perspective-taking recruit distinct neural circuits and can be discerned already during early and throughout adult development. Both capacities also vary substantially between situations and people. Differences can be systematically related to situational characteristics as well as personality traits and mental disorders. The clear distinction of affect sharing from other social emotions like compassion and from cognitive perspective-taking, argues for a clear-cut terminology to describe these constructs. In our view, this speaks against using empathy as an umbrella term encompassing all affective and cognitive routes to understanding others. Unifying the way we speak about these phenomena will help to further research on their underlying mechanisms, psychopathological alterations, and plasticity in training and therapy.