In listening to multi-part music, auditory streams can be attended to either selectively or globally. More specifically, musicians rely on prioritized integrative attention which incorporates both stream segregation and integration to assess the relationship between concurrent parts. In this fMRI study, we used a piano duet to investigate which factors of a leader-follower relationship between parts grab the listener's attention and influence the perception of multi-part music. The factors considered included the structural relationship between melody and accompaniment as well as the temporal relationship (asynchronies) between parts. The structural relationship was manipulated by cueing subjects to the part of the duet that had to be prioritized. The temporal relationship was investigated by synthetically shifting the onset times of melody and accompaniment to either a consistent melody or accompaniment lead. The relative importance of these relationship factors for segregation and integration as attentional mechanisms was of interest. Participants were required to listen to the cued part and then globally assess if the prioritized stream was leading or following compared to the second stream. Results show that the melody is judged as more leading when it is globally temporally ahead whereas the accompaniment is not judged as leading when it is ahead. This bias may be a result of the interaction of salience of both leader-follower relationship factors. Interestingly, the corresponding interaction effect in the fMRI-data yields an inverse bias for melody in a fronto-parietal attention network. Corresponding parameter estimates within the dlPFC and right IPS show higher neural activity for attending to melody when listening to a performance without a temporal leader, pointing to an interaction of salience of both factors in listening to music. Both frontal and parietal activation implicate segregation and integration mechanisms and a top-down influence of salience on attention and the perception of leader-follower relations in music.