In pull-based Peer-to-Peer video streaming systems, peers exchange buffer maps to reveal the availability of video chunks in their buffer. When collecting these buffer maps, a malicious party can infer the system's overlay structure and even identify head nodes, the direct communication partners of the stream's source. Attacking these head nodes can isolate peers from the source resulting in a disruption of the video dissemination for most peers in the system. We introduce a lightweight SWAP scheme, which allows peers to proactively change their partners, to reduce the chance of head nodes to be identified by such an inference attacker. Extensive simulation studies demonstrate that our scheme effectively undermines the attack's accuracy in identifying head nodes. So, SWAP lowers the chunk miss ratio while causing only a slight increase in signaling overhead.