The substitution of scandium in fullerene single-molecule magnets (SMMs) DySc₂N@C₈₀ and Dy₂ScN@C₈₀ by lutetium has been studied to explore the influence of the diamagnetic metal on the SMM performance of dysprosium nitride clusterfullerenes. The use of lutetium led to an improved SMM performance of DyLu₂N@C₈₀, which shows a higher blocking temperature of magnetization (T_B=9.5 K), longer relaxation times, and broader hysteresis than DySc₂N@C₈₀ (T_B=6.9 K). At the same time, Dy₂LuN@C₈₀ was found to have a similar blocking temperature of magnetization to Dy₂ScN@C₈₀ (T_B=8 K), but substantially different interactions between the magnetic moments of the dysprosium ions in the Dy₂MN clusters. Surprisingly, although the intramolecular dipolar interactions in Dy₂LuN@C₈₀ and Dy₂ScN@C₈₀ are of similar strength, the exchange interactions in Dy₂LuN@C₈₀ are close to zero. Analysis of the low-frequency molecular and lattice vibrations showed strong mixing of the lattice modes and endohedral cluster librations in k-space. This mixing simplifies the spin–lattice relaxation by conserving the momentum during the spin flip and helping to distribute the moment and energy further into the lattice.