Phase transitions in flexible metal-organic frameworks or soft porous crystals are mediated by low-frequency phonons or rigid-unit modes. The alteration of specific building blocks may change the lattice dynamics of these frameworks, which can influence the phase transition mechanism. In this work, the impact of building block substitution on the rigid-unit modes in flexible MIL-53 analogs with a winerack topology will be investigated via ab initio lattice dynamics calculations. First, the accuracy of the theoretical simulations is verified via experimental Raman measurements, which provide unique fingerprint vibrations in the terahertz range to characterize the phase transition. Following analysis of the low-frequency vibrations shows that there exists a set of universal rigid-unit modes inducing translations and/or rotations of the building blocks. The theoretical results demonstrate that linker substitutions have a large effect on the rigid-unit mode frequencies, whereas this is less so for inorganic chain substitutions. These findings may help to rationally tune the phonon frequencies in soft porous crystals.