The crystal family of potassium titanyl phosphate (KTiOPO4) is a promising material group for applications in quantum and nonlinear optics. The fabrication of low-loss optical waveguides, as well as high-grade periodically poled ferroelectric domain structures, requires a profound understanding of the material properties and crystal structure. In this regard, Raman spectroscopy offers the possibility to study and visualize domain structures, strain, defects, and the local stoichiometry, which are all factors impacting device performance. However, the accurate interpretation of Raman spectra and their changes with respect to extrinsic and intrinsic defects requires a thorough assignment of the Raman modes to their respective crystal features, which to date is only partly conducted based on phenomenological modelling. To address this issue, we calculated the phonon spectra of potassium titanyl phosphate and the related compounds rubidium titanyl phosphate (RbTiOPO4) and potassium titanyl arsenate (KTiOAsO4) based on density functional theory and compared them with experimental data. Overall, this allows us to assign various spectral features to eigenmodes of lattice substructures with improved detail compared to previous assignments. Nevertheless, the analysis also shows that not all features of the spectra can unambigiously be explained yet. A possible explanation might be that defects or long range fields not included in the modeling play a crucial rule for the resulting Raman spectrum. In conclusion, this work provides an improved foundation into the vibrational properties in the KTiOPO4 material family.