We report the detailed high-field and high-frequency electron spin resonance spectroscopic study of the single-crystalline van der Waals compounds Mn2P2S6 and MnNiP2S6. Analysis of magnetic excitations shows that in comparison to Mn2P2S6 increasing the Ni content yields a larger magnon gap in the ordered state and a larger g-factor value and its anisotropy in the paramagnetic state. The studied compounds are found to be strongly anisotropic having each the unique ground state and type of magnetic order. Stronger deviation of the g factor from the free electron value in the samples containing Ni suggests that the anisotropy of the exchange is an important contributor to the stabilization of a certain type of magnetic order with particular anisotropy. At temperatures above the magnetic order, we have analyzed the spin-spin correlations resulting in a development of slowly fluctuating short-range order. They are much stronger pronounced in MnNiP2S6 compared to Mn2P2S6. The enhanced spin fluctuations in MnNiP2S6 are attributed to the competition of different types of magnetic order. Finally, the analysis of the temperature-dependent critical behavior of the magnon gaps below the ordering temperature in Mn2P2S6 suggests that the character of the spin wave excitations in this compound undergoes a field-induced crossover from a 3D-like toward a 2D XY regime.