The Mermin-Wagner theorem forbids spontaneous symmetry breaking of spins in one/two-dimensional (2D) systems at a finite temperature and rules out the stabilization of this ordered state. However, it does not apply to all types of phase transitions in low dimensions, such as the topologically ordered phase rigorously shown by Berezinskii-Kosterlitz-Thouless (BKT) and experimentally realized in very limited systems such as superfluids and superconducting thin films. Quasi-2D van der Waals magnets provide an ideal platform to investigate the fundamentals of low-dimensional magnetism. We explored the quasi-2D honeycomb antiferromagnetic single crystals of (Ni _x Fe_1−x )₂P₂S₆ (x = 1, 0.7, 0.5, 0.3, and 0) using in-depth temperature-dependent Raman measurements supported by first-principles calculations of the phonon frequencies. Quite surprisingly, we observed renormalization of the phonon modes much below the long-range magnetic ordered temperature attributed to the topological ordered state, namely the BKT phase, which is also found to change as a function of doping. The extracted critical exponent of the order-parameter (spin-spin correlation length, ξ ( T ) ) evinces the signature of a topologically active state driven by vortex-antivortex excitations. As a function of doping, a tunable transition from paramagnetic to antiferromagnetic ordering is shown via phonons reflected in the strong renormalization of the self-energy parameters of the Raman active phonon modes. The extracted exchange parameter (J) is found to vary by ∼100% by increasing the value of doping, ranging from ∼6 meV (for x = 0.3) to 13 meV (for x = 1).