We investigate the role of quantum fluctuations in topological quantum phase transitions of quantum spin-Hall insulators and quantum anomalous-Hall insulators. Employing the variational cluster approximation to obtain the single-particle Green's function of the interacting many-body system, we characterize different phases by direct calculation of the recently proposed topological order parameter for interacting systems. We pinpoint the influence of quantum fluctuations on the quantum spin Hall to Mott insulator transition in several models. Furthermore, we propose a general mechanism by which a topological quantum phase transition can be driven by the divergence of the self-energy induced by interactions.