Is a periodic orbit underlying a periodic pattern of spikes in a heterogeneous neural network stable or unstable? We analytically assess this question in neural networks with delayed interactions by explicitly studying the microscopic time evolution of perturbations. We show that in purely inhibitorily coupled networks of neurons with normal dissipation (concave rise function), such as common leaky integrate-and-fire neurons, all orbits underlying non-degenerate periodic spike patterns are stable. In purely inhibitorily coupled networks with strongly connected topology and normal dissipation (strictly concave rise function), they are even asymptotically stable. In contrast, for the same type of individual neurons, all orbits underlying such patterns are unstable if the coupling is excitatory. For networks of neurons with anomalous dissipation ((strictly) convex rise function), the reverse statements hold. For the stable dynamics, we give an analytical lower bound on the local size of the basin of attraction. Numerical simulations of networks with different integrate-and-fire type neurons illustrate our results.