We study collective synchronization of pulse-coupled oscillators interacting on asymmetric random networks. We demonstrate that random matrix theory can be used to accurately predict the speed of synchronization in such networks in dependence on the dynamical and network parameters. Furthermore, we show that the speed of synchronization is limited by the network connectivity and remains finite, even if the coupling strength becomes infinite. In addition, our results indicate that synchrony is robust under structural perturbations of the network dynamics.