Swarmalators constitute a paradigmatic model for understanding the collective dynamics of coupled moving agents, integrating both internal and spatial degrees of freedom. Empirical evidence from systems such as bird flocks and living matter highlights the relevance of topological, metric-free coupling, but their impact on swarmalator dynamics remains largely unknown to date. Here, we present and analyze a topological swarmalator model in which the units interact topologically, on Delaunay networks. We find intriguing self-organized collective dynamics, including patterns with local vortices and unprecedented spatiotemporal patterns absent in metric-based models. Identifying three order parameters to quantify synchrony, spatial order, and vortex formation, we map the phase diagram that classifies these diverse patterns. Notably, we uncover a first-order transition even if the phases of all units are frozen, a dynamics inverted relative to the classical Kuramoto model. These insights not only advance our theoretical understanding of locally coupled systems of moving agents, but also offer key guidelines for their control.