Altermagnetism, a type of magnetic order that combines properties of ferro- and antiferromagnets, has stirred great interest lately, not only as a promising source of spintronics applications, but also as a potential gateway to exotic phases of matter. Here, we demonstrate how to generalize collinear altermagnetism to SU(N) magnets with N>2. Guided by symmetry principles, we present a recipe to construct Heisenberg models for such generalized altermagnets and apply it explicitly for N=3, 4. Using flavor-wave theory, we compute the excitation spectrum of a two-dimensional SU(3) model and show that it exhibits magnon bands with altermagnetic splitting according to magnetic quantum numbers; we connect this quantum-number splitting to the frequently used concept of magnon chirality. We also compute the electronic band structure for a metallic system of the same symmetry and map out the polarization of the resulting flavor-split bands.