We present a framework for the multiscale modeling of finite strain magneto-elasticity based on physics-augmented neural networks (NNs). By using a set of problem specific invariants as input, an energy functional as the output and by adding several non-trainable expressions to the overall total energy density functional, the model fulfills multiple physical principles by construction, e.g., thermodynamic consistency, material symmetry and a stress-free and non-magnetized unloaded configuration. Three NN-based models with varying requirements in terms of an extended polyconvexity condition and the growth condition of the magneto-elastic potential are presented. First, polyconvexity, which is a global concept, is enforced via input convex neural networks (ICNNs). Afterwards, we introduce the new concept of a relaxed local version of the polyconvexity and fulfill it in a weak sense by adding a tailored loss term. As an alternative, a loss term to enforce the weaker requirement of strong ellipticity locally is proposed, which can be favorable to obtain a better compatibility with data. Databases for training of the models are generated via computational homogenization for both compressible and quasi-incompressible magneto-active polymers (MAPs). To reduce the computational cost, 2D statistical volume elements and an invariant-based sampling technique for the pre-selection of relevant states are used. All models are calibrated by using the database, whereby interpolation and extrapolation are considered separately. Furthermore, the performance of the NN models is compared to a conventional model from the literature. The numerical study suggests that the proposed NN approach is advantageous over the conventional model for MAPs. Thereby, the two more flexible NN models in combination with the weakly enforced local polyconvexity lead to good results, whereas the model based only on ICNNs has proven to be too restrictive.