We present a data-driven framework for the multiscale modeling of anisotropic finite strain elasticity based on physics-augmented neural networks (PANNs). Our approach allows the efficient simulation of materials with complex underlying microstructures which reveal an overall anisotropic and nonlinear behavior on the macroscale. By using a set of invariants as input, an energy-type output and by adding several correction terms to the overall energy density functional, the model fulfills multiple physical principles by construction. The invariants are formed from the right Cauchy-Green deformation tensor and fully symmetric 2nd, 4th or 6th order structure tensors which enables to describe a wide range of symmetry groups. Besides the network parameters, the structure tensors are simultaneously calibrated during training so that the underlying anisotropy of the material is reproduced most accurately. In addition, sparsity of the model with respect to the number of invariants is enforced by adding a trainable gate layer and using lp regularization. Our approach works for data containing tuples of deformation, stress and material tangent, but also for data consisting only of tuples of deformation and stress, as is the case in real experiments. The developed approach is exemplarily applied to several representative examples, where necessary data for the training of the PANN surrogate model are collected via computational homogenization. We show that the proposed model achieves excellent interpolation and extrapolation behaviors. In addition, the approach is benchmarked against an NN model based on the components of the right Cauchy-Green deformation tensor.