Herein, we investigate the structure-property relationships of soft magnetorheological elastomers (MREs) filled with remanently magnetizable particles. The study is motivated from experimental results which indicate a large difference between the magnetization loops of soft MREs filled with NdFeB particles and the loops of such particles embedded in a comparatively stiff matrix, e.g. an epoxy resin. We present a microscale model for MREs based on a general continuum formulation of the magnetomechanical boundary value problem which is valid for finite strains. In particular, we develop an energetically consistent constitutive model for the hysteretic magnetization behavior of the magnetically hard particles. The microstructure is discretized and the problem is solved numerically in terms of a coupled nonlinear finite element approach. Since the local magnetic and mechanical fields are resolved explicitly inside the heterogeneous microstructure of the MRE, our model also accounts for interactions of particles close to each other. In order to connect the microscopic fields to effective macroscopic quantities of the MRE, a suitable computational homogenization scheme is used. Based on this modeling approach, it is demonstrated that the observable macroscopic behavior of the considered MREs results from the rotation of the embedded particles. Furthermore, the performed numerical simulations indicate that the reversion of the sample’s magnetization occurs due to a combination of particle rotations and internal domain conversion processes. All of our simulation results obtained for such materials are in a good qualitative agreement with the experiments.