A thermomagnetic generator is a promising technology to harvest low-grade waste heat and convert it into electricity. To make this technology competitive with other technologies for energy harvesting near room temperature, the optimum thermomagnetic material is required. Here we compare the performance of a state of the art thermomagnetic generator using gadolinium and La-Fe-Co-Si as thermomagnetic material, which exhibit strong differences in thermal conductivity and type of magnetic transition. gadolinium is the established benchmark material for magnetocaloric cooling, which follows the reverse energy conversion process as compared to thermomagnetic energy harvesting. Surprisingly, La-Fe-Co-Si outperforms gadolinium in terms of voltage and power output. Our analysis reveals the differences in thermal conductivity are less important than the particular shape of the magnetization curve. In gadolinium an unsymmetrical magnetization curve is responsible for an uncompensated magnetic flux, which results in magnetic stray fields. These stray fields represent an energy barrier in the thermodynamic cycle and reduce the output of the generator. Our detailed experiments and simulations of both, thermomagnetic materials and generator, clearly reveal the importance to minimize magnetic stray fields. This is only possible when using materials with a symmetrical magnetization curve, such as La-Fe-Co-Si.