We present results of experimental and theoretical study of the effect of external magnetic field on elastic properties of dense soft magnetic polymers filled with micron-sized magnetizable particles. The samples were cured without a magnetic field, thus with isotropic internal morphology of the particles disposition. Experiments demonstrate that under quite moderate magnetic fields the shear elastic modulus of the studied composites increased more than two orders of magnitudes. Decrease in the modulus with the value of the global shear deformation is detected. We explain these effects by adhesive isotropic agglomeration of the particles, at the stage of the composite polymerization. Under magnetic field, these agglomerates are magnetized and aggregate into chains, oriented in the field direction. The chain length is determined by competition between magnetic attraction of the agglomerates and the host medium elastic resistance to their translocation. Theoretical results of the proposed model quantitatively reproduce data’s of measurements of the composite elastic modulus.