We analyze the magnetostriction effect occurring in the magneto-sensitive elastomers (MSEs) containing inhomogeneously magnetized particles. As it was shown before, the expression for the interaction potential between two magnetic spheres, that accounts for their mutual inhomogeneous magnetization, can be obtained from the Laplace equation. We use this potential in the approximation formula form to construct magnetic energy of the sample in terms of the pairwise interactions of the particles. We show that this form of magnetic energy leads to the same demagnetizing factor as predicted by the continuum mechanics, confirming that only dipole–dipole magnetic interactions are important on a large scale. As the next step, we examine the role played by the particles arrangement on the magnetostriction effect. We consider different spatial distributions of the magnetic particles: a uniform one, as well as several lattice-type distributions (SC, BCC, HCP and FCC arrangements). We show that the particles arrangement affects significantly the magnetostriction effect if the separation between them became comparable with the particles dimensions. We also show that, typically, this contribution to the magnetostriction effect is of the opposite sign to the one related with the initial elastomer shape. Finally, we calculate the magnetostriction effect using the same interaction potential but expressed in a form of a series expansion, qualitatively confirming the above findings.