The Casimir effect, a two-body interaction via vacuum fluctuations, is a fundamental property of quantum systems. In solid state physics it emerges as a long-range interaction between two impurity atoms via virtual phonons. In the classical limit for the impurity atoms in D dimensions the interaction is known to follow the universal power law U(r)∼r-D. However, for finite masses of the impurity atoms on a lattice, it was predicted to be U(r)∼r-2D-1 at large distances. We examine how one power law can change into another with an increase of the impurity mass and in the presence of an external potential. We provide the exact solution for the system in one dimension. At large distances indeed U(r)∼r-3 for finite impurity masses, while for the infinite impurity masses or in an external potential it crosses over to U(r)∼r-1. At short distances the Casimir interaction is not universal and depends on the impurity mass and the external potential.