The stability criterion for the magnetic separation of rare-earth ions is studied, taking dysprosium Dy(iii) ions as an example. Emphasis is placed on quantifying the factors that limit the desired high enrichment. During magnetic separation, a layer enriched in Dy(iii) ions is generated via the surface evaporation of an aqueous solution which is levitated by the Kelvin force. Later, mass transport triggers instability in the enriched layer. The onset time and position of the instability is studied using an interferometer. The onset time signals that an advective process which significantly accelerates the stratification of enrichment is taking place, although the initial phase is quasi-diffusion-like. The onset position of the flow agrees well with that predicted with a generalized Rayleigh number (Ra∗=0) criterion which includes the Kelvin force term acting antiparallel to gravity. Further three-dimensional analysis of the potential energy, combining magnetic and gravitational terms, shows an energy barrier that has to be overcome to initiate instability. The position of the energy barrier coincides well with the onset position of the instability.