This article showcases the development of a dynamic Arbitrary Lagrangian Eulerian (ALE) formulation to account for inelastic material models within a finite element framework. Such a formulation is commonly utilized in research domains like fluid mechanics, fluid-structure interaction, quasi static remeshing techniques, and quasi static load movement. The work at hand describes the application of the ALE formulation to efficiently analyse structures subjected to moving loads in the field of transient inelastic solid mechanics. In particular, structures such as pavements, gantry crane girders etc., which are subjected to moving loads, can be numerically simulated, and their transient response in the relevant region around the load can be obtained without relying on moving loads. The focus of this article is to facilitate the treatment of history variables stemming from inelastic material models. Of particular interest is the advection procedure required to transport the history variables through the mesh, as the material appears to flow through it. The mathematical framework necessary to treat this advection process is described in detail, considering a nonlinear viscoelastic material model on a neo-Hookean base at finite deformations. Then, four methods for numerically achieving the advection are implemented within a transient finite element ALE formulation. These methods are compared against each other, and additionally with the conventional Lagrangian method for validation. The results demonstrate satisfactory agreement with conventional simulation methods, while offering a significant improvement in terms of computation speed. With the work at hand, the dynamic response of inelastic materials subjected to moving loads can be numerically simulated in a computationally efficient manner.