In this contribution, a physically motivated continuum mechanical model for asphalt concrete and stone mastic asphalt (asphalt material) at large deformations is proposed. The rheological model consists of a short-term and a long-term part connected in series governing the description of the short-term and the long-term behavior (volume preserving) of the asphalt material, respectively. The rheological element assembly of the short-term part is physically motivated by the particle size distribution of the asphalt material under investigation (choice of several networks formed by different size fractions of the aggregate mixture) and represents the asphalt material's mainly viscoelastic behavior to short-term loading, e.g. during contact with a rolling tire. The long-term part captures large inelastic (accumulated) deformations due to repetitive loading and enables to represent e.g. rut formation during a structural pavement analysis. An experimental testing program including digital image correlation (DIC) for strain measurements on cylindrical samples was set up and enabled the identification of the model parameters. Furthermore, specific model parameters have been identified from the information provided by the manufacturer's data sheet of the asphalt material based on the particle size distribution of aggregates, the binder content and the void content.