The microlayer framework is an innovative and powerful approach for the numerical simulation of heterogeneous materials, such as aggregate-matrix composites across multiple scales. In this study, the microlayer framework is extended for the first time to account for viscoelastic-elastoplastic material behavior. The kinematics of the representative volume element (RVE) at the microscale are designed to accurately capture the behavior of typical composites, such as asphalt or concrete. The constitutive equations at the microscale are developed independently of the macroscale, ensuring the necessary conditions for proper computational homogenization. The thermodynamically motivated scale transition is carried out using the principle of multiscale virtual power (PMVP). In numerical studies, it is shown by embedding classical material models at the micro level that homogenization leads to physically meaningful triaxial mechanical behavior at the macro level. It is demonstrated that with a suitable choice of microlayer geometry, the tensile-compressive anomaly of the stress-strain behavior observed in aggregate-matrix composites can be modeled without modifying the material model. Finally, the quality of the microlayer framework is shown by validating a triaxial test of an asphalt specimen with a complex cyclic harmonic axial and radial loading regime.