In dentistry, endodontic instruments made of metallic and polymer materials are used for the disinfection of infected root canal systems. Due to their beneficial material properties, the use of polymers offers the advantages of a minimally invasive operation and a lower risk of breakage. In contrast, the metallic instruments used have an improved cleaning efficiency due to high-frequency oscillations. A simulation-based development method for the design of optimized polymer-based instruments that clean effectively is therefore being worked out. As starting point, an engineering analysis of the methodological and experimental fundamentals of the overall system has been carried out. The description of the instrument’s transient vibration behavior is performed by dynamic finite element analyses using a viscoelastic material model. The required material behavior of the selected polymer polyetheretherketone is characterized with the aid of a newly developed test setup. The resulting simulation model allows for the first time an analysis of the contact mechanical behavior of polymeric miniaturized structures under high-frequency vibration excitation. As a result, this model provides a realistic description of the vibration behavior and the stresses that occur. The knowledge gained will make a significant contribution to the targeted, material-specific and vibration-optimized design of future dental instruments for root canal irrigation.