Elevated resting heart rate variability (HRV) in the high frequency range has been proposed to be protective against worrying when facing environmental stressors. Yet, prospective studies using real-life stressors are still scarce. The present study set out to replicate the previous finding of reduced resting HRV predicting COVID-19-associated worries in a larger, more homogenous sample over a longer period of time (N = 123; age: 42.32 [SD:10.72]; 65.9 % female; average time lag: six years). In addition, we were interested in investigating the specificity of this effect with respect to worry content, other physiological markers of autonomic functions, and additional potentially relevant covariates with a special focus on a potential moderating effect of sex on this association. In regression analyses adjusting for age, sex, BMI and smoking status, the interaction between HRV and sex was significant, with women depicting a stronger association between HRV and COVID-19 associated worries. Further sensitivity analyses revealed the specificity of the effect for HRV as distinct from mean heart rate, as well as its dependence on previous COVID-19 infection, but not COVID-19 vaccination status and chronic stress level. These data are in line with theories that propose that higher HRV levels can be protective against the deleterious effects of real-life environmental stressors. However, our results also point to the specificity of this effect, especially with respect to worry content and sex.