Current evidence suggests that severity and mortality of COVID-19 is higher in men than in women, whereas women might be at increased risk of COVID-19 reinfection and development of long COVID. Differences between sexes have been observed in other infectious diseases and in the response to vaccines. Sex-specific expression patterns of proteins mediating virus binding and entry, and divergent reactions of the immune and endocrine system, in particular the hypothalamic-pituitary-adrenal axis, in response to acute stress might explain the higher severity of COVID-19 in men. In this Personal View, we discuss how sex hormones, comorbidities, and the sex chromosome complement influence these mechanisms in the context of COVID-19. Due to its role in the severity and progression of SARS-CoV-2 infections, we argue that sexual dimorphism has potential implications for disease treatment, public health measures, and follow-up of patients predisposed to the development of long COVID. We suggest that sex differences could be considered in future pandemic surveillance and treatment of patients with COVID-19 to help to achieve better disease stratification and improved outcomes.