Despite significant therapeutic advances in recent years, treatment of metastatic prostate cancer (PCa) remains palliative, owing to the inevitable occurrence of drug resistance. There is increasing evidence that epithelial glucocorticoid receptor (GR) signaling and changes in the tumor-microenvironment (TME) play important roles in this process. Since glucocorticoids (GCs) are used as concomitant medications in the course of PCa treatment, it is essential to investigate the impact of GCs on stromal GR signaling in the TME. Therefore, general GR mRNA and protein expression was assessed in radical prostatectomy specimens and metastatic lesions. Elevated stromal GR signaling after GC treatment resulted in altered GR-target gene, soluble protein expression, and in a morphology change of immortalized and primary isolated cancer-associated fibroblasts (CAFs). Subsequently, these changes affected proliferation, colony formation, and 3D-spheroid growth of multiple epithelial PCa cell models. Altered expression of extra-cellular matrix (ECM) and adhesion-related proteins led to an ECM remodeling. Notably, androgen receptor pathway inhibitor treatments did not affect CAF viability. Our findings demonstrate that GC-mediated elevated GR signaling has a major impact on the CAF secretome and the ECM architecture. GC-treated fibroblasts significantly influence epithelial tumor cell growth and must be considered in future therapeutic strategies.