Adipose progenitor cells, or preadipocytes, constitute a small population of immature cells within the adipose tissue. They are a heterogeneous group of cells, in which different subtypes have a varying degree of commitment toward diverse cell fates, contributing to white and beige adipogenesis, fibrosis or maintenance of an immature cell phenotype with proliferation capacity. Mature adipocytes as well as cells of the immune system residing in the adipose tissue can modulate the function and differentiation potential of preadipocytes in a contact- and/or paracrine-dependent manner. In the course of obesity, the accumulation of immune cells within the adipose tissue contributes to the development of a pro-inflammatory microenvironment in the tissue. Under such circumstances, the crosstalk between preadipocytes and immune or parenchymal cells of the adipose tissue may critically regulate the differentiation of preadipocytes into white adipocytes, beige adipocytes, or myofibroblasts, thereby influencing adipose tissue expansion and adipose tissue dysfunction, including downregulation of beige adipogenesis and development of fibrosis. The present review will outline the current knowledge about factors shaping cell fate decisions of adipose progenitor cells in the context of obesity-related inflammation.