Collective cell migration plays a pivotal role in the formation of organs, tissue regeneration, wound healing and many disease processes, including cancer. Despite the considerable existing knowledge on the molecular control of cell movements, it is unclear how the different observed modes of collective migration, especially for small groups of cells, emerge from the known behaviors of individual cells. Here we derive a physical description of collective cellular movements from first principles, while accounting for known phenomenological cell behaviors, such as contact inhibition of locomotion and force-induced cell repolarization. We show that this theoretical description successfully describes the motion of groups of cells of arbitrary numbers, connecting single cell behaviors and parameters (e.g., adhesion and traction forces) to the collective migration of small groups of cells and the expansion of large cell colonies. Specifically, using a common framework, we explain how cells characterized by contact inhibition of locomotion can display coherent collective behavior when in groups, even in the absence of biochemical signaling. We find an optimal group size leading to maximal group persistence and show that cell proliferation prevents the buildup of intercellular forces within cell colonies, enabling their expansion.