Evolution by natural selection has resulted in a remarkable diversity of organism morphologies that has long fascinated scientists and served to establish the first relations among species. Despite the essential role of morphology as a phenotype of species, there is not yet a formal, mathematical scheme to quantify morphological phenotype and relate it to both the genotype and the underlying developmental genetics. Herein we demonstrate that the morphological diversity in the beaks of Darwin's Finches is quantitatively accounted for by the mathematical group of affine transformations. Specifically, we show that all beak shapes of Ground Finches (genus Geospiza) are related by scaling transformations (a subgroup of the affine group), and the same relationship holds true for all the beak shapes of Tree, Cocos, and Warbler Finches (three distinct genera). This analysis shows that the beak shapes within each of these groups differ only by their scales, such as length and depth, which are genetically controlled by Bmp4 and Calmodulin. By measuring Bmp4 expression in the beak primordia of the species in the genus Geospiza, we provide a quantitative map between beak morphology and the expression levels of Bmp4. The complete morphological variation within the beaks of Darwin's finches can be explained by extending the scaling transformations to the entire affine group, by including shear transformations. Altogether our results suggest that the mathematical theory of groups can help decode morphological variation, and points to a potentially hierarchical structure of morphological diversity and the underlying developmental processes.