Eukaryotic cells produce over 1,000 different lipid species that tune organelle membrane properties, control signalling and store energy^1,2. How lipid species are selectively sorted between organelles to maintain specific membrane identities is largely unclear, owing to the difficulty of imaging lipid transport in cells³. Here we measured the retrograde transport and metabolism of individual lipid species in mammalian cells using time-resolved fluorescence imaging of bifunctional lipid probes in combination with ultra-high-resolution mass spectrometry and mathematical modelling. Quantification of lipid flux between organelles revealed that directional, non-vesicular lipid transport is responsible for fast, species-selective lipid sorting, in contrast to the slow, unspecific vesicular membrane trafficking. Using genetic perturbations, we found that coupling between energy-dependent lipid flipping and non-vesicular transport is a mechanism for directional lipid transport. Comparison of metabolic conversion and transport rates showed that non-vesicular transport dominates the organelle distribution of lipids, while species-specific phospholipid metabolism controls neutral lipid accumulation. Our results provide the first quantitative map of retrograde lipid flux in cells⁴. We anticipate that our pipeline for mapping of lipid flux through physical and chemical space in cells will boost our understanding of lipids in cell biology and disease.