Obese individuals exhibit an increase in pancreatic beta cell mass; conversely, scarce nutrition during pregnancy has been linked to beta cell insufficiency in the offspring [reviewed in 1, 2]. These phenomena are thought to be mediated mainly through effects on beta cell proliferation, given that a nutrient sensitive beta cell progenitor population in the pancreas has not been identified. Here, we employed the fluorescent ubiquitination-based cell-cycle indicator system to investigate beta cell replication in real time and found that high nutrient concentrations induce rapid beta cell proliferation. Importantly, we found that high nutrient concentrations also stimulate beta cell differentiation from progenitors in the intrapancreatic duct (IPD). Furthermore, using a new zebrafish line where beta cells are constitutively ablated, we show that beta cell loss and high nutrient intake synergistically activate these progenitors. At the cellular level, this activation process causes ductal cell reorganization as it stimulates their proliferation and differentiation. Notably, we link the nutrient-dependent activation of these progenitors to a downregulation of Notch signaling specifically within the IPD. Furthermore, we show that the nutrient sensor mechanistic target of rapamycin (mTOR) is required for endocrine differentiation from the IPD under physiological conditions as well as in the diabetic state. Thus, this study reveals critical insights into how cells modulate their plasticity in response to metabolic cues and identifies nutrient-sensitive progenitors in the mature pancreas.