Searches are performed for nonresonant and resonant di-Higgs boson production in the bb¯γγ final state. The dataset used corresponds to an integrated luminosity of 139 fb-1 of proton-proton collisions at a center-of-mass energy of 13 TeV recorded by the ATLAS detector at the CERN Large Hadron Collider. No excess above the expected background is found and upper limits on the di-Higgs boson production cross sections are set. A 95% confidence-level upper limit of 4.2 times the cross section predicted by the Standard Model is set on pp→HH nonresonant production, where the expected limit is 5.7 times the Standard Model predicted value. The expected constraints are obtained for a background hypothesis excluding pp→HH production. The observed (expected) constraints on the Higgs boson trilinear coupling modifier κλ are determined to be [-1.5,6.7] ([-2.4,7.7]) at 95% confidence level, where the expected constraints on κλ are obtained excluding pp→HH production from the background hypothesis. For resonant production of a new hypothetical scalar particle X (X→HH→bb¯γγ), limits on the cross section for pp→X→HH are presented in the narrow-width approximation as a function of mX in the range 251 GeV≤mX≤1000 GeV. The observed (expected) limits on the cross section for pp→X→HH range from 640 fb to 44 fb (391 fb to 46 fb) over the considered mass range.