A search for the nonresonant production of Higgs boson pairs in the Formula Presented channel is performed using Formula Presented of proton-proton collisions at a center-of-mass energy of 13 TeV recorded by the ATLAS detector at the CERN Large Hadron Collider. The analysis strategy is optimized to probe anomalous values of the Higgs boson self-coupling modifier Formula Presented and of the quartic Formula Presented (Formula Presented) coupling modifier Formula Presented. No significant excess above the expected background from Standard Model processes is observed. An observed (expected) upper limit Formula Presented is set at 95% confidence-level on the Higgs boson pair production cross section normalized to its Standard Model prediction. The coupling modifiers are constrained to an observed (expected) 95% confidence interval of Formula Presented (Formula Presented) and Formula Presented (Formula Presented), assuming all other Higgs boson couplings are fixed to the Standard Model prediction. The results are also interpreted in the context of effective field theories via constraints on anomalous Higgs boson couplings and Higgs boson pair production cross sections assuming different kinematic benchmark scenarios.