We report an observation of a giant magneto-optical (MO) effect quadratic in magnetization (Cotton-Mouton effect) in a 50 nm thick layer of yttrium-iron garnet (YIG). By a combined theoretical and experimental approach, we managed to quantify both linear and quadratic MO effects in the studied material. We show that the quadratic MO signal in the thin YIG film can exceed the linear MO response, reaching values of 450 μrad that are comparable with Heusler alloys or ferromagnetic semiconductors. This relative enhancement of the quadratic MO response relative to the linear MO response is attributed to antiferromagnetic coupling of two Fe sublattices. Furthermore, we demonstrate that a proper choice of experimental conditions, particularly with respect to the used wavelength, is crucial for optimization of the quadratic MO effect, which can be used very efficiently for magnetometry measurement.