Despite the fact that the observation of cononsolvency was reported as early as four decades ago, its phase-transition mechanism is still under debate. In this work, we provided a comprehensive study of the phase behaviors of poly(N-isopropylacrylamide) (PNiPAAm) in sulfoxide or sulfone aqueous solutions. We observed a sharp collapse transition of PNiPAAm brushes in sulfoxide but not in sulfone aqueous solutions by equilibrium measurements of in situ spectroscopic ellipsometry. We found that the hydrogen-bond formation between sulfoxide oxygens and amide hydrogens of the polymer chains plays a critical role in regulating the cononsolvency of PNiPAAm. We have extended the concept of preferential adsorption by taking into account hydrophobic interactions between cosolvent molecules, which are adsorbed on the polymer by hydrogen bonds. This can explain the experimental observations of PNiPAAm brushes in these solvent mixtures and sheds light on understanding the phase behaviors of polymer solutions where hydrogen bonds and hydrophobic interaction play a critical role. Our results can also be of interest for the liquid-liquid phase separation in the living cell where the condensation of proteins bound to large biomacromolecules plays an essential role.