The tetraboride HoB₄ crystallizes in a tetragonal structure (space group P4/mbm), with the Ho atoms realizing a Shastry-Sutherland lattice. It orders antiferromagnetically at T_N1=7.1 K and undergoes a further magnetic transition at T_N2=5.7 K. The complex magnetic structures are attributed to competing order parameters of magnetic and quadrupolar origin with significant magnetoelastic coupling. Here, we investigate the response of the lattice of HoB₄ across the antiferromagnetic phase transitions by using low-temperature powder X-ray diffraction and ultrasound-velocity measurements, supported by crystal electric field (CEF) calculations. Below T_N2, the crystal structure of HoB₄ changes to monoclinic (space group P2₁/b) as a macroscopic manifestation of the quadrupolar ordering. Between 300 and 3.5 K, the total distortion amplitude is 0.46 Å and the relative volume change is 3.5×10⁻³. This structural phase transition is compatible with the huge softening of the modulus C₄₄ observed around T_N2 due to ferroquadrupolar order. A lattice instability developing immediately below T_N1 is seen consistently in X-ray and ultrasound data. The CEF analysis suggests a quasi-degenerated ground state for the Ho³⁺ ions in this system.