CeRh2As2 has recently been reported to be a rare case of a multiphase unconventional superconductor close to a quantum critical point (QCP). Here, we present a comprehensive study of its normal-state properties and of the phase (I) below T0≈0.4 K which preempts superconductivity at Tc=0.26 K. The second-order phase transition at T0 presents signatures in specific heat and thermal expansion but none in magnetization and ac susceptibility, indicating a nonmagnetic origin of phase I. In addition, an upturn of the in-plane resistivity at T0 points to a gap opening at the Fermi level in the basal plane. Thermal expansion indicates a strong-positive-pressure dependence of T0, dT0/dp=1.5 K/GPa, in contrast to the strong-negative-pressure coefficient observed for magnetic order in Ce-based Kondo lattices close to a QCP. Similarly, an in-plane magnetic field shifts T0 to higher temperatures and transforms phase I into another nonmagnetic phase (II) through a first-order phase transition at about 9 T. Using renormalized band-structure calculations, we find that the Kondo effect (TK≈30 K) leads to substantial mixing of the excited crystalline-electric-field states into the ground state. This allows quadrupolar degrees of freedom in the resulting heavy bands at the Fermi level which are prone to nesting. The huge sensitivity of the quadrupole moment on hybridization together with nesting causes an unprecedented case of phase transition into a quadrupole-density-wave state at a temperature T0≪TK, which explains the nature of phases I and II.