We demonstrate how non-Abelian geometric phases can be used to universally process a spin qubit in heavy hole quantum dots in the absence of magnetic fields. A time-dependent electric quadrupole field is used to perform any desired single-qubit operation by virtue of non-Abelian holonomy. During the proposed operations, the degeneracy of the time-dependent two-level system representing the qubit is not split. Since time-reversal symmetry is preserved and hyperfine coupling is known to be weak in spin qubits based on heavy holes, we expect very long coherence times in the proposed setup.