Three-dimensional gapless semiconductors with quadratic band touching, such as HgTe, α-Sn, or Pr2Ir2O7, are believed to display a non-Fermi-liquid ground state due to long-range electron-electron interaction. We argue that this state is inherently unstable towards spontaneous formation of a (topological) excitonic insulator. The instability can be parameterized by a critical fermion number Nc. For N<Nc the rotational symmetry is spontaneously broken, the system develops a gap in the spectrum, and features a finite nematic order parameter. To leading order in the 1/N expansion and in the static approximation, the analogy with the problem of dynamical mass generation in (2+1)-dimensional quantum electrodynamics yields Nc=16/[3π(π-2)]. Taking the important dynamical screening effects into account, we find that Nc≥2.6(2) and therefore safely above the physical value of N=1. Some experimental consequences of the nematic ground state are discussed.