Recent research has pushed the efficiency of colloidal quantum dot solar cells toward a level that spurs commercial interest. Quantum dot/metal oxide bilayers form the most efficient colloidal quantum dot solar cells, and most studies have advanced the understanding of the quantum dot component. We study the interfacial recombination process in depleted heterojunction colloidal quantum dot (QD) solar cells formed with ZnO as the oxide by varying (i) the carrier concentration of the ZnO layer and (ii) the density of intragap recombination sites in the QD layer. We find that the open-circuit voltage and efficiency of PbS QD/ZnO devices can be improved by 50% upon doping of the ZnO with nitrogen to reduce its carrier concentration. In contrast, doping the ZnO did not change the performance of PbSe QD/ZnO solar cells. We use X-ray photoemission spectroscopy, ultraviolet photoemission spectroscopy, transient photovoltage decay measurements, transient absorption spectroscopy, and intensity-dependent photocurrent measurements to investigate the origin of this effect. We find a significant density of intragap states within the band gap of the PbS quantum dots. These states facilitate recombination at the PbS/ZnO interface, which can be suppressed by reducing the density of occupied states in the ZnO. For the PbSe QD/ZnO solar cells, where fewer intragap states are observed in the quantum dots, the interfacial recombination channel does not limit device performance. Our study sheds light on the mechanisms of interfacial recombination in colloidal quantum dot solar cells and emphasizes the influence of quantum dot intragap states and metal oxide properties on this loss pathway.