We demonstrate reversible voltage control of perpendicular exchange bias via H+ pumping in a NiO/Pd/Co/Pd/Gd(OH)3/Au heterostructure at room temperature. The perpendicular exchange bias results from a tailored layer structure consisting of an antiferromagnetic NiO layer and a ferromagnetic Co layer, stabilized by an ultrathin Pd interlayer. Voltage mediated H+ pumping through the Gd(OH)3 layer and subsequent H absorption at the Pd/Co interface leads to a decrease in the perpendicular anisotropy. In consequence, also the perpendicular exchange bias vanishes upon voltage application (3V). During voltage switch-off, this process reverses and perpendicular exchange bias recovers. The first voltage switching cycle shows relatively slow kinetics and an inverse relation of exchange bias and coercivity changes. We discuss these features with regard to an H-induced crystallization of the initially amorphous Pd/Co/Pd trilayer, which is revealed by transmission electron microscopy. With subsequent voltage switching steps, a decrease of the exchange bias field in the voltage switch-off state is observed, which levels off with increasing cycle numbers. A reversible setting of exchange bias field values is achieved when a magnetic field (+/- 2 kOe) is superposed during the H loading step. In this case, the shift of the exchange bias field can be controlled by the direction of the applied magnetic field. These results open an innovative route to electrically control exchange bias.