Organic photovoltaics based on non-fullerene acceptors (NFAs) show record efficiency of 16 to 17% and increased photovoltage owing to the low driving force for interfacial charge-transfer. However, the low driving force potentially slows down charge generation, leading to a tradeoff between voltage and current. Here, we disentangle the intrinsic charge-transfer rates from morphology-dependent exciton diffusion for a series of polymer:NFA systems. Moreover, we establish the influence of the interfacial energetics on the electron and hole transfer rates separately. We demonstrate that charge-transfer timescales remain at a few hundred femtoseconds even at near-zero driving force, which is consistent with the rates predicted by Marcus theory in the normal region, at moderate electronic coupling and at low re-organization energy. Thus, in the design of highly efficient devices, the energy offset at the donor:acceptor interface can be minimized without jeopardizing the charge-transfer rate and without concerns about a current-voltage tradeoff.