The introduction of nonfullerene acceptors (NFA) facilitated the realization of high-efficiency organic solar cells (OSCs); however, OSCs suffer from relatively large losses in open-circuit voltage (V_OC) as compared to inorganic or perovskite solar cells. Further enhancement in power conversion efficiency requires an increase in V_OC. In this work, we take advantage of the high dipole moment of twisted perylene-diimide (TPDI) as a nonfullerene acceptor (NFA) to enhance the V_OC of OSCs. In multiple bulk heterojunction solar cells incorporating TPDI with three polymer donors (PTB7-Th, PM6 and PBDB-T), we observed a V_OC enhancement by modifying the cathode with a polyethylenimine (PEIE) interlayer. We show that the dipolar interaction between the TPDI NFA and PEIE─enhanced by the general tendency of TPDI to form J-aggregates─plays a crucial role in reducing nonradiative voltage losses under a constant radiative limit of V_OC. This is aided by comparative studies with PM6:Y6 bulk heterojunction solar cells. We hypothesize that incorporating NFAs with significant dipole moments is a feasible approach to improving the V_OC of OSCs.