Stability of organic photovoltaic devices is a limiting factor for their commercialization and still remains a major challenge whilst power conversion efficiencies are now reaching the minimum requirements. The inverted organic solar cell architecture shows promising potential for improving significantly the cells working lifetime however, often when solution processed ZnO is used as electron extraction layer (EEL), a light soaking step is required before the device reaches a non-permanent maximum performance. Here we show that by doping ZnO with Sr or Ba using sol-gel processing the light-soaking step is circumvented. In a model poly [3-hexylthiophene] (P3HT): [6, 6]-Phenyl C60 butyl acid methyl ester (PCBM) system we obtain EQE ~55% before UV exposure for ZnSrO or ZnBaO EELs as compared to 10% for undoped ZnO EEL. We have investigated the origin of this improvement by comparing the response to UV light of doped and undoped ZnO. Characterization includes electrical conductivity and x-ray photoemission spectroscopy studies on thin films, current-voltage experiments and electroabsorption (EA) spectroscopy to probe the built-in field in the devices. We will discuss how the results obtained and in particular the higher effective built-in field in doped ZnO devices (1.5V) compared to a ZnO device (0.5V) can help interpret the mechanism behind the device performance improvement with Sr and Ba doping of ZnO.