This invited paper reviews a framework to assist in mitigating societal risks that software can pose. This is to promote effective human oversight, which is a central requirement enforced by the European Union’s upcoming AI Act [29]. The paper advertises fragments of an upcoming journal publication [12], and as such is itself low in genuine originality. Yet it offers a specific perspective on that original work. Extrapolating earlier work on software doping, we report on the combination of established techniques for runtime monitoring and for probabilistic falsification to arrive at a black-box analysis technique for identifying undesired effects of software. We describe its application to high-risk systems that evaluate humans in a possibly unfair or discriminating way. The approach can assist humans-in-the-loop to make better informed and more responsible decisions. Our technical contribution is complemented by juridically, philosophically, and psychologically informed perspectives on the potential problems caused by AI systems.